#Opinions on pipeline code reusability and project structure

1 messages · Page 1 of 1 (latest)

unborn river
#

Hi everyone. I am tinkering with dagger, already powering some projects with it and I am still a bit unsure about how to reuse pipelines and code written in dagger in other repositories that can basically follow same instructions.

The environment here is around 20 applications/microservices written in .NET, now being migrated to .NET 6. I have a generic pipeline in dagger that builds and pushes the image where I tell it to, from what images and git repository. I have built the pipelines to binaries, and included them in the runners that run the pipelines as a CLI tool

What I mostly see in other projects (for example https://github.com/thechangelog/changelog.com) is the dagger code included in the repository and pipeline running the code directly. The difference is that this is probably the only repository where this specific pipeline is being used, so my usecase is quite different.

I'm afraid that even if I make my pipelines into a package and include that into projects where it's being used I would have more unnecessary boilerplate being copypasted everywhere. But what I have to do now is distribute the executable everywhere where it's used (which is now only like 3 runners so no big deal).

What do you guys think?

brittle wave
#

Hey! My immediate goto, in any SDK/language is to create an installable package, which can be private. When working on Python apps, I had a few shared packages that were published in the company's private pypi instance (python package index). Can also be installed from a private git repo. So sharing pipelines or helpers in an installable dependency to reduce boilerplate is a natural thing for me.

However, if the CLI tool you built works well for your use case and leads to even less boilerplate, then that's great! I'm curious about how that looks in practice. Can you share a simple example? 🙂

unborn river
#

Sure! Don't know if it will be simple but I will try to provide context and share some parts of code that will make sense

#

We have self-hosted GitLab, all is in internal network so no access from the internet, private within internal CA. I have one repository that acts as a template holder, other repositories contain the .NET applications each.

We then have self-hosted gitlab runners, one runner specifically uses shell executor but has docker installed on the server with custom dagger engine running as docker-container://dagger-runner. So whenever the runner runs dagger code it accesses this engine based on env variable _EXPERIMENTAL_DAGGER_RUNNER_HOST

The Templates repository has yaml specifcations for each pipeline that gitlab runners can understand and when it's building the images I call the dagger code which is built into a executable called horse (Horse, because it's stable - pun intended), it takes in some flags with OCI registry credentials, GIT oauth2 token, repository link, what images, where to publish the solution and so on.. And what I can do in the .NET repositories is just specify simple short .gitlab-ci.yml that includes those templates and just pass in variables that define the specific project path and so on.

#

For example this is tag.yaml job that runs only in git tags

build-oci-image:
  stage: build
  rules:
    - if: $CI_COMMIT_TAG != null
  script:
    - >
      horse build-dotnet -D
      -r $LOCAL_OCI_REGISTRY_URL
      -u $LOCAL_OCI_REGISTRY_USERNAME
      -p $LOCAL_OCI_REGISTRY_PASSWORD
      -g $LOCAL_GIT_REPOSITORY_URL
      -t $LOCAL_GIT_TAG
      -c $LOCAL_CSPROJ_PATH
      -B $LOCAL_OCI_BUILD_IMAGE
      -R $LOCAL_OCI_RUNTIME_IMAGE
      -S -P $LOCAL_OCI_PUBLISH_TAG_IMAGE
    - >
      horse build-dotnet -D
      -r $LOCAL_OCI_REGISTRY_URL
      -u $LOCAL_OCI_REGISTRY_USERNAME
      -p $LOCAL_OCI_REGISTRY_PASSWORD
      -g $LOCAL_GIT_REPOSITORY_URL
      -t $LOCAL_GIT_TAG
      -c $LOCAL_CSPROJ_PATH
      -B $LOCAL_OCI_BUILD_IMAGE
      -R $LOCAL_OCI_RUNTIME_IMAGE
      -S -P $LOCAL_OCI_PUBLISH_LATEST_IMAGE

  tags:
    - dagger
#

Some of the variables are specified in variables.yaml and others are in the repository with project itself. The .NET .gitlab-ci.yml looks like this for example

include:
  - project: "cz/it/gitops/templates"
    ref: "master"
    file:
      - "/pipelines/dotnet/stages.yaml"
      - "/pipelines/dotnet/variables.yaml"
      - "/pipelines/dotnet/merge-request.yaml"
      - "/pipelines/dotnet/default-branch.yaml"
      - "/pipelines/dotnet/tag.yaml"

variables:
  LOCAL_GIT_REPOSITORY_URL: "https://oauth2:$GRP_CICD_RUNNER_GIT_PASSWORD@git.COMPANYgrp.com/cz/it/COMPANY.sql.service.git"
  LOCAL_CSPROJ_PATH: "COMPANY.SQL/COMPANY.SQL.csproj"
  LOCAL_OCI_BUILD_IMAGE: "oci.git.COMPANYgrp.com/cz/it/gitops/templates/dotnet/sdk:6.0"
  LOCAL_OCI_RUNTIME_IMAGE: "oci.git.COMPANYgrp.com/cz/it/gitops/templates/dotnet/aspnet-krb5:6.0"

build-manifests:
  rules:
    - when: never

.deploy-manifests: &deploy-manifests
  rules:
    - when: never

I have excluded some jobs with the rule: - when: never override

#

as you can see on top, I include the tag.yaml and the job just takes the variables provided in the variables: section, which is project-specific

#

tags: - dagger say to run the pipeline only on runners where dagger tag is present

#

So what happens is, when git tag is pushed to the repository with .NET project, it runs the build-oci-image job (and plenty others I've excluded for this example), fills in the variables for horse build-dotnet command and dagger pipeline gets executed inside, builds OCI image and pushes it to registry

#

and this happens in many other repositories and all it changes is some rule overrides and variables specifications

#

horse is written in go and currently does three things build-dotnet builds the .NET image in generic way that is applicable to all our .NET codebase and pushes the image, semver checks if the last git tag follows semantic version and returns new tag based on keyword specified in commit message, and also I have some git-download which downloads directory from some git repo, that's what I use for kubernetes deployment where I only download kustomize files for that specific project and deploy

#

All I do is here and there extend the functionality of the horse tool and upload new version so it's available for the runner

#

TLDR: it's complicated 😅

distant yoke
unborn river