Multi-repo builds: package/deploy steps?

[opaque pulsar]

We have a set of interconnected library and application Gitlab projects running in Gitlab CI. We have another project that contains the others as submodules (for better or worse) and owns the top-level docker-compose files, developer scripts, as well as the parts of CI/CD pipeline that assemble the libraries and push Docker images that should be deployed.

So far we've spiked out Dagger in a few of the library level projects, but we're about to try the first application one (which depends on some of the libraries and ultimately creates a Docker image). My question is: should Dagger also do the package and/or container deployment steps? Or should I leave that to the existing Gitlab CI jobs?

Our goals are:

1. Have local development and CI builds work more or less the same (which Dagger is doing a great job of)
2. Have efficient but complete build pipelines from anywhere in the stack, whether local or on CI. If only application level code changes then no libraries should be rebuilt. If a low-level library is changed, projects above should be rebuilt or at least retested.

Our proposed solution:

We have been thinking of keeping steps to push packages and containers to their registries in Gitlab CI since they should never actually be executed in local development and only should be run for protected branches.

But in that case, we still need the local pipelines to get the packages from lower level projects if they're being run on a developer's local machine as part of this top-level project's pipelines. I had been considering some Dagger code that would check those conditions and if they're not in CI, pull them from the build directories in the filesystem.

Does that sound right? Or should I write the deployments into our Dagger code? (Or something else entirely...)

[magic vale]

Hey @opaque pulsar 👋

That's an interesting problem! I'm not sure if there's a 100% "correct" answer since both/all paths will surely bring some degree of success. Some notes:

I had been considering some Dagger code that would check those conditions and if they're not in CI, pull them from the build directories in the filesystem.
I've seen this in the wild before and it's definitely an option. Most CI platforms will set CI=true in the environment and a lot of other dev tools tend to rely on that. You could also toggle the release based on certain credentials being present or a cli flag

we still need the local pipelines to get the packages from lower level projects if they're being run on a developer's local machine as part of this top-level project's pipelines
This part sounds like the tough one to crack for me, without seeing exactly how the projects are set up. Do developers typically have all of the code and associated pipelines locally, or just the project they're focused on?

[opaque pulsar]

Do developers typically have all of the code and associated pipelines locally, or just the project they're focused on?

Typically all (through the containing project with all of the submodules). Although, in theory, I'd like to make it work standalone as well (i.e., it would fetch from the registry if it doesn't see the build directories where it would expect). Also, when the pipelines run on CI, they're be standalone (unless I switch to only running CI in the containing project... which I've considered).

[agile chasm]

Couple of ideas, since we have similar problems at my place of work:

1. gate the publish behind both permissions (devs can't push to the repository, for instance) and either a ci flag e.g. --publish or an environment variable like PUBLISH=true

2. do as you say, leave publish and deploy in your CI implementation of choice. I feel like it violates the principle of having ci/cd all in one contained thing though, and introduces complexities like you mentioned

Generally we have gravitated towards the first approach, making publish a part of the pipeline and just skipping that step for local runs as well as making it so it's quite hard to perform the publish from an unintended location like gating publish permissions behind a specific account or similar

[pastel knot]

With my solution, i've gravitated towards No. 1 but with a small addition. We do allow devs to publish their images/artifacts in local but make sure that the artifacts published by local builds go to a user specific folder/location and have user specific tags+properties to make it very apparent that the artifact came from a development machine and not CI. When run in CI they have the regular versioning+tagging.

It's useful for developers to sometimes publish their builds so they can test them in an actual managed environment instead of just locally.

[opaque pulsar]

That’s a good point @pastel knot … it would be advantageous to be able to publish development packages. The reason I had gravitated toward keeping publishing in Gitlab was that I found myself testing against and requiring enough Gitlab specific environment variables that it felt like I was going to have to add an abstraction layer for configuration/args/env vars to keep the build scripts CI-agnostic and usable by humans.

That felt like the tool was pushing back against me and when I thought about it, I wasn’t sure if it was worth the effort and complexity. But I wasn’t too satisfied with the idea of splitting the responsibilities either.