How should you handle monorepo builds with dagger?

[safe shadow]

Build systems like bazel use dependency graphs to determine what needs to be rebuilt. I've used GHA triggers to skirt this problem but I'd like to do it the right way, similar to bazel.

[winged zealot]

@safe shadow - Not sure what you mean by "skirt this problem". Even with Github actions, you should still be allowing Bazel to do the tasks it can do best.

So, Bazel is for instance, your build system. It understands your monorepo's dependencies, generates precise build actions, and ensures determinism. Dagger would be your CI orchestration layer (or more specifically, a portable build engine). Dagger's role is to provide the environment (a container), manage dependencies for that environment (e.g., install Bazel, JDK, Git), handle caching, and execute the Bazel commands.

Think of it like this: You use a screwdriver (Bazel) to screw in a screw. Dagger provides the workbench, the lighting, and perhaps even the robotic arm to hold the screwdriver, ensuring it's always in the right place with the right conditions.

The bigger difference is that Dagger makes your CI pipelines themselves programmable and testable locally, and more importantly, outside of GitHub (or any specific CI vendor).

[halcyon grove]

the best way to manage a monorepo with dagger is to attach a dagger module to each component of the monorepo, then model the dagger module dependencies after the logical dependencies between components. A major strength of dagger here is that it can model dependencies across platforms. For example our docs builder is js, and our cli is go. But the docs build includes a generated cli reference, and that is accurately modeled as a dependency between the cli and docs modules.

[halcyon grove]

Now, on the topic of determining what needs to be rebuilt, this happens in two layers: CI events, and build cache.

1. Build cache

This is where Dagger helps most. If your build cache is good enough, you don't need your events to be too smart: even if you trigger unnecessary CI jobs on every change, most of those jobs finish almost instantly because dagger quickly determines there is nothing to run. The remaining overhead of spawning VMs is small potatoes compared to the efficiency gains of not running them for dozens of minutes per job. The CI configuration also becomes much smaller and less brittle, because you don't need to manually keep your CI infra configuration in sync with application-specific caching logic.

The major caveat here is that distributing dagger cache data across a cluster of CI runners, although possible, requires hands-on infrastructure work. We are working on major improvements in this area, by the fall we will have a completely revamped caching system, with full decoupling of data and compute. This will make just clear how far you can scale a dagger monorepo build without fancy event filtering.

2. Events

Even though CI event filtering isn't as important with Dagger because of better caching, in some cases it can be useful to use event filters as a second line of defense. This can be done pragmatically using your usual CI configuration. Those are not portable like dagger, and don't cleanly separate infra and app concerns, so should be used sparingly, as a performance optimization of last resort.

We will eventually provide a "smart events" feed in Dagger Cloud, to ingest raw git events from a repo, and emit change events only for individual dagger modules that have actually changed. Then you can hook up CI runners directly to those events, and bypass your legacy CI control plane. Basically rebuild your own lightweight CI from Dagger and commodity infra.

But we're going to solve distributed caching first 🙂 Fingers crossed this smart events feed will ship this year. We already built the first layer with Module Insights which shipped last month: https://dagger.io/blog/module-catalog-insights

[safe shadow]

I do not use bazel yet, but I want something more automated and principled than GHA trigger rules.

[dense spruce]

I've had a couple of use-cases in PR pipelines where I wanted to only do something if certain (subset of) files changed, but really did not want to maintain that as separate GHA workflows. I built a little module that diffs two git refs and gives you the list of changed files. I usually just check these to see if I should end a function early if nothing relevant changed. This is inherently slower, though, since it needs to wait for dagger to install/load. How much slower depends on your runner setup. For me it's been more about the simplicity than speed so far.
https://daggerverse.dev/mod/github.com/valorl/daggerverse/git-files-changed

[dense spruce]

Interesting. Is there an issue / more info around the caching revamp. I'm planning to move our setup away from cloud cache and ephemeral GH runners in the near future. Might inform my design.

[terse oxide]

My take at monorepo is to have a top-level custom cli that uses the dagger sdk. We have only a single go.mod, so the module based approach does not work with this constraint. We've paired this with CUE to create a BuildPack like experience for 6 products and 100+ services within a single git repo.

To deal with what needs to be built, we have a system that compares the current commit with a target commit, gets the diff, and then calculates the files needed for a component, finally looking for matches between the two lists. One of the issues you'll discover are the "hidden" inter-service dependencies, because you'll want to load only the minimal file-system for a build, and then they fail, and then you have to add ways to specify these in the CUE based software catalog...

[pastel niche]

@terse oxide Could you elaborate on the role of CUE in your setup? (Disclaimer: I don't have any experience with it.) I'm asking because apart from CUE our setup seems similar: We, too, have a custom CLI (basically a Dagger custom application) which loops over all projects in our monorepo and builds them one-by-one, provided they have changed. We don't look at the diff / git history to identify changes, though, but simply rely on the image layer cache to trigger a build as necessary. As it so happens, we also ran into the exact same issue you mentioned w.r.t. minimal file systems and inter-service dependencies.

[terse oxide]

• we run on Jenkins, with a tree of subjobs, so we use git to decide what to build. There are certain branches where we build everything, and the layer cache disappears with the workers (scale to zero setup)
• We use CUE for two systems BFG (branch config) and INV (inventory / software catalog)
• BFG has 3 main things: branch info, org wide versions, and CI config/overrides... things that are specific to a branch and changes that happen at that level. We actually CI the code as it would look like after a merge, not before, because this catches more bugs and regressions. This requires we know what the target branch is before a PR is created. Every branch gets a bfg/<branch>.json that contains this information, is merged with the CUE schema, and gets some extra runtime (CI time) values. This is passed down through all of our builds and dagger
• INV is a tree of products / services / components / etc... and that part is a lot like buildpack, but also includes any exceptions to the rules, like inter-service dependencies. We use this system to collect all the services that go into a product, so that we can loop over them in Dagger (and more places). The benefit of having this outside of the build system is that the info is useful in more places and we can use the CUE CLI from python or Make in example. A large part of this decision was to avoid product-service list duplication, which caused a lot of headaches when one list was updated and another was not.
• we chose CUE because (1) I'm one of the biggest fans and an expert (2) It's a great language for configuration, especially when you want to merge or import. It also has a Go SDK, which we can pair with the Dagger Go SDK in our Go CLI. It has allowed us to move a bunch of stuff out of code and into the config layer

[pearl basalt]

Are there any code examples you all could share, please?

[terse oxide]

I've been meaning to write mine up, it's currently in a private org I contract with, but at least I do have permission to write about it.

[pearl basalt]

just the file structure would already be a thing... and only for two modules

maybe the new blueprint feature that's baking will help

[halcyon grove]

FYI the blueprint feature shipped 🙂

[terse oxide]

In our monorepo, we have but a single go.mod, adding more back for modules is not political capital I can afford

[halcyon grove]

you can share a go.mod across dagger modules. the default layout with dagger init --sdk=go creates a standalone go.mod but you can just remove it

[terse oxide]

Can we invoke and use modules via the SDK?