General understanding on intended Dagger use cases

[fair hedge]

I've done a few experiments with writing some Dagger tasks and in general really like how its scriptable and flexible. However, I'm having issues understanding how I can really run them in a CI pipeline actually triggered on git pushes in a repository.

From what I can gather in the documentation the current best practice is to sort of use your existing pipeline provider (e.g. Gitlab) as just a launcher for the containerized Dagger task. I am finding it a little difficult to see how things fit together nicely given this assumption.

My specific situation is that I have a new monorepo using Gitlab CI with self-hosted runners using their Kubernetes runner. The code and pipelines are small, but will be rapidly getting more complex as I want to implement logic for running only specific tasks in CI depending on what actually changed in the repo. Ultimately I would rather not have Gitlab CI/CD at all and just use Dagger, however currently to just use Gitlab CI as a connector to the git repo and launcher of Dagger tasks is fine. However, the problem is that I still need to implement most of the workflow level logic in my Gitlab CI/CD pipeline (i.e. run this job under this condition; run this task before this task, this task can fail) because if implement all my logic in python code around Dagger it will all run in the same job. This is bad primarily because I would normally get a new pod in the cluster created for each Gitlab job that runs in parallel. Secondarily I don't get the visualization of the pipeline. AFAIU there is some work being done for a paid offering to replace e.g. Gitlab as the launcher.

The other issue is just the extra layers of containers that comes along with piggybacking on another systems runners. For instance the Gitlab runner for Kubernetes will launch each job in a new pod with the container image for that job. To use Dagger I need to run podman (since its not very easy/secure to run Docker in K8s) and have Dagger launch another container.

My question then is just to ask how others have put Dagger into use, what is the expected "user story", does it match to my expectations, and maybe the status of the project on the path to that goal?

I'd like to adopt Dagger because I can see the potential of being able to run scripted dynamic pipelines that can easily be replicated locally, but I don't currently see that as possible atm.

[gusty dome]

@fair hedge in my opinion there are 2 benefits of using dagger:

• testing the pipeline in local, aka avoid empty git commit to try and dev the pipeline itself
• create an abstraction between the pipeline and the cicd, you just have to do few yaml lines in order to run dagger, in the futur if you change to gitlab to github as example you just need to rewrite the few lines of yaml code which lunch dagger

In your case you could diff changes in the code itself

Like if "dir X of previous commit" != "Dir X now" continue, else do nothing more

[pastel halo]

Hi @fair hedge , I agree with @gusty dome that those are two major benefits of using Dagger.

The questions you're asking yourself about best practices - "what's the correct way to use Dagger?" - are very reasonable, and in fact quite common. We are working on improving our website, docs and UX so that the answer is more obvious. I will try to address some of your questions below.

[pastel halo]

I still need to implement most of the workflow level logic in my Gitlab CI/CD pipeline (i.e. run this job under this condition; run this task before this task, this task can fail) because if implement all my logic in python code around Dagger it will all run in the same job

My answer to this is "yes and no" 🙂

It's true that Dagger doesn't (yet) support dispatching jobs across multiple machines on its own: you have to configure the underlying runner for that. However, that doesn't mean you can't move all that pipeline logic to Dagger.

The trick is to decouple pipeline logic ("run this job under this condition; run this task before this task, this task can fail") from runner infrastructure (how much compute and CPU to allocate to which part of your pipelines, and how).

The approach we recommend:

1. Choose how much of your pipeline logic you want to write in code. Sometimes the answer is "all of it", sometimes it's "only this job over there". It depends on your project. It boils down to (a) benefits of writing pipeline logic in code, vs (b) inconvenience of changing something that works. Once you identify the parts where the benefits of code outweigh the inconvenience of change: that's the part you should port to Dagger 🙂

2. Write all the code in the language of your choice, using the corresponding Dagger SDK. This includes "this task after that", "this task can fail", etc). Dagger gives you a way to handle all that in code. Make sure it all works locally. Don't worry about CI job breakdown for now.

3. Test it in CI as a single big CI job. Yes that means it's all running on one machine. That may or may not cause performance issues, depending on your particular workload. The point is, you need to try, because sometimes performance bottlenecks in your current CI will melt away with Dagger - because of caching, and because of the switch to a new, more efficient programming model. NOTE: this step requires good caching and visualization, which we are currently building. DM me for early access 🙂

4. Once everything runs end-to-end in a single job, THEN you can easily optimize infrastructure concerns. In practice that means identifying parts of your pipeline that really do need scale-out CPU, even if it means less cache reuse. Typically that's CPU-intensive or GPU-intensive test suites, or perhaps inference or training steps if you're dealing with AI workloads.

Note: this approach requires us (Dagger) to provide two important features for production CI deployments:

1. CI-independent visualization. You should have high-quality visualization of your pipeline, regardless of where they run, and regardless of how many CI jobs they're split up into.

2. Intelligent caching. CI runners are stateless, so scaling them out naively will destroy your cache rate: often this means spending more money to throw more CPU at your pipelines will result in them running slower. The solution is to hook up all runners to a centralized service that can orchestrate caching decisions to maximize the chances of getting a cache hit across a dynamic pool of machines with unpredictable cache state.

We are building both as part of Dagger Cloud 🙂

[fair hedge]

Thanks for the answer that clears it up. I think the answer for me is "use Dagger cloud". Most of my tasks are simple so most of the scripting I would want is around orchestrating those tasks.

[pastel halo]

If you DM @vestal tinsel he can hook you up 😁

[hollow oriole]

I still don't understand how dagger can be used to test pipelines locally. Real pipelines most of the time have dependencies on external services which are not available locally. It could be secrets service, distributed cache, vulnerability scanner and so on. If the services are not available and there are no mocks for them, running locally is not an option and therefore there is still a need to do things using services provided by the platform.
Also there are differences in architectures. Production can run on x86 while development can happen on mac using arm64 chips. The networking can be provisioned differently. The local kube installation on mac is in the VM which means dagger build should run on that VM as well.

Is there any resources describing how to setup macOS for local development of complex pipelines which has integration test stages? The integration stages involve fault injection frameworks which often hard to setup locally in VMs.

[gusty dome]

@hollow oriole Hi, the goal is (in my opinion) to be cicd agnostic

so for secrets you should use an external secret manager/vault and not the build it in the current cicd platform

for vulnerability scanner until you have a oci image there is no issue at all ? (never worker with this kind of tools)

also for cpu arch dagger use qemu I believe

you just have to specify the platform in the SDK

[pastel halo]

That’s right 🙂 Sometimes your pipeline has external service dependencies - in that case it’s valuable to parametrize the dependency: API credentials (via pluggable secrets), API endpoints and instance/team/workspace/whatever. Then you can easily reuse the same logic in different environments: dev, staging, one-off integration etc

Sometimes your dependencies can be mocked in a container. Dagger supports that natively with service containers. That’s a lot of environment-specific glue scripts that you can replace with clean reusable code. Again across dev, staging etc.

Also now you can seamlessly mix mocked and external dependencies. So over time you can add mocking to services that initially didn’t support it, for example. All within the same dev and operational workflow

On platform differences: that is a very valid point. Dagger API supports multi-platform containers natively. See https://docs.dagger.io/406009/multiplatform-support/

There is an operational caveat which is the use of qemu for platforms not native to the current host. We will add support for native performance across all platforms. The plumbingis there (buildkit + docker ecosystem) but we need to package and integrate it. Note that when we do, this will be transparent to your pipeline code since the APIs are already in place. One day you’ll upgrade dagger and your multi-platform pipelines will just be much faster 🙂

[fair hedge]

For "local" execution the feature that you really want is to be able to kick off the execution from the current files you have and not have to go through a git commit, whether its remote or local doesn't really matter or depends on the specific constaints of your pipeline. Most of the annoyance with debugging CI/CD pipelines is having to commit the tiniest little typos over and over again then have to rebase it all later. Of course startup is a big factor too though.

Of course something like QEMU could be cool if it works seamlessly and is opt in.