Dagger and Jenkins

[half nexus]

Hey folks, I had some questions around Dagger and Jenkins. Admittedly, some of them might be stupid questions but I'm still trying to get my head around Dagger and how it might slot into our current infrastructure / eco system. Below is a simplified version of one of our current Jenkins pipelines:

Checkout the Microservice Repo
Gradle Build / Test
Sonar Scan
Docker Build / Push to Registry
Run Kustomize to create K8’s deployment yaml
Checkout a separate repo K8's repo (monitored by ArgoCD) and merge the deployment.yaml

We also do some other small things which probably aren’t relevant in this context.

I’m just trying to understand, in terms of the Jenkins context, how Dagger would slot in here. The Dagger engine would be running on our Jenkins nodes - but do we still need a traditional pipeline with stages and steps; but instead of gradle and docker commands for example, we just replace with dagger commands? Or would it be more granular than this?

The local testing element is a big draw for us in terms of Dagger, but obviously if it’s a mix and match approach in terms of Jenkins there will be some elements that still aren’t testable at a local level.

Thanks in advance!

[low saddle]

Hey @half nexus!

There is quite a few ways of going about this. When it comes to where you have to run the dagger engine, you don't have to run on the same jenkins nodes. You definitely can and it might be the best approach for your setup, we can talk more about that if you want!

When it comes to the workflow itself. The "pipelines with stages and steps" depends entirely on how you want to orchestrate everything. There are quite a few ways to go about doing this, I'll share a few:

Mix of Application modules + Infra modules in a step by step workflow
In this setup you could leverage different modules for each part of your pipeline, and as you said in your message, replace each step with a dagger function call. For example, for buliding/testing the Gradle application you could use a module such as https://daggerverse.dev/mod/github.com/matipan/daggerverse/gradle. For sonar you could use: https://daggerverse.dev/mod/github.com/fluent-ci-templates/sonar-pipeline. And for buliding, pushing, kustomizing and merging into a new repository you could develop your own infra module that takes care of this process. If you want developers to have control of how their applications get built and tested then you could use a dagger module that is a part of the microservice repo. In this setup, the overall workflow works exactly the same as the current one, but you leverage dagger for each step and you can test that locally.

Mix of modules orchestrated from a single Dagger Function
If we build from above, we have a few different modules:

• Gradle to build/test
• Sonar
• Infra for build,push, kustomize, etc
  You could orchestrate all of this modules from a single function call that imports every single one of them and puts everything into a single dagger function. Meaning that from Jenkins perspective your pipeline is now a single command, calling the dagger function of your orchestrating module that pulls every module together

When it comes to benefits/drawbacks, it really depends on how heavy the use of jenkins is and how your o11y story works. Having each step of the pipeline be a separate function call will make it easier to understand where things are failing. Having everything in a single dagger function will require an initial step of checking which step failed.

My advise would be to start small. Tackle one step of the pipeline at a time and see how well it works for you. If you go through that migration, then putting every single dagger function call into a single orchestrating module is going to be quite a simple task! And whether it's necessary or not will depend on how your migration goes

[half nexus]

@low saddle thank you so much for your reply; this is really comprehensive and exactly the kind of information I was looking for! I was really struggling to get my head around joining the dots in terms of Jenkins and Dagger - but it sounds like breaking up an existing pipeline and "Daggerizing" some of the steps is a good place to start. Especially from a PoC point of view so I can sell it / justify it in the first instance 🙂

I guess I'd like to get to the point where we're mostly decoupled from Jenkins so that we're a bit more portable - that is to say, if we decide to move away from Jenkins in the future we're setting ourselves up for success and reducing potential headaches down the line.

Thanks again for your help🙌