#Dagger + Gitlab CI/CD components
upper cave
Hello @broken knoll ! I'm not a Gitlab expert, but I believe Dagger and Gitlab CI/CD components will work together just fine. Do you have a particular use case in mind for combining them?
broken knoll
Hi @upper cave . Damn you are fast! - I work for an org that is heavily invested in Gitlab and have a big piplines with a lot of bash scripts (and all its glory). We really want to rewrite it (hopefully in dagger).
The teams that use these pipelines what it to be more "modular" in terms of composing the pipeline themselves. What steps they want and in what order. Maybe they have a mono-repo with multiple project etc. They look at CI/CD components (CI components are just smaller CI pieces with input that can then be composed) are their saviour, but for maintainability I really want to move towards dagger. Do you have any experience in this very modular requirement and using dagger?
Maybe i am asking the wrong questions and looking at this the wrong way? 🤷♂️
upper cave
Yes Dagger is basically a more portable alternative to Gitlab CI/CD components
If you're already using components, then Dagger can fit in (as just another component, with its own component system - russian dolls). But if you're writing from scratch, better to avoid the russian dolls and go straight to Dagger, assuming it meets your requirements.
broken knoll
If you're already using components, then Dagger can fit in (as just another comp...
Yeah - that makes sense. Then we would just have a component calling a component. I will look at the case studies. Thank you for your time
mellow veldt
🙌 @broken knoll I'm working on all most identical scenario at the moment. We have big heavy mono repo with modular gitlab configuration (300+ unique jobs).
I would recommend looking into
for CI part.
Also dagger would be ideal if you want to create self serving DX, when you provide customized and maintained daggerverse repo (set of modules your devs need, replacement of that glorious scripts)
I'm currently preparing internal demo for dev teams to push dagger forward
broken knoll
Uh that sounds exactly like my scenario. What are you doing in terms of gitlab-runners? Are you running them in a kubernetes cluster or docker-in-docker.
mellow veldt
we're running them in k8s at the moment. For demo, I'm installing dagger-engine as daemon set to cluster and add to runners as described in guides. It's working pretty okay.
For CI part, just created a template similar to this example to install dagger to use in jobs
stages:
- test
.dagger:
image: alpine:latest
stage: test
interruptible: true
rules:
- when: always
retry:
max: 2
when:
- runner_system_failure
tags:
- dagger
variables:
GIT_DEPTH: 1
before_script:
- apk add curl docker
- curl -L https://dl.dagger.io/dagger/install.sh | BIN_DIR=/tmp/bin sh
dagger:build:
extends: .dagger
script:
- dagger call container-echo --string-arg "hello world" stdout
dagger:lint:
extends: .dagger
script:
- dagger call container-echo --string-arg "hello world" stdout
Okay - cool. Do you get the benefits of caching and such?
mellow veldt
Yes, as long as your node exist and have disc space dagger would use it's cache
broken knoll
I got it to work with dind, but it has to pull everytime it runs
mellow veldt
but each node going to have it's own cache, this is down side of the engine at the moment but for this you could check dagger cloud and magic cache
broken knoll
Yeah sure, but at least some of the time we have cache
mellow veldt
I got it to work with dind, but it has to pull everytime it runs
that would work but, it'll be slower than your current ci/cd and it's too hard to sell with gitlab env to dev teams
broken knoll
that would work but, it'll be slower than your current ci/cd and it's too hard t...
Yes indeed. We were actually looking at dagger for optimizing the run times for our pipelines.
Did you do anything other than what is mentioned in the guides?
mellow veldt
Yes and No, I'm already using dagger for a long time, it's just my current company doesn't use it. I'm preparing a small demo with suitable highlights for them. But generally, after setting up stable ci/cd with acceptable caching strategy and writing couple of example pipelines would be enough for show case
You can write your questions to #gitlab and ping me if you need something extra 🙇
For starting point first k8s guideline to install dagger-engine to cluster => than gitlab ci integration to configure runner would be enough for you to start
broken knoll
Great, thank you so much for the help and pointers 😉