How to visualize dagger steps in gitlab (or any other) ?

[dusk niche]

Hello, I wonder if it is possible to visualize steps in our runners ?

The exemple shown is injecting dagger in a step build and call build from dagger.

It feels like we are missing the point of dagger to create dag? Or we are repeating the steps created from dagger to just create step in Gitlab.

Exemple, if I am adding a step in my CI, I need to add it to dagger, then adding it to gitlab CI. What happened if I am not writting the steps in the right order in gitlab CI ?

Can dagger create a gitlabCI directly ?

[round lava]

Hello! Dagger makes it possible collapse many steps of your CI configuration into a single step that just says "call this dagger pipeline". It's up to you how much you want to collapse.

If you collapse many steps into one, as you said, the CI will not be able to visualize them separately. To solve this, we offer Dagger-specific visualization on Dagger Cloud: https://dagger.cloud (there is a free plan).

This allows you to visualize your Dagger pipelines no matter where they run: in CI, or in development. There is a Github integration so the status of Dagger pipelines is added to the right pull request as a check.

[shy sandal]

I end up running multiple modules (in Jenkins stages) just to get the Jenkins visualization. It's not ideal. I also don't have the ability to use dagger.cloud.

The experience for the developer running locally doens't have to be ther same. You can still create a parent module that runs all other modules which developers can run locally.

[mental yarrow]

yeah, I was scratching my head about this for a while as well. At first, I created all the per-step dagger functions (build, test, lint, oci-build, oci-push, etc.), binding them to respective BB CI jobs. That was a PAIN, because I had to use 2 repository providers: BitBucket (because my current project is hosted in BB) and GitHub (because Dagger only allows fetching modules from GH 🙄 ). That was inconvenient, so I tried storing modules locally, basically a copy of the same dagger code in every repo of the project. Talk about DRY coding.... a maintenance hell.

Eventually, I ended up using a Wolfi module exposing *Container to the dagger CLI that I can use to spin up custom containers and pass them instructions I'd otherwise pass using Golang (module code). e.g.

: "Run SAST scans"
              out_file="trivy-report.out"
              cmd="trivy fs -c trivy.yaml --output ${out_file} --exit-code 0 ./"

              dagger -m github.com/shykes/daggerverse/wolfi call container \
              from --address aquasec/trivy \
              with-directory --path /target/ --directory ./ \
              with-workdir --path /target/ \
              with-exec --skip-entrypoint --args sh,-c,"${cmd}" \
              file --path ./${out_file} --output ./

binding each action to a respective BB CI job. This way I can maintain BB CI visualisation and have reusable, though sometimes terribly long, dagger CLI commands to launch my CI jobs. However, this means that each BB CI job, running in isolation, has to download and install and start dagger engine separately, adding 1-2 minutes of overhead, which is very inconvenient (dagger does not provide an OCI image and I haven't made mine yet).
Also, this approach prevents me from having a master CI job, allowing me/developers to run the whole pipeline locally. But that's a tradeoff I can live with. And there's the caveat of passing []string values as CSV, introducing issues like https://github.com/dagger/dagger/issues/7678.

[shy sandal]

I feel your pain! Thankfully we moved off of BB to GH so I've been able to just use a GH repo as an internal "Daggerverse". It's not ideal because I can't yet import from a private GH repo. Fix is in the works though. Also a fix for vcs other than GH is almost released so relief is coming there for sure.

Neither one fixes the CI visualization issue. I am still forced to use multiple stages and multiple dagger calls but at least in my use case that works pretty well. I can't take advantage of buildkit caching though, so every job run, I pull the engine and initialize it. It's not demonstrably worse that what we have today (without dagger), but it's not better either so it will be a hard sell.

The bigger selling point is local runs. My plan is to create a "full pipeline" module for developers to run locally. That produces a compromise where, what we run in the CI job is not the same as what's being run on local dev. I don't think there's any way around that at the moment.

[mental yarrow]

@shy sandal well, at least you know (almost) for sure that the separate dagger-based steps (functions) you're running will run identically on both local and remote (GH CI) setups. Regardless of whether ran individually or triggerred by a master CI function (for devs).

yeah, it's a hard sell. That's why I try to stay silent about using dagger, at least for now 🙂 My current pitch is "it's easier for me, a devops, to build reliable pipelines locally that will run OK remotely". However, occasional sporadic dagger failures on remote don't help much to support the "reliable" angle 😄

[shy sandal]

Similar boat here. I haven't run into sporadic failures on CI though. It's been pretty stable when I get it to work.

[mental yarrow]

As for the visualization in remote CI, I doubt there's any way for the dagger team to integrate with the remote CI's UI. I think you're stuck with having a CI job == dagger function call association here (i.e. separate function calls for separate CI steps). And if you really want to -- a master CI job for local env. But then again, the CI sequence not always can be identical for remote CI runs and local ones. E.g. I really don't want developers overwriting my OCI images in the registry (images the dev/qat/other environments are using) with their half-baked WIP features that they're running local CI against just to see if it builds and tests pass OK. So it doesn't always make sense to have an identical CI sequence everywhere