Dagger "execution environment"

[hearty wasp]

Hi everyone 🙂 Newbie question here 👋.

How does the execution environment of a Dagger pipeline look like? Can I use a cloud provider to spin up containers from a Dagger pipeline?

Currently, when using either GitLab CI or GitHub actions I know that every job is executed on a separate runner that can be a separate machine, thereby allowing to scale build runners horizontally e.g. using cloud VMs (which we currently do). Reading Daggers documentation, I couldn't find any information that would allow me to close the mental gap that I have in my mind.

Background of this question: I recently heard about Dagger on the Changelog podcast. I have been frustrated with the de-facto vendor lock-in for CI/CD (currently on GitLab, considering migration to GitHub). Just the fact that it would take considerable development efforts to migrate something "simple" as a CI/CD pipeline is kind of hindering us to do a migration at this point. Adopting Dagger could allow us to get out of this vendor lock-in trap AFAIU.

[jagged furnace]

@hearty wasp 👋 Currently Dagger pipelines you write in one of the SDKs (e.g. Go, Node.js, Python) can be executed anywhere you can run containers. That could be your laptop with docker desktop (or podman, or nerdctl/lima), or it could be a CI runner VM with docker support (e.g. hosted/managed runners in GitHub Actions or CircleCI or GitLabCI) or Kubernetes runners (e.g. self-hosted runners in GHA, Circle, GitLab that you run in your cloud account like AWS, GCP, Azure, etc).

In CI, we currently use the control plane of the CI provider to detect a change in GitHub/GitLab and to trigger the execution of a Dagger pipeline on a runner.

Here's a dagger run kicked off from your terminal (top), Jenkins (lower left) and GitHub Actions (lower right).

We have users on many different CI platforms using CI-provider-managed runners and self-managed runners.

What does your current CI environment look like?

[hearty wasp]

Hi @jagged furnace , thanks for getting back to me! I haven't set up our CI but from what I can tell we use "GitLab Runner Docker Machine Autoscaler (GA)" in combination with Google Compute Engine. So AFAIU, we have a "runner manager" that handles the scaling of runners.
Back to Dagger, I believe this is where my mental model has a gap: which entity would manage the autoscaling when executing a Dagger pipeline? Is Dagger itself doing that?

[jagged furnace]

Thanks, @hearty wasp 🙂 Dagger doesn't handle runner management, but works with whatever CI system you have to give you the ability to code pipelines (in your favorite languages) and run those locally (e.g. dev laptops) and in CI; as well as give you CI portability (of course the caching, visibility, and collaboration are pretty great).

In your case, you could continue using your current GitLab setup to manage the scaling of runners and just have Dagger execute in those runners (using the shared Dagger distributed cache, ideally). You could also move to a model where you autoscale GitLab (or another CI's runners) in k8s.

I'll have @sand compass reach out in a DM so we can share more with you 🙂

[hearty wasp]

Thanks @jagged furnace . I'll hopefully continue to evaluate Dagger in the future, but there's no concrete time plan yet. So far, I'm mostly trying to grasp the big picture on a conceptual level. Maybe my questions help to inform you which questions could be better clarified on your website.

Btw... for us as organization, probably the best DX would be an automated migration from GitHub or GitLab to Dagger.io. Not sure this is achievable at all, but it would definitely lead to the lowest possible inhibition threshold.