👀

[midnight lake]

The reason I ask is because we use a Go program right now to run our CI steps instead of bash, but having both a Go program and a giant yaml file results in a lot of drift between the two, and thus a lot of errors that i think could have been prevented.

Like, for example, the arguments change on one of the commands but the yaml doesn't get updated everywhere.

[lost geyser]

Hi, your new assumption is totally correct 😇

In previous versions of Dagger, we did not generate that config, no. It was the same as the Go SDK, but our messaging wasn't clear at that time 😁

We were just betting on the fact that people would love writing their pipelines in Cue, and run it the same way we do today

[midnight lake]

Gotya that makes sense! And so you ge around all the different requirements of every provider by just running everything somewhere else (and more consistent), right? 😂

[lost geyser]

Dagger currently supports all CIs that are compatible with containers

We currently rely on docker to spawn our dagger engine, but this is an implementation detail.

Under the hood, our dagger engine embeds an OCI container runtime for consistency across all providers.

We had some users that preferred to have a dagger engine running on a VPS they controlled, and their pipelines would interact with it. It worked quite well too, especially regarding the caching

If this is the path you want to take, I think it currently works with the go SDK, it's just not documented

Do not hesitate to ping for any question on the subject 😇

[midnight lake]

We have a lot of jobs that are really CPU intensive so we rely pretty havily on using multiple nodes. I remember seeing a conversation about clustering mode somewhere, is that what would allow the dagger engine to run jobs on separate machines?

[lost geyser]

Totally, here is an issue explaining in more detail the current architecture : https://github.com/dagger/dagger/issues/3595

The clustering is a next step, and its architecture is currently being discussed. Feel free to create an issue to ask all your questions on the subject, it would help on the architectural design 🙏

[glacial kindle]

I generally like to build my CI jobs (wherever they run) in a structure, so I can use my own solution (eg. invoke one make target, one "dagger pipeline" to run everything), but can also utilize some of the execution environment features.

For example, I still use build matrices to parallelize various builds. In the context of Dagger/Make/whatever it would mean that I have a single build/test/whatever command, but there are "subactions" that I can call one by one (eg. one for each supported Go version) that I can call from a GitHub Action with a build matrix for Go versions.

This way I have my own, portable pipeline that I can run locally, but can also use some of the features provided by the execution environment.

In a sense I've been doing this long before Dagger came along, but Dagger makes it a lot easier and build pipelines are way more expressive in Cue (or Go) than in a Makefile for example.

[sullen plume]

@midnight lake I'd love to learn more about your CI setup and collect your requirements, if you're open to it 🙂

I'll send you a friend request.