GitHub Runners, and multiple versions of the Dagger Engine?

[sharp pecan]

Hi all -- I'm deploying the Dagger Helm Chart to an AKS cluster, which has a dedicated nodepool for runners. I also have a GitHub Workflows repository that has tagged releases, and a Daggerverse repo that also has tagged releases.

I've hit a scenario where I'd like to update the engine to Dagger v0.15.2, but I still need to have some overlap in support for v0.14.0 as I won't be able to cut all consuming repos over to the new GHA at the same time, over multiple major releases of the re-usable workflows.

Is there a pattern to follow to deploy multiple versions of the Dagger Engine on a single nodepool? Then I could have Runner Pools for a specific Dagger Socket, and then label the runners with that version of Dagger.

I hope I've been able to describe the issue clearly. Let me know if I haven't. 🫠

[sand yoke]

I'm not sure exactly how to do this in the same nodepool since you still only want one engine per node (I'm sure there's a way). But the way you're thinking of it with labels is definitely the way to go

I think if it's deployed as a daemonset then maybe it can't be in the same nodepool?

Actually maybe I'm wrong about one engine per node. It might be fine since the GHA runner would know which socket to point to based on it's label

[boreal gazelle]

@sharp pecan the engine has a compatibility mode where you can still run modules with an older version of the engine as long as the dagger.json references the correct version. I'd try just deploying a single 0.15.2 engine and validate that your workflows work untouched. If that's the case, you can simply upgrade your entire nodepool and bump your modules progressively

[sharp pecan]

It's more around CLI/Engine compatibility. The version of the CLI we're using is set in the workflow either by a configuration option, or by the version of the Dagger GHA.

[boreal gazelle]

using the latest CLI + Engine should be the most common use case because as mentioned before, if any of the upstream modules that your pipeline uses has an older version of Dagger, generally the compatibility module will cover those cases

there are exceptions, that's why it's advisable to test your pipelines with the latest Dagger CLI + Engine to make sure compat mode will work ok

Unless of course there's a breaking change in the CLI and you need different CLI's depending on the workflow that you're running but I don't think there's been a major break between v0.14 and 0.15 in the CLI

so v0.14 pipelines should work with v0.15's CLI

[sharp pecan]

Ah -- I think I misunderstood the CLI/Engine relationship. So I can have a v0.14.0 dagger CLI connect to a v0.15.2 Dagger Engine?

[boreal gazelle]

Ideally the CLI and the Engine should be on the same version. So in your case it's advisable that you bump both to the latest version since pipelines which have v0.14 in their dagger.json should still work as it'll use the compat mode I've described above

Make sure your pipelines still work in v0.15.x before upgrading as sometimes there are some edge cases

[sharp pecan]

Got it. I think I'll run multiple nodepools. One for the current version of engine, and one for the previous version. Then I'll label the runners with the version the workflow should execute on, then that'll least give projects some time to migrate to a later release of my teams re-usable workflows.

It makes upgrading a little more work, but not too much. Thanks @boreal gazelle, I think I have a way forward.

[boreal gazelle]

Sure, that also works. We implemented the compat mode for these cases where a newer CLI/Engine is still able to run "older" dagger.json pipelines. In theory, the multiple nodepool approach shouldn't be needed but it's surely more reliable to do it that way

[sharp pecan]

I think we might be talking cross-concepts. Let me know if you'd like a call so I can show you the situationship between my workflows, CLI, and Engine.

[boreal gazelle]

That'd be great. I'm AFK now, maybe @sand yoke is still around to take a quick look

[sharp pecan]

Or @boreal bramble if Kyle's not around 🤣

[boreal gazelle]

Just trying to understand if compat mode would be an option here Kyle instead of having separate node pools

[sand yoke]

I'm not around to dig in atm but

So I can have a v0.14.0 dagger CLI connect to a v0.15.2 Dagger Engine

like @boreal gazelle said ideally these are the same version, however the 3rd dimension is the version of your module, and that does not need to be the same as CLI + Engine. So you could have 0.15.2 CLI + engine, and a 0.14.0 module should work fine

[sharp pecan]

I'm up for a call tomorrow. I'm in SoCal, but work Central Time. I'm around from 6.30am (PST) onwards.