Removing the /usr/local/bin/dagger symlink hack

[serene hound]

https://docs.dagger.io/541047/alternative-runtimes

This page shows a user how to use Dagger with an alternative container runtime. The method is a hack to symlink an alternative CLI to the Dagger CLI.

We also want to be agnostic and Andrew thought we could skip the hack by:

1. use Go's exec.LookPath
2. have switch and var so we can exec which ever directly
3. have flag/env-var to let the user force a runtime

This is specificially the "runtime" as in CLI Dagger and Hof exec out to in order to perform certain actions.

I think Dagger should do this too, what do you think?

WIP: https://github.com/hofstadter-io/hof/pull/238/files#diff-71e78c449fe5b64e3a88478cd5c134fbdf8891d973ddcbf436cdef5fa797b2ca

[jagged mist]

we could do that, but that would mean that we need to add native runtime support in dagger's CLI. AFAIK the "ultimate" solution is not to depend in runtime at all and just "embed" the runtime in the dagger CLI.

I'm not sure if it's worth adding this now particularly given that we know that depending on docker run to bootstrap the engine is not what we want in the long term

so basically delegating this responsbility to the user by asking them to create some symlinks we think it's a good enough trade-off for now

[terse grotto]

The most reliable way to customize engine provisioning is to just provision it directly. Run registry.dagger.io/engine any way you need to, then point the CLI at that

in production you don’t want an engine to be auto-installed anyway. So in production it doesn’t matter how customizable the auto-installer is, since you shouldn’t use it anyway.

[jagged mist]

is that so? I'd have thought that for some users, even in production, it'd be ok for them to rely in an auto-installer if they're using ephemeral runners. Otherwise, they'd have to add some more glue to their pipelines to figure out a way to provision the engine themseleves. i.e: a company that's using gitlab self-hosted runners and podman as a current solution to orchestrate their CI/CD pipelines. When they decide to migrate to Dagger, they'll have to modify those runners so Dagger can auto-provision the engine accoringly.

[terse grotto]

Anyone using self-hosted gitlab runners, presumably on kubernetes, should install dagger as a daemonset so that there is always an engine running on each node of the cluster. So auto-provisioning is not desirable in that scenario.

In the case of, say, Github Actions managed runners (so ephemeral machines), I see two options:

1. You don’t care about customizing: in that case you just use whatever the default config does. Today that’s auto-provisioning with standard docker client. In the future it could be using a dagger action from the github marketplace that handles provisioning. TBD.

2. You want to customize. So you disable auto-provisioning and add glue in your runner config that uses podman or whatever. Which is probably what you want if you’re the type to customize your CI runner anyway.

[jagged mist]

Yeah I agree.. still users relying on Gitlab or Github's hosted runners might actually find the auto-installer quite handy without docker if they're using something else like podman or whatever... ?

edit: you beat me to it 😄

[terse grotto]

For “I like auto-provisionet but use podman” the symlink is all you need IMO. Podman does claim to be a drop-in replacement… It’s part of the appeal.

One hairy problem with auto-provision is that it puts engine lifecycle in the hands of the pipeline dev. If there are multiple pipelines by multiple devs, it’s not impossible that multiple different versions of the engine will be installed. You can’t centrally upgrade to the latest engine across your entire CI just by changing one config file. That’s not a very exciting prospect for operators.

But maybe that can be addressed by always relying on the dagger CLI wrapper. And locking engine version to CLI version. Which is what you mentioned earlier with the “embedding” idea… 🤔

engine version management gives me headaches