#Is it possible to run multiple engines (different versions) at the same time?

1 messages · Page 1 of 1 (latest)

tranquil oyster
#

I'm facing an issue where I am using dagger in multiple different projects and they are all using different dagger engine versions.

The problem is my CI build agents now have to switch (recreate the dagger-engine container) every time a new build comes in with a different version than the one's running in docker.

Is it possible, through some settings, or some shell/docker hackery to run multiple versions of the dagger engine in parallel?

Or maybe I need to configure projects to not recreate the engine container somehow?

ancient ginkgo
#

Unfortunately, I don't think so - the default behavior is to clear up any others versions of the dagger engine that were running.
That said, I've been annoyed by this behavior in the past, so an issue on github would be most welcome! Even if we had an option to just simply disable the garbage collection of old engines, that would be an improvement IMO

#

I'd love to get rid of the garbage collection behavior in general, but this would require that engines shut themselves down "after some time" - and there's some technical reasons that make that really tricky.

hallow widget
#

This garbage collection should only happen during automatic provisioning. Does it still clean up when using dagger run with Go?

#

You're free to download the CLI yourself and put where you want it. That's actually required with dagger run since you need the CLI before running dagger. In this case it won't be cleaned.

#

That's for the CLI. To run different engine versions at the same time (or independently), you can also do it. Just make sure to set _EXPERIMENTAL_DAGGER_RUNNER_HOST pointing to the right container for each, before the dagger command.

#

However, you have to start the engine and stop it manually (docker run, docker stop).

whole rapids
# tranquil oyster I'm facing an issue where I am using dagger in multiple different projects and t...

Yep, so like Helder mentions, you could do this sort of thing. It's really important that the different dagger engine containers have different names and different volume names like I have below, otherwise the other engine containers won't start due to conflicts like these.

From docker run with dup name:

docker: Error response from daemon: Conflict. The container name "/dagger-engine-097" is already in use...

Dup volume name, From docker logs <failed container id>;

buildkitd: could not lock /var/lib/dagger/buildkitd.lock, another instance running?

Download CLIs:

mkdir -p /tmp/env095 && cd /tmp/env095 && curl -L https://dl.dagger.io/dagger/install.sh | DAGGER_VERSION="0.9.5" sh;

mkdir -p /tmp/env096 && cd /tmp/env096 && curl -L https://dl.dagger.io/dagger/install.sh | DAGGER_VERSION="0.9.6" sh;

mkdir -p /tmp/env097 && cd /tmp/env097 && curl -L https://dl.dagger.io/dagger/install.sh | DAGGER_VERSION="0.9.7" sh;

/tmp/env095/bin/dagger version
/tmp/env096/bin/dagger version
/tmp/env097/bin/dagger version

# start engine
docker run \
--name dagger-engine-095 \
--privileged \
--stop-signal SIGTERM \
-d \
-v dagger-engine-095:/var/lib/dagger \
registry.dagger.io/engine:v0.9.5 > /tmp/env095/id

# using #project-zenith func
_EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger-engine-095 /tmp/env095/bin/dagger call -m github.com/shykes/daggerverse/hello shout

# if you need to stop engine
# docker stop `cat /tmp/env095/id`
# 
# start engine
docker run \
--name dagger-engine-096 \
--privileged \
--stop-signal SIGTERM \
-d \
-v dagger-engine-096:/var/lib/dagger \
registry.dagger.io/engine:v0.9.6 > /tmp/env096/id

# using #project-zenith func
_EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger-engine-096 /tmp/env096/bin/dagger call -m github.com/shykes/daggerverse/hello shout

# if you need to stop engine
# docker stop `cat /tmp/env096/id`
# 
# start engine
docker run \
--name dagger-engine-097 \
--privileged \
--stop-signal SIGTERM \
-d \
-v dagger-engine-097:/var/lib/dagger \
registry.dagger.io/engine:v0.9.7 > /tmp/env097/id

# using #project-zenith func
_EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger-engine-097 /tmp/env097/bin/dagger call -m github.com/shykes/daggerverse/hello shout

# if you need to stop engine
# docker stop `cat /tmp/env097/id`
whole rapids
# 
~ ➤ docker ps
CONTAINER ID   IMAGE                              COMMAND                  CREATED          STATUS          PORTS     NAMES
fc7dce9cc46b   registry.dagger.io/engine:v0.9.7   "dagger-entrypoint.sh"   2 seconds ago    Up 1 second               dagger-engine-097
e7f72049116e   registry.dagger.io/engine:v0.9.6   "dagger-entrypoint.sh"   24 seconds ago   Up 22 seconds             dagger-engine-096
ae7302fc35c9   registry.dagger.io/engine:v0.9.5   "dagger-entrypoint.s…"   32 seconds ago   Up 31 seconds             dagger-engine-095
# 
┃ HELLO, WORLD!!!!!!!
• Engine: ae7302fc35c9 (version v0.9.5)
⧗ 3.78s ✔ 41 ∅ 6

✔ dagger call shout [0.30s]
┃ HELLO, WORLD!!!!!!!
• Engine: e7f72049116e (version v0.9.6)
⧗ 30.11s ✔ 131

✔ dagger call shout [0.31s]
┃ HELLO, WORLD!!!!!!!
• Engine: fc7dce9cc46b (version v0.9.7)
⧗ 28.60s ✔ 95
tranquil oyster
#

That works thanks!

I'll see if I can raise a GitHub issue to make this a more proper feature request.