HELP- Dagger Engine failure | Dagger | Page 1

surreal trout Oct 9, 2025, 12:41 AM

#

Seems to never connect and when I abend it shows
moby.buildkit.v1.Control/Info 0.0s ERROR : can only create exec sessions on running containers: container state improper?

I have had it running previously along side K0s as standalone client and within the cluster , per install instructions .

Starting having issues sever updgrades ago?

PODMAN
Client: Podman Engine
Version: 4.9.3
API Version: 4.9.3
Go Version: go1.22.2
Built: Wed Dec 31 17:00:00 1969
OS/Arch: linux/amd64

K0s
sudo k0s status
Version: v1.33.4+k0s.0
Process ID: 2175
Role: controller
Workloads: true
SingleNode: true
Kube-api probing successful: true
Kube-api probing last error:

obtuse oasis Oct 9, 2025, 12:51 AM

#

Hi 👋
Can you please explain me how I can repro from scratch 👀🙏

cloud drift Oct 9, 2025, 2:22 PM

#

@surreal trout I have a strong opinion that it probably has to do with some podman configuration. If you run docker/podman ps -a you should see the engine container in a stopped and exited state

cloud drift Oct 9, 2025, 6:22 PM

#

would you mind running docker logs on the failed engine container and posting the output here please?

surreal trout Oct 10, 2025, 4:22 PM

#

from what i see, with the prompt from your question, it doesn't appear to be interacting well with crun, see attached

#

Nothing appears when executing docker/podman logs 57473577ec37

cloud drift Oct 10, 2025, 5:43 PM

#

surreal trout from what i see, with the prompt from your question, it doesn't appear to be int...

as you can see from the logs, when dagger starts it tries to start or create the engine container with docker run or docker start. Seems like something in your system with podman is prevening the engine to start successfuly.

Have you checked this guide? https://docs.dagger.io/reference/container-runtimes/podman/

Podman | Dagger

Learn how to use Dagger with Podman.

#

I also see that even though you're using podman, dagger is trying to start the engine by using the docker command. Do you have some sort of symlink from docker to podman or something?

#

could you try removing that?

surreal trout Oct 10, 2025, 6:12 PM

#

OK, I changed things up realizing Im working against myself. I may revisit the guide above (thank you) however now i have removed podman altogether and installed docker.
Client: Docker Engine - Community
Version: 28.5.1
API version: 1.51
Go version: go1.24.8
Git commit: e180ab8
Built: Wed Oct 8 12:17:26 2025
OS/Arch: linux/amd64
Context: rootless

I suspect "rootless" may be an issue, however, I was prompted for sudo priv's but I noticed it seemed to imply removing a container?

in any case I seem to be have OCI shim issues now? See Attached

#

#

Failed to mention I removed dagger and reinstalled also after the change from PM to Docker

cloud drift Oct 10, 2025, 6:21 PM

#

surreal trout

Thx for sharing. Mind if I ask under which setup you're running this?

Which linux distro are you using?

surreal trout Oct 10, 2025, 6:22 PM

#

Ubuntu 24.04

cloud drift Oct 10, 2025, 6:22 PM

#

surreal trout OK, I changed things up realizing Im working against myself. I may revisit the ...

Rootless won't work for sure as mentioned in our docs. That's likely the issue you're hitting now

surreal trout Oct 10, 2025, 6:23 PM

#

Im not even sure how that happened, lol

cloud drift Oct 10, 2025, 6:23 PM

#

surreal trout Im not even sure how that happened, lol

Lol yeah.. strange. I'm using Ubuntu 24.04 myself and works fine here with docker

#

Maybe you accidentally installed the rootless package?

surreal trout Oct 10, 2025, 6:25 PM

#

I'll revisit

surreal trout Oct 10, 2025, 7:57 PM

#

THINK I GOT IT - So many layers in the onion here, lol - I had some stuff lingering around but I believe I now have a clear slate with my machine now in rootful mode

Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /home/n30:/home/n30

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:

    podman machine set --rootful

API forwarding listening on: /home/n30/.local/share/containers/podman/machine/qemu/podman.sock
You can connect Docker API clients by setting DOCKER_HOST using the
following command in your terminal session:

    export DOCKER_HOST='unix:///home/n30/.local/share/containers/podman/machine/qemu/podman.sock'

Machine "podman-machine-default" started successfully
n30@gri ~/proj> [13:48:56]>
n30@gri ~/proj> [13:49:59]> podman machine set --rootful
Error: cannot change settings while the vm is running, run 'podman machine stop' first
n30@gri ~/proj> [13:50:23]> podman machine stop
Waiting for VM to exit...
Machine "podman-machine-default" stopped successfully
n30@gri ~/proj> [13:50:53]> podman machine set --rootful
n30@gri ~/proj> [13:50:58]>
n30@gri ~/proj> [13:51:01]> podman machine start
Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /home/n30:/home/n30
API forwarding listening on: /home/n30/.local/share/containers/podman/machine/qemu/podman.sock
You can connect Docker API clients by setting DOCKER_HOST using the
following command in your terminal session:

    export DOCKER_HOST='unix:///home/n30/.local/share/containers/podman/machine/qemu/podman.sock'

Machine "podman-machine-default" started successfully

surreal trout Oct 10, 2025, 8:15 PM

#

NOPE still no success

#

surreal trout Oct 10, 2025, 8:49 PM

#

FINALLY !!!!
K0s running a standalone local lightweight master/worker
Docker v27.5.1

#

Dagger v0.19.2

#

For my next TRICK .... Dagger running in the Cluster - Stay Tuned

cloud drift Oct 10, 2025, 11:28 PM

#

Nice!

#

What was the Dagger issue?

#

Anything we could add to the docs to improve this?

#

@surreal trout

surreal trout Oct 11, 2025, 8:22 PM

#

Im not really sure, most likely a biological user interface error, lol - on my part. I have had a lot of config changes across different versions while trying different CNI implementations along with Podman -desktop etc. Podman didnt even really seem to go into rootfull mode when set as such so I think I had some layered things going on that never really cleared things out . I followed these instructions for a total docker purge and when I substituted podman found all kinds of things lingering around. I think I had two podman setups one as root and another as a user In any case I have things working with Docker now. Soon I will be introducing dagger into a setup with 9 nodes and 3 controllers and working on integrating Fortify tools in a Dagger pipeline . May also use a gitlab runner but I would prefer not to if possible .

cloud drift Oct 12, 2025, 3:16 PM

#

surreal trout Im not really sure, most likely a biological user interface error, lol - on my ...

Sweet, awesome!

We're around if you have any questions!

south quiver Oct 14, 2025, 5:16 PM

#

I'm having issues with the latest release on NixOS.

exec /usr/local/bin/dagger-engine --config /etc/dagger/engine.toml "$@"
time="2025-10-14T17:14:55Z" level=info msg="detected mtu 1500 via interface eth0"
time="2025-10-14T17:14:55Z" level=debug msg="engine name: 3d0b3a4b1f8e"
time="2025-10-14T17:14:55Z" level=debug msg="creating engine GRPC server"
time="2025-10-14T17:14:55Z" level=debug msg="creating engine lockfile"
time="2025-10-14T17:14:55Z" level=debug msg="creating engine server"
dnsmasq[40]: started, version 2.91 cachesize 150
dnsmasq[40]: compile time options: IPv6 GNU-getopt no-DBus no-UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset no-nftset auth no-DNSSEC loop-detect inotify dumpfile
dnsmasq[40]: warning: interface dagger0 does not currently exist
dnsmasq[40]: using only locally-known addresses for dagger.local
dnsmasq[40]: reading /etc/dnsmasq-resolv.conf
dnsmasq[40]: using nameserver 100.100.100.100#53
dnsmasq[40]: using only locally-known addresses for dagger.local
dnsmasq[40]: read /etc/hosts - 18 names
dnsmasq[40]: read /var/run/containers/cni/dnsname/dagger/addnhosts - 0 names
time="2025-10-14T17:14:55Z" level=info msg="auto snapshotter: using overlayfs"
time="2025-10-14T17:14:55Z" level=warning msg="failed to release network namespace \"yiddtgdnakd66jsaxmle4ao3s\" left over from previous run: plugin type=\"loopback\" failed (delete): unknown FS magic on \"/var/lib/dagger/net/cni/yiddtgdnakd66jsaxmle4ao3s\": 9123683e"
dagger-engine: failed to create engine: failed to create network providers: CNI setup error: plugin type="bridge" failed (add): failed to list chains: running [/usr/local/sbin/iptables -t nat -S --wait]: exit status 3: iptables v1.8.11 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

#

The engine constantly restarts now

cloud drift Oct 14, 2025, 7:27 PM

#

south quiver I'm having issues with the latest release on NixOS. ``` exec /usr/local/bin/dag...

hey Ed! seems related to: https://docs.dagger.io/reference/troubleshooting/#dagger-restarts-with-a-cni-setup-error

Troubleshooting | Dagger

Learn how to troubleshoot common issues with Dagger, including unresponsive functions, network errors, and more.

south quiver Oct 15, 2025, 10:53 AM

#

cloud drift hey Ed! seems related to: https://docs.dagger.io/reference/troubleshooting/#dagg...

Thanks. My NixOS system has switched over fully to nftables so can't even get the modules for iptables installed unfortunately...