#v0.11.7 stopped working with iptables in legacy mode

1 messages · Page 1 of 1 (latest)

clear hound
#

Hey folks,

I just tried upgrading my Dagger playground to v0.11.7 and ran into an interesting issue with dagger init hanging indefinitely:

$ docker ps -a
CONTAINER ID   IMAGE                               COMMAND                  CREATED          STATUS                         PORTS     NAMES
5fe9bffc797e   registry.dagger.io/engine:v0.11.7   "dagger-entrypoint.s…"   17 seconds ago   Restarting (1) 2 seconds ago

A brief investigation showed that the engine container is in a crash loop because the iptables command fails:

...
dnsmasq[35]: using nameserver 1.1.1.1#53
dnsmasq[35]: using only locally-known addresses for dagger.local
dnsmasq[35]: read /etc/hosts - 18 names
dnsmasq[35]: read /var/run/containers/cni/dnsname/dagger/addnhosts - 0 names
time="2024-06-15T14:00:52Z" level=info msg="auto snapshotter: using overlayfs"
time="2024-06-15T14:00:52Z" level=warning msg="failed to release network namespace \"akq0o8xxuugnzhyux411vqg56\" left over from previous run: plugin type=\"loopback\" failed (delete): unknown FS magic on \"/var/lib/dagger/net/cni/akq0o8xxuugnzhyux411vqg56\": ef53"
time="2024-06-15T14:00:52Z" level=debug msg="shutting down otel pub/sub"
time="2024-06-15T14:00:52Z" level=debug msg="shutting down otel pub/sub"
buildkitd: failed to create worker: failed to create network providers: CNI setup error: plugin type="bridge" failed (add): running [/sbin/iptables -t nat -A POSTROUTING -s 10.87.0.11 -j CNI-fb0ef6dbec94aecf623cd11e -m comment --comment name: "dagger" id: "eeyq02izuj1a98shgcga38wwo" --wait]: exit status 4: iptables v1.8.10 (nf_tables):  CHAIN_ADD failed (Not supported): chain POSTROUTING

The host is freshly upgraded Ubuntu 22.04 with iptables in the legacy mode (and a custom kernel build 🙈 ):

$ iptables --version
iptables v1.8.7 (legacy)

You can easily repro it here https://labs.iximiuz.com/playgrounds/dagger - just upgrade the currently installed v0.11.6 to the latest v0.11.7.

iximiuz Labs
Dagger playground | iximiuz Labs

A 4-in-1 playground (Docker, Podman, containerd, and Kubernetes hosts) to experiment with Dagger - a powerful development workflow automation engine. Starts instantly. No paywall.

hazy pollen
#

hey @clear hound ! looks like the environment has a particular configuration on kernel + os version

#

I see that /etc/debian_version shows bookworm/sid but apt packages seem to be from ubuntu/jammy

#

on top of that, kernel version shows 5.10 when bookworm should have 6.1 by default?

#

I have debian bookworm stable in my machine and I'm not running into any issues

clear hound
# 
docker run ubuntu:22.04 cat /etc/debian_version
bookworm/sid

🙂

#

Honestly, idk why ubuntu image has this file 🙂 But it doesn't really matter. Something changed in 0.11.7 that made it use the nf_tables version of iptables.

hazy pollen
#

my impression is that it might be a combination of newer buildkit CNI version with the environment specific config

clear hound
#

The "problem" with my environment is that it doesn't support nf_tables

#

I don't know how many environments out there still default to the legacy version of iptables. But docker and vanialla containerd still seem to support this mode - just tried with the fresh versions.

hazy pollen
#

thx for reporting

clear hound
#

Also, just tried building an image with nerdctl (hence, buildkit), and it also works with the legacy version 🙂

hazy pollen
#

yes, thing is nerdctl and docker don't use buildkit's CNI feature

#

that's why you're very likely not seeing the error there

clear hound
#

Please let me know if you decide to support only the nf_tables mode - I'll finally have a good excuse to reconfigure the kernel and maybe even bump up the version 🙂

hazy pollen
#

I don't think that's the case since it'll very likely break for a lot of users

#

I think the issue might be that we bumped buildkit's default alpine image which might have changed the iptables version

clear hound
clear hound
hazy pollen
# 
cat: can't open '/etc/alpine_release': No such file or directory
root@docker:~# docker run --rm --entrypoint /bin/sh registry.dagger.io/engine:v0.11.6 -c "cat /etc/alpine-release"
3.18.6
root@docker:~# docker run --rm --entrypoint /bin/sh registry.dagger.io/engine:v0.11.6 -c "cat /etc/alpine-release && iptables --version"
3.18.6
iptables v1.8.9 (legacy)
root@docker:~# 
root@docker:~# 
root@docker:~# docker run --rm --entrypoint /bin/sh registry.dagger.io/engine:v0.11.7 -c "cat /etc/alpine-release && iptables --version"
3.20.0
iptables v1.8.10 (nf_tables)
root@docker:~# ```
#

yep, that's the case

#

will open a fix in a bit 🙏

clear hound
#

Could you tag me on the GH issue? I'll keep an eye on it and upgrade the dagger playground as soon as the fix released

hazy pollen
#

will do. Thx again for reporting 🙏

clear hound
#

Thanks for a great tool! And for the quick reply 🙂

#

Speaking of the fix, I remember there was a lengthy discussion in https://github.com/kubernetes/kubernetes/ on a very similar issue. Something along the lines of what kube-proxy should do if its iptables version differs from the host's version.

#

IIRC, the TL;DR is that it's not a trivial choice 🙂

hazy pollen
hazy pollen