NixOS + dagger + podman

[gaunt shoal]

I am on NixOS, and I wanted to do some local development work on my PC; installed dagger using the flake as input into my devshell. I don't use docker, but podman on my PC.

dagger fails to initialize because I don't have docker installed:

    ╰─✘ moby.buildkit.v1.Control/Info 0.0s ERROR
      ! connection error: desc = "error reading server preface: command [docker exec -i dagger-engine-v0.18.18
        buildctl dial-stdio] has exited with exit status 1, make sure the URL is valid, and Docker 18.09 or later is
        installed on the remote host: stderr=Error: dial unix /run/dagger/engine.sock: connect: no such file or
        directory\nUsage:\n  dial-stdio [flags]\n\nFlags:\n      --addr string   The address serving the grpc api
        (default \"unix:///run/dagger/engine.sock\")\n  -h, --help          help for dial-stdio\n      --timeout int
        The timeout in seconds for connecting to the grpc api (default 5)\n\n"

I did find this page for setting up podman, but this is not helpful in NixOS.

[gaunt shoal]

Created an issue: https://github.com/dagger/nix/issues/31

[sacred moss]

Thanks for opening the issue! Your config looked correct to me. Maybe @iron palm has a better idea?

[midnight thicket]

@gaunt shoal can you check if by any chance your dagger engine failed to start? If you run docker ps -a (or podman's equivalent command) do you see an exited dagger engine container?

If so, running docker logs $ctr should show more information about what could have happened

[gaunt shoal]

Thanks all! You are very constructive, deeply appreciated! @midnight thicket, splendid idea in troubleshooting: https://github.com/dagger/nix/issues/31#issuecomment-3303592496

[midnight thicket]

yep, that's what I thought. You probably need to do this as well: https://docs.dagger.io/reference/troubleshooting/#dagger-restarts-with-a-cni-setup-error

[gaunt shoal]

Great!

On NixOS this is just: boot.kernelModules = [ "iptable_nat" ];

Will test now, let's see

[midnight thicket]

issue renamed

[gaunt shoal]

iptables have been removed from nixos, boot.kernelModules = ["nf_nat"]; is the new way

but this doesn't work for me @midnight thicket

https://github.com/dagger/dagger/issues/11694

[frosty prism]

just FYI I sent out a new fix here: https://github.com/dagger/dagger/pull/11731

[midnight thicket]

testing this really quick in a >= 6.17 kernel image

[midnight thicket]

it doesn't seem to work Erik. Installing ip6tables makes iptables fallback to the legacy version . You can check with iptables -V it should print (nf_tables) when it's working with nftables.

I just did a quick check and iptables-nft effectively installs the ip6tables binary with the nft variant. So apparently this could be some sort of performance regression when the iptables-nft packages works in legacy mode somehow

[frosty prism]

Thanks for testing! I have a timebox with this but I'll poke a little bit more

[frosty prism]

I pushed another attempt (details here). If you still have the test setup for 6.17, could you give it another try when you have time?

[midnight thicket]

sure, in a bit

[midnight thicket]

still fails in 6.17 when the engine initializes Erik

it's correctly detecting the nftables backend but seems like the CNI driver is still trying to set some things via iptables-legacy which fail?

in this case it seems like it's not the nat table anymore but the filter one 🤔

[frosty prism]

oh that's the firewall plugin, which is separate from bridge, I bet it just needs to be configured similar to bridge

[midnight thicket]

https://github.com/containernetworking/plugins/blob/main/plugins/meta/firewall/firewall.go#L121 🤔

[frosty prism]

Nah, that would have been way too easy

[midnight thicket]

doesn't seem it supports nft?

[frosty prism]

Yep, it's a nightmare

What we'd need to do is install iptables-nft, so then it can call to iptables but actually hit nftables

But then we go back to the perf regression

[midnight thicket]

lol, this. I was about to type that

[frosty prism]

It's definitely possible to get it all working with the right symlinks setup in the right places

[midnight thicket]

it's a bummer they don't allow setting the iptables binary name so that way we could have both installed

[midnight thicket]

but we'd have to mangle those at the engine startup?

[frosty prism]

Ya, which is not the end of the world

That's essentially what this does I think: https://github.com/kubernetes-sigs/iptables-wrappers/tree/master

But we can probably get away with something a little simpler iiuc

[midnight thicket]

https://tenor.com/bWQSW.gif

[frosty prism]

I hit my timebox on this for today, I'll revisit again tomorrow

[midnight thicket]

yeah, same. I'll add a comment to the issue for visibility

[frosty prism]

Yeah the fact that xtables is disabled by default on 6.17+ seems like quite the iceberg we're all heading directly towards

[midnight thicket]

https://tenor.com/gJXmmYrDhWB.gif

[frosty prism]

I already started typing this out

if you want to save time

[midnight thicket]

sweet, thx 🙏

[frosty prism]

@midnight thicket were you just using latest arch to test this? I'll set it up quick so we don't have to ping-pong again

[midnight thicket]

yep, I downloaded their qcow2 image and started it with libvirt locally

https://gitlab.archlinux.org/archlinux/arch-boxes/-/package_files/10897/download

[frosty prism]

The deep irony of the fact that in my attempts to test this I can't even start dockerd due to:

[root@alarm ~]# /usr/bin/dockerd
/usr/bin/dockerd: /usr/lib/libc.so.6: version `GLIBC_2.38' not found (required by /usr/lib/libnftables.so.1

[frosty prism]

Sorry to ping again but when you have time can you give it a shot with whatever setup you had? I finally got one installed and actually working but then discovered the kernel was actually built with xtables enabled still? It's on linux 6.18 but there's still a kernel compile option to enable it.

I have to use ARM arch, so might be a difference there. I have no clue though and don't want to lose the rest of the day to this nonsense.

It's not rush at all. I built a container off my PR and pushed it, so you should only need to set this env var: export _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-image://eriksipsma/dagger-test:test

[midnight thicket]

sorry, missed this

[frosty prism]

no problem, I was still trying at that point, before I lost all hope

[midnight thicket]

sure, I'll check it in a bit 🙏

[midnight thicket]

@frosty prism is there a way you can push en x86 verison of this engine "relatively" quickly?

[frosty prism]

oh shit sorry, that makes sense

[midnight thicket]

I have to build it from scratch from a cold cache and it might take a bit here

[frosty prism]

Yeah I can do it quick

[midnight thicket]

since I'm on a slowish wifi ATM

[frosty prism]

pushed

[midnight thicket]

ref?

[frosty prism]

same one

_EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-image://eriksipsma/dagger-test:test

[midnight thicket]

woot, missed it above

checking now

all good Erik!

I'll go ahead and approve the PR

[frosty prism]

https://klipy.com/gifs/gerald-ford-president-ford

[frosty prism]

Thank you!!