Docker can not pull images until `docker kill dagger-engine...`

[thorny ore]

#1 resolve image config for docker.io/library/alpine:latest
#1 ERROR: failed to do request: Head "https://registry-1.docker.io/v2/library/alpine/manifests/latest": dial tcp: lookup registry-1.docker.io on 10.87.0.1:53: read udp 10.87.0.1:52572->10.87.0.1:53: i/o timeout
2023/08/08 08:39:45 input:1: container.from failed to do request: Head "https://registry-1.docker.io/v2/library/alpine/manifests/latest": dial tcp: lookup registry-1.docker.io on 10.87.0.1:53: read udp 10.87.0.1:52572->10.87.0.1:53: i/o timeout

dagger-engine log-output attached.

Once I kill the dagger-engine container, and re-run the build it works as expected.

My suspicion is that on a desktop, which is regularly rebooted, and is using a vpn, the networking changes in ways where the network interfaces are not as the container created previously expects.

Just saw i had the dagger package version 0.7, will upgrade to 0.8 and test again.

With 0.8 it does not properly use the dns for the vpn I am using, causing resolution of ORG dns not to work.

Due to internal names I can not share the full error:

ERROR: failed to fetch remote https://...........git: git error: exit status 128
stderr:
fatal: unable to access 'https://..........git/': Could not resolve host: org.domain.com

Engine starts with the wrong dns, which won't work within my org:

dnsmasq[40]: started, version 2.89 cachesize 150
dnsmasq[40]: compile time options: IPv6 GNU-getopt no-DBus no-UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset no-nftset auth no-cryptohash no-DNSSEC loop-detect inotify dumpfile
dnsmasq[40]: warning: interface dagger0 does not currently exist
dnsmasq[40]: using only locally-known addresses for dagger.local
dnsmasq[40]: reading /etc/dnsmasq-resolv.conf
dnsmasq[40]: using nameserver 212.186.211.21#53
dnsmasq[40]: using nameserver 195.58.161.123#53
dnsmasq[40]: using only locally-known addresses for dagger.local
time="2023-08-08T07:04:36Z" level=info msg="detected mtu 1500 via interface eth0"

killing the engine and starting a new build works.

docker kill dagger-engine...
docker rm dagger-engine...

[distant dew]

👋 @thorny ore have you found a deterministic way to repro this? Can you provide more information about your current environment? My first impression is that this is very likely not to be Dagger but Docker related.

The Dagger daemon is automatically configured to autostart once your machine reboots. So what I supose it's very likely happening is that the engine gets started before your VPN setup is established which casues Docker to break the network configuration probably

[distant dew]

after reading this message this seems to be the case. This is related to this issue / discussion: https://github.com/dagger/dagger/issues/5583.

Already updated the issue to reflect this scenario

[thorny ore]

I am using fedora 38, docker, and the GoLang SDK V0.8.1. sadly I can not give a easy way to reproduce this issue at the moment, just to reboot my machine. 😄
Maybe I can think of something to reproduce this in a vm context I could provide.

docker version
Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:37:15 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:40 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[distant dew]

@thorny ore can you check if this reproduces the issue?

1. Run any Dagger pipeline (which starts the engine)
2. Make sure you're not connected to the VPN
3. Restart the docker service (systemctl restart docker?)
4. Re-connect to the VPN
5. Try running Dagger again. (I'd expect it to fail).

[thorny ore]

I will try when I have some time, currently I had to finish a pipeline using dagger, and am running in many issues.
Maybe have jumped the shark going with dagger, without more experience.

Will report back when I have more Information.

[distant dew]

don't hesitate to ping if we can help out in any way. Happy to jump into a quick #911305510882513037 session to speed things up a bit if required. cc @north fox

[thorny ore]

Someone giving it a quick look would be very much appreciated, maybe I am just missing something.

[distant dew]

@thorny ore I'm around if you can chat for a bit now

[thorny ore]

That would be great, I've got most of it working but would like some feedback

[distant dew]

2 minutes and I'm there 🙏

ok, free now. Going to #911305510882513037

[distant dew]

just showed René some nsenter foo to get unblocked to troubleshoot his pipelines 💪

[thorny ore]

thanks a lot for that, that makes debugging much easier.
i went as far as to use ngrok to tunnel outside 🙂

[distant dew]

anytime!

[thorny ore]

sorry for the very late response to your test instructions.

I've now tested it and as you expected it it fails.
after disconnecting and restarting docker + vpn the dagger-engine container can does not use the nameservers given from the vpn.

[distant dew]

ok, this makes sense in that case. It's not a Dagger error, but a "common" problem that all containers have which dns gets resolved at container creation time

so the "fix" here is to make sure you remove your dagger engine container after connecting to the VPN

so you're sure that next time a Dagger pipeline runs, the engine will be re-started with the correct config

same thing should happen when disconnecting from the VPN as well

[thorny ore]

yep, will add this to my connect script.

thanks for the assistance 🙂

[distant dew]

sure! np!

cc @rare arch where would it be the right place to document this?

in the FAQ maybe?

[rare arch]

I guess here: https://docs.dagger.io/235290/troubleshooting

However can you clarify the "problem"? Is it simply "DNS errors with Dagger" or how would you describe it?

[thorny ore]

Dagger Engine, or any docker-containers, won't get notified if the network-configuration of the host changes, including nameservers, which the docker-container gets set upon creation. If you for example move between connections, locations, vpns which cause dns changes these won't update containers.

I duno, maybe that is something the dagger-cli could check for and throw a "warning" that there is a dns-mismatch

[rare arch]

FYI https://github.com/dagger/dagger/pull/5717

[distant dew]

@thorny ore if you have some time, can you validate if docker restart $dagger_engine_id also fixes the issue?

[thorny ore]

sure

[distant dew]

bringing this up since @rare arch's PR mentions stopping and "rmming" the engine and I think a restart might be faster / easier

[rare arch]

Thanks @distant dew

[thorny ore]

jup restart does work

and it is much faster 🙂

[distant dew]

awesome, thx 🙏