Quickstart DNS issue | Dagger | Page 1

sturdy heath Mar 18, 2023, 6:18 PM

#

I'm trying to test dagger but I'm stuck with this error:

go run ./main.go
#1 resolve image config for docker.io/library/golang:1.19
#1 ERROR: failed to do request: Head "https://registry-1.docker.io/v2/library/golang/manifests/1.19": dial tcp: lookup registry-1.docker.io on 10.87.0.1:53: server misbehaving
------
 > resolve image config for docker.io/library/golang:1.19:
------
panic: input:1: container.from failed to do request: Head "https://registry-1.docker.io/v2/library/golang/manifests/1.19": dial tcp: lookup registry-1.docker.io on 10.87.0.1:53: server misbehaving

Please visit https://dagger.io/help#go for troubleshooting guidance.

goroutine 1 [running]:
main.main()
        /home/XX/hello-dagger/ci/main.go:28 +0x3e9
exit status 2

I don't understand why the container is using this dns address, I tried to docker system prune as I found this address in /var/lib/docker/overlay2 path with no improvement...

Has anyone encountered this issue already?

#

docker version 23.0.1
/etc/resolv.conf :

# Generated by NetworkManager
nameserver 1.1.1.1
nameserver 8.8.8.8

/etc/hosts

# Host addresses
127.0.0.1  localhost
127.0.1.1  mycomputer
::1        localhost ip6-localhost ip6-loopback
ff02::1    ip6-allnodes
ff02::2    ip6-allrouters

fallen kindle Mar 18, 2023, 9:21 PM

#

👋 can you check if setting _EXPERIMENTAL_DAGGER_SERVICES_DNS=0 as an env variable work?

neon beacon Mar 20, 2023, 10:47 PM

#

Running into this too, @fallen kindle this env var does not seem to change anything

#

cc @wide pawn

#

using v0.5.1 and brand new buildkit container

fallen kindle Mar 20, 2023, 10:49 PM

#

cc @strange lotus

neon beacon Mar 20, 2023, 10:57 PM

#

looks like if i delete the buildkit container again, export the env var, and then run dagger it creates it with the variable set and future dagger commands work

strange lotus Mar 20, 2023, 11:04 PM

#

that's the intended method for setting it 👍

not sure why it didn't work with services DNS though. hmm

#

if you're willing to flip the env var back on and troubleshoot, can you cat /etc/resolv.conf and /etc/resolv.conf.upstream in the engine container?

neon beacon Mar 20, 2023, 11:07 PM

#

/etc/resolv.conf contains

/ # cat /etc/resolv.conf
# dagger dnsmasq server
nameserver 10.87.0.1

and resolv.conf.upstream is my host's resolv.conf

#

which has

nameserver 10.200.128.5

which is my local network's dns server

#

(it also runs dnsmasq, if that is of interest)

strange lotus Mar 20, 2023, 11:09 PM

#

interesting. is 10.200.128.5 reachable from the engine container? (nslookup google.com 10.200.128.5)

neon beacon Mar 20, 2023, 11:09 PM

#

it is not

#

/ # nslookup 10.200.128.5
Server:         10.87.0.1
Address:        10.87.0.1:53

** server can't find 5.128.200.10.in-addr.arpa: REFUSED

#

oh sorry,

#

/ # nslookup google.com 10.200.128.5
Server:         10.200.128.5
Address:        10.200.128.5:53

Non-authoritative answer:
Name:   google.com
Address: 142.250.65.174

Non-authoritative answer:
Name:   google.com
Address: 2607:f8b0:4006:80e::200e

yes it is

strange lotus Mar 20, 2023, 11:11 PM

#

hm interesting, let me see if dnsmasq is logging anywhere useful

neon beacon Mar 20, 2023, 11:11 PM

#

I dont see anything in /var/log

strange lotus Mar 20, 2023, 11:11 PM

#

and what does nslookup google.com say? (trying to use the 10.87 DNS here)

neon beacon Mar 20, 2023, 11:12 PM

#


# nslookup -debug google.com 
Server:         10.87.0.1
Address:        10.87.0.1:53

Query #0 completed in 0ms:
** server can't find google.com: REFUSED

Query #1 completed in 0ms:
** server can't find google.com: REFUSED

#

sorry lol pasted the wrong one

strange lotus Mar 20, 2023, 11:12 PM

#

lol np. hmm...

#

welp, time for the big guns: apk add strace; strace -f -p $dnsmasq_pid &; nslookup google.com

neon beacon Mar 20, 2023, 11:14 PM

#

unfortunately apk add runs into a similar issue... 😆

strange lotus Mar 20, 2023, 11:14 PM

#

oh, true

neon beacon Mar 20, 2023, 11:14 PM

#

classic dns

strange lotus Mar 20, 2023, 11:14 PM

#

is there anything else in /etc/resolv.conf.upstream? or is it literally just that nameserver line?

neon beacon Mar 20, 2023, 11:15 PM

#

just a search line

strange lotus Mar 20, 2023, 11:17 PM

#

ok, new idea: pkill dnsmasq; /usr/sbin/dnsmasq -u root --conf-file=/run/containers/cni/dnsname/dagger/dnsmasq.conf -d --log-debug &

#

and then try another nslookup

neon beacon Mar 20, 2023, 11:20 PM

#

/var/log # pkill dnsmasq; /usr/sbin/dnsmasq -u root --conf-file=/run/containers/cni/dnsname/dagger/dnsmasq.conf -d --log-debug &
/var/log # dnsmasq: started, version 2.87 cachesize 150
dnsmasq: compile time options: IPv6 GNU-getopt no-DBus no-UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset no-nftset auth no-cryptohash no-DNSSEC loop-detect inotify dumpfile
dnsmasq: using only locally-known addresses for dagger.local
dnsmasq: failed to read /etc/resolv.conf.upstream: Permission denied
dnsmasq: no servers found in /etc/resolv.conf.upstream, will retry
dnsmasq: read /run/containers/cni/dnsname/dagger/addnhosts - 16 addresses

🤔

/var/log # ls -al /etc/resolv.conf.upstream
-rw-r--r--    1 root     root            82 Mar 20 23:06 /etc/resolv.conf.upstream
/var/log # whoami
root

strange lotus Mar 20, 2023, 11:21 PM

#

not_sure_if

#

...selinux? apparmor?

neon beacon Mar 20, 2023, 11:22 PM

#

no selinux, but yeah it's likely apparmor

#

i suppose the container needs a cap

strange lotus Mar 20, 2023, 11:23 PM

#

yeah maybe. we do a kind of funny thing with that file, it's originally mounted in to /etc/resolv.conf via Docker and then we "move" the mount to a new path by bind-mounting it again and unmounting the original path. (which apparently works.)

#

not sure if that matters

#

i mean, you can cat it, so it's kind of odd that dnsmasq can't read it

#

but i'm not sure what apparmor's rules are

neon beacon Mar 20, 2023, 11:24 PM

#

yeah interesting that i can cat it

#

it is possible that dnsmasq is forking itself and dropping permissions

strange lotus Mar 20, 2023, 11:28 PM

#

yeah. it should be running as root though, and /proc/<pid>/status seems to have the same Cap* 🤔

#

i don't know much about apparmor, is there anything you can do on your side to troubleshoot there?

neon beacon Mar 20, 2023, 11:29 PM

#

oh, yeah there we go

[26303.838148] audit: type=1400 audit(1679354910.133:934): apparmor="DENIED" operation="open" profile="dnsmasq" name="/etc/resolv.conf.upstream" pid=3018223 comm="dnsmasq" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

strange lotus Mar 20, 2023, 11:30 PM

#

nice

#

where does this get enforced? i'm guessing outside of the container at the kernel level? is there a host-side config that lets it reach /etc/resolv.conf or something, and the paths just match up inside/outside?

#

oh, i guess we don't even run dnsmasq with services DNS disabled (so disabling services isn't evidence of /etc/resolv.conf even being allowed)

neon beacon Mar 20, 2023, 11:34 PM

#

i will have to look this up

strange lotus Mar 20, 2023, 11:35 PM

#

i guess apparmor is configured to care about execve(/usr/sbin/dnsmasq) or something, and the rule fires even for containers, not caring about mount namespaces since it's enforced at the kernel level? you mentioned using dnsmasq on the host too so that would explain why those rules might be configured

neon beacon Mar 20, 2023, 11:36 PM

#

looks like there is a policy specifically for /usr/sbin/dnsmasq that restricts the files it can read

#

yeah there are several dnsmasqs, on the host there's NetworkManager

strange lotus Mar 20, 2023, 11:36 PM

#

interesting, that's annoying. i mean, we could rename it to try to dodge apparmor rules 😂

neon beacon Mar 20, 2023, 11:36 PM

#

lolol

#

there are a few files/directories allowed such as

 /etc/dnsmasq.d/ r,                       
 /etc/dnsmasq.d/* r,                      
 /etc/dnsmasq.d-available/ r,             
 /etc/dnsmasq.d-available/* r,            
 /etc/ethers r,                           
 /etc/NetworkManager/dnsmasq.d/ r,        
 /etc/NetworkManager/dnsmasq.d/* r,       
 /etc/NetworkManager/dnsmasq-shared.d/ r, 
 /etc/NetworkManager/dnsmasq-shared.d/* r,
 /etc/dnsmasq-conf.conf r,                
 /etc/dnsmasq-resolv.conf r,

strange lotus Mar 20, 2023, 11:39 PM

#

makes sense. wonder where that's configured, maybe just package distribution defaults?

neon beacon Mar 20, 2023, 11:40 PM

#

yup default from arch

strange lotus Mar 20, 2023, 11:40 PM

#

i guess we could use one of those filenames, depending on how widespread those defaults are

neon beacon Mar 20, 2023, 11:41 PM

#

those are going to be the defaults everywhere

#

https://gitlab.com/apparmor/apparmor/-/blob/master/profiles/apparmor.d/usr.sbin.dnsmasq

GitLab

profiles/apparmor.d/usr.sbin.dnsmasq · master · AppArmor / apparmor...

The AppArmor user space development project.

strange lotus Mar 20, 2023, 11:42 PM

#

oh nice

#

opened https://github.com/dagger/dagger/pull/4799 for this, thanks for troubleshooting! 👍

neon beacon Mar 20, 2023, 11:48 PM

#

np!

sturdy heath Mar 22, 2023, 1:43 PM

#

Thank you guys

fallen kindle Mar 22, 2023, 1:48 PM

#

closing this

#Quickstart DNS issue