DNS Resolution Failing | Dagger | Page 1

fallow tinsel Mar 7, 2024, 5:17 PM

#

I'm having some errors with java tests failing to resolve DNS and so trying to understand DNS resolution in dagger a little better.

I exec'd into my dagger engine, and then into the target container namespace via runc

I see the following in DNS settings.

root@buildkitsandbox:/src# cat /etc/resolv.conf
search tg71nu3c0flau.dagger.local buildkit.svc.cluster.local svc.cluster.local cluster.local us-west-2.compute.internal
nameserver 10.87.0.1
options ndots:5

root@buildkitsandbox:/src# cat /etc/hosts
127.0.0.1    localhost buildkitsandbox
::1    localhost ip6-localhost ip6-loopback

10.87.0.18    redis-cluster

10.87.0.20    db

10.87.0.19    localstack

I can resolve redis-cluster via ping, but not nslookup

root@buildkitsandbox:/src# ping redis-cluster
PING redis-cluster (10.87.0.18) 56(84) bytes of data.
64 bytes from redis-cluster (10.87.0.18): icmp_seq=1 ttl=127 time=0.034 ms
64 bytes from redis-cluster (10.87.0.18): icmp_seq=2 ttl=127 time=0.046 ms
64 bytes from redis-cluster (10.87.0.18): icmp_seq=3 ttl=127 time=0.032 ms
root@buildkitsandbox:/src# nslookup
> redis-cluster
Server:        10.87.0.1
Address:    10.87.0.1#53

** server can't find redis-cluster: NXDOMAIN

Why would nslookup not work in this case? This behavior mirrors the problems I'm seeing in my tests, some libraries work for resolution (database), but others do not (redis), indicating they are using different resolution methods somewhere.

A search in discord here shows some previous mentions to an expermental flag, but that appears missing from the current codebase in a search.

All that said, any ideas? Perhaps a team member could help provide some detail about the dns internals?

#

Minor context/progress, on a hunch I searched dnsmasq in the codebase and found this commit.

Trying an update to engine 0.10.1

GitHub

better resolv.conf/dnsmasq setup (#4658) · dagger/dagger@a379b81

Previously we had a kind of wonky dnsmasq + resolv.conf setup that
seemed to work in local testing but appears to cause problems in other
environments.

I haven't reproduced those i...

#

after update to 10.1 (both client and server) I see the same behavior

91: [265.6s]     redis.clients.jedis.exceptions.JedisConnectionException: Failed to connect to any host resolved for DNS name.

sharp shoal Mar 7, 2024, 5:39 PM

#

fallow tinsel I'm having some errors with java tests failing to resolve DNS and so trying to u...

nslookup only does direct DNS resolution, which is significantly different from the Name Service Switch subsystem that your other applications use; that is to say nslookup ignores /etc/hosts and mDNS.

#

AFAIK the java resolution system should use /etc/hosts also to resolve names

#

what's the part in your java test that's failing particularly?

#

doing a quick google search: https://gist.github.com/rajkrrsingh/b4cd654739425acabdd8c3086e00e6a1, Java default resolution system should do proper /etc/hosts lookup

fallow tinsel Mar 7, 2024, 5:42 PM

#

redis tests. frustratingly, I can connect via redis-cli

#

redis cluster instantiation looks like this FWIW

  companion object {
    private val redis by lazy {
      val hosts = "redis-cluster:30000,redis-cluster:30001,redis-cluster:30002,redis-cluster:30003,redis-cluster:30004,redis-cluster:30005"
      JedisCluster(
        hosts.toTestHosts().toHostPorts(), # just splits the string
        DefaultJedisClientConfig
          .builder()
          .build()
      )
    }
  }

sharp shoal Mar 7, 2024, 5:44 PM

#

fallow tinsel redis cluster instantiation looks like this FWIW ```kotlin companion object ...

JedisCluster.. I'm wondering which poor soul had to write that https://github.com/redis/jedis/commit/a8987ed865e1034f83d1e6a3ef681582ccb8a614 🙉

GitHub

Add first test to thorw MovedDataException when receiving MOVED fro...

…a cluster node

fallow tinsel Mar 7, 2024, 5:47 PM

#

hah! nice

#

If it helps, our current setup just runs docker-compose with a redis cluster, and that is able to resolve no issue

sharp shoal Mar 7, 2024, 5:50 PM

#

@fallow tinsel which Java docker image are you using?

fallow tinsel Mar 7, 2024, 5:50 PM

#

coretto 8

#

err.... its like debian:bullseye + coretto

#

root@d5995a003654:/src# java -version
openjdk version "1.8.0_402"
OpenJDK Runtime Environment Corretto-8.402.08.1 (build 1.8.0_402-b08)
OpenJDK 64-Bit Server VM Corretto-8.402.08.1 (build 25.402-b08, mixed mode)

sharp shoal Mar 7, 2024, 5:57 PM

#

strange.. error comes from here: https://github.com/redis/jedis/blob/ea76b84d1dfd5eada31468e166be1f47f93b86f7/src/main/java/redis/clients/jedis/DefaultJedisSocketFactory.java#L63 and we are effectively using getAllByName: https://github.com/redis/jedis/blob/ea76b84d1dfd5eada31468e166be1f47f93b86f7/src/main/java/redis/clients/jedis/DefaultJedisSocketFactory.java#L58
so it "should" work

GitHub

jedis/src/main/java/redis/clients/jedis/DefaultJedisSocketFactory.j...

Redis Java client. Contribute to redis/jedis development by creating an account on GitHub.

#

is the java docker image you're using public @fallow tinsel ?

fallow tinsel Mar 7, 2024, 6:01 PM

#

no :/

sharp shoal Mar 7, 2024, 6:02 PM

#

ok, I'll do a quick test with the same JDK version of a public image

#

and see what happens

fallow tinsel Mar 7, 2024, 6:02 PM

#

which jdk image are you gonna use I could try the same, just rebasing ours off that

sharp shoal Mar 7, 2024, 6:05 PM

#

was looking at this: https://hub.docker.com/_/amazoncorretto

amazoncorretto - Official Image | Docker Hub

Corretto is a no-cost, production-ready distribution of the Open Java Development Kit (OpenJDK).

#

they seem to be based on alpine though 😕

fallow tinsel Mar 7, 2024, 6:07 PM

#

damn.

before it's lost to my terminal history here's a little more context.. I found this issue which seems somewhat similar, and mentions /etc/nsswitch.conf was missing.. that wasn't the case for me, though I dont know if there's some misconfiguration with the content

root@buildkitsandbox:/src# vim /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         files
group:          files
shadow:         files
gshadow:        files

hosts:          files dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

sharp shoal Mar 7, 2024, 6:08 PM

#

hosts seems to be ok

#

I'll try with this image and see what happens: https://hub.docker.com/layers/library/amazoncorretto/8-al2023-jdk/images/sha256-1879611244fa9512018135c8e0ac096863e42fbdad5747945d641f152d85770f?context=explore

sharp shoal Mar 7, 2024, 6:34 PM

#

so @fallow tinsel this seems to work:

func main() {
    ctx := context.TODO()

    redis := dag.Container().From("redis").WithExposedPort(6379).AsService()

    dag.Container().From("amazoncorretto:8-al2023-jdk").
        WithWorkdir("/app").
        WithFile("DNSClient.java", dag.Host().File("DNSClient.java")).
        WithServiceBinding("redis", redis).
        WithExec([]string{"javac", "DNSClient.java"}).
        WithExec([]string{"java", "DNSClient", "redis"}).
        Stdout(ctx)
}

import java.net.InetAddress;
import java.net.UnknownHostException;

public class DNSClient {
    public static void main(String[] args) {
        InetAddress address = null;
        try {
            address = InetAddress.getByName(args[0]);
            System.out.println("host address: "+address.getHostAddress());
        } catch (UnknownHostException e) {
            throw new RuntimeException(e);
        }

    }
}

#

trying your image now

fallow tinsel Mar 7, 2024, 6:35 PM

#

is zenith going to behave different here somehow? I'm not using it

sharp shoal Mar 7, 2024, 6:35 PM

#

nope, it's the same codepath for service resolution

#

that code snippet seems to work with your image as well

#

interesting.. I'll try an actual JedisCluster example a bit later

fallow tinsel Mar 7, 2024, 8:22 PM

#

Here's a full reproduction - https://github.com/stobias123/dagger-dns-troubleshoot

fallow tinsel Mar 15, 2024, 5:41 PM

#

Hate to nag but were you able to try the reproduction?

#

I simplified it even and am still completely stumped. redis cli works fine, but jedis can't handle it

fallow tinsel Mar 15, 2024, 5:57 PM

#

yeah like I even just added a test to test dns resolution and pinging all ports successfully....

34: [54.6s] JedisTest > test redis-cluster IP resolution() STANDARD_OUT
34: [54.6s]     IP addresses for redis-cluster:
34: [54.6s]     10.87.0.81
34: [54.6s]     Ping result for port 30000: PONG
34: [54.6s] 
34: [54.6s]     Ping result for port 30001: PONG
34: [54.6s] 
34: [54.6s]     Ping result for port 30002: PONG
34: [54.6s] 
34: [54.6s]     Ping result for port 30003: PONG
34: [54.6s] 
34: [54.6s]     Ping result for port 30004: PONG
34: [54.6s] 
34: [54.6s]     Ping result for port 30005: PONG
.....

34: [54.6s] JedisTest > test a simple redis ping() STANDARD_OUT
34: [54.6s]     2024-03-15 17:54:50 DEBUG ConnectionFactory:62 - Error while makeObject
34: [54.6s]     redis.clients.jedis.exceptions.JedisConnectionException: Failed to connect to any host resolved for DNS name.

so must be something in jedis I guess, hard to know where to go from here.

perhaps the redis cluster is telling it to go to some node by hostname that isn't the service alias. edit: not sure this is it, using cluster mode on redis cli doesn't experience this behiavior

sharp shoal Mar 17, 2024, 11:23 PM

#

fallow tinsel yeah like I even just added a test to test dns resolution and pinging all ports ...

Hey Steven! I'll check out tomorrow. Between the latest releases and my Kubecon demo haven't had a lot of time to dig into this 🙏

fallow tinsel Mar 18, 2024, 7:02 PM

#

no worries - kind of half rubber ducking here hah. appreciate the help though

sharp shoal Jun 29, 2024, 4:38 PM

#

hey @fallow tinsel ! did you ever get to overcome this networking issue? catching up with old threads, I realized I never came back to this one 🙏

#DNS Resolution Failing