Networking between containers stopped working in 0.18.17

[half summit]

I recently tried upgrading dagger and noticed my pipeline stopped working. Specifically the containers that I launch no longer can network between each other.

I've tested that the last version that is still working is 0.18.16.

I launch 2-3 containers and I manually assign them DNS names by writing out each of their /etc/hosts files. I basically have a process that looks at the initial /etc/hosts file to discover the containers assigned IP address, and then update each /etc/hosts file so that all containers know the DNS of each other.

This initial discovery process still works fine, but it appears that networking between the containers is just broken.

On 0.18.17 I see error messages like this:

lookup ujfvap1713d22.cen23482dp2bg.dagger.local on 10.87.0.1:53: no such host
lookup ujfvap1713d22.ave6bb9s0j7m4.cen23482dp2bg.dagger.local on 10.87.0.1:53: no such host

https://dagger.cloud/conor/traces/1d3df6d4a3fdca58448fd2607cd6b4c1?span=349e0d678a857952

When trying on 0.19.6, it seems it gets farther & DNS works, but the networking just fails.

dial tcp 10.87.0.80:26656: i/o timeout

(sorry this one is pretty noisy)
https://dagger.cloud/cordialsys/traces/7dc112f62f3a277410c24b21de5c52a9?span=cfb7f2344f703c27#cfb7f2344f703c27:L144

Is there a better way that I can ensure networking between containers, or can test this issue?

[hollow osprey]

@half summit mind sharing what's the main reason of doing this manual /etc/hosts dancing and not using either the "WithServiceBinding" or "WithHostname" native service discovery mechanisms?

manually fiddling with the containers /etc/hosts file is not something that we officially support so it's hard to know what could be broken there.

[half summit]

Before I tried to use services but it did not work due to cyclic dependency (multiple containers need to reach each other for p2p networking).

Perhaps this is no longer the case, I can try again.

These containers are also not services in a strict sense - they are what I'm testing and P2P networking is a part of it. I'm regularly exec'ing commands on them to progress through tests.

Can I create a service and still exec commands on it while it's running?

[hollow osprey]

the cyclic dependency part as been addressed with the new WithHostname call which allows you to set a global name to your services without the WithServiceBinding deadlock.

in regards to execute commands in a service while it's running no, it's not possible but you also need to be carful while doing this with the container.WithExec API because each WithExec starts a new container and you can't have guarantess that it'll re-use the same IP that it used the last time

so if you do something like:

ctr := dag.Container()
ctr.WithExec([]string{"hostname", "-I")
ctr.WithExec([]string{"hostname", "-I")

^ taking aside the caching aspects, we don't guarantee that both of those WithExecs will yield to the same IP

so I'm a bit curious about how you're doing that today in your /etc/hosts workflow

[half summit]

yeah it's a bit tedious, but every exec I have a script that syncronously re-discovers the IPs for /etc/hosts everytime. I think i'll have to try to re-write using services.

[hollow osprey]

if there's any way we can help by supporting this use-case better, happy to chat about it.

If by any chance you could share a simple example of what you're trying to achieve and where services fall short, that will probably trigger some fruitful conversations about supporting the use-case better

we're all about making Dagger the best workflow/test runner out there so there's surely something we can do to make you a happier user 😄

[half summit]

thanks that's appreciated. what i did was pretty complex, so i will try to re-architect based on services before i try to explain it haha.