#Dagger Service Port Mapping

1 messages · Page 1 of 1 (latest)

azure ivy
#

Trying to write a utility to consume a docker-compose file, and inject as dagger service bindings.... but I can't see how to map a port?

I can expose a port, like this...

    // Database service used for application tests
    database := client.Container().From("postgres:15.2").
        WithEnvVariable("POSTGRES_PASSWORD", "test").
        WithExec([]string{"postgres"}).
        WithExposedPort(5432)

But how can I map this port, like I do in a docker compose file?

I was hoping for something like this...

    // Database service used for application tests
    database := client.Container().From("postgres:15.2").
        WithEnvVariable("POSTGRES_PASSWORD", "test").
        WithExec([]string{"postgres"}).
        WithExposedPort("<MY_PORT>:5432")
grand shuttle
#

We haven't implemented host-to-container networking yet. It's planned

#

There are problably workarounds though

azure ivy
#

yeah, I think I just need to think of it different.

grand shuttle
#

Sorry - what you're looking for is 100% something we want to offer

#

One workaround, is that you can proxy the local port to a unix socket on the host; then mount that unix socket into the container, and proxy that to your service. It adds glue on the host and in the DAG, but it will work

#

postgres client on host --> socat on host ----- (unix socket) ---> socat in container --> postgres in container

azure ivy
#

no just the way you phrased it helped me shift a little. port mapping is for host networking, I shouldn't really need to fuss with that.

#

there's vars to overwrite the port my tests expect PG to be on, so it should be fine to default it back to 5432

#

but this also assumes that you can run multiple service binds in parallel binding to the same port? I haven't dug into the services implementation too much

grand shuttle
#

all services are containerized (they’re actually “service containers”) so they each get their own network namespace

#

upside: no port conflicts
downside: no connectivity to/from host by default

heady saffron
#

@azure ivy @calm thicket recently put together a demo where he parsed a compose file and translate that to to a Dagger pipeline. Here's the repo for inspiration purposes: https://github.com/vito/dagger-compose

#

IIRC he even implemented the host <> container proxy thing Solomon was mentioning

calm thicket
atomic oasis
#

Say I run docker run -p 3000:3000 ... from withing a docker-in-dagger container, and then a curl localhost:3000 from within that container as well.

Will this not work as well?

#

It seems it does not, which makes sense

grand shuttle
#

if the docker daemon itself is inside the container, then it should work (if the docker cli is also in the same container)

#

can you share the code, or a run link?

atomic oasis
#

I'm passing the daemon socket through

#

I could run the daemon in the container, but that comes with other issues too

grand shuttle
#

airbyte does it in their CI

#

or, maybe bypass the docker run entirely, and run a dagger service directly?

atomic oasis
#

Ideally we could start the containers once and then use them in multiple tests, instead of needing to life-cycle for each test

grand shuttle
#

i’m guessing you already thought of that last one though

grand shuttle
atomic oasis
#

our CLI is pulling and running containers in the background of other commands we are testing

atomic oasis
#

startup time is a concern

grand shuttle
#

I see

#

we’re happy to help if you want to dig into optimizations

atomic oasis
#

we are accepting that our tests will be slower in Dagger, but more correct as the tradeoff

grand shuttle
#

but I understand why there’s docker runs in the mix as a pragmatic first step

grand shuttle
atomic oasis
#

minimal, correct, fast... in that order... iirc the saying

#

How are you reliably testing the (revamped) Dagger CLI which needs to talk to container runtimes? Do you test the docker symlink setup for alternatice runtimes?

I'm willing to consider full VMs to run the gamut 😔

#

Is it possible to do something like:

  1. build a base image with docker daemon and our CLI
  2. container.WithExec(<docker image stuff>)
  3. for test in tests { container.WithFile, container.WithExec }
#

How would we exec hof ... in a container with a docker daemon? What ensures the daemon is running? (b/c containers typically don't use init.d or systemd to babysit processes)

atomic oasis
grand shuttle
azure ivy
#

yooo @heady saffron this is dope, I'll check it out.

#

A follow up question here, I'm debating this type of thing with a coworker...

Say I'm running buildkitd/dagger on K8s. I start a pipeline. Will containers started w/ dagger have access to k8s service/pods/networking?

grand shuttle
#

I can’t help but notice the dedicated Service type is making a comeback, making my “proxy service” proposal viable once again 😇

atomic oasis
grand shuttle
#

it would make composition much easier

#

especially since running more than one docker engine in dagger currently causes issues (I believe)

atomic oasis
#

Yes, this seems to be almost working for me, but hitting the following error, any insights?

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: no cgroup mount found in mountinfo: unknown.

It is failing at the docker run in the client container. (line 113 from the following link)

Self-contained reproducer: https://github.com/hofstadter-io/hof/blob/gen-path-simplifications/test/dagger/main/dockerd-in-dagger.go 🤞

grand shuttle
#

are you running privileged?

#

you need that extra opt in WithExec

atomic oasis
#

on the client or just the server?

#

it's on the server (daemon)

#

docker pull works, but not run (from the client)

grand shuttle
#

client does not need privileged

#

but it does need to set DOCKER_HOST to the alias of the daemon - and for the daemon to listen on tcp

#

either that or unix socket mounted across containers, does that even work? cc @calm thicket

#

I cannot wait for this to be an extension 😁

atomic oasis
#

using the DOCKER_HOST & tcp

#

basically following what Airbyte is doing, but in Go

grand shuttle
atomic oasis
#

If someone else could run that reproducer, maybe it is something on my underlying system missing or in the way

grand shuttle
atomic oasis
#

gonna spin up a vm in gcp and test as well

calm thicket
atomic oasis
#

so DOCKER_HOST w/ tcp and the Dagger Service pattern is the way to go

#

I would guess the issue lies with the daemon container being used as a service, that's where the run would happen and need the right linux setup & packages

grand shuttle
#

are you using the same image as airbyte for dockerd?

steep cliff
atomic oasis
#

yes, but not accounting for moving image tags, it's docker:24-dind

#

ok, it was suuuuper easy to make the docker version configurable and retest!

same error though...

limber harness
#

I'm not seeing that error on my end, but still working on getting it working completely

#

ok got the repro working! I didn't run into the error you had though. The only change I made was to install curl and point curl at global-dockerd instead of localhost

atomic oasis
#

arg, docker hub 503 is breaking my dagger builds...

#

do you see any overlay FS errors? (about not being available)

atomic oasis
#

getting a different error on a debian cloud VM at the docker pull step

115.4 Error response from daemon: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n"

Interestingly, the inner docker daemon is getting the same cgroup settings as the root host.

  • locally, cgroup v1, where I get ano cgroup mount found in mountinfo: unknown error at docker run, can pull
  • cloudly, cgroup v2, error above at pull or run

I just need the best of both :[

#

local also has big warnings

#3 1.486 time="2023-06-23T02:15:19.823684955Z" level=warning msg="WARNING: No memory limit support"
#3 1.486 time="2023-06-23T02:15:19.823687726Z" level=warning msg="WARNING: No swap limit support"
#3 1.486 time="2023-06-23T02:15:19.823689967Z" level=warning msg="WARNING: No kernel memory TCP limit support"
#3 1.486 time="2023-06-23T02:15:19.823692107Z" level=warning msg="WARNING: No oom kill disable support"
#3 1.486 time="2023-06-23T02:15:19.823694251Z" level=warning msg="WARNING: No cpu cfs quota support"
#3 1.486 time="2023-06-23T02:15:19.823696337Z" level=warning msg="WARNING: No cpu cfs period support"
#3 1.486 time="2023-06-23T02:15:19.823698540Z" level=warning msg="WARNING: No cpu shares support"
#3 1.486 time="2023-06-23T02:15:19.823700688Z" level=warning msg="WARNING: No cpuset support"
#3 1.486 time="2023-06-23T02:15:19.823702743Z" level=warning msg="WARNING: No blkio throttle.read_bps_device support"
#3 1.486 time="2023-06-23T02:15:19.823704834Z" level=warning msg="WARNING: No blkio throttle.write_bps_device support"
#3 1.486 time="2023-06-23T02:15:19.823707007Z" level=warning msg="WARNING: No blkio throttle.read_iops_device support"
#3 1.486 time="2023-06-23T02:15:19.823709193Z" level=warning msg="WARNING: No blkio throttle.write_iops_device support"
#3 1.486 time="2023-06-23T02:15:19.823711463Z" level=warning msg="WARNING: bridge-nf-call-iptables is disabled"
#3 1.486 time="2023-06-23T02:15:19.823714364Z" level=warning msg="WARNING: bridge-nf-call-ip6tables is disabled"
heady saffron
atomic oasis
#

I'll be curious to see if it works in GitHub actions

heady saffron
#

@surreal flower you might want to also add a WithMountedCache to the dind service so you don't run into funky overlay-in-overlay issues and also keep the daemon cache so you don't re-pull images constantly.

diff --git a/main.go b/main.go
index 24a9a46..0506ba1 100644
--- a/main.go
+++ b/main.go
@@ -65,8 +65,7 @@ func main() {
 }
 
 func (R *runtime) baseContainer() (*dagger.Container, error) {
-    c := R.client.Container().From(dockerVer + "-cli").
-        WithExec([]string{"apk", "add", "curl"})
+    c := R.client.Container().From(dockerVer + "-cli")
     c = c.Pipeline("base/image")
 
     return c, nil
@@ -78,7 +77,6 @@ func (R *runtime) daemonContainer() (*dagger.Container, error) {
 
     c = c.WithMountedCache("/tmp", R.client.CacheVolume("shared-tmp"))
     c = c.WithExposedPort(2375)
-    c = c.WithMountedCache("/var/lib/docker", R.client.CacheVolume("shared-docker"))
 
     c = c.WithExec(
         []string{
@@ -121,7 +119,6 @@ func (R *runtime) dockerTest(c *dagger.Container) error {
     t = t.WithExec([]string{"docker", "pull", "nginxdemos/hello"})
     t = t.WithExec([]string{"docker", "images"})
     // t = t.WithExec([]string{"ls", "-l", "/sys/fs/cgroup"})
-    t = t.WithEnvVariable("CACHE", time.Now().String())
     t = t.WithExec([]string{"docker", "run", "-p", "4000:80", "-d", "nginxdemos/hello"})
     t = t.WithExec([]string{"curl", "global-dockerd:4000"})
 
@@ -144,4 +141,3 @@ func (R *runtime) daemonTest(c *dagger.Container) error {
 
     return err
 }
-

That's a reversed diff of the things that I added locally

#

also added the CACHE bust to re-trigger multiple times

heady saffron
#

let's see if it works on the first try 🤞

heady saffron
# heady saffron https://github.com/marcosnils/test/actions/runs/5352327837 👀 
0.158 <p class="smaller"><span>Date:</span> <span>23/Jun/2023:02:48:32 +0000</span></p>
#8 0.158 <p class="smaller"><span>URI:</span> <span>/</span></p>
#8 0.158 </div>
#8 0.158 <div class="check"><input type="checkbox" id="check" onchange="changeCookie()"> Auto Refresh</div>
#8 0.158     <div id="footer">
#8 0.158         <div id="center" align="center">
#8 0.158             Request ID: 260e0b0d8b1d1a027e4b273aaa7cbc33<br/>
#8 0.158             &copy; NGINX, Inc. 2018
#8 0.158         </div>
#8 0.158     </div>
#8 0.158 </body>
#8 0.158 </html>
#8 0.158 ```
:party_parrot:
#

GHA pipeline exited with status code 1 though.. even though the dagger part worked. That's something to check

atomic oasis
#

I'm getting the following on my debian 11/12 instances, at the container pull step

#3 2.023 time="2023-06-23T04:37:53.105559348Z" level=info msg="API listen on [::]:2375"
#3 2.082 time="2023-06-23T04:37:53.164598564Z" level=debug msg="Calling HEAD /_ping"
#3 2.084 time="2023-06-23T04:37:53.166969527Z" level=debug msg="Calling POST /v1.43/images/create?fromImage=hello-world&tag=latest"
#3 2.140 time="2023-06-23T04:37:53.221940034Z" level=debug msg="Trying to pull hello-world from https://registry-1.docker.io"
#3 ...

#3 daemon/image
#3 73.82 time="2023-06-23T03:53:27.763679360Z" level=error msg="Handler for POST /v1.43/images/create returned error: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: \"<html><body><h1>408 Request Time-out</h1>\\nYour browser didn't send a complete request in time.\\n</body></html>\\n\""
#3 ...

#8 client/test
#8 60.82 Error response from daemon: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n"
#8 ERROR: process "docker-entrypoint.sh docker pull hello-world" did not complete successfully: exit code: 1
input:1: container.from.pipeline.withExec.pipeline.withEnvVariable.withServiceBinding.withMountedCache.pipeline.withExec.withExec.withExec.withExec.withEnvVariable.withExec.withExec.stdout process "docker-entrypoint.sh docker pull hello-world" did not complete successfully: exit code: 1
#

This would seem to indicate something from the client made it to dockerd service, but did not complete?

Are there network settings to be aware of, at the host, buildkit, or dagger?

atomic oasis
#

maybe it's DNS...?

grand shuttle
#

can the dockerd container ping or curl docker hub?

#

maybe run dockerd with max verbosity and look at its logs?

#

also stuck on the fact that docker run times out but docker pull works fine, is that still the case?

atomic oasis
#

no, this is a new env, debian 12 in GCP, with the logs up, it looks like the daemon cannot reach the public registry

I can exec/curl the registry just before exec/dockerd and I get a 404, so it seems to be reaching

heady saffron
#

@atomic oasis can you try checking what's the MTU on the host machine and the dockerd container?

#

you can check the MTU of the dockerd container by doing something like:

pgrep dockerd # (grab the PID of the dockerd running in dagger)
nsenter -t $PID -n ip a s
limber harness
# atomic oasis do you see any overlay FS errors? (about not being available)

yeah I usually see these and everything still works fine

level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2
level=error msg="exec: \"fuse-overlayfs\": executable file not found in $PATH" storage-driver=fuse-overlayfs
level=error msg="AUFS was not found in /proc/filesystems" storage-driver=aufs
level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay
level=error msg="Failed to built-in GetDriver graph devicemapper /var/lib/docker"
heady saffron
limber harness
atomic oasis
#

can confirm the /var/lib/docker caching works as you describe, makes sense

atomic oasis
# heady saffron you can check the MTU of the `dockerd` container by doing something like: ``` ... 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP group default qlen 1000
    link/ether 42:01:0a:80:00:30 brd ff:ff:ff:ff:ff:ff
    altname enp0s4
    inet 10.128.0.48/32 metric 100 scope global dynamic ens4
       valid_lft 86323sec preferred_lft 86323sec
    inet6 fe80::4001:aff:fe80:30/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ca:80:49:ef brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:caff:fe80:49ef/64 scope link 
       valid_lft forever preferred_lft forever
5: veth7a04dc9@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default 
    link/ether 0a:3e:82:6f:a6:5d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::83e:82ff:fe6f:a65d/64 scope link 
       valid_lft forever preferred_lft forever
#

Am I able to turn up Dagger logs in the Go SDK to see this MTU line you linked?

#

or is that going to be in the engine container?

heady saffron
atomic oasis
#

this is the host system by the way, the issue I have is in the nested dockerd (in-docker)

heady saffron
atomic oasis
#

right... 🤦

heady saffron
#

you need to get the PID of the dagger-in-dagger dockerd, not the one in the host

atomic oasis
#

I still need coffee

heady saffron
atomic oasis
#

ok, so I

  1. create the dockerd container
  2. attach this as a service to client container
    (dockerd starts at this point?, or more when the client container needs to start)

How can I run a command in the dockerd container while it is running?

Will I see 2 dockerd PIDs when looking on the GCP VM host?

#

ok, there are 3 PIDs on the host now...

#

oh, maybe that is just pgrep picking up something extra

# 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth1@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether aa:95:40:eb:53:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.87.0.130/16 brd 10.87.255.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a895:40ff:feeb:53e1/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:24:7a:f5:06 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
#

So is that 1460 on the ens4 an issue?

heady saffron
#

🤔 so ens4 is 1460 but everything else is 1500..that might be an issue. cc @calm thicket wondering why Dagger didn't pick that one

atomic oasis
#

Would manually setting the nested dockerd mtu help test?

heady saffron
#

I don't think so since the eth1 is being set by dagger. I think there's a way to set the mtu through a cni config IIRC

calm thicket
heady saffron
atomic oasis
#

So I want a /etc/docker/daemon.json with that setting?

calm thicket
heady saffron
#

strange that docker0 didn't pick the correct mtu of the host iface though 🤔

#

brb 🦮

atomic oasis
#

Maybe I need a restart for it to take effect

#

After making this change, docker0 should be 1500? (on the host)

heady saffron
#

just recall that the mtu will change once an interface is attached to the bridge network

#

so it's ok if you still see 1500 if no containers are running

#

once you run a container, the mtu should change.. try with docker run --rm nginx and 👀

atomic oasis
#

but only when a container is running, yea?

#

or maybe exposes a port

heady saffron
#

exposing a port shouldn't be necessary. I'm also thinking that you will likely have to change the mtu as well from the docker-in-docker daemon.. Since that will start with 1500 as well

atomic oasis
#

it works!

heady saffron
#

🚀

atomic oasis
#

thank you so much!

atomic oasis
#

it's so nice to be unblocked before the weekend :]

#

now the question is... can we get nerdctl and podman in docker, like dind?

Then we can test our CLI against all 3 tools using Dagger

heady saffron
#

Podman should also work.. nerdctl is just a client tool it doesn't spin any daemon AFAIK

atomic oasis
#

yes, this is my expectations as well, we just need to get the dagger code in place to realize this

#

did you ever figure out your exit(1) in GHA?

heady saffron
atomic oasis