Kind cluster with docker socket

[viscid turtle]

I did it a while ago but never committed it anywhere. I remember that I made it worked. What's you error?

[light helm]

it seems I deleted to error logs. I'll reproduce the error and share log

[light helm]

#12 640.7   [FAILED] in [BeforeAll] - /src/pkg/kubernetes/kubernetes_test.go:40 @ 01/16/23 16:26:13.452
#12 640.7   << Timeline
#12 640.7 
#12 640.7   [FAILED] Expected success, but got an error:
#12 640.7       <*fmt.wrapError | 0xc0006660c0>: {
#12 640.7           msg: "failed to create kind cluster: failed to generate kubeadm config content: failed to get kubernetes version from node: file should only be one line, got 0 lines",
#12 640.7           err: <*errors.withStack | 0xc00118a630>{
#12 640.7               error: <*errors.withMessage | 0xc00118e7c0>{
#12 640.7                   cause: <*errors.withStack | 0xc00118a600>{
#12 640.7                       error: <*errors.withMessage | 0xc00118e7a0>{
#12 640.7                           cause: <*errors.fundamental | 0xc00118a5d0>{
#12 640.7                               msg: "file should only be one line, got 0 lines",
#12 640.7                               stack: [0x22c7037, 0x22c6ffb, 0x22dfb3c, 0x22df8e5, 0x22a0989, 0x110e981],
#12 640.7                           },
#12 640.7                           msg: "failed to get kubernetes version from node",
#12 640.7                       },
#12 640.7                       stack: [0x22dfb99, 0x22dfb9a, 0x22df8e5, 0x22a0989, 0x110e981],
#12 640.7                   },
#12 640.7                   msg: "failed to generate kubeadm config content",
#12 640.7               },
#12 640.7               stack: [0x22dfa65, 0x22dfa66, 0x22a0989, 0x110e981],
#12 640.7           },
#12 640.7       }
#12 640.7       failed to create kind cluster: failed to generate kubeadm config content: failed to get kubernetes version from node: file should only be one line, got 0 lines
#12 640.7   In [BeforeAll] at: /src/pkg/kubernetes/kubernetes_test.go:40 @ 01/16/23 16:26:13.452```

I'm getting this error. failed to create kind cluster: failed to generate kubeadm config content: failed to get kubernetes version from node: file should only be one line, got 0 lines" didn't able to understand why I'm getting this error

[light helm]

I just find out when I result of the kind export logs is different based on execution env. if I execute from my local host everything is working however, if I execute command from dagger container. Some contents are missing like kubernetes-version.txt from logs

[viscid turtle]

i dig a little into kind code but it seems it cannot cat /kind/version inside node. but can't find how is it trying to do that

[light helm]

https://github.com/kubernetes-sigs/kind/blob/4a0de915dedf571e36409a3ff02ee6ae1e1e85a9/pkg/cluster/nodeutils/util.go#L36-L44 that's the code for getting version

this is the docker command builder https://github.com/kubernetes-sigs/kind/blob/4a0de915dedf571e36409a3ff02ee6ae1e1e85a9/pkg/cluster/internal/providers/docker/node.go#L103-L109

it seems they're generating cmd based on their provider and reading it's output lines https://github.com/kubernetes-sigs/kind/blob/4a0de915dedf571e36409a3ff02ee6ae1e1e85a9/pkg/exec/helpers.go#L78 to find these values or export data from inside

I'm suspecting something blocking docker exec --privileged commands inside of the container

[viscid turtle]

Oh there is a nested privileged argument to dagger withExec. Try setting it to true

[light helm]

I tried but didn't changed the result. I'll try to look more.

[light helm]

interesting, container = container.WithExec([]string{"ash", "-c", "docker exec -t --privileged dagger-engine-8d8bdc86d448fd7b ls"}, dagger.ContainerWithExecOpts{ExperimentalPrivilegedNesting: true}) this is not returning anything to console

however, executing same command from my local host returns.

note: privileged not change the result of the pipeline

[viscid turtle]

Well, I hand it over to the dagger team, I did reproduce it tough. If I have time I'll look into it

<@&946480760016207902>

[void cove]

I can help here. I'll check in ~30m

[sand tiger]

@light helm Is there somewhere we could see the code?

[sand tiger]

You're going to have to do something like:

ctr.WithUnixSocket("/var/run/docker.sock", c.Host().UnixSocket("/var/run/docker.sock))

in order to passthrough the unix socket to the container

    out, err := dg.Container().From("docker:cli").
        WithUnixSocket("/var/run/docker.sock", dg.Host().UnixSocket("/var/run/docker.sock")).
        WithExec([]string{"docker", "ps"}).
        Stdout(ctx)

    if err != nil {
        panic(err)
    }
    fmt.Println(string(out))

this works on my machine

[novel arrow]

You may also be hitting the issue described here that required an upstream fix to buildkit: https://github.com/dagger/dagger/issues/4073#issuecomment-1361996260

That problem resulted in output streams from docker containers to be closed early due, which seems to possibly align with the error message of should only be one line, got 0 lines

There is also a follow up fix to that which is going to go into buildkit v0.11.1 (https://github.com/moby/buildkit/pull/3506), last I heard that patch release is coming out today, so once that's out we can update dagger to use the new release and get the fix out

[light helm]

I’m already have docker socket mounted and it’s working

I’ll check the upstream issue. That could be related

[light helm]

This is also work for me

But docker exec don’t

[sand tiger]

Maybe it's the -t of docker exec?

[light helm]

Tried all combinations, result is same

Just try to run a command in a existing container. You should able to replicate the behaviour

From my tests, commands using directly docker cli working however if command needs to some kind of forwarding or connections don’t

[light helm]

I’ll try to look closer buildkit PR

[sand tiger]

Indeed.

Ok so I tried to WithExec([]string{"docker", "run", "--rm", "hello-world"}). -- I don't get the output either, but I do see the messages from the docker CLI pulling the image

so the docker CLI is exec'ed, and it works -- somehow the container output (and only that) is lost

[novel arrow]

The context is here: https://github.com/dagger/dagger/issues/4073#issuecomment-1361967787

It has to do with the fact that the docker cli closes the write side of its socket when reading stdout+stderr streams from containers, but then expects to continue reading data. Buildkit's internal mechanisms for proxying sockets didn't handle this correctly though and just closed the whole thing when this happened. I fixed it upstream but we want to wait for v0.11.1, should be soon

I don't know why docker does that... it seems to be causing issues for others too e.g. here https://github.com/apocas/dockerode/issues/534#issuecomment-1219960665

[sand tiger]

👍 thanks

@light helm In short, docker run / exec does work, however the output doesn't come back. It's a bug upstream which @novel arrow fixed, waiting for the next release

As a workaround, using docker run THEN docker logs does the trick for me:

WithExec([]string{"sh", "-c", "docker run --name foo hello-world && docker logs foo"}).

[light helm]

Great news thanks for looking into it.

Unfortunately this exec part of the KinD. I just created that snippet for debugging issue

[void cove]

were you able to create the kind cluster finally? Do you have some code to share so we can try repro?

[light helm]

unfortunately, I didn't able to create kind cluster. Kind uses docker exec internally for able to get some information from k8s cluster and as @sand tiger explained docker run / exec not return output in builtkit at the moment and it prevent kind to creating configuration files. It seem @novel arrow already had fix for this issue and we're waiting for release.

for reproducing issue running WithExec([]string{"docker", "run", "--rm", "hello-world"}) would be enough. it won't print any output

[void cove]

oh, I see.. Yeah, I'm aware of the issue. Surprised that kind actually runs docker exec instead of using the docker API directly.

[light helm]

same here, I'm surprised as well :). at least we have a fix for it

[light helm]

it seems latest release fixed the kind cluster creation issue. However, networking still has issues. I'll make more test but it seems default kubeconfig trying to reach it from local host and getting timeout

I'll make more test and create a gist for the issue

[void cove]

you can change the kubeconfig hostname so you can go through the default gateway hostname. That way it should work. I recall we discussed that in a thread somewhere around here. Let me look for it really quick

[light helm]

That's my guess as well, didn't had a chance to look this into more.

[sand tiger]

Glad that it fixed it!

Regarding the networking issue — @gleaming lance is currently working (and has a working PR) of dagger services / networking! Allows to explicitly define network connections between containers

Perhaps we can see if that helps? Services/networking are meant to start things like a Postgres containers in your pipelines — starting a Kube cluster is a boss level test for this 🙂

[gleaming lance]

yep, I'm starting on container-to-container networking, PR is here: https://github.com/dagger/dagger/pull/4505 - what kind of networking are you trying to do with kind? sounds more like host => container? if so my PR might actually be a step back since it introduces a bridge network, so localhost won't reach ports bound by containers anymore 🤔

[swift sentinel]

They still could, with H2C proxy?

[gleaming lance]

yeah, just haven't gotten to that part yet 🙂 assuming you mean this part: https://github.com/dagger/dagger/pull/4163/files#diff-4242183254eaee9649c3ed82114bd292d61bd23a0d7b2ff701c3bb16d3fb8fab - but there's also an extra wrinkle here where kind is talking to the outer Docker which blows through Dagger's abstractions

[novel arrow]

If all of the inner buildkit containers are isolated w/ CNI I think it might be okay to change it so we run the outer engine docker container w/ --net=host by default? Need to think through more but I can't immediately think of why that would be a bad idea

[novel arrow]

Well I guess the downside would be that all the bridges and stuff are in the host's network namespace, which feels sort of risky if we could be impacted by the iptables and other network config there, so maybe not

[gleaming lance]

yeah i think they'd also need to add the cni dns server to their host for that to work, vs right now where part of my PR changes the docker run to configure dns flags: https://github.com/dagger/dagger/pull/4505/files#diff-67f7d61dad599f723bac51a4d5b212bbe37c4525422adbc6515f9bbed0ad87bdR133

[light helm]

it's great to hear we're working on networking for containers. I'll try to look into that PR and to see what kind of workaround we can could find.

[light helm]

KinD has two different types of kubeconfig. What we normally use is external config uses local port to connect k8s control plane. second one is contains internal host address and this can be usable inside of the same docker network

if we can connect two network together, we could use internal kubeconfig to connect cluster easily. However, these are stll assumptions based on docker networks. Not sure if any specifics involved with buildkit

[swift sentinel]

Is there a way to either:

1. Bring the docker engine inside (run it as a service), then kind can just talk to that

or

2. Configure kind to not require the docker engine

or

3. Use something than kind, to run kubernetes entirely inside dagger?

[light helm]

actually we're using some hacky solution run KinD. We're using kind as a lib. Maybe as 4 option we can add custom kind impl as extension. WDYT @swift sentinel ?

it would be really neat feature, running a kubernetes cluster with dagger directly.

[novel arrow]

If we want to run docker or kubernetes entirely in dagger, we just need Alex's PR + unix socket services and privileged exec support.

@light helm do you need linux capabilities (i.e. --privileged) for using KinD as a lib like you described? Or some other special configs/features?

[swift sentinel]

maybe I’m getting ahead of my skis here, but doen the road assuming we can run kubernetes in dagger directly with privileged containers (plenty of work), could we go even further and have it use a special cri-dagger adapter that runs pods with dagger calls, removing the need for privileged kubelet? 😁

[light helm]

Not exactly, we're creating or destroying clusters programmatically with custom kind configs.

only special thing I can count is multi cluster setups with different network combinations. however, as long as we have access docker socket, we shouldn't have issues

[light helm]

I would say, wrapping KinD or some alternative solutoin to dagger interim solution would give same feature much less complexity. However, as an idea, that would be neat feature for us to use

[void cove]

@light helm aside from the networking thread. Did you manage to make kind the way you needed with the current version of dagger?

last thing I recall was that you needed to change kind API server hostname so Dagger can reach to it