Small kubernetes setup

[uneven ore]

What I want:

Build Docker images with a Node JS app that runs on a kubernetes cluster

What I have so far:

I have created a dagger engine DaemonSet with this command:

helm upgrade --create-namespace --install --namespace dagger dagger oci://registry.dagger.io/dagger-helm

In a different namespace I have a deployment of my Node JS app that automatically builds images with dagger.

I would like the Node JS app to communicate with the separate dagger engine. (does this even make sense?) so the builds are executed in the separated pod.

I think I would have to use the _EXPERIMENTAL_DAGGER_RUNNER_HOST variable but Im not sure how.

Thx for your thoughts and help!

Feel free to ask me any questions about my setup if you need more details.

[uneven ore]

Is there something wrong with my post? If so let me know and I will do my best to improve it!

[humble magnet]

hey! apologies for the delay. We're a kubecon this week and a big part of the team is busy running that event.

[humble magnet]

so now that you have applied the helm chart, this creates a daemonset in all your nodes running the dagger engine. The next thing to do for your dagger pipelines running in pods to use that engine, is to start the the pod with a volumeMount using the dagger socket and setting an env variable. That's how it looks like:

      env:
        - name: _EXPERIMENTAL_DAGGER_RUNNER_HOST
          value: unix:///var/run/dagger/buildkitd.sock
      volumeMounts:
        - name: dagger-socket
          mountPath: /var/run/dagger
      volumes:
        - name: dagger-socket
          hostPath:
            path: /var/run/dagger

that shoud be all you need. Let us know how it goes

[uneven ore]

Ahhh I thought its maybe some sort of holiday 🙂 Thanks for reaching out! I will test your answer, I have been working with this _EXPERIMENTAL_DAGGER_RUNNER_HOST=kube-pod://dagger-dagger-helm-engine-n5ddh?namespace=dagger so far but I could not get it to work. Would this also work some way?

[humble magnet]

_EXPERIMENTAL_DAGGER_RUNNER_HOST=kube-pod://dagger-dagger-helm-engine-n5ddh?namespace=dagger so far but I could not get it to work.

the only way this would work is if you have the kubectl binary installed in the pod where you're running your dagger pipelines along with the correct service-account to be able to perform a kubectl proxy. I'd suggest trying the _EXPERIMENTAL_DAGGER_RUNNER_HOST=unix:///var/run/dagger/buildkitd.sockroute

[humble magnet]

hey @uneven ore , let us know if you still need some help here 🙏

[uneven ore]

Hey, So I have mounted the the volume but I dont see the build kitd.sock in it. Is there anything I have to check besides running the Helm chart and mounting the volume. Pipeline Pod:

[humble magnet]

@uneven ore around? Happy to jump into a quick #911305510882513037 call to check it out together if you can

[humble magnet]

It'd also check if the node where that pod is scheduled also has a dagger DaemonSet pod running in there

the Dagger DaemonSet should be the one creating the buildkitd.sock in the host machine

[uneven ore]

I will check that. No sorry I was at work. What Timezone are you in? Maybe on Fraiday?

[humble magnet]

I'm in GMT -3. Sure, ping me whenever you're around and we can double check together

[uneven ore]

Okey, I will Ping you tomorrow.

[humble magnet]

I'll be here

[uneven ore]

Hey, @humble magnet I would be available from now on for some time.

[humble magnet]

yep, I'm here

wanna jump to #911305510882513037 quickly?

[uneven ore]

sure

[humble magnet]

hey, issue here was that the helm chart was installing an old engine version. This will be addressed by https://github.com/dagger/dagger/pull/5957/. cc @broken salmon

[misty apex]

hey, question
is unix:///var/run/dagger/buildkitd.sock correct?

if I launch a dagger engine on k8s (using just pods, not helm) dagger goes to infinite loop trying to connect to the engine

changing _EXPERIMENTAL_DAGGER_RUNNER_HOST to unix:///var/run/buildkit/buildkitd.sock seems to fix the issue, maybe the helm chart should use that path instead?

I'm using v0.9.3

[humble magnet]

is unix:///var/run/dagger/buildkitd.sock correct?

hi, this path is the default one the chart issues. Did you deploy your engines using the chart? https://github.com/dagger/dagger/blob/main/helm/dagger/templates/engine-daemonset.yaml#L83-L85

[humble magnet]

@misty apex just checking here. Let us know if you have further quesitons

[misty apex]

@humble magnet Thanks, I haven't actually used the helm chart yet, only ran the dagger engine as a sidecar in jenkins pipeline that utilizes podTemplate for k8s, since I used the standalone image the socket was launched in /var/run/buildkit instead of /var/run/dagger

way I made it work is by using _EXPERIMENTAL_DAGGER_RUNNER_HOST="unix:///var/run/buildkit/buildkitd.sock" on the second container that runs the dagger cli program

when I get the chance to try and run dagger engine using helm chart I'll update

[humble magnet]

thx for the update. That makes sense, as you can see from the chart, the volumemount in the container is effectively /var/run/buildkit, but we then set it in the host as /var/run/dagger

[neon fossil]

Hey !! having a similar issue here, just to re-confirm, have my new pods spinning up with the below config ?

env:
        - name: _EXPERIMENTAL_DAGGER_RUNNER_HOST
          value: unix:///var/run/dagger/buildkitd.sock
      volumeMounts:
        - name: dagger-socket
          mountPath: /var/run/dagger
      volumes:
        - name: dagger-socket
          hostPath:
            path: /var/run/dagger

I have dagger running in a daemonset in a namespace, in that same namespace, jenkins deploys pod templates containers which start building code (this container, has dagger installed by default, so from this container, needs the above config to work?) Thanks