Help running Dagger on Kuberenetes

[ocean moth]

Submission from @fringe prism - moved from orgnianl post in demo forum - https://discord.com/channels/707636530424053791/1245068814589235291

---

I have followed the instruction on setting up dagger Engine inside, it is running but having issue connecting from the git . here is the video https://www.youtube.com/watch?v=sogSICwyg0Y . I also rwady instruction from dagger.io but could not get it working.

➜  hello-dagger git:(main) kubectl get pods -n dagger
NAME                              READY   STATUS    RESTARTS      AGE
dagger-dagger-helm-engine-4dz5t   1/1     Running   0             55m
dagger-dagger-helm-engine-5r5k2   1/1     Running   0             55m
dagger-dagger-helm-engine-tth5r   1/1     Running   0             55m
gitlab-runner-65dcb59c85-tgvp6    1/1     Running   2 (63m ago)   68m
rockylinux                        1/1     Running   0             6m57s

✗  hello-dagger git:(main) echo $env.DAGGER_ENGINE_POD_NAME
dagger-dagger-helm-engine-4dz5t
➜  hello-dagger git:(main) echo $env._EXPERIMENTAL_DAGGER_RUNNER_HOST
kube-pod://dagger-dagger-helm-engine-4dz5t?namespace=dagger

but still dagger refuse to connect

here is my namespace configuration

apiVersion: v1
kind: Namespace
metadata:
creationTimestamp: "2024-05-28T16:15:31Z"
labels:
kubernetes.io/metadata.name: dagger
pod-security.kubernetes.io.warn: privileged
pod-security.kubernetes.io/audit: privileged
pod-security.kubernetes.io/enforce: privileged
name: dagger
resourceVersion: "2255"
uid: 2a7f969a-ddec-4db0-8308-922318a6b0d9
spec:
finalizers:

• kubernetes
  status:
  phase: Active

and this is the command i use in installing it

helm upgrade --install --namespace=dagger --create-namespace dagger oci://registry.dagger.io/dagger-helm
but i get this error,

`➜ hello-dagger git:(main) dagger init --sdk=go
● connect 0.1s
2024/05/28 11:28:30 http2: server: error reading preface from client localhost: rpc error: code = Unknown desc = failed to get client metadata for session call: failed to JSON-unmarshal x-dagger-client-metadata: json: cannot
∅ connect 14.6s
┃ Failed to connect; retrying... name:"error" value:"make request: Post "http://dagger/query\": rpc error: code = Unknown desc = failed to get client metadata for session call: failed to JSON-unmarshal x-dagger-client-metada
┃ : json: cannot unmarshal array into Go struct field ClientMetadata.labels of type telemetry.Labels"
┃ Failed to connect; retrying... name:"error" value:"make request: Post "http://dagger/query\": rpc error: code = Unknown desc = failed to get client metadata for session call: failed to JSON-unmarshal x-dagger-client-metada
┃ ta: json: cannot unmarshal array into Go struct field ClientMetadata.labels of type telemetry.Labels"
┃ Failed to connect; retrying... name:"error" value:"make request: Post "http://dagger/query\": rpc error: code = Unknown desc = failed to get client metadata for session call: failed to JSON-unmarshal x-dagger-client-metada
┃ ta: json: cannot unmarshal array into Go struct field ClientMetadata.labels of type telemetry.Labels"
┃ Failed to connect; retrying... name:"error" value:"make request: Post "http://dagger/query\": rpc error: code = Unknown desc = failed to get client metadata for session call: failed to JSON-unmarshal x-dagger-client-metada
┃ ta: json: cannot unmarshal array into Go struct field ClientMetadata.labels of type telemetry.Labels"

Error: context canceled`

@stiff rapids @pallid sapphire This user was watching your video https://www.youtube.com/watch?v=sogSICwyg0Y, so you might have some pointers for them.

[azure viper]

Hey @fringe prism can you validate that your dagger CLI is the same version as the engine deployed in k8s?

[fringe prism]

yes they are

[root@rockylinux ~]# dagger version
dagger v0.11.5 (registry.dagger.io/engine) linux/amd64

and engine versiob number is

➜  ~/.kube/cache kubectl get ds dagger-dagger-helm-engine -n dagger -o yaml | grep version
    app.kubernetes.io/version: v0.11.5

[fringe prism]

i have tried, real kubernetes cluster running in remote with dagger cli on my local machine
i have also tried a vm in the same namespace with the enginer, but still cannot establish connection to the engine
currently also trying it on minikube

but it works if i am running dagger engine directly on on my localhost docker service and dagger cli also on local system

[azure viper]

are you running the dagger CLI in the same machine where you have kubectl installed?

cc @fringe prism

the kube-pod scheme requires kubectl with a valid context and kubeconfig file to work

since it uses kubectl exec under the hood to open the connection to the engine

I'm asking because I see the prompt of the two snippets you shared are different

[fringe prism]

yes, i also tried that

[azure viper]

is kubectl in the PATH? I see you're calling ./kubectl

[fringe prism]

in the above screenshot i am inside the rockylinux pod

[fringe prism]

from my local machine it works, and i made sure i delete the local engine.

[azure viper]

try moving ./kubectl to /usr/bin inside the user's PATH in the rocky linux host

[fringe prism]

the reason is because i exec into the rockylinux pod, i am trying to test it from there, i downloaded the kubectl binary and the config that was why i called it like ./kubectl

oh i see if i call it ./kubectl, the dagger cli would not be able to access it becauase it is not in the path.Dammit

[azure viper]

right. Sorry about that. I'm already opening an issue to improve this error handling case 🙏

[fringe prism]

you are the man

[azure viper]

https://tenor.com/5gMZ.gif

[fringe prism]

lol

@azure viper please can i ask 2 more questions?

[azure viper]

yeah, sure

https://github.com/dagger/dagger/issues/7490

[fringe prism]

okay thanks

DAGGER_ENGINE_POD_NAME="$(kubectl get pod \
    --selector=name=dagger-dagger-helm-engine --namespace=dagger \
    --output=jsonpath='{.items[0].metadata.name}')"
export DAGGER_ENGINE_POD_NAME

_EXPERIMENTAL_DAGGER_RUNNER_HOST="kube-pod://$DAGGER_ENGINE_POD_NAME?namespace=dagger"
export _EXPERIMENTAL_DAGGER_RUNNER_HOST

if you have 3 worker node, so you get 3 engine pods, but from the above snippet from the dagger on kubernetes only the first detected pod will be in use. does dagger cli or sdk detected the other behine the scene from that one pod name?

i am asking because if it was a service, i would have understood it will loadbalace between the pods, but in this case if that pod goes down, what happens next

[azure viper]

this snippet is only to test that your connection to the engine actually works. What you want to do to run your workloads is basically schedule your pods in the same physical nodes that the Dagger daemonset is running and use the same node where the dagger pipeline pod is running

we already expose the engine socket to the host in the daemonset, the only thing you need to do is to create your pods with that socket mounted. Let me get you an example.

cc @worthy token and @stiff rapids. Just realized we're not mentioning this in our kubernetes guide: https://docs.dagger.io/integrations/104820/kubernetes

[fringe prism]

yes, infact that is was i just got working, using the unix socket, mounted it to the test rockylinux pod,

that was my second question

thank you very much, now i understand it

[azure viper]

perfect! you have everything you need the to rock it 🤘

https://tenor.com/buwcP.gif

[fringe prism]

https://tenor.com/view/brucelee-bow-master-smile-gif-15509287684475744673

@azure viper thank you again