#Remote Modules on K8s Pod using Dagger Helm Chart

1 messages · Page 1 of 1 (latest)

bold berry
#

I am using Dagger in a Cluster environment.

  • Dagger engine is installed as DaemonSet using the Dagger Helm chart
  • Dagger CLI is downloaded in the target POD as initContainer
  • setting env var _EXPERIMENTAL_DAGGER_RUNNER_HOST with value: unix:///var/run/dagger/engine.sock

It used to work, but now I have a strange error.
Local modules do work, Remote ones don't

/home/dagger-run # dagger call -m github.com/shykes/daggerverse/hello hello --giant --name internet
✔ connect 0.0s
✘ load module: github.com/shykes/daggerverse/hello 0.6s ERROR
┇ initializing module › ModuleSource.asModule › 
✘ Container.file(path: "typedefs.json"): File! 0.0s ERROR
! failed to add directory "/": failed to copy source directory: failed to copy xattrs: failed to set xattr
  "security.selinux" on /var/lib/dagger/worker/snapshots/snapshots/100/fs/hello: permission denied

This does not happen if I clone the repo and run it locally everything is OK.

I am not getting what I am missing since I have not changed anything, I suspect the problem is with the node in EKS, also because the engine is referred from an hostPath volume.
Especially,

  • why SELinux issue since I am in alpine
  • why it is referring to root folder.

Is anyone experiencing this?

bold berry
#

UPDATE: same error switching to ubuntu

wicked forge
#

We also ran into this recently trying to get our Dagger experimental setup going again. We ended up creating a headless service and the runners use tcp connections for their "dedicated" engine.

This is an example of the service for the engines:

apiVersion: v1
kind: Service
metadata:
  name: dagger-engines 
  namespace: dagger
spec:
  clusterIP: None
  selector:
    app.kubernetes.io/name: dagger
  ports:
    - name: dagger
      port: 1234
      targetPort: 1234

and we set the DAGGER_HOST env variable in the runner to something like:

- name: DAGGER_HOST
              value: "tcp://dagger-engine-0.dagger-engines.dagger.svc.cluster.local:1234"

dagger-engine-0 is the name of the engine pod.

Important note: We are deploying the engines using a StatefulSet (not a standard Deployment via a DaemonSet). This is required so that the pods get predictable names (like dagger-engine-0), which allows the Headless Service DNS to resolve correctly.

We also make sure the runner pods always get the same URL config'ed too, so the connection per runner pod is "sticky" and the build cache can be used properly. It's all experimental/POC currently, but it's what we've come up with!

bold berry
#

That's weird because I used to have this DaemonSet setup and it worked, I don't see any specific change. I am wondering if it is smt related to the underlying BottelrocketOS

wicked forge
#

Same here. But then it didn't work. 🤷🏻

lavish flame
bold berry
#

@lavish flame amazing catch. I swear I already used Dagger + BottlerocketOS. But first time I see this. I'll dig around, you are a lifesaver

lavish flame
#

@bold berry by any chance are you doing multi-platform builds?

bold berry
#

wait... sending you full example

lavish flame
bold berry
#

My test to reproduce is dead simple...

I have

  • dagger-engine as DaemonSet
  • dagger-client installed within the pod
  • dagger engine referred as hostPath, using Unix socket (it used to work smoothly)

Here is my test pod

apiVersion: v1
kind: Pod
metadata:
  name: ubu
spec:
  initContainers:
  - name: dagger-cli
    image: ubuntu:24.04
    command:
      ... install Dagger cli
    volumeMounts:
      - name: dagger-cli
        mountPath: /opt/dagger/bin
  containers:
    - name: ubuntu
      image: ubuntu:24.04
      command:
        - bash
        - -exc
        - |-
          cp /opt/dagger/bin/dagger /bin/dagger
          apt update
          apt install -y --no-install-recommends git-core curl ca-certificates
          sleep 3600
      env:
        - name: _EXPERIMENTAL_DAGGER_RUNNER_HOST
          value: unix:///var/run/dagger/engine.sock
      volumeMounts:
        - name: dagger-cli
          mountPath: /opt/dagger/bin
        - name: dagger-engine
          mountPath: /var/run/dagger
  volumes:
  - name: dagger-cli
    emptyDir: {}
  # Dagger engine installed from Helm Dagger Helm chart is configured 
  # to use hostPath for run vol
  - name: dagger-engine
    hostPath:
      path: /var/run/dagger-dagger-engine-dagger-helm

To test I am getting into the container in the Pod itself.

Running a remote module like

dagger call -m github.com/shykes/daggerverse/hello hello --giant --name internet

but ....

git clone https://github.com/shykes/daggerverse.git
cd daggerverse/hello
dagger call hello --giant --name internet

succeeds, and this suggest Dagger engine is working... For example if I unset the env variable also this run won't work.

I have tried with old version of BottlerocketOS (from October, longer before I used this), and the error is still there....
TBH I have not tried yet old version of Dagger client or Dagger Helm chart
Could be a good try

Also trying locally with KinD or Ks3 can be good... Or K3s in Dagger itself ❤️

bold berry
#

Just to confirm my suspects...
Using Dagger Engine v.18.14 the issue disappears.

I also tried various combination of versions (latest, v19, v18) in different environments (k3s,Kind) and I can confirm the issue is confined in the combination BottlerocketOS (selinux) + Dagger engine v.19

@lavish flame is there a place where we can track this in the Dagger issue tracker, or the issues you pointed out are enough?

lavish flame
bold berry
flat monolith
lavish flame
flat monolith
#

So making it universally ignored will squash it once and for all, in theory

lavish flame
flat monolith
bold berry
#

@flat monolith Please ping me when the bug is handled or even better when it is included in a future release of Dagger 🙏

dark flint
#

Hi I am looking for a good resource to help me get started on how to deploy a Helm chart to a Kubernetes cluster. If I had a "hello world helm" example that is probably all I need. Thanks

lavish flame