#Dagger on kub & argo
shell elbow
Seems that isn't a setup you have all really looked at yet.
not yet anyway 🙂
Can you tell me more about the use case?
I can at the very least paint you a picture of where the architecture is headed and what an integration would look like. Seems very doable at first glance.
graceful viper
We are just starting to re-architect our CI/CD pipeline. The current idea is we have a fairly standardize setup for the CI portion. We are currently hoping to define a dagger.io pipeline in a given repo. When a PR is created Argo Events will pick that up and start an Argo-Workflow job that will run the dagger pipeline to create a docker container. I think we also want that pipeline to scan the container and other steps too. Once its complete it will upload the container to an ECR repo. Once there, our CD tool (called keel) will update to k8s deploymnet.
What I want to know is what we need to do to get dagger to build a container while running in a k8s pod on an EKS cluster.
Once I understand that I can create a simple one step argo-workflow template create the pod and run the dagger pipeline.
At least, thats the idea anyway.
shell elbow
Sure, I understand. This is a pretty common use case. The CI platform varies but the kubernetes part remains roughly the same
You can approach this in two stages:
-
A first stage that is good for developing and proof-of-concept; doesn’t require deploying anything new to your cluster
-
A second stage which adds performance optimizations for production - via an additional daemonset deployed on your cluster. The dagger pipeline code is not changed so you can drop this in later
Stage 1 consists simply of installing the dagger CLI in your CI environment (in your case, that argo workflow pod), and making sure it can run a container. From there it can bootstrap itself. The default method is to give it access to a docker engine (eg. by mounting a docker socket in the pod) but you can also use podman or any other oci-compatible container runner. Most of these approaches require the pod to be privileged although I believe there are possible rootless configurations (I’m not up to date on those)
grim badger
Stage 1 consists simply of installing the `dagger` CLI in your CI environment (i...
There are possibilities around this... but we haven't explored them much. What I remember from what @floral sentinel told me/us about them is that they require specific things from the host Linux kernel... which are in newer kernels, but those kernels are not in most of the popular distros yet.
hearty oyster
I think @half saddle has looked at Argo
grand holly
adding @cedar knot to this thread 🙂
cedar knot
thanks @grand holly
@graceful viper I'll have some time soon to try a prototype, my setup now installs both argo and dagger in Kind, just have to give it a whirl and see what happens
cedar knot
Close!
having issues connecting to the dagger engine
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: dagger-in-argo-
spec:
entrypoint: dagger-test
volumes:
- name: dagger-test
configMap:
name: dagger-test
templates:
- name: dagger-test
container:
image: golang:1.21.0-bookworm
command: ["sh", "./dagger.sh"]
workingDir: /work
env:
- name: "_EXPERIMENTAL_DAGGER_RUNNER_HOST"
value: "kube-pod://dagger-engine-24fjp?namespace=dagger&container=dagger-engine"
volumeMounts:
- name: dagger-test
mountPath: /work
package main
import (
"context"
"fmt"
"os"
"dagger.io/dagger"
)
func main() {
if err := build(context.Background()); err != nil {
fmt.Println(err)
}
fmt.Println("Done!")
}
func build(ctx context.Context) error {
fmt.Println("Building with Dagger")
// initialize Dagger client
client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
if err != nil {
return err
}
defer client.Close()
client.
Pipeline(fmt.Sprintf("golang")).
Container().
From("golang:1.21.0")
return nil
}
It could be a k8s issue, pods are in different namespaces
looks to be the same even if they are in the same namespace
Does _EXPERIMENTAL_DAGGER_RUNNER_HOST only work with the dagger cli?
cedar knot
ah, extra info when running via the dagger cli
new client: buildkit failed to respond: failed to list workers: Unavailable: connection error: desc = "transport: error while dialing: exec: \"kubectl\": e
xecutable file not found in $PATH"
time="2023-08-12T03:50:24 UTC" level=info msg="sub-process exited" argo=true error="<nil>"
Error: exit status 1
half saddle
Does _EXPERIMENTAL_DAGGER_RUNNER_HOST only work with the dagger cli?
works with just SDK too
cedar knot
ok, I just need to know what to set it to in k8s to get it to work now, tried many things
PAG! got it working based on #1136128194710356038 message
🤦♂️ I was one / short
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: dagger-in-argo-
spec:
entrypoint: dagger-test
volumes:
- name: dagger-test
configMap:
name: dagger-test
- name: dagger-conn
hostPath:
path: /var/run/dagger
templates:
- name: dagger-test
container:
image: golang:1.21.0-bookworm
command: ["bash", "./dagger.sh"]
workingDir: /work
env:
- name: "_EXPERIMENTAL_DAGGER_RUNNER_HOST"
value: "unix:///var/run/buildkit/buildkitd.sock"
volumeMounts:
- name: dagger-test
mountPath: /work
- name: dagger-conn
mountPath: /var/run/buildkit
#!/bin/bash
set -euo pipefail
apt-get update -y
apt-get install -y wget
cd /tmp
wget https://github.com/dagger/dagger/releases/download/v0.8.2/dagger_v0.8.2_linux_amd64.tar.gz -O dagger.tar.gz
tar -xf dagger.tar.gz
ls
mv dagger /usr/local/bin/dagger
cd /work
go mod download
echo "starting dagger"
dagger run go run main.go
Here is the DaemonSet for completeness
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: dagger-engine
labels:
k8s-app: dagger-engine
spec:
selector:
matchLabels:
name: dagger-engine
template:
metadata:
labels:
name: dagger-engine
spec:
securityContext:
runAsUser: 0
runAsGroup: 1001
fsGroup: 1001
fsGroupChangePolicy: OnRootMismatch
initContainers:
- name: download-bin
image: registry.dagger.io/engine:v0.8.2
command:
- sh
- -c
- wget https://github.com/moby/buildkit/releases/download/v0.12.1/buildkit-v0.12.1.linux-amd64.tar.gz -O /shared-bin/buildkit.tar.gz && tar -xvzf /shared-bin/buildkit.tar.gz --strip-components 1 bin/buildctl -C /shared-bin
volumeMounts:
- name: shared-bin
mountPath: /shared-bin
containers:
- name: dagger-engine
image: registry.dagger.io/engine:v0.8.2
imagePullPolicy: Always
args:
- --oci-max-parallelism
- num-cpu
securityContext:
privileged: true
capabilities:
add:
- ALL
readinessProbe:
exec:
command:
- /shared-bin/buildctl
- debug
- workers
initialDelaySeconds: 5
periodSeconds: 5
volumeMounts:
- name: varlibdagger
mountPath: /var/lib/dagger
- name: varrundagger
mountPath: /var/run/buildkit
- name: shared-bin
mountPath: /shared-bin
terminationGracePeriodSeconds: 300
volumes:
- name: shared-bin
emptyDir: {}
- name: varlibdagger
hostPath:
path: /var/lib/dagger
- name: varrundagger
hostPath:
path: /var/run/dagger
My argo setup (workflows,cd,events) is the default from the community maintained argo-helm charts
cedar knot
I can no longer be distracted from a CUE take on Helm!
simple jasper
@cedar knot Cool. How is the performance? Because I'm running a similar setup on my OpenShift cluster, but got some performance issues while connecting/creating the session via socket. (See here https://github.com/dagger/dagger/issues/5625)
did you experience something similar? What Kubernetes and Linux kernel version is running in your cluster?
