#Kubernetes in Dagger

1 messages · Page 1 of 1 (latest)

vestal tiger
#

👋 I can help you getting ublocked here

#

what SDK are you trying Dagger with?

short tusk
#

The Go SDK

#

Can we document a working example of spinning up a Kind or K3d cluster, and then running kubectl or helm against it via Dagger?

#

My biggest concern is the privileged requirements could make it hard to get working in CI like CircleCI or Github Actions

vestal tiger
#

both Cirlce CI and GHA allows running privileged Docker containers, so we should be able to make it run correctly there

short tusk
# 
❯ docker run --rm -it --privileged rancher/k3s:v1.27.2-k3s1 server
❯ docker exec -it cbb kubectl get node
/ # kubectl get node
NAME           STATUS   ROLES                  AGE   VERSION
cbb6907615a8   Ready    control-plane,master   14s   v1.27.2+k3s1

The above works great. But I can't get the equivalent working using Dagger's Go SDK. It keeps blocking on the k3s container probably blocked waiting for port 6443 to be live?

vestal tiger
#

yep, just managed to get this one working @short tusk. Here's an initial snippet you can use as a starting point:

package main

import (
    "context"
    "fmt"
    "os"
    "time"

    "dagger.io/dagger"
)

func main() {
    ctx := context.Background()

    // create Dagger client
    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
    if err != nil {
        panic(err)
    }
    defer client.Close()

    cache := client.CacheVolume("k3s")

    // create k3s service container
    httpSrv := client.Container().
        From("rancher/k3s").
        WithMountedCache("/etc/rancher/k3s", cache).
        WithEntrypoint([]string{"sh", "-c"}).
        WithExec([]string{"k3s server --snapshotter native --bind-address $(ip route | grep src | awk '{print $NF}')"}, dagger.ContainerWithExecOpts{InsecureRootCapabilities: true}).
        WithExposedPort(6443)

    val, err := client.Container().
        From("bitnami/kubectl").
        WithMountedCache("/cache/k3s", cache).
        WithServiceBinding("k3s", httpSrv).
        WithEnvVariable("CACHE", time.Now().String()).
        WithUser("root").
        WithExec([]string{"cp", "/cache/k3s/k3s.yaml", "/.kube/config"}, dagger.ContainerWithExecOpts{SkipEntrypoint: true}).
        WithExec([]string{"cat", "/.kube/config"}, dagger.ContainerWithExecOpts{SkipEntrypoint: true}).
        WithExec([]string{"chown", "1001:0", "/.kube/config"}, dagger.ContainerWithExecOpts{SkipEntrypoint: true}).
        WithUser("1001").
        WithEntrypoint([]string{"sh", "-c"}).
        WithExec([]string{"while true; do kubectl get nodes -o wide; sleep 2; done"}).
        Stdout(ctx)
    if err != nil {
        panic(err)
    }

    fmt.Println(val)
}
#

take into account that this is a rough example, it'd be great to see the final helm pipeline that you end up coming up with so we can use that in our examples

#

as you can see from above, this pipeline does the following:

  • Start a k3s service container using the native snapshotter and the service internal bind-address so kube can be accessible from the pipeline containers
  • Write the generated k3s config files to a shared cache so then the pipeline container can get the k3s.yaml and use it as kube.config
  • Preform a while true at the very end since the control-plane node generally takes a bit to become ready after the API server starts to listen.
#

let us know how this goes. I'llc reate a follow-up issue to document this officially. cc @knotty summit@pine drift

short tusk
vestal tiger
#

nit: both chown and mkdir accept variadic files and directories respectively, so you can avoid doing those multiple WithExec calls and just send everything as argument

short tusk
#

There are some helm-isms that need to get resolved with file permissions but overall it’s going well I think

vestal tiger
hidden sandal
wind cloak
#

Marcos, looping back from your comment about moving the WithEnvVariable("CACHE", time.Now().String()) elsewhere for caching the folder and file s copy.

So now I dont have any issue to run it... so I guess something was wrong with my docker (probably needed a restart).
The thing that I am noticed, is that the file and folders are not being cached, e.g:

func (k *K8sInstance) exec(name, command string) (string, error) {
    parentDir, err := filepath.Abs("..")
    if err != nil {
        panic(err)
    }
    return k.client.Pipeline(name).Pipeline(command).Container().
        From("dtzar/helm-kubectl:3.12").
        WithMountedCache("/cache/k3s", k.cache).
        WithDirectory("/host", k.client.Host().Directory(parentDir)).
        WithServiceBinding("k3s", k.container).
        WithUser("root").
        WithExec([]string{"mkdir", "-p", "/.kube"}, dagger.ContainerWithExecOpts{SkipEntrypoint: true}).
        WithExec([]string{"mkdir", "-p", "/.config/helm"}, dagger.ContainerWithExecOpts{SkipEntrypoint: true}).
        WithExec([]string{"mkdir", "-p", "/.cache/helm"}, dagger.ContainerWithExecOpts{SkipEntrypoint: true}).
        WithExec([]string{"cp", "/cache/k3s/k3s.yaml", "/.kube/config"}, dagger.ContainerWithExecOpts{SkipEntrypoint: true}).
        WithExec([]string{"cat", "/.kube/config"}, dagger.ContainerWithExecOpts{SkipEntrypoint: true}).
        WithExec([]string{"chown", "1001:0", "/.kube/config"}, dagger.ContainerWithExecOpts{SkipEntrypoint: true}).
        WithExec([]string{"chown", "1001:0", "/.config/helm"}, dagger.ContainerWithExecOpts{SkipEntrypoint: true}).
        WithExec([]string{"chown", "1001:0", "/.cache/helm"}, dagger.ContainerWithExecOpts{SkipEntrypoint: true}).
        WithUser("1001").
        WithEnvVariable("CACHE", time.Now().String()).
        WithEntrypoint([]string{"sh", "-c"}).
        WithExec([]string{command}).
        Stdout(k.ctx)
}

as you can see in the attached screenshot.

#

not sure if i am doing this properly or if this is even doable, but I am really wondering if we could cache it 🤔

vestal tiger
#

Kubernetes in Dagger