#Post-Kubecon love

1 messages ยท Page 1 of 1 (latest)

timber estuary
#

Nice ๐Ÿ™‚ Starting a thread to pick your brain on what you're building!

#

cc @keen finch @merry willow @little pollen

supple pulsar
#

Hey @timber estuary ใ€ฐ๏ธ

timber estuary
#

Mostly I'm curious on the caching piece. which we started discussing in person, but didn't get a chance to finish.

supple pulsar
#

So caching is out to our Object Store with the _EXPERIMENTAL_DAGGER_CACHE_CONFIG flag

#

Full spec that I'm deploying is here:

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dagger-engine
  namespace: dagger-engine
  labels:
    k8s-app: dagger-engine
  annotations:
    argocd.argoproj.io/sync-wave: "10"
spec:
  selector:
    matchLabels:
      name: dagger-engine
  template:
    metadata:
      labels:
        name: dagger-engine
    spec:
      containers:
        - name: dagger-engine
          image: registry.dagger.io/engine:v0.9.3
          imagePullPolicy: Always
          args:
            - "--oci-max-parallelism"
            - "num-cpu"
          env:
            - name: _EXPERIMENTAL_DAGGER_CACHE_CONFIG
              value: type=s3,mode=max,endpoint_url=.....
          securityContext:
            privileged: true
          resources:
            limits:
              cpu: 3
              memory: 12Gi
          readinessProbe:
            exec:
              command: ["buildctl", "debug", "workers"]
            initialDelaySeconds: 5
            timeoutSeconds: 30
            periodSeconds: 15
            successThreshold: 1
            failureThreshold: 10
          volumeMounts:
            - name: varlibdagger
              mountPath: /var/lib/dagger
            - name: varrundagger
              mountPath: /var/run/buildkit
            - name: dagger-engine-config
              mountPath: /etc/dagger/engine.toml
              subPath: engine.toml
      terminationGracePeriodSeconds: 300
      volumes:
        - name: varlibdagger
          hostPath:
            path: /var/lib/dagger
            type: DirectoryOrCreate
        - name: varrundagger
          hostPath:
            path: /var/run/dagger
            type: DirectoryOrCreate
        - name: dagger-engine-config
          configMap:
            name: engine-config
            items:
              - key: engine.toml
                path: engine.toml
#

that's going out to 3 nodes at the moment.

I can roll those pods as much as I want, and the cache between restarts and each deploy means that I get the benefits as if it were a local single node cache

timber estuary
#

So it's a standard issue Dagger engine (no custom buildkit), with experimental buildkit cache export using the S3 adapter? And S3 API compat in the Civo object storage?

supple pulsar
#

So it's a standard issue Dagger engine

Yes - I in was running a buildkit deployment 0.5.x that did the same thing

This now seems to work in the std dagger engine

#

I have some fun when using our k3s / alpine backed clustres (cgroupv1 / v2 issues with builkit), but it worked out of the box with our Talos clusters

timber estuary
#

It does have limitations though, cache volumes not persisted for example.

supple pulsar
#

I've not played with cache volumes - just files uploaded files for a quick yamllint test

timber estuary
#

At some point we should talk about upgrading you to something even better ๐Ÿ™‚ But this is a great incremental step, I'm glad you're no longer dealing with a custom buildkit, not being able to upgrade to latest Dagger etc.

supple pulsar
# 
    client := c.Pipeline("yaml").Pipeline("lint")

    golang := client.Container().
        From("registry.gitlab.com/pipeline-components/yamllint:0.29.0").
        WithDirectory(".", client.Host().Directory(".",
            dagger.HostDirectoryOpts{
                Include: []string{".yamllint", "**/*.yaml", "**/*.yml"},
                Exclude: []string{"**/node_modules", ".cache"},
            },
        )).
        WithExec([]string{"yamllint", "."})

    version, err := golang.Stdout(ctx)
    if err != nil {
        panic(err)
    }

    // print output
    fmt.Println("Hello from Yaml Lint and " + version)
#

I'll start playing with cache volumes over the next few days and I'll report back how it goes

timber estuary
#

Awesome progress

#

FYI you kicked off an internal thread about how to support Civo Storage in our Distributed Cache service in the future ๐Ÿ™‚

supple pulsar
#

our internal testing of it and to our own clusters has shown that it's lightning fast out of the box - we have even more performance to add on our roadmap as well

It's been great for our general gitlab ci backed cache builds, and our ML pipelines as well

#

we have a container registry internally under R&D as well that may be a useful cache layer

#

I can likely get the dagger team some very early alpha access to that if you think it's worth discussing further

timber estuary
#

Thanks @supple pulsar ! I think the blob storage is more likely integration point given how Dagger Cloud works. That said, we should make sure there is a great Dagger Module for every Civo Service, including your alpha registry! ๐Ÿ™‚