Configuring properly a self-hosted Github runner (Scaler set)

[drowsy hatch]

Hi,

I'm currently trying to setup dagger in a self-hosted runner.
I deployed this scale-set in order to test with a local dagger cache.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: dagger-gha-runner-scale-set
spec:
  releaseName: dagger-gha-runner-scale-set
  chart:
...
  values:
    runnerGroup: "default"
    githubConfigUrl: "https://github.com/Smana/demo-cloud-native-ref"
    githubConfigSecret: gha-runner-scale-set
    maxRunners: 5

    containerMode:
      type: "dind"

Then this scaler-set is used in a workflow as follows:

...
  test-cache:
    name: Testing in-cluster cache
    runs-on: dagger-gha-runner-scale-set
    container:
      image: ghcr.io/actions/actions-runner:latest
    env:
      _EXPERIMENTAL_DAGGER_RUNNER_HOST: "tcp://dagger-engine:8080"
      cloud-token: ${{ secrets.DAGGER_CLOUD_TOKEN }}

    steps:
      - name: Simulate a build with heavy packages
        uses: dagger/dagger-for-github@v5
        with:
          version: "latest"
          verb: call
          module: github.com/shykes/daggerverse.git/wolfi@dfb1f91fa463b779021d65011f0060f7decda0ba
          args: container --packages "python3,py3-pip,go,rust,clang"

I get the error

...
  VERSION=latest
  if [[ "$VERSION" == "latest" ]]; then
    VERSION=
  fi
  
  # The install.sh script creates path ${prefix_dir}/bin
  cd "$prefix_dir" && { \
  curl -sL https://dl.dagger.io/dagger/install.sh 2>/dev/null | \
  DAGGER_VERSION=$VERSION sh; }
  shell: bash --noprofile --norc -e -o pipefail {0}
  env:
    _EXPERIMENTAL_DAGGER_RUNNER_HOST: tcp://dagger-engine:8080
    cloud-token: ***

The default container image used is ghcr.io/actions/actions-runner:latest and I guess there are preparation steps missing.

The archived documentation (https://archive.docs.dagger.io/0.9/934191/eks-github-karpenter/) uses a deprecated method to run self-hosted runners.

Any guidance would be welcome 🙂

[viscid notch]

cc @near copper 🙏

[drowsy hatch]

my experimentations can be found here: https://github.com/Smana/demo-cloud-native-ref/pull/311

[drowsy hatch]

I'm gonna try building an image, but it seems the installation process is part of the github action. Anyway, I'll let you know

[near copper]

Hey @drowsy hatch! Sorry for the delay. I don't see an error in what you sent, could it be that its missing or am I missing something?

[drowsy hatch]

the error is not relevant actually, its an exit code 127, but when I ran the script on my laptop I noticed that the image is pretty light, without curl for instance.

[drowsy hatch]

here it is https://github.com/Smana/demo-cloud-native-ref/actions/runs/9651388987/job/26619126704

[drowsy hatch]

Well using another images with more tools (e.g. curl) is better, now I really have an error:
1 : connect
2 : connecting to engine
2 : connecting to engine ERROR [10m0.0s]
2 : ! new client: context deadline exceeded
1 : connect ERROR [10m0.0s]
1 : ! start engine: new client: context deadline exceeded
Error: start engine: new client: context deadline exceeded
Error: Process completed with exit code 1.

I'm troubleshooting, that might be caused by network policies or missconfiguration

I didn't see any drop in Hubble though (Cilium).

I think I found the culprit: the service is not properly set
kg ep -n tooling
NAME ENDPOINTS AGE
dagger-engine <none> 17m

Great 🎉 , that worked
First run took > 3m

Second

my initial prob was caused by the default image, which is secure and very light. Now I need to find the best tradeoff in terms of security. Do you recommend a docker image?

[near copper]

Are you referring to the runners image?

[drowsy hatch]

I'm refering to the image used in the workflow, but honestly I need to dig further in order to understand, when I checked the pod it is composed of 2 containers runner and dind.
The image I used is smana/nettools , I think I can use an lighter image configured with just the dagger cli. I'll try to confirm that

name: Cache testing

on:
  pull_request:
  push:
    branches: ["main"]

jobs:

  test-cache:
    name: Testing in-cluster cache
    runs-on: dagger-gha-runner-scale-set
    container:
      image: smana/nettools:latest
    env:
      _EXPERIMENTAL_DAGGER_RUNNER_HOST: "tcp://dagger-engine:8080"
      cloud-token: ${{ secrets.DAGGER_CLOUD_TOKEN }}

    steps:
      # - run: echo "🎉 This job uses runner scale set runners!"
      - name: Simulate a build with heavy packages
        uses: dagger/dagger-for-github@v5
        with:
          version: "latest"
          verb: call
          module: github.com/shykes/daggerverse.git/wolfi@dfb1f91fa463b779021d65011f0060f7decda0ba
          args: container --packages "python3,py3-pip,go,r`

[near copper]

Internally (and for workflows running in dagger/dagger) we use the default runner image of the runner container and we mount the dagger CLI there. We don't want to build and maintain a separate image, so instead we added an initContainer that downloads the dagger CLI binary with the requested version and then we mount that into the runner container. When it comes to installing dependencies, we just run apt install in the exec for the runner container. There are very few additional tools we want to install since almost everything runs within a dagger call.

If you are curious, we docummented with a lot of technical detail how we do CI internally here: https://dagger.io/blog/argo-cd-kubernetes

If you don't care about dynamic runners (meaning: runners that show up automagically as new dagger versions are released, e.g dagger-v0-11-9), you can ignore the Argo CD part