Dagger Engine DockerHub Auth

[tidal lake]

Hey Team,

I'm running a long runner dagger engine in k8s for our centralised CI pipelines. We've just started hitting the following docker rate limit error:

429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

I have a docker account that I can use, but I can't work out how to get the dagger-eninge to pick this up. I have the config.json with valid creds set within the dagger enigne container:

cat $DOCKER_CONFIG/config.json
{"auths":{"https://registry-1.docker.io/v2/": REDACTED }

notes from build kit would suggest this should work:

If credentials are required, buildctl will attempt to read Docker configuration file $DOCKER_CONFIG/config.json. $DOCKER_CONFIG defaults to ~/.docker.

https://github.com/moby/buildkit

Has anyone been able to get this working?
Is there a different env var / engine.toml setting I need to use?

[carmine crystal]

Heya, what in the contents of the auths key?

the full format should look like this:

{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "<redacted>"
        }
    }
}

so you'd need the auth part inside auths

it's also needs to be on the dagger client (wherever you're calling dagger run/call from)
not the dagger server

[tidal lake]

auths has auth, is there anyway to have it on the dagger server rather than the client? I can then manage that centrally rather than for each job ...

unless I find a way to inject it on the other side

reads the gitlab runner docs

[carmine crystal]

(ignore my last message, that's not right)

[tidal lake]

looks like that's for certs 🙂 i was excited for a second 😂

[carmine crystal]

ugh, i was looking for an issue for this a while back - i know we've discussed this before, i just can't find it

[tidal lake]

looks like I can mount a secret in all gitlab runners as they come up, which is good. I guess it's the same thing I've tried on the engine side, a config.json and an env var for DOCKER_CONFIG to the correct path

I'll give that a try and report back

[carmine crystal]

the tl;dr of it as i remember was - we need to make some changes in buildkit to allow config server side - i don't think there's any objections to it, it just needs to be prioritized and actually done 😦

[tidal lake]

from my (quick) research, buildkit do seem to support passing this config file in already

https://github.com/moby/buildkit/issues/4456

https://github.com/moby/buildkit/issues/2136

however it's not worked for me /shrug

[carmine crystal]

i don't think these issues are related.
buildctl-daemonless is quite a different setup to how dagger's buildkit works, so doesn't really apply here
about halfway through the second one, it's mentioned that:

Credentials are client side

[tidal lake]

hmm, still no luck. This is on the client side now:

# env | grep DOCKER
DOCKER_CONFIG=/dockercreds
# cat /dockercreds/config.json
{"auths":{"https://registry-1.docker.io/v2/":{"username":"","password":"","email":"","auth":""}}}

Values removed

creds are valid (sanity checking for my side :p)

❯ docker login --username <USERNAME>
Password:
Login Succeeded

Logging in with your password grants your terminal complete access to your account.
For better security, log in with a limited-privilege personal access token. Learn more at https://docs.docker.com/go/access-tokens/

unless, the client needs the file in a different place

https://docs.gitlab.com/ee/ci/docker/authenticate_registry.html#kubernetes

Testing without the DOCKER_CONFIG env var now

ok - that's worked!!

Thanks @carmine crystal for the pointers

[carmine crystal]

eyy awesome!

[tidal lake]

for anyone else who finds this in the future:

apiVersion: v1
kind: Secret
metadata:
  name: dockerhub-credentials
  namespace: gitlab-runner
data:
  .dockerconfigjson: <Base 64 encoded config>

gitlab config.toml snippet

                [[runners.kubernetes.volumes.secret]]
                  name = "dockerhub-credentials"
                  mount_path = "/root/.docker"
                  read_only = true
                  [runners.kubernetes.volumes.secret.items]
                    ".dockerconfigjson" = "config.json"

oh, myabe not. I think my timeout window rolled 😂

It's failing again now 😦

[tidal lake]

missed a run as root ... added

[runners.kubernetes]
privileged = true

to the gitlab runner conig. testing now, but if it works ... do I trust it?

indeed - just hit the rate limit again

[jagged rampart]

we're doing this ourselves for our github runners

let me share how we're doing this

it's basically very similar to this approach

        initContainers:
        - command:
          - sh
          - -c
          - ' cp /tmp/docker/config.json /home/runner/.docker/config.json'
          image: summerwind/actions-runner-dind
          name: init-myservice
          resources: {}
          volumeMounts:
          - mountPath: /tmp/docker
            name: dockerconfig
          - mountPath: /home/runner/.docker
            name: runnerdockercfg
        labels:
        - dagger-runner-8c-32g
        organization: dagger
        resources:
          requests:
            cpu: "8"
            memory: 8Gi
        securityContext:
          fsGroup: 1001
          fsGroupChangePolicy: OnRootMismatch
        tolerations:
        - effect: NoSchedule
          key: actions-runner
          operator: Exists
        volumeMounts:
        - mountPath: /var/run/buildkit
          name: varrundagger
        - mountPath: /home/runner/.docker
          name: runnerdockercfg
        volumes:
        - hostPath:
            path: /var/run/dagger
          name: varrundagger
        - name: dockerconfig
          secret:
            items:
            - key: .dockerconfigjson
              path: config.json
            secretName: regcred
        - emptyDir: {}
          name: runnerdockercfg

so, similar to your approach, we have a regcred secret where the dockerconfigjson secret lives and then we have an initContainer that runs before the dind runner container starts which puts the docker/config.json in the right place and with the proper permissions

we had to do this because otherwise the config.json would be created as readonly inside the dind container which caused issues in some steps of the pipeline doing a docker login against other registries

hope this helps!

[tidal lake]

Thanks - I'm going to get some food and then come back with fresh view - I wonder if there is an auth endpoint on dockerhub I can write a quick dagger pipeline to check ...

[jagged rampart]

to check if your engine is authenticated you mean? Well.. you can try pulling a private image

that will definitely tell you if your engine has access to the secrets

[tidal lake]

thanks, this is where I'm at ...

func dockerAuthTest(ctx context.Context, client *dagger.Client) error {
    // Fetch the current user
    homeDir, _ := os.UserHomeDir()
    fmt.Println("Home dir: " + homeDir)
    // check if the docker config file exists
    _, err := os.Stat(homeDir + "/.docker/config.json")
    if err != nil {
        fmt.Println("Docker config file does not exist")
    }

    fmt.Println("Fetching opencp shim container")
    out, err := client.Container().
        From("PRIVATE/IMAGE").
        WithEntrypoint([]string{"ls", "/"}).
        Stdout(ctx)

    if err != nil {
        return err
    }

    fmt.Printf("Output: %s\n", out)

    return nil
}

Back to the days of this sort of commit history:

* 88f1cd99 (HEAD -> dagger-sidecar, origin/dagger-sidecar) fix
* fb1d40cc fix
* 0cb6bc0d fix
* 71be8c1e fix
* 772bdf85 updates
* 71f44c4e updates
* 9f25bf33 add test

one day this will be a thing of the past ...

[jagged rampart]

seems ok to me

[tidal lake]

sigh turns out ...

https://index.docker.io/v2/ != https://index.docker.io/v1/

works with the above url when set on the client side. For complteness on this issue, I'm going to get the original DOCKER_CONFIG on the engine side

[jagged rampart]

IIRC the correct URL should be v1?

[tidal lake]

yeah - v1 works

and just confirmed that setting it on the dagger engine contianer does not work

Thanks @jagged rampart and @carmine crystal

[jagged rampart]

awesome!