Pull images from private docker registry

[night plover]

Hello,
Due to the Docker Hub rate limitation, we are forced to copy the common images (like python image) to our private container registry. However, I can't find a way to efficiently login to the private registry in the Dagger engine.
I saw the documentation about Custom Registry Mirrors but it doesn't include any information about authentication so I suppose it is not where I'll find my solution.
I'm deploying the dagger pipelines on Kubernetes in the following way:

• Init container: run the docker login command to generate the file in /root/.docker/config.json having the authentication to the private container registry
• Sidecar: Dagger engine where I mount the same volume containing the at /root/.docker (I verified, the file is here and contains the authentication to the private registry)
• Main container: Dagger client running my dagger call ... command to run the pipeline.

Here is a very simplified version of my pod YAML. Everything is running fine, until the moment I try to pull a image from my registry.

volumes:
  - name: dagger-socket
    emptyDir: {}
  - name: docker-configuration
    emptyDir: {}
templates:
  initContainers:
    - name: dind-login
      image: docker:27-rc
      command:
        - sh
        - -c
        - "docker login <registry_url> -u <username> -p <password>"
      volumeMounts:
        - mountPath: /root/.docker
          name: docker-configuration
  sidecars:
    - name: dagger-engine
      image: "registry.dagger.io/engine:v0.14.0"
      volumeMounts:
        - mountPath: /var/run/buildkit
          name: dagger-socket
        - mountPath: /root/.docker
          name: docker-configuration
  containers:
    - image: alpine:latest
      command:
        - sh
        - -c
        - "dagger call ..."
      env:
        - name: "_EXPERIMENTAL_DAGGER_RUNNER_HOST"
          value: "unix:///var/run/dagger/buildkitd.sock"
      volumeMounts:
        - mountPath: /var/run/dagger
          name: dagger-socket

At the end, when I'm doing that in my pipeline:

dag.container().from_("<my private regitry>/python:3.12.7-bullseye")

I have the following result:

Stderr:
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ Function execution error: resolve: failed to resolve image                   │
│ <my private registry>/python:3.12.7-bullseye: failed to resolve source         │
│ metadata for <my private registry>/python:3.12.7-bullseye: failed to           │
│ authorize: failed to fetch anonymous token: unexpected status from GET       │
│ request to                                                                   │
│ https://<my private registry>/oauth2/token?...: 401 Unauthorized                         │
╰──────────────────────────────────────────────────────────────────────────────╯

Just like I have no authentication at all

What I know so far:

• I can't use with_registry_auth as I don't have a Container object before pulling the image
• No documentation is provided for authentication using Custom Registry Mirrors
• The /root/.docker/config.json is present in the container with the Dagger engine and contains the right authentication.
• Doing the same authentication process locally with the dagger engine on my computer works fine.

Does anybody has an idea what am I missing?

[wise cedar]

are you logged in on your client machine with docker login? The dagger CLI should automatically proxy docker credentials

[night plover]

As I said, I'm on Kubernetes. It means that the dagger engine run in a Docker-in-Docker environment, which make impossible to docker login on it, this is why I have mounted the .docker/config.json in the pod.
Dagger engine image doesn't have the docker client installed, so I can't run docker login

[brazen crater]

@night plover not sure I follow the with_registry_auth comment

you can do dag.container().with_registry_auth()._from()

and that will use the correct credentials

[night plover]

Oh, I didn't know it was possible to do that ! 😅 Is it possible to do the same with docker_build() function? We are both pulling image in _from() function and in our Dockerfiles

[brazen crater]

let me check really quick

having said that, the .docker/config,json approach should have worked also if you prefer to go that way

@night plover yes, this works:

    s := dag.SetSecret("dockerhub", "$TOKEN")
    dag.Container().WithRegistryAuth("index.docker.io", "$USER", s).Sync(context.Background())
    return src.DockerBuild()

^ that's Go code but the same Dagger API you have in Python

just make sure to await on the container().sync()function call

so the secret gets set in the engine side

you should also pass the secret via function arguments and not hardcode it in the code 🙏

[night plover]

Thank you very much! I'm going to try it!

[night plover]

@brazen crater Do you think it would be a good idea to give the opportunity to the users to set the docker authentication on the Engine level to avoid the authentication redundance when building multiple Dockerfiles?

[brazen crater]

@night plover as I mentioned above, the .docker/config.json should also work. I was reading your first post in this help thread and I think I know why that didn't work the first time. the .docker/config.json file should be located in the main container which actually runs the dagger call, not in the engine container. Have you tried that?

[night plover]

That is exactly what I thought too, I just finished implementing it in a way that the Dagger client has the authentication and not the Engine and it works! I slightly changed my approach by using the docker:cli image to run the dagger client, which allowed me to directly do the docker login in the container that run the client.
Thank you a lot for your help!

[brazen crater]

👍

[wise cedar]

👆

[knotty steppe]

@brazen crater tried that code in python but doesn't seem to have the expected effect: #1404414159549894696 message

[brazen crater]

@knotty steppe have a trace URL to check the error by any chance?

[knotty steppe]

That's the connecting to dagger servers? We did not set that up.

In the dagger logs I see things like:

resolving docker.artifacts.airamed.local/ghcr.io/astral-sh/uv:0.7.19
85 : ┆ ┆ HTTP GET
85 : ┆ ┆ HTTP GET DONE [0.0s]
86 : ┆ ┆ remotes.docker.resolver.HTTPRequest
87 : ┆ ┆ HTTP HEAD
87 : ┆ ┆ HTTP HEAD ERROR [3.9s]
87 : ┆ ┆ ! context canceled
86 : ┆ ┆ remotes.docker.resolver.HTTPRequest ERROR [3.9s]
86 : ┆ ┆ ! context canceled
84 : ┆ ┆ resolving docker.artifacts.airamed.local/ghcr.io/astral-sh/uv:0.7.19 ERROR [4.0s]
84 : ┆ ┆ ! failed to do request: Head "https://docker.artifacts.airamed.local/v2/ghcr.io/astral-sh/uv/manifests/0.7.19": context canceled
88 : ┆ ┆ resolving docker.artifacts.airamed.local/python:3.10-slim-bullseye
89 : ┆ ┆ HTTP GET
89 : ┆ ┆ HTTP GET DONE [0.0s]
90 : ┆ ┆ remotes.docker.resolver.HTTPRequest
90 : ┆ ┆ remotes.docker.resolver.HTTPRequest DONE [1.1s]
88 : ┆ ┆ resolving docker.artifacts.airamed.local/python:3.10-slim-bullseye ERROR [4.0s]
88 : ┆ ┆ ! failed to copy: httpReadSeeker: failed open: unexpected status code https://docker.artifacts.airamed.local/v2/python/manifests/sha256:f1fb49e4d5501ac93d0ca519fb7ee6250842245aba8612926a46a0832a1ed089: 403 Forbidden - Server message: unknown: Forbidden
91 : ┆ ┆ resolving docker.artifacts.airamed.local/nvidia/cuda:11.3.0-cudnn8-runtime-ubuntu20.04
91 : ┆ ┆ resolving docker.artifacts.airamed.local/nvidia/cuda:11.3.0-cudnn8-runtime-ubuntu20.04 DONE [1.0s]
75 : ┆ Directory.dockerBuild ERROR [4.0s]
75 : ┆ ! docker.artifacts.airamed.local/python:3.10-slim-bullseye: failed to resolve source metadata for docker.artifacts.airamed.local/python:3.10-slim-bullseye: failed to copy: httpReadSeeker: failed open: unexpected status code https://docker.artifacts.airamed.local/v2/python/manifests/sha256:f1fb49e4d5501ac93d0ca519fb7ee6250842245aba8612926a46a0832a1ed089: 403 Forbidden - Server message: unknown: Forbidden

In our Artifactory logs I saw that it's trying to connect via "anonymous" user, (which is the non-authenticated user) and then Artifactory rejects with 403

[brazen crater]

yes, it's usually easier to understand what's happening there 🙏

would it be possible if you could share the complete trace? Since I don't see the WithRegistryAuth above 🙏

[knotty steppe]

Here the whole trace

I see that for the withRegistryAuth step it says "CACHED"

[brazen crater]

yeah.. I was seeing that. Mind adding a CACHE buster to that call to see if that fixes it?

await dag.container()
    .with_env_variable("CACHEBUSTER", str(datetime.now()))
    .with_registry_auth(
        "docker.artifacts.company.local", registry_username, registry_password
    )
    .sync()

[knotty steppe]

Indeed that helped.

I wonder, should I leave that in? Will that only apply to that one specific container step or will that apply for the whole pipeline?

[brazen crater]

the registry auth is global per session (dagger call). If any other parts of the pipeline try to access the docker.artifacts.company.local registrty, it'll use the credentials registered in that step

@knotty steppe also bare in mind that dagger is compatible with docekr's .config.json for registry auth as well so if you do docker login in the same environment where you run dagger call, dagger will pick those credentials automatically

[knotty steppe]

It's running in Kubernetes, so that docker login might be a bit more tricky? I know I could also mount a docker config, and I was considering that but then I thought that I already have the login data as part of my dagger module and staying with a pipeline which does the same thing locally as on the cluster sounded better to me, hence trying to solve it within the dagger module.

The question with leaving it in wasn't on the withRegistryAuth but on the CACHEBUSTER

[brazen crater]

yep, solving withing the dagger module makes sense in this case 🙏

[brazen crater]

it'll only apply to that with_registry_auth step so it's safe to leave it here 🙏

[knotty steppe]

Perfect, thanks!

Such a great help here!

I think some of that info could be included in the docs, avoiding others running into the same.

[brazen crater]

agreed, I was just relaying this over to the docs team. cc @fierce harbor

[fierce harbor]

Thanks, opened an internal issue to add this to docs