Docker registry mirror using dagger engine in k8s | Dagger | Page 1

thin tree Oct 21, 2024, 10:03 AM

#

IHello!

I am trying out dagger in k8s and have installed from the helm chart and included config to use docker mirror. Our mirror is plain http and insecure. I am using this config:

data:
  engine.toml: |
    debug = true
    insecure-entitlements = ["security.insecure"]
    [registry."docker.io"]
      mirrors = ["registry-mirror.mycompany.local"]
      http = true
      insecure = true

I then try to confirm that the mirror is being used by this command:

dagger query --progress=plain <<< '{ container { from(address:"hello-world") {  } } }' -vv

In the output I get this:

...

27  :     HTTP HEAD
27  :     HTTP HEAD ERROR [0.1s]
27  :     ! tls: failed to verify certificate: x509: certificate is valid for xxxx, not registry-mirror.mycompany.local
26  :     remotes.docker.resolver.HTTPRequest ERROR [0.1s]
26  :     ! tls: failed to verify certificate: x509: certificate is valid for xxx, not registry-mirror.mycompany.local
28  :     remotes.docker.resolver.HTTPRequest
29  :     HTTP HEAD
29  :     HTTP HEAD ERROR [0.5s]
28  :     remotes.docker.resolver.HTTPRequest ERROR [0.5s]
30  :     HTTP GET
30  :     HTTP GET DONE [0.4s]
31  :     remotes.docker.resolver.HTTPRequest
32  :     HTTP HEAD
32  :     HTTP HEAD DONE [0.1s]
31  :     remotes.docker.resolver.HTTPRequest DONE [0.1s]
25  :     resolving docker.io/library/node:20-alpine@sha256:df01469346db2bf1cfc1f7261aeab86b2960efa840fe2bd46d83ff339f463665 DONE [1.1s]
24  :     Container.from DONE [1.1s]
23  :     Directory.withDirectory DONE [1.1s]
22  :     generatedCode DONE [1.1s]
...

It seems to me it is falling back to using docker hub instead of the mirror because it tries with HTTPS/TLS even though the config says plain HTTP.
If I put an invalid mirror address then I get an error that it does not exist, so the mirror config is obviously being consumed.
Is there some more config I need in order to use a plain HTTP registry mirrror?

#

Btw, I use the env variable to use the k8s engine: export _EXPERIMENTAL_DAGGER_RUNNER_HOST="kube-pod://dagger-dagger-helm-engine-xxx?namespace=xxxx"

#

And I am using the instruction from here for the config: https://docs.dagger.io/configuration/custom-registry/

thin tree Oct 21, 2024, 10:26 AM

#

Read some more on buildkit and seems insecure = true is for accepting self-signed certs, so removed that and tried with only http=true but still same issue.

hasty mulch Oct 21, 2024, 1:17 PM

#

thin tree Read some more on buildkit and seems insecure = true is for accepting self-signe...

hey Jonas! strange, http=true should to the trick. Let me try a quick repro

hasty mulch Oct 21, 2024, 3:22 PM

#

hey Jonas! So, you need to add an extra config to your engine.toml that defines only the http=true config for your registry

#

so something like this:

data:
  engine.toml: |
    debug = true
    insecure-entitlements = ["security.insecure"]
    [registry."docker.io"]
      mirrors = ["registry-mirror.mycompany.local"]
    ["registry-mirror.mycompany.local"]
      http = true

#

the config that gets set for the registry is the one that's set at the registty level

#

just tested that locally and it works as expected

thin tree Oct 22, 2024, 6:17 AM

#

Hey, thanks for the quick support, that makes a lot of sense and seems to work!

thin tree Oct 22, 2024, 6:42 AM

#

Spoke a bit too soon, I was actually running without _EXPERIMENTAL_DAGGER_RUNNER_HOST since I started a new shell since yesterday 🙂

#

Got it working in the end and your config was very helpful. The final piece was that

    ["registry-mirror.mycompany.local"]
      http = true

Should be

    [registry."registry-mirror.mycompany.local"]
      http = true

#

I was able to verify in the registry mirror log that it was hit so fully working now!
Thanks again for the awsome support! 😄

#Docker registry mirror using dagger engine in k8s