How to mount Docker Registry Cert on ARGO Workflows?

[indigo cargo]

I struggled a lot with getting Dagger to publish a container image to Artifactory as we need to provide the Cert of the Docker registry. I was unable to get it to work via the Python SDK. The only thing that worked locally was to mount the cert to the engine container via a manual run (mounting the cert to /usr/local/share/ca-certificates/ca.crt).

Now I tried to do the same in the ARGO Workflow, but I can not get it to work.

[indigo cargo]

I reduced it to

• name: artifactory-docker-cert
  mountPath: /usr/local/share/ca-certificates/ca.crt
  subPath: ca.crt

as that is basically the same I have on my local system. But it's just not working.

I don't know what else to try.

[young glen]

@indigo cargo if you exec into the engine sidecar have you validated that the cert effectively exists at /usr/local/share/ca-certificates/ ?

[indigo cargo]

I did exec into the build container, and it was there. Would assume the same is true for the sidecar.
To clarify, the cert only needs to go into the sidecar container, not in the build container?

[young glen]

yes, correctly

[indigo cargo]

I can try to check the sidecar specifcially (need to see where I can actually find a sidecar container)

[indigo cargo]

I checked, and yes, the cert is available in the sidecar:

kubectl -n argo-workflows --as=cluster-admin exec -it build-monorepo-js5cs-dagger-publish-inner-1294142309 -c dagger-engine -- /bin/sh
/ # ls /usr/local/share/ca-certificates/
ca.crt

Any idea is appreciated.

[young glen]

1 sec

[indigo cargo]

If the password is a "Secret" in the python SDK, i.e. the parameter is a env var key, is the env var going to be read from the sidecar or the build container?

Have to admit that I do not have a good understanding of what the sidecar is doing.

[young glen]

the sidecar is te Dagger engine

Dagger has a simiar architecture to Docker

it's basically a client/server arch

the sidecar is the Dagger server which effectively runs your pipeline

the client is the dagger CLI which communicates to the server and send all the necessary context (files, secrets, etc) so the server can do its job

@indigo cargo re: the certificate issue. Is there any chance you can run ls -la /etc/ssl/certsin the engine sidecar and check if your certificate is there? If your cert is called ca.crt in /usr/local/share/ca-certificates , you shoul see ca-cert-ca.pem in /etc/ssl/certs

[indigo cargo]

Yes, it's in there.

I think I have a first lead.

The logs are print something like this.

DBG executing query query="query ....

this does print the password into the log

but the password it prints is actually not the password

At least it's a base64 encoded something

The password is mounted in via secret as env var:

env:
- name: ARTIFACTORY_PASSWORD
valueFrom:
secretKeyRef:
name: artifactory-credentials
key: password

This is the same way we are doing it for a while in other pipelines.

[young glen]

ok the cert should be working then

[indigo cargo]

But would be strange if it displays

tls: failed to verify certificate: x509: certificate signed by unknown authority

When actually the password is wrong.

When I do

echo "$ARTIFACTORY_PASSWORD"

prior to the dagger call, this prints the password correctly.

The dagger logs print

password: secret(uri: "env://ARTIFACTORY_PASSWORD"): Secret!

But

DBG executing query query="query
contains a base64 encoded value for the password.

[young glen]

are you sure the IT team put the right certificate there for whatever proxy is needed?

the cert is being installed and present in the sidecar engine

[indigo cargo]

The same cert is working with Kaniko

Let me compare it against the cert I have locally

[young glen]

that's expected You're probably seeing dagger's internal representation of the secret given that Dagger masks that so it doesn't leak in te logs

if you're seeing the verify certificate errors, then it's definitely related to certs somehow

we just confirmed the cert is effectively being mounted and installed in the sidecar. At this stage everything points at not being the correct cert for whatever proxy server it's being used

[indigo cargo]

I compared the cert content of what is mounted in the sidecar and it is the same cert I used locally, which worked with dagger engine connecting to our docker registry.

Only difference is that the cert stored in the container has no line breaks.

I copied the ca.crt out of the sidecar container and will use that to test the local execution

[young glen]

that makes sense

I assume there's a global proxy enforcement and both your local machine as well as the kube server are using the same proxy server to reach to the internet?

[indigo cargo]

btw. the local execution I'm doing via:

docker run -d
--name "$CONTAINER_NAME"
--privileged
-v /var/lib/dagger
-v "$CA_CERT_DIR:/usr/local/share/ca-certificates/ca.crt"
$DOCKER_IMAGE

[indigo cargo]

The docker registry is hosted internally

okay, the local execution fails as well:

tls: failed to verify certificate: x509: certificate signed by unknown authority

[young glen]

🎉

[young glen]

🎉

[indigo cargo]

As I said, the only difference is that the cert file does not have line breaks

[young glen]

according to gemini they are required.

[indigo cargo]

So might be that Kaniko had no problem handling the cert file without line breaks and that's why we didn't notice this requirement

[young glen]

interesting.. specially when Kaniko and Dagger are both written in Go should use the same underlying TLS library

🤷‍♂️

infra is esotheric

[indigo cargo]

🥳

I updated the sealed secret with proper line breaks

and the publish went through

@young glen thanks for your help!

[young glen]

Happy to help! I'm around if you have further questions

We're open to feedback if there's anything we could do to make Dagger better