failed to verify certificate: x509: certificate signed by unknown authority

[stuck agate]

Please i am trying to use dagger with certificate from no so popular CA authority. according to this documentation Custom CA Certs we can either build the engine with the certificate baked in, i did that and it worked. the same doc also mention that we can just mount the certificate inside the engine in this folder /etc/ssl/certs/, using this method does not work. please not i tried the .crt and .pem extension

.gitlab-ci.yml file

.docker:
  image: python:3.11-alpine
  services:
    - name: docker:${DOCKER_VERSION}-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_VERIFY: '0'
    DOCKER_TLS_CERTDIR: '/certs'
    DOCKER_CERT_PATH: '/certs/client'
    DOCKER_DRIVER: overlay2
    DOCKER_VERSION: '24.0.5'
    DAGGER_ENGINE_CONTAINER_NAME: 'dagger-engine'
    _EXPERIMENTAL_DAGGER_RUNNER_HOST: 'docker-image://registry.dagger.io/engine:v0.8.4'

.dagger:
  extends: [ .docker ]
  before_script:
    - apk add --no-cache docker-cli
    - echo ${CA_CERTIFICATE} > /ca.crt
    - if [[ $(docker ps -a --filter="name=$DAGGER_ENGINE_CONTAINER_NAME" --filter "status=running" | grep -w "$DAGGER_ENGINE_CONTAINER_NAME") ]]; then echo "Dagger engine already running, not starting ..."; else echo "No dagger engine container running, starting ..."; docker run --rm --privileged --name $DAGGER_ENGINE_CONTAINER_NAME -d -v dagger-engine:/var/lib/dagger -v /ca.crt:/etc/ssl/certs/ca.pem  registry.dagger.io/engine:v0.8.4; fi
    - cp ${CONFIGURATION_FILE} conf/clm.toml
    - cp ${DEVELOPMENT_ENV_FILE} .env
build:
  extends: [ .dagger ]
  script:
    - pip install -r requirements.txt
    - python pipeline.py test

after running my dagger python code

24: > in from registry.****************.net/devops/rust-dev-env
24: resolve image config for registry.**********************.net/devops/rust-dev-env:latest ERROR: failed to do request: Head "https://registry.************.net/v2/devops/rust-dev-env/manifests/latest": tls: failed to verify certificate: x509: certificate signed by unknown authority

here is my python code

[vapid wagon]

@stuck agate 👋. If you're starting the engine yourself, you need to change the _EXPERIMENTAL_DAGGER_RUNNER_HOST variable to use docker-container:// instead of docker-image:// so it uses the container you started before kicking the pipeline

[stuck agate]

thanks, i corrected the yaml file, but still failing with x509 error
here is the updated yaml

.docker:
  image: python:3.11-alpine
  services:
    - name: docker:${DOCKER_VERSION}-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_VERIFY: '0'
    DOCKER_TLS_CERTDIR: '/certs'
    DOCKER_CERT_PATH: '/certs/client'
    DOCKER_DRIVER: overlay2
    DOCKER_VERSION: '24.0.5'
    DAGGER_ENGINE_CONTAINER_NAME: 'dagger-engine'
    _EXPERIMENTAL_DAGGER_RUNNER_HOST: docker-container://${DAGGER_ENGINE_CONTAINER_NAME}

.dagger:
  extends: [ .docker ]
  before_script:
    - apk add --no-cache docker-cli
    - cat ${CA_CERTIFICATE} > /usr/local/share/ca-certificates/ca.crt && update-ca-certificates
    - if [[ $(docker ps -a --filter="name=$DAGGER_ENGINE_CONTAINER_NAME" --filter "status=running" | grep -w "$DAGGER_ENGINE_CONTAINER_NAME") ]]; then echo "Dagger engine already running, not starting ..."; else echo "No dagger engine container running, starting ..."; docker run --rm --privileged --name $DAGGER_ENGINE_CONTAINER_NAME -d -v dagger-engine:/var/lib/dagger -v /usr/local/share/ca-certificates/ca.crt:/etc/ssl/certs/ca.crt  registry.dagger.io/engine:v0.8.4; fi
    - cp ${CONFIGURATION_FILE} conf/clm.toml
    - cp ${DEVELOPMENT_ENV_FILE} .env
build:
  extends: [ .dagger ]
  script:
    - pip install -r requirements.txt
    - python pipeline.py test

[vapid wagon]

nvm.. just saw it is

@stuck agate can you run a docker ps after the python pipeline.py test to validate if only one dagger engine is effetively running?

[stuck agate]

running it now, waiting for the result

[vapid wagon]

not sure if circle's script continues on errors or not since your test.py script might be short-circuiting the build

[stuck agate]

no it errored out on python pipeline test, but i can comment it out and just run the docker ps. Alos i was thinking the below line should have prevented multiple dagger run

    - if [[ $(docker ps -a --filter="name=$DAGGER_ENGINE_CONTAINER_NAME" --filter "status=running" | grep -w "$DAGGER_ENGINE_CONTAINER_NAME") ]]; then echo "Dagger engine already running, not starting ..."; else echo "No dagger engine container running, starting ..."; docker run --rm --privileged --name $DAGGER_ENGINE_CONTAINER_NAME -d -v dagger-engine:/var/lib/dagger -v /usr/local/share/ca-certificates/ca.crt:/etc/ssl/certs/ca.crt  registry.dagger.io/engine:v0.8.4; fi

[vapid wagon]

don't comment it because that's what want to see

[stuck agate]

okay

[vapid wagon]

can you modify it to python pipline.py test | true

[vapid wagon]

so, the way it works is that if _EXPERIMENTAL_DAGGER_RUNNER_HOST is present (which seems it is), the dagger SDK should use that engine instead of starting a new one. I'm just trying to verify if this is the case or if effectively there's something wrong with the engine configuration

[stuck agate]

@vapid wagon , I've finally got it sorted out! It was a bit tricky.

Since the Docker daemon is running as a remote server, using -v /usr/local/share/ca-certificates/ca.crt:/etc/ssl/certs/ca.crt didn't work as expected. Upon investigation, I discovered that ca.crt is actually a directory inside the Dagger engine. I came across this Stack Overflow post single-file-volume-mounted-as-directory-in-docker that explains why a single file can be mounted as a directory in Docker.

To understand why this issue was occurring in the CI environment and not on my local host, I found another Stack Overflow post A case where Docker might not find the file that clarified that when mounting in Docker, it uses the filesystem where the daemon is running, not where the docker-cli is running. and my DOCKER_HOST is tcp://docker:2376

Here's how I worked around it:

I started the engine without binding any volume.
Then, I used docker cp to copy the certificate file.
Finally, I ran update-ca-certificates.
Everything's working smoothly now!**

here is the update gitlab-ci.yml

.docker:
  image: python:3.11-alpine
  services:
    - name: docker:${DOCKER_VERSION}-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_VERIFY: '0'
    DOCKER_TLS_CERTDIR: '/certs'
    DOCKER_CERT_PATH: '/certs/client'
    DOCKER_DRIVER: overlay2
    DOCKER_VERSION: '24.0.5'
    DAGGER_ENGINE_CONTAINER_NAME: 'dagger-engine'
    _EXPERIMENTAL_DAGGER_RUNNER_HOST: docker-container://${DAGGER_ENGINE_CONTAINER_NAME}

.dagger:
  extends: [ .docker ]
  before_script:
    - apk add --no-cache docker-cli
    - cat ${CA_CERTIFICATE} > /ca.crt
    - if [[ $(docker ps -a --filter="name=$DAGGER_ENGINE_CONTAINER_NAME" --filter "status=running" | grep -w "$DAGGER_ENGINE_CONTAINER_NAME") ]]; then echo "Dagger engine already running, not starting ..."; else echo "No dagger engine container running, starting ..."; docker run --rm --privileged --name $DAGGER_ENGINE_CONTAINER_NAME -d -v dagger-engine:/var/lib/dagger registry.dagger.io/engine:v0.8.4; fi
    - docker cp /ca.crt ${DAGGER_ENGINE_CONTAINER_NAME}:/usr/local/share/ca-certificates/ca.crt
    - docker exec ${DAGGER_ENGINE_CONTAINER_NAME} /bin/sh -c "update-ca-certificates"
    - cp ${CONFIGURATION_FILE} conf/clm.toml
    - cp ${DEVELOPMENT_ENV_FILE} .env
build:
  extends: [ .dagger ]
  script:
    - pip install -r requirements.txt
    - python pipeline.py test

[vapid wagon]

That makes total sense

Glad you managed to figure it out