dagger develop not working with python SDK (v0.12.4) and Custom CA | Dagger | Page 1

ruby chasm Aug 6, 2024, 8:10 AM

#

Hello,

I am unable to run dagger develop anymore. uv does not seem to respect the custom ca certificate I put in the dagger engine, nor does it respect our internal registry configuration configured in a custom python base image.

python -m codegen generate -i /schema.json -o /gen.py" did not complete successfully: exit code: 2
Stderr:
error: Failed to prepare distributions
  Caused by: Failed to fetch wheel: graphql-core==3.2.3
  Caused by: Request failed after 3 retries
  Caused by: error sending request for url (https://files.pythonhosted.org/packages/f8/39/e5143e7ec70939d2076c1165ae9d4a3815597019c4d797b7f959cf778600/graphql_core-3.2.3-py3-none-any.whl)       
  Caused by: client error (Connect)
  Caused by: invalid peer certificate: UnknownIssuer

engine.Dockerfile

FROM debian:bookworm-slim as cacert
RUN apt-get update && apt-get install -y curl \
    && curl -sSL https://nexus.company.com/repository/raw/ldnca-rca.crt -o \
       /usr/local/share/ca-certificates/ldnca-rca.crt

FROM registry.dagger.io/engine:v0.12.4
COPY --from=cacert /usr/local/share/ca-certificates/ldnca-rca.crt \
    /usr/local/share/ca-certificates/ldnca-rca.pem
COPY engine.toml /etc/dagger/engine.toml

python.Dockerfile

FROM python:3.11-slim

COPY --from=cacert /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

ENV PIP_INDEX_URL=https://nexus.company.com/repository/pypi-group/simple \
    UV_INDEX_URL=https://nexus.company.com/repository/pypi-group/simple \
    REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt \
    SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

[project]
name = "main"
version = "0.0.0"
dependencies = []

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.dagger]
base-image = "nexus.company.com/docker/dagger-python-base:0.2.2"
use-uv = false

[tool.uv.pip]
index-url = "https://nexus.company.com/repository/pypi-group/simple"

ruby chasm Aug 6, 2024, 8:26 AM

#

I am confused as to why:

the uv download does not go through our internal pypi proxy when setting UV_INDEX_URL in the custom python base image.
Why the CA Certificate is ignored and I get the invalid peer certificate: UnknownIssuer.
uv is still running even though I put use-uv = false in the pyproject.toml.

night wraith Aug 6, 2024, 9:49 AM

#

Hey! Did it work for you in 0.12.3? Can you tell which version broke it for you?

night wraith Aug 6, 2024, 9:49 AM

#

ruby chasm I am confused as to why: 1. the uv download does not go through our internal pyp...

I'm not familiar with the specifics on the custom certificates, maybe @vernal citrus can help but I'll need to investigate.

As for why uv is being used, the setting use-uv = false only applies when installing your module's code dependencies. Uv is still being used regardless on codegen, which is where you're failing apparently. The pipeline responsible for it is in https://github.com/dagger/dagger/blob/a70d6ac50b11faabeb02e8bcc601c2692c6ca540/sdk/python/runtime/main.go#L291-L300. If you're willing to build a dev version of the engine, you could edit that command and add --verbose to uv run so we get more information. I'll do it, but will take me longer.

#

@ruby chasm, can I ask you why you disabled uv? It helps me understand the use cases 🙂

ruby chasm Aug 6, 2024, 12:43 PM

#

Hi @night wraith , thanks for the response. I am happy to use uv I was just trying to see if it would work if I disabled it. Is it possible to configure the UV_INDEX_URL when it is being used for codegen?

#

I will try to have a look at building a dev dagger engine. Thanks for the tip!

night wraith Aug 6, 2024, 2:03 PM

#

ruby chasm Hi <@768585883120173076> , thanks for the response. I am happy to use `uv` I was...

Yeah, but it seems you're already doing it in your base image. The codegen step uses the same base container.

#

You seem to be failing on the certificates. Should you be seeing the package being downloaded from your mirror or does the mirror return to the client a public URL?

I'm asking because uv is trying to download its (only) dependency from pythonhosted.org:

https://files.pythonhosted.org/packages/f8/39/e5143e7ec70939d2076c1165ae9d4a3815597019c4d797b7f959cf778600/graphql_core-3.2.3-py3-none-any.whl

ruby chasm Aug 6, 2024, 2:42 PM

#

Perhaps uv run and uv pip install behave differently when pulling dependencies. Either way, the CA certificate is not being used, otherwise it would be trusted when downloading from files.pythonhosted.org. I will continue playing around, thank you!

ruby chasm Aug 6, 2024, 3:24 PM

#

@night wraith So I got it working... perhaps my config was too complicated. This is my custom base image now:

FROM debian:bookworm-slim as cacert
RUN apt-get update && apt-get install -y curl ca-certificates \
    && curl -sSL https://nexus.company.com/repository/raw/ldnca-rca-2030.crt -o /usr/local/share/ca-certificates/ldnca-rca.crt \
    && update-ca-certificates

FROM python:3.11-slim

COPY --from=cacert /usr/local/share/ca-certificates/ldnca-rca.crt \
    /usr/local/share/ca-certificates/ldnca-rca.pem

ENV PIP_INDEX_URL=https://nexus.company.com/repository/pypi-group/simple \
    UV_INDEX_URL=https://nexus.company.com/repository/pypi-group/simple

#

Instead of copying a full ca bundle over, I just add the one CA cert to the base image, and remove the SSL_CERT_FILE env var

night wraith Aug 6, 2024, 3:34 PM

#

That's great! Is copying the certs to the base image really necessary though? Doesn't the engine do that automatically?

#

As described in https://docs.dagger.io/manuals/administrator/custom-ca#configuration-applied-to-user-containers:

[...] there is best-effort support for automatically installing the Engine's custom CAs in all Containers created by user pipelines (i.e. those created via a withExec API call). [...] Currently, the Engine supports automatically installing custom CAs in images with the following base distributions:

Alpine

Debian-based (e.g. debian and ubuntu)

Redhat-based (e.g. rhel, fedora, centos, etc.)

ruby chasm Aug 7, 2024, 7:04 AM

#

I have tried without copying the cert in the python base image but get the same issue as before:

Stderr:
error: Failed to prepare distributions
  Caused by: Failed to fetch wheel: graphql-core==3.2.3
  Caused by: Request failed after 3 retries
  Caused by: error sending request for url (https://files.pythonhosted.org/packages/f8/39/e5143e7ec70939d2076c1165ae9d4a3815597019c4d797b7f959cf778600/graphql_core-3.2.3-py3-none-any.whl)
  Caused by: client error (Connect)
  Caused by: invalid peer certificate: UnknownIssuer

python.Dockerfile

FROM python:3.11-slim

ENV PIP_INDEX_URL=https://nexus.company.com/repository/pypi-group/simple \
    UV_INDEX_URL=https://nexus.company.com/repository/pypi-group/simple

engine.Dockerfile

FROM debian:bookworm-slim as cacert
RUN apt-get update && apt-get install -y curl \
    && curl -sSL https://nexus.company.com/repository/raw/ldnca-rca.crt -o \
       /usr/local/share/ca-certificates/ldnca-rca.crt

FROM registry.dagger.io/engine:v0.12.4
COPY --from=cacert /usr/local/share/ca-certificates/ldnca-rca.crt \
    /usr/local/share/ca-certificates/ldnca-rca.pem
COPY engine.toml /etc/dagger/engine.toml

So either the certfificate is not actually being propagated to the python base image.. or I am doing something wrong with the paths or filenames of the certs. But according to that linked doc it should be placed under /usr/local/share/ca-certificates/

#

I believe it used to generate a ca-certificates.crt for all certs placed under /usr/local/share/ca-certificates/ then I imagine it gets propagated to the python base image. Then I saw that UV_NATIVE_TLS is enabled. So I think with the above configuration it should work...

night wraith Aug 7, 2024, 9:00 PM

#

@vernal citrus, does anything jump out to you?

vernal citrus Aug 7, 2024, 10:15 PM

#

ruby chasm I have tried without copying the cert in the python base image but get the same ...

One quick thing to try, can you see if it works when you change the engine.Dockerfile to do:

COPY --from=cacert /usr/local/share/ca-certificates/ldnca-rca.crt \
    /usr/local/share/ca-certificates/ldnca-rca.crt

Difference being that the final file retains the .crt extension instead of switching to .pem

I saw that discrepancy and tried using a .pem extension instead of .crt in our integ tests for CA certs and got similar errors with .pem (https://github.com/sipsma/dagger/blob/d9a15262229f1dba0aa53dbd0216908438331995/core/integration/cacert_test.go#L543-L543)

ruby chasm Aug 7, 2024, 10:15 PM

#

Sure thing, will try that tomorrow and let you know!

ruby chasm Aug 10, 2024, 11:52 AM

#

Hey, sorry for the delay.. Good news, using the .crt extension works 🚀

#

I have removed the cert from the python.Dockerfile and only include it in the engine. Codegen is working as well as calling modules! Thanks a lot for the investigation. Would you like me to contribute/open a PR for the docs? Might be worth highlighting the expected file extension here: https://docs.dagger.io/manuals/administrator/custom-ca/#configuration-applied-to-user-containers

Configure the Engine to use Custom Certificate Authorities | Dagger

The Dagger Engine can be configured to use custom certificate authorities (CAs) when communicating with external services like container registries, Git repositories, etc.

#

also simply adding:

[tool.uv.pip]
index-url = "https://nexus.company.com/repository/pypi-group/simple"

to the pyproject.toml of the module enables us to use the dagger base python image and leverage our internal registry (for internal packages and proxying packages) which gives much better performance within our corporate network. (This is more of an FYI if someone faces a similar issue)

mossy onyx Oct 31, 2024, 3:05 PM

#

Hi! I am inside a corporate network and having trouble getting my IDE to understand the import dagger. I am seeing import "dagger" could not be resolved. I am not a python expert but I am trying to create some example modules to share.

dagger develop runs fine. dagger call [function] also runs fine. I am using a custom dagger engine with my CA certs setup. however, I am not really where where my company private PIP registry goes. I tried adding it like

[tool.uv.pip]
index-url = "https://company.artifactory.com/artifactory/api/pypi/pypi-prereleases/simple"

But that didn't make a difference.

night wraith Oct 31, 2024, 4:29 PM

#

This is changing in https://github.com/dagger/dagger/pull/8772, but currently it's still:

[tool.uv]
index-url = "https://company.artifactory.com/artifactory/api/pypi/pypi-prereleases/simple"

mossy onyx Oct 31, 2024, 4:59 PM

#

What if I have to pass in credentials with the URL?

#

My local pip.conf looks like this

[global]
index-url = https://[user]@[pass]@company.registry.com/artifactory/api/pypi/pypi-prereleases/simple

#

I tried switching the URL. Still seeing this. I don't think it's an issue with pulling dependencies because I can dagger call [function] fine.

#

dagger develop is successful too.

#

I tried both in vscode and my neovim (lazyvim) setup.. Same error

night wraith Oct 31, 2024, 5:09 PM

#

Oh, if it's just locally, with your IDE, just follow their docs: https://docs.astral.sh/uv/configuration/indexes/

#

Now the index in [tool.uv] is only understood by Dagger. You need the more recent [[tool.uv.index]] locally.

#

I'm assuming you've run uv run vim or uv sync locally. That's what installs the package in .venv.

mossy onyx Oct 31, 2024, 5:12 PM

#

night wraith I'm assuming you've run `uv run vim` or `uv sync` locally. That's what installs ...

oh! I have not. Is this documented in dagger docs?

night wraith Oct 31, 2024, 5:12 PM

#

Yes, it is

#

https://docs.dagger.io/api/ide-integration

mossy onyx Oct 31, 2024, 5:17 PM

#

Ooh that's pretty clear! I should have searched better. I didn't think to look in IDE setup

mossy onyx Oct 31, 2024, 8:11 PM

#

So I was able to get code completion in vscode by adding extraPaths to the .vscode/settings.json file. I don't see that as a requirement in dagger doc. Am i doing something wrong? Even then, I still couldn't get it to work on my neovim setup.

#dagger develop not working with python SDK (v0.12.4) and Custom CA