Module init errors with python SDK and internal ca certificate

[median oriole]

Using podman, I am running a custom engine version v0.11.8 with our internal ca cert mounted in /usr/local/share/ca-certificates/.

I am able to successfully initialize modules for both Go and TypeScript SDKs but am facing an error when initializing a module with the python SDK with dagger init --sdk=python.

codegen is failing at the installation of uv via pip

Collecting uv==0.2.11 (from -r /reqs.txt (line 1))
Stderr:
...'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c.10006)

🤔 I'm wondering if the internal CA cert is not getting applied to the codegen container as I'm seeing entries in the engine logs like :

level=debug msg="> creating tw29lol6zrzv... [update-ca-certificates]" span=[internal] exec /runtime"
level=debug msg="> container done tw29lol6zrzv... [update-ca-certificates]" error="<nil" span="[internal] exec '/runtime"

Is anyone else running a custom engine with internal CA certs running into module initialization issues with the Python SDK?

[dire forge]

Hi @median oriole I believe this issue is that pip (or uv pip) does not use the system certificate store by default (https://pip.pypa.io/en/stable/topics/https-certificates/#using-a-specific-certificate-store), but the bundled CA certificate store from certifi. In my case, I created a custom python image which sets the CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE environment variables equal to the system trust store /etc/ssl/certs/ca-certificates.crt

FROM debian:bookworm-slim as cacert
COPY /your/ca-cert.crt /usr/local/share/ca-certificates/ca-cert.crt
RUN apt-get update && apt-get install -y ca-certificates \
    && update-ca-certificates

FROM python:3.11-slim
COPY --from=cacert /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ENV CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt \
    REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

docker build . -t registry.company.com/dagger-python-base:0.1.0

Then in the pyproject.toml:

[project]
name = "main"
version = "0.0.0"
dependencies = []

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.dagger]
base-image = "registry.company.com/dagger-python-base:0.1.0"

Finally run: dagger init --name my-cool-module --sdk python --source . within the directory of the pyproject.toml.

Hope this helps!

[astral badge]

@dire forge are you also using the python SDK? I'm not entirely sure if this is the same issue as Jeremy is talking about our python runtime container.

silently pinging @feral swallow

[dire forge]

Hey @astral badge I am using the Python within a corporate environment. We also configure our own internal pypi proxy with Nexus and it works fine with PIP_INDEX_URL and UV_INDEX_URL in the runtime container. But perhaps it is different if collecting uv is happening within the engine container

[astral badge]

Ok. I see. Did you change your dagger.json so it uses your custom python image instead of the default one?

[dire forge]

I wasn’t aware of the dagger.json field, in the docs for python sdk it says to use the pyproject.toml file mentioned above

[astral badge]

Oh right, I missed the base-image field in the pyproject.toml. Yes, this definitely fixes the issue. Sorry for the confusion here

[dire forge]

No worries, went through this already in https://discord.com/channels/707636530424053791/1246470396086321162 , thanks to your help 😅

Might be worth setting these env vars in the engine container to have python leverage the appropriate root certs

[median oriole]

Hi @dire forge Thank you for sharing! I like your multi-stage solution for loading the CA bundle into the custom runtime. I can certainly try initializing with a customized runtime as you have done but was hoping I wouldn't have a need to deviate from the default base image if possible.

With custom CA support outlined in https://github.com/dagger/dagger/issues/6599 ( CA cert/bundle mounted in /usr/local/share/ca-certificates/) and https://github.com/dagger/dagger/pull/7356 I am under the impression the CA certs would be loaded and used in the spawned runtime containers -including any needed during code gen to support module initialization...

I can confirm our custom CA certs are getting loaded into the custom engine but I'm still seeing certificate issues only when initializing a new python module with dagger init --sdk=python. One thing I'm doing with our custom engine that I am unable to do for the runtime container created during init is pass CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE env vars. Your solution definitely enables that through the custom runtime image override.

I've been taking additional looks at https://docs.dagger.io/manuals/developer/module-structure/ more closely and one thing that isn't clear to me is what file(s) might be referenced during module initialization. I didn't expect dagger init would require or reference any host files and instead generate everything it needs for me to start with. @astral badge are the docs missing this info or can you confirm the expected behavior?

[feral swallow]

Hey @median oriole, can you show more of the error? I wonder where it fails. Is it the OS or uv? Does it change if you switch to pip?

[median oriole]

Hi @feral swallow it seems to be failing trying to pip install uv.

This is within an empty directory

[feral swallow]

Ah, right. Current SDK version installs uv using pip, but I'm going to submit an update soon to install uv using the OCI image. uv has an option to use the system's certificates instead of the cargo library it comes with. Not sure if that'll fix it or not, but relying on the OS's certificates seems like the safe bet.

[median oriole]

That sounds promising and worth a try! Until then, it seems the best path forward may be trying the custom runtime as proposed above.

And relying on the OS's certs should then include any custom CA certs loaded is that correct?

Wondering if we may still be dependent on any SSL cert related env vars...

[median oriole]

@dire forge are you using a custom runtime solely for controlling custom CA config or are there additional requirements which need continued use of the custom runtime you're supplying?

[dire forge]

Hey Jeremy, it is purely to tell the pip operations to use the CA cert that dagger loads and injects

[feral swallow]

Can you guys submit an issue? I wasn't aware of this. The Python SDK can do more to ensure the system certificates are used, like setting env vars that are generally needed here.

[median oriole]

Sure thing, I'll get an issue submitted:)

[median oriole]

Here's the issue: https://github.com/dagger/dagger/issues/7774

[feral swallow]

@median oriole, can you try again with v0.12?