🐛 PSA: Dagger + testcontainers failing on GitHub Actions after Nov 12 containerd v2 upgrade

[serene ember]

Hey team! 👋

Wanted to share an issue affecting Dagger pipelines with testcontainers on GitHub Actions ubuntu-22.04:

Problem:

• GitHub updated ubuntu-22.04 runners on Nov 12 with containerd v2.1.5 (was v1.7.28)
• Dagger pipelines using testcontainers now fail with: Docker API responded with status code=InternalServerError, failed to mount /tmp/containerd-mount: err: invalid argument

Environment:

• Working: ubuntu22/20250929.88 (containerd v1.7.28) ✅
• Failing: ubuntu22/20251112.150 (containerd v2.1.5) ❌
• Tested: Dagger v0.19.2 and v0.19.6 (both fail)

Issue opened: https://github.com/dagger/dagger/issues/11427

Has anyone else hit this or found a workaround? The issue seems specific to Docker-in-Docker scenarios (testcontainers inside Dagger containers) with containerd v2's overlay filesystem changes.

Implementation details:
We're using the Daggerverse testcontainers module (github.com/vito/daggerverse/testcontainers@1464b057adc8229e79fcf9ab4b96670517e70a8e) which spawns a docker-in-dagger service via dag.Docker().Daemon().Service(). The Docker daemon service is where the containerd v2 overlay issue occurs when testcontainers tries to create Redis containers for tests.

Code: https://github.com/Wakam-Insurance/PricingRenewalSchedulerForGithubExploration/blob/main/Infra/GraphSessionRenewalScheduler/pipeline/src/index.ts#L105

Has anyone else hit this or found a workaround?

Thanks! 🙏

[clear thunder]

@serene ember how are you running testcontainers within Dagger? Are you passing the docker socket to your module? Or are you spawning a docker-in dagger service in your pipeline?

[serene ember]

testcontainers withing dagger

[clear thunder]

Oh, you're using the testcontainers module here?

https://daggerverse.dev/mod/github.com/vito/daggerverse/testcontainers@776610fc77055f983a3dca99f5f55946616c624e?

[serene ember]

yes this one : github.com/vito/daggerverse/testcontainers

@clear thunder ?

[clear thunder]

by any chance do you know what docker version is installed in the ubuntu runner?

ok, probably 29?

I'm installing docker 29.0.0 in my system and checking now

I'd assume it's related to that since docker 29 has that cgroupv1 deprecation

[serene ember]

GitHub Actions ubuntu-22.04:
• Working (20250929.88): Docker 28.0.4 + containerd v1.7.28 ✅
• Failing (20251112.150): Docker 28.0.4 + containerd v2.1.5 ❌

[clear thunder]

ok.. interesting. Usually the version of containerd is bundled with docker

in my machine, Docker 28.5.1 should be using containerd v1.7.28

have you checked if there's an issue in the GH repo?

since docker 28.x.x shouldn't be using containerd v2

seems more like a docker issue to me

[serene ember]

Just verified - you're absolutely right! Docker 28.x should be using containerd v1.7.28, not v2.

But here's the key: The GitHub Actions runner itself has Docker 28.0.4, but Dagger's engine container is using Docker 29.0.1 internally:

[clear thunder]

yep, makes sense That's because the testcontainer module is using the latest docker:dind image here: https://github.com/vito/daggerverse/blob/776610fc77055f983a3dca99f5f55946616c624e/docker/daemon.go#L28

let me check very quickly if that can be overriden

[clear thunder]

yep. You'll need to add a WithDocker to your testcontainers call.

so testcontainers.WithDocker(dockerService).Setup

[serene ember]

work great locally i try on github right now

[clear thunder]

and you'll have to create a dockerservice from here: https://daggerverse.dev/mod/github.com/vito/daggerverse/docker@73fe87189cc73d472393dcab1b84f586dc3b2c9d

and you need to pass the 28 version

[serene ember]

test in progress thx

[clear thunder]

👍

@serene ember even though the GH runner is using Docker 28, have you verified that it's effectively using containerd v1.7.28? Mostly asking given that the issue could be that GH installed the wrong containerd version in their runners

[serene ember]

Work well 🙂 on Github 🙂

Hum ok let's try to find out which version of contianerd

From the runner:

• Docker version: 28.0.4
• containerd version: v2.1.5 (commit: fcd43222d6b07379a4be9786bda52438f0dd16a1)

[clear thunder]

ok I see.. that also works

ok.. seems like the issue was the inner docker version being 29.0.1 and your testcontainers library not being supported

https://github.com/dagger/dagger/issues/11427#issuecomment-3539705896

^ feel free to close if that sounds good to you

[serene ember]

https://github.com/dagger/dagger/issues/11427#issuecomment-3539708216

[clear thunder]

@serene ember I don't think it's a dagger issue since dagger is not doing anything fancy here

I'd say it's the testcontainers .NET SDK you have pinned with docker 29

[serene ember]

right more testcontainers-dotnet repo

[clear thunder]

👍

[serene ember]

thx you for the troubleshooting !