How to mount Dagger-generated files in Docker-in-Docker containers? | Dagger | Page 1

reef fog Aug 12, 2025, 7:59 PM

#

I want to use Docker-in-Docker to run an existing docker-compose file, while mounting files that were generated within the Dagger pipeline. This would allow me to reuse existing Docker Compose workflows without having to rewrite them entirely in Dagger.

What I've Tried

I have four test methods that demonstrate my progression. Each can be invoked from the command line as follows:

dagger call dind-example - Basic Docker-in-Docker (fails)
dagger call dind-example-with-docker-socket --docker /var/run/docker.sock - Works with host socket
dagger call dind-example-with-file-mount --docker /var/run/docker.sock - Fails to mount generated file
First create a test file: echo "hello from host" > /tmp/test-host-file.txt, then dagger call dind-example-with-host-file-mount --docker /var/run/docker.sock --host-file-path /tmp/test-host-file.txt - Successfully mounts host file

#

// Basic Docker-in-Docker - this fails (See "Actual Output" below)
func (h *Headway) DindExample(ctx context.Context) string {
    out, err := dag.Container().
        From("docker:dind").
        WithExec([]string{"docker", "run", "debian:bookworm-slim", "ls", "/"}).
        Stdout(ctx)
    if err != nil {
        panic(fmt.Errorf("failed to run dind example: %w", err))    
    }
    return out
}   

// Using host Docker socket - this works for basic commands
func (h *Headway) DindExampleWithDockerSocket(ctx context.Context, docker *dagger.Socket) string {
    out, err := dag.Container().
        From("docker:dind").
        WithUnixSocket("/var/run/docker.sock", docker).
        WithExec([]string{"docker", "run", "debian:bookworm-slim", "ls", "/"}).
        Stdout(ctx)
    if err != nil {
        panic(fmt.Errorf("failed to run dind example: %w", err))    
    }
    return out
}   

// What I want to achieve - mount a Dagger-generated file in DinD container
func (h *Headway) DindExampleWithFileMount(ctx context.Context, docker *dagger.Socket) string {
    out, err := dag.Container().
        From("docker:dind").
        WithUnixSocket("/var/run/docker.sock", docker).
        WithFile("/data/hello.txt", h.GenerateFile()).
        WithExec([]string{"docker", "run", "-v", "/data/hello.txt:/data/hello.txt", "debian:bookworm-slim", "cat", "/data/hello.txt"}).
        Stdout(ctx)
    if err != nil {
        panic(fmt.Errorf("failed to run dind example: %w", err))    
    }
    return out
}

#

func (h *Headway) DindExampleWithHostFileMount(ctx context.Context, docker *dagger.Socket, hostFilePath string) string {
    out, err := dag.Container().
        From("docker:dind").
        WithUnixSocket("/var/run/docker.sock", docker).
        WithExec([]string{"docker", "run", "-v", hostFilePath + ":/data/hello.txt", "debian:bookworm-slim", "cat", "/data/hello.txt"}).
        Stdout(ctx)
    if err != nil {
        panic(fmt.Errorf("failed to run host file mount example: %w", err))    
    }
    return out
}

func (h *Headway) GenerateFile() *dagger.File {
    return dag.Directory().WithNewFile("hello.txt", "hello world").File("hello.txt")
}

#

The Problem

The third approach fails because when using the host's Docker socket, bind mount paths are resolved relative to the host filesystem, not the Dagger container's filesystem. The file I created with WithFile() exists in the Dagger container but not on the host.

#

Actual Output

Test 1 (dagger call dind-example) fails:

docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Test 2 (dagger call dind-example-with-docker-socket --docker /var/run/docker.sock) works correctly:

bin
boot
dev
etc
home
lib
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var

Test 3 (dagger call dind-example-with-file-mount --docker /var/run/docker.sock) fails with:

cat: /data/hello.txt: Is a directory

Test 4 (dagger call dind-example-with-host-file-mount --docker /var/run/docker.sock --host-file-path /tmp/test-host-file.txt) works correctly:

hello from host

This confirms our hypothesis: when using the host's Docker socket, bind mounts must reference files that exist on the host filesystem, not within the Dagger container's filesystem. Test 3 fails because /data/hello.txt doesn't exist on the host (only in the Dagger container), while Test 4 succeeds because /tmp/test-host-file.txt exists on the host.

#

Questions

Is there a recommended pattern for using existing docker-compose files within Dagger pipelines?
How can I make files generated in a Dagger pipeline available for bind mounting in Docker-in-Docker scenarios?

Use Case Context

I'm migrating from Earthly to Dagger and have complex docker-compose files for data import processes (Pelias geocoding). Rather than rewriting all the orchestration logic in Dagger, I'd prefer to reuse the existing compose files while still benefiting from Dagger's pipeline capabilities for generating configuration files and managing dependencies.

sturdy silo Aug 13, 2025, 3:42 PM

#

cc @glossy iron related to our recent discussions.

Hey Michael!

Thanks for the detailed question.

Is there a recommended pattern for using existing docker-compose files within Dagger pipelines?

There is no great solution for this at the moment, the way that docker-compose works is not 100% compatible with Dagger out of the. box.

How can I make files generated in a Dagger pipeline available for bind mounting in Docker-in-Docker scenarios?

I think there is an answer here but it may require you to rethink your approach. Dagger in Dagger or Docker in Dagger works out of the box and allows you to infinitely nest things. The whole point of dagger is treat types like directory, file, and container and first-class objects that you can pass around in your code.

To me this means instead of treating dagger pipeline and docker in docker stuff as two different things, you should consider modeling out the docker-in-docker stuff using dagger as well. If you do, then getting things to work will be as simple as passing an argument into a function.

FWIW since dagger is a container runtime, you technically can call docker-compose up within a dagger container and it should work as expected.

reef fog Aug 13, 2025, 4:46 PM

#

Thanks for the response. I'm having trouble applying it though.

You say docker-in-docker works "out of the box", but I'm struggling to use it as shown in my examples (none of which use docker compose btw).

Probably I'm doing something wrong - I guess my brain is stuck in the old model and I'm having a hard time understanding how to solve my problem in the dagger paradigm.

In particular, how can I make files generated in a Dagger pipeline available for bind mounting in Docker-in-Docker scenarios?

sturdy silo Aug 13, 2025, 5:35 PM

#

reef fog Thanks for the response. I'm having trouble applying it though. You say docker...

No worries, there are some slight differences and it takes some getting used to.

The main idea is that dagger works closer to docker build than it does to docker run.

Dagger is able to run an infinitely nested set of containers.

If you want to do docker in docker you need to pass a service that is running docker first. Here is a code example based on your first try that should work.

This example uses the docker module, you can look at the source code to see exactly what is happening.

func (h *Dinddemo) DindExample(ctx context.Context) string {
    out, err := dag.Docker().
        Cli().
        Container().
        WithExec([]string{"docker", "run", "debian:bookworm-slim", "ls", "/"}).
        Stdout(ctx)
    if err != nil {
        panic(fmt.Errorf("failed to run ls example: %w", err))
    }
    return out
}

https://github.com/shykes/daggerverse/tree/c9a80c9eac0675a53a7e052da3594207f4235988/docker

https://daggerverse.dev/mod/github.com/shykes/daggerverse/docker@c9a80c9eac0675a53a7e052da3594207f4235988

Using that same setup with docker engine running as a service you can call docker-compose too

func (h *Dinddemo) DcExample(ctx context.Context) string {
    out, err := dag.Docker().
        Cli().
        Container().
        WithExec([]string{"docker-compose"}).
        Stdout(ctx)
    if err != nil {
        panic(fmt.Errorf("failed to run ls example: %w", err))
    }
    return out
}

You would use these same ideas and building blocks to pass files into docker compose.

GitHub

daggerverse/docker at c9a80c9eac0675a53a7e052da3594207f4235988 · s...

My personal collection of Dagger modules. Contribute to shykes/daggerverse development by creating an account on GitHub.

docker :: Daggerverse

A Dagger Module for integrating with the Docker Engine

reef fog Aug 13, 2025, 5:39 PM

#

Ah ok, I haven't looked into anything external stuff on daggerverse yet. I'll give docker as a Service a try and see if things fall into place.

sturdy silo Aug 13, 2025, 5:40 PM

#

reef fog Ah ok, I haven't looked into anything external stuff on daggerverse yet. I'll gi...

Its not required to use the external docker service but its the simplest way to get this working. If you want to understand how it works under the hood I'd suggest looking at that source code because you will see all the missing flags and things that you did not have in your example. 🙂

reef fog Aug 13, 2025, 8:55 PM

#

The shykes docker-in-docker stuff is nice!

(As a side note, I'd previously looked for some docker-in-docker stuff on daggerverse, but there were a lot of options: https://daggerverse.dev/search?q=docker Because there was no basis for popularity, I didn't now how to evaluate which one would be a good one to try, and gave up before I saw one written by a name I recognized)

I'm able to start a dind more succinctly with the shykes/docker stuff, and the namespaces persistence is nice, but I am ultimately hitting the same wall:

func (h *Headway) DindExampleWithDockerServiceWithFileMount(ctx context.Context) string {
    out, err := dag.Docker().
        Cli().
        Container().
        WithFile("/data/hello.txt", h.GenerateFile()).
        WithExec([]string{"docker", "run", "-v", "/data/hello.txt:/data/hello.txt", "debian:bookworm-slim", "cat", "/data/hello.txt"}).
        Stdout(ctx)
    if err != nil {
        panic(fmt.Errorf("failed to run dind example: %w", err))    
    }
    return out
}

cat: /data/hello.txt: Is a directory

I have generated a * dagger.File, and I want a docker-in-docker container to read it. I guess maybe I need to get the file into the container running the docker daemon, rather than mounting into the CLI container, but the daemon container is opaque to me (It's given to me AsService), so I can't manipulate it.

I'm not sure if I'm on the right track and close to a solution or just fundamentally doing this wrong. I wonder why earthly made this easier for me - it must have warped my mind. 😵‍💫

docker :: Daggerverse

Search Daggerverse modules.

sturdy silo Aug 14, 2025, 1:23 PM

#

reef fog The shykes docker-in-docker stuff is nice! (As a side note, I'd previously loo...

I think the main difference is the built in default module system makes an extremely secure and reproducible sandbox, but this has tradeoffs.

Let me look into this a bit more and get back to you soon.

#

Also thanks for feedback on Daggerverse cc @latent knot

We’re working on improving this experience so it’s more obvious which modules are ready out of the box

crude tusk Aug 14, 2025, 4:10 PM

#

hey Michael! the reason why this is not working is because when you use -v with a remote engine, the engine has to actually mount the files from the host where the target engine lives, not the client. That's why you see the is a directory error because in the Dagger engine service container, /data/hello.txt doesn't exist.

Fortunantely, you can still make this work by leveraging cache volumes to share whatever data you need from the client and engine contaienrs. Having said that, there's an extra step you need to make for this to work which is copying the data from your dagger.File or dagger.Directry into the cache volume via a userland cp command given that you can't directly add them via the With{File,Directory} commands.

Here's a Go example putting everything together.

func (m *Dind) Test(ctx context.Context) *dagger.Container {
    tmp := dag.CacheVolume("temp")
    ctr := dag.
        Container().
        From("index.docker.io/docker:24.0-dind").
        WithMountedCache("/temp", tmp).
        WithMountedCache("/var/lib/docker", dag.CacheVolume("dind-lib-docker")).
        WithoutEntrypoint().
        WithExposedPort(2375)

    return dag.Container().From("docker:cli").
        WithServiceBinding("docker", ctr.AsService(dagger.ContainerAsServiceOpts{
            Args: []string{
                "dockerd",
                "--host=tcp://0.0.0.0:2375",
                "--host=unix:///var/run/docker.sock",
                "--tls=false",
            },
            InsecureRootCapabilities: true,
        })).
        WithEnvVariable("DOCKER_HOST", "tcp://docker:2375").
        WithMountedCache("/temp", tmp).
        WithNewFile("foo.txt", "Hello").
        WithExec([]string{"cp", "foo.txt", "/temp/foo.txt"}).
        WithExec([]string{"docker", "run", "-v", "/temp/foo.txt:/foo.txt", "alpine", "cat", "/foo.txt"})

}

#

I had to use my custom dockerd service since the one in the daggerverse doesn't currently allow passing extra cache volumes to mount in the engine container

#

cc @reef fog

reef fog Aug 14, 2025, 4:24 PM

#

That's the ticket — thank you @crude tusk!

#How to mount Dagger-generated files in Docker-in-Docker containers?

What I've Tried

The Problem

Actual Output

Questions

Use Case Context