Dagger docker compose

[azure nymph]

I had great luck with @vocal geode's docker module's compose function: https://github.com/vito/daggerverse/tree/main/docker

I had to fork it to support services.*.env_file for my use-case (PR: https://github.com/vito/daggerverse/pull/10), but if that's not a feature that you need, then something like this should give you a service that you can bind to a container via withServiceBinding

dagger -m github.com/vito/daggerverse/docker@v0.1.1 -c 'compose $(git https://github.com/pelias/docker | head | tree | directory projects/texas) | all'

Dagger docker compose

[maiden shard]

My dream would be for Dagger to expose an official Docker Engine API facade

it wouldn't support 100% of the Docker engine API, but it could support enough for the major clients to work. Specifically:

1. docker-compose
2. testcontainers
3. dockerslim
4. CNCF buildpack
5. ?

[near mountain]

devconatiners for sure

[azure nymph]

@maiden shard I've been looking for a way to get involved with Dagger, and would enjoy implementing #1. I think vito's implementation is pretty close to ideal, but I have a few ideas for improvements.

[maiden shard]

To be clear, with a Docker API facade, we should get 1-5 all at once, or pretty close

This is a different approach than targeting docker-compose compat directly (which is also useful)

[tough spire]

I think my challenge was a conflagration of things:

1. running docker compose (which required docker in docker)
2. starting a service with docker compose
3. the other containers started by docker compose needed to mount a file created earlier in the build (which meant funneling that file to the docker engine host)
4. exporting data from the service started by docker compose (which meant mounting a persistent cache on the service)

It may be that these things individually are all simple, but it's a recurring phenomenon for me in software, especially when working with a new system, where I can get all the individual "hello worlds" to work, but then when I try to tie them together it's chaos.

...add to that, that half the time the problem was that one of my golang fields was mis-capitalized or something. 🤦

[maiden shard]

yeah that's a lot of unknowns all at once... Sorry that it was such a grind. But you made it to the other side 🙂

[azure nymph]

like the Dagger engine implements the same interface as the Docker daemon? e.g. I could DOCKER_HOST=tcp://some-dagger-engine docker pull busybox through Dagger? Or just a dag.Docker() builtin module?

[maiden shard]

It doesn't... But it could 🙂 that's the feature I'd like to implement

(I prototyped it a couple years ago. should be much easier now, the engine came a long way)

[azure nymph]

If you did that, how would you distribute the engine? Currently, the CLI defaults to relying on something that already implements the Docker API (er, I guess the CLI, which just so happens to usually mean the API as well), so the engine implementing that same API is a bit redundant, right?

It's cool regardless of redundancy, just curious what your drive is 🙂

[maiden shard]

The Dagger engine has zero runtime dependency on Docker.

What you're talking about is a small convenience in the dagger CLI, which supports different methods for provisioning the engine - one of those methods being docker.

But at the moment, if you write a Dagger module that needs access to a Docker API endpoint, there's no portable way to do that. Each end user must provide an external Docker engine, which is a major external dependency.

Even if you do have a Docker engine available, it will be a black box: you won't get deep traces, you won't get caching, when we introduce auto scale-out of the engine, it won't be available; unified secrets management; etc

[azure nymph]

I see, so the Dagger Engine implementing the Docker API is meant to benefit processes running inside the engine, not outside. That was my misunderstanding

[maiden shard]

yes exactly. But you can also expose it to the outside. Just less useful I think (since as you said, I might as well just install docker)

[azure nymph]

Is this prototype still around anywhere?

[maiden shard]

Oh man, I'd have to track it down

HA! Found it. https://github.com/shykes/docker2dagger

3 years old

built on top of a "virtual docker" scaffolding by @open condor

[open condor]

i think it's private

[azure nymph]

yeah 404 for me

[open condor]

@maiden shard ^

[maiden shard]

@azure nymph now public 🙂

[azure nymph]

@maiden shard I see that your prototype is just using in-memory maps to remember images/containers, and I see that dagger.io/dagger.Client still doesn't really have any list query capabilities. Does the that kind of functionality exist in the engine and it's just not exposed in the client?

If not, does Dagger have existing "internal" storage that would be a better option than in-memory maps, or would this kind of storage be a new concept?

[maiden shard]

You can use a cache volume for persistence. Or, you can write files, export images or connect to services in the client's context. That will be as persistent as your client context. That could range from very persistent (if client context is your host machine), to ephemeral (if your client context is a dagger function)

[azure nymph]

Copy that.

Different topic: is it possible to get ahold of stdin/stdout/stderr streams to a *dagger.io/dagger.Container? I assume so since *dagger.io/dagger.Container.Terminal() exists, but I was hoping that you could point me in the right direction for that.

[maiden shard]

you can get the buffered stdout / stderr (or combined) after execution is complete, but not the streams

Terminal connects the streams directly to the client's, but the dagger API doesn't have a type for the streams themselves

[azure nymph]

I took a peak at dagger and saw that it uses github.com/dagger/dagger/engine/client.Client which does seem to expose raw streams. Would that be useable from inside of Dagger? i.e. if I have something call github.com/dagger/dagger/engine/client.Connect() inside of Dagger, will it connect to the Engine that its running inside of?

If not, I see that dagger.io/dagger.ContainerWithExecOpts has RedirectStd__ for each of the streams. Would it be possible to mount some files into the container from the host, redirect the streams to those files, and use the files as the streams?

If not, then I guess the only path forward would be to just not implement any of the Docker API's streaming endpoints

[maiden shard]

There are several options:

1. Like you saw, the client library lets you connect stdin/ stdout / stderr. This is a global redirect for the whole session though: you'll get streams for all containers executed in the session ('Container.withExec, Container.asService, Container.Terminal`), plus output stream for dagger itself. You could open a new client session for each docker run.

2. You can redirect stdout/stderr to a file then dump their contents. But no streaming (a long command will show nothing until it's over).

3. You could use cache volumes to redirect stdout/stderr to named pipe or unix socket and access the same cache volumes to stream.

4. You could wrap a dagger client and intercept the otel data, then print the otel logs. Dagger converts all events to standard otel, and honors standard variables for otel collectors.

5. You could add support for streams in Dagger API, then do everything cleanly 🙂