docker-bake with dagger in Gitlab CI

[royal fiber]

Looking into how to do a docker-bake build in dagger with Gitlab CI, or locally.
I was able to get this working by having the Gitlab runner mount /var/run/docker.sock and passing the socket into a dagger module build function. Unfortunately using that socket is not very scalable.
I'd prefer to get this working with a docker-in-docker service and using tcp://docker:2376 for the connection but when running the build it can't connect to that address due to 'no such host'

I'm wondering if there is a way to get this to work with a docker-in-docker service?

If not is using something like BuildKit the recommended approach instead or is there a better way to be doing this? I'd like the Gitlab CI to be able to run docker builds from bake files in a scalable way. I don't currently have a kubernetes cluster set up, our runners are just docker.

Suggestions appreciated!

docker-bake with dagger in Gitlab CI

[vale geyser]

Hey there! running docker-in-docker is totally possible. Here's a Go example on how you can achieve that

func (m *Dagger) Build(source *dagger.Directory, tag string) (string, error) {
    container := dag.Container().
        From("docker").
        WithDirectory(".", source).
        WithEnvVariable("DOCKER_HOST", "tcp://docker:2375").
        WithServiceBinding("docker", dag.Docker().Engine()).
        WithExec([]string{"docker", "build", "-t", tag, "."})

    return container.Stdout(context.Background())
}

The example above shows a basic docker build example by leveraging the docker module here https://daggerverse.dev/mod/github.com/shykes/daggerverse/docker@c9a80c9eac0675a53a7e052da3594207f4235988

[royal fiber]

Thanks! That's very similar to what I'm doing but I then get
ERROR: error during connect: Head "https://docker:2376/_ping": dial tcp: lookup docker on <redacted>: no such host

Note that I am using port 2376 and have also tried with 2375 with the same results. But I do have TLS certs working all the way down to inside the container in the Build function, I don't think thats the issue.

Am I missing something about how to get the "docker" host to resolve inside the container?

Oh I didn't see this WithServiceBinding("docker", dag.Docker().Engine()). Maybe that's what I'm missing

dag.Docker is undefined when using WithServiceBinding("docker", dag.Docker().Engine()).
Note that I'm using the import of dagger/myModule/internal/dagger if that matters

  ctx context.Context,
  bakeFile string,
  targets[] string,
  source *dagger.Directory,
  certs *dagger.Directory, // For TLS
) (string, error) {
  if len(targets) == 0 {
    targets = []string{"default"}
  }
  ctr := dag.Container().
    From("docker:24.0-cli").
    WithMountedDirectory("/certs", certs). // Can I make this a default instead of passing it in?
    WithServiceBinding("docker", dag.Docker().Engine()).
    WithEnvVariable("DOCKER_HOST", "tcp:/docker:2376").
    WithEnvVariable("DOCKER_TLS_VERIFY", "1").
    WithEnvVariable("DOCKER_CERT_PATH", "/certs/client").
    WithMountedDirectory("/src", source).
    WithUser("root").
    WithWorkdir("/src")

    cmd := append([]string{"docker", "buildx", "bake", "-f", bakefile}, targets...)
    out, err := ctr.WithExec(cmd).Stdout(ctx)
    if err != nil {
      return "", err
    }
    return out, nil
}```

And that's what gives the dag.Docker undefined (type *dagger.Client has no field or method Docker

[vale geyser]

sorry I was AFK. I'm at the computer now

let me check really quick

@royal fiber the example I've shared above works for me installing this module dagger install github.com/shykes/daggerverse/docker@v0.4.3

[vale geyser]

this example won't work as defined here becaue the dag.Docker().Engine() module that you're using, doesn't the engine in TLS mode (https://github.com/shykes/daggerverse/blob/c9a80c9eac0675a53a7e052da3594207f4235988/docker/main.go#L57). You'd have to find a module that either support that, or create your own functions / module which starts the engine in TLS mode 🙏

[royal fiber]

I was able to get this working by passing in the a service *dagger.Service to the function and using WithServiceBinding("docker", service).

So now the docker bake command is executing, yay!

I'm running into a new issue and I can open a new help question for that but I'll put it here for starters.

The intent of this function is to provide a reusable function/module for building images based on a docker-bake file. I'm trying to replace our existing Gitlab CI pipeline jobs that do this with a dagger based pipeline.

The problem I'm now running into is that certain environment that are used in Gitlab CI are not available when the container build happens within dagger. And passing them each into the function is not practical because 1. We won't ever know in advance the variables a particular project might use in its Dockerfile for building and 2. passing each variable individually is cumbersome.

What's the right approach for this situation?

[vale geyser]

@royal fiber passing env variables is tricky because we need to find a way to avoid accidentally leaking variables into modules which could potentially exfiliate data. More info about this here:
https://github.com/dagger/dagger/issues/6112