#Docker in Docker for Localstack

1 messages · Page 1 of 1 (latest)

gleaming relic
#

I'm migrating my team's CI workflows from Earthly to Dagger, and have been successful up until this point where I'm migrating our integration test workflows. Specifically, with Earthly, our integration tests run alongside Localstack in a dind container:

    FROM earthly/dind:alpine

    COPY tests/integration/aws/docker-compose.yml .

    RUN --secret LOCALSTACK_AUTH_TOKEN=+secrets/LOCALSTACK_AUTH_TOKEN \
        echo "LOCALSTACK_AUTH_TOKEN=$LOCALSTACK_AUTH_TOKEN" > .env
    # Consider adding a setup service to create initial resources
    WITH DOCKER --load=+integration-snapshot-image \
            --compose docker-compose.yml \
            --service localstack
        RUN docker run --network=host lacework/sidekick-integrationtest:snapshot-latest
    END

As documented here, WITH DOCKER ... END starts a Docker daemon to execute the contained commands against it.

Using Docker in Earthly | Earthly
#

So far, here's what I have in Dagger:

          dag.Container().
        From("docker:dind").
        WithExec([]string{"apk", "add", "--no-cache", "docker-compose", "curl", "bash"}).
        WithFile("/docker-compose.yml", dockerCompose).
        WithFile("/snapshot.test", testBinary).
        WithExec([]string{"chmod", "+x", "/snapshot.test"}).
        WithSecretVariable("LOCALSTACK_AUTH_TOKEN", localstackToken).
        // Start docker daemon
        WithExec([]string{
            "sh", "-c", "dockerd --storage-driver=vfs &",
        }, dagger.ContainerWithExecOpts{
            InsecureRootCapabilities: true,
        }).
        // DEBUG: Start interactive shell
        Terminal(dagger.ContainerTerminalOpts{
            InsecureRootCapabilities: true,
        }).
        // Wait for docker daemon to be ready
        WithExec([]string{
            "sh", "-c", "while ! docker info >/dev/null 2>&1; do sleep 1; done",
        }).
        // Create volume directory for localstack
        WithExec([]string{
            "mkdir -p ./volume/localstack",
        }).
        // Start localstack
        WithExec([]string{
            "docker-compose -p test up -d localstack",
        }).
        // Wait for localstack to be ready
        WithExec([]string{
            "sh", "-c", 
            `for i in $(seq 1 60); do
                if curl -s http://localhost:4566/_localstack/health 2>/dev/null | grep -q '"ec2": "available"'; then
                    echo "LocalStack is ready!"
                    break
                fi
                echo "Waiting for LocalStack... ($i/60)"
                sleep 1
            done`,
        })

The issue I'm having is dockerd doesn't seem to be starting properly - when I terminal in immediately after the exec statement running dockerd --storage-driver=vfs &, I run ps and dockerd isn't running. However, I execute the same command in the terminal and dockerd seems to run fine.

#

Here's my question: Am I doing something wrong here, or is what I'm here doing an antipattern in dagger or something?

#

For context on why I'm looking to run localstack in dind: I first tried to simply use Localstack as a dagger service with my host's docker socket mounted to Localstack, but this didn't work because it failed to create emulated EC2 instances due to port conflicts on the host machine. That's when I realized that our earthly implementation necessarily runs everything in a dind container for isolation.

// IntegrationTestSnapshot tests snapshot functionality
func (s *Ci) IntegrationTestSnapshot(
    ctx context.Context,
    source *dagger.Directory,
    // +required
    localstackToken *dagger.Secret,
    // +required
    dockerSocket *dagger.Socket,
) *dagger.File {
    // Start LocalStack using the official module with Docker socket support
    // Don't specify SERVICES config to avoid parsing issues - let it use defaults
    localstackService := dag.Localstack().
        Start(dagger.LocalstackStartOpts{
            AuthToken:     localstackToken,
            Configuration: "DEBUG=1",
            DockerSock:    dockerSocket,
        })
    
    // Run test directly with coverage
    coverageFile := "coverage_snapshot.out"
    
    return s.BuildImage(ctx, source).
        WithDirectory(fmt.Sprintf("%s/tests", SIDEKICK_SRC_PATH), source.Directory("tests")).
        WithServiceBinding("localstack", localstackService).
        WithEnvVariable("AWS_ENDPOINT", "http://localstack:4566").
        WithEnvVariable("S3_BUCKET", "bucket").
        WithWorkdir(fmt.Sprintf("%s/tests/integration/aws/snapshot", SIDEKICK_SRC_PATH)).
        WithExec([]string{
            "go", "test",
            "-v",
            "-timeout", "10m",
            "-coverprofile", coverageFile,
            "-coverpkg", "github.com/lacework-dev/sidekick/...",
            "-tags", GO_BUILD_TAGS,
            "./...",
        }).
        File(coverageFile)
}
#

as you can see in this docker-compose.yml, Localstack mounts a docker socket which it requires to emulate EC2 instances

version: "3.7"
services:
  localstack:
    image: localstack/localstack-pro
    container_name: localstack
    network_mode: bridge
    ports:
      - "127.0.0.1:4566:4566"
      - "127.0.0.1:4571:4571"
      - "127.0.0.1:8055:8055"
      - "127.0.0.1:53:53"                # DNS config (required for Pro)
      - "127.0.0.1:53:53/udp"            # DNS config (required for Pro)
      - "127.0.0.1:443:443"              # LocalStack HTTPS Gateway (required for Pro)
    environment:
      - SERVICES=s3,ec2,secretsmanager,ecs,iam,sts,fis
      - DEBUG=1
      - DATA_DIR=/var/lib/localstack/data
      - HOST_TMP_FOLDER=${LOCALSTACK_VOLUME_DIR:-./volume}/localstack
      - DOCKER_HOST=unix:///var/run/docker.sock
      - LOCALSTACK_AUTH_TOKEN=${LOCALSTACK_AUTH_TOKEN}
    volumes:
      - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
Docs
Elastic Compute Cloud (EC2)

Get started with Amazon Elastic Compute Cloud (EC2) on LocalStack

verbal mulch
#

Hey! Welcome, thanks for sharing this issue.

I think even if you are using dind you still need to run it as a service. There is no other way to have a long running container.

safe fox
gleaming relic
#

Hi @verbal mulch, thanks for the pointer. I'm a little unclear on what that means in my case - are you saying I create a service from the dind image and run localstack within it?

In that case, wouldn't I still have the same issue of not being able to start dockerd in the dind container?

safe fox
#

by quickly skimming through localstack's official module, I see here that the way they're support docker compatible services like ECS, Lambda, etc is by mounting the host's docker socket within the module (https://github.com/localstack/localstack-dagger-module/tree/6e2683e20e54e7ceab130ebe2abed050b6670ed2?tab=readme-ov-file#mounting-docker-socket) instead of starting a docker engine within the dagger pipeline which seems to be closer to what Earyle's WITH DOCKER did

GitHub
GitHub - localstack/localstack-dagger-module at 6e2683e20e54e7ceab1...

A Dagger module for running LocalStack as a service within your Dagger pipelines. - GitHub - localstack/localstack-dagger-module at 6e2683e20e54e7ceab130ebe2abed050b6670ed2

gleaming relic
#

Hi @safe fox, thanks for sharing - yes, I did see that - that's what I used to run localstack as a service in the latest example I shared

safe fox
#

so you basically have two options:

  • You can start a docker engine within dagger as a *dagger.Service as Lev is mentioning
  • You pass your host's docker.sockto your dagger module which will use your host's docker engine to run things
gleaming relic
#

the issue I'm having is that I don't want to mount my host's docker socket because that results in issues due to port conflicts on the host when Localstack creates emulated EC2 instances - as such, I'm looking to run Localstack in a dind container, mounting the dind container's docker socket

The issue I'm having is that I'm unable to start the docker daemon in the dind container

safe fox
safe fox
gleaming relic
#

gotcha, if I create a docker engine using dagger's docker module, how would I get mount its docker socket to localstack?

safe fox
gleaming relic
safe fox
#

usually it's via setting DOCKER_HOST=tcp://$host since all the docker SDK's currently respect that

gleaming relic
#

got it, thank you @safe fox !

safe fox
gleaming relic
#

@safe fox you were completely right - Localstack does support the DOCKER_HOST envvar (ref)

Thanks so much for your help!

Docs
Configuration

Overview of configuration options in LocalStack.

#

For posterity, here's what ended up working for me:

// createLocalstackService configures and runs LocalStack as a dagger service
func createLocalstackService(localstackToken *dagger.Secret) *dagger.Service {
    return dag.Container().
        From("localstack/localstack-pro:latest").
        WithSecretVariable("LOCALSTACK_AUTH_TOKEN", localstackToken).
        WithEnvVariable("SERVICES", "s3,ec2,secretsmanager,ecs,iam,sts,fis").
        WithEnvVariable("AWS_DEFAULT_REGION", "us-east-1").
        WithEnvVariable("DEBUG", "1").
        WithExec([]string{"mkdir", "-p", "/var/lib/localstack/data"}).
        WithEnvVariable("DATA_DIR", "/var/lib/localstack/data").
        // Bind docker engine service as `docker``
        WithServiceBinding("docker", dag.Docker().Engine()).
        // Have localstack interact with docker engine service created above
        WithEnvVariable("DOCKER_HOST", "tcp://docker:2375").
        WithExposedPort(4566).
        WithExposedPort(53).
        WithExposedPort(53, dagger.ContainerWithExposedPortOpts{Protocol: dagger.NetworkProtocolUdp}).
        WithExposedPort(443).
        AsService()
}
lofty cloak
#

We're going to flip it on as a partner module in Daggerverse.

#

Thanks for sticking with it @gleaming relic 💪

gleaming relic