BigTesty project with Dagger and packaged in Docker : issue with Docker Dind and Cloud Build | Dagger | Page 1

unkempt fiber Dec 12, 2023, 10:21 PM

#

Hello everyone,

Few months ago I presented to the Dagger community meetup a project called BigTesty.

This library proposes integration testing with BigQuery on Google Cloud with an ephemeral infra and it's based on Dagger for the CI logic.

Currently the orientation is to package the project on Docker, because we want to be able to use BigTesty in each CI tool (Cloud Build, Gitlab CI, Github Actions...). Each of these tools is based on Docker.

Today when I build the image locally and execute it with Docker (build and run), it works correctly.

Dagger depends on Docker and I need to deal with Docker Dind but it works well locally.

I haven't tested yet with Github actions and Gitlab CI, but I think it should work because I saw some examples of how to use Dagger in an existing CI.

In these examples, the execution is done with Docker Dind in a privileged mode.

I have an issue using BigTesty and deal with Docker in Docker with Cloud Build.

There is no example in the Google Cloud or Dagger doc to use Dagger in Cloud Build.

I wanted to ask some help from the Dagger community, maybe the approach used today is not the best approach (BigTesty packaged in Docker to be used in each CI tool), please feel free to give me your advice, feedback and propositions.

To prevent the use of Docker Dind in the CI tools (painful with Cloud Build), I also tested to run the Dagger code of BigTesty with Podman and rootless mode but I have some issues :

I share with you the link of the project from Github :
https://github.com/tosun-si/bigtesty

The Dagger logic is written in this file :
https://github.com/tosun-si/bigtesty/blob/main/main.go

The packaging of the Dagger app in Docker is written is this file :
https://github.com/tosun-si/bigtesty/blob/main/Dockerfile

Contributions are also welcome to help me to release an alpha version.

Thanks for your help.

GitHub

GitHub - tosun-si/bigtesty: BigTesty is a framework that allows to ...

BigTesty is a framework that allows to create Integration Tests with BigQuery on a real and short lived Infrastructure. - GitHub - tosun-si/bigtesty: BigTesty is a framework that allows to create I...

GitHub

bigtesty/main.go at main · tosun-si/bigtesty

BigTesty is a framework that allows to create Integration Tests with BigQuery on a real and short lived Infrastructure. - tosun-si/bigtesty

GitHub

bigtesty/Dockerfile at main · tosun-si/bigtesty

BigTesty is a framework that allows to create Integration Tests with BigQuery on a real and short lived Infrastructure. - tosun-si/bigtesty

wide finch Dec 13, 2023, 2:20 PM

#

Dagger depends on Docker and I need to deal with Docker Dind but it works well locally.

Docker is used generally as a convenient way to start the Dagger engine but the dagger engine doesn't have a dependency on docker. You can totally start the Dagger engine without docker.

#

Dagger depends on Docker and I need to deal with Docker Dind but it works well locally.

why is it that you need to deal with dind? Not sure I'm following

#

IIUC is the question how to make Dagger run in Cloud Build?

unkempt fiber Dec 13, 2023, 3:54 PM

#

Hi @wide finch thanks for your help.

Yes, BigTesty is only a wrapper on top of Dagger.

If I want to include a Dagger app to Cloud Build or Gitlab CI, I am in a Docker in Docker situation and need to deal with Docker Dind.

#

In Gitlab CI, it's simple to pass the params for Docker Dind.

In Cloud Build, unfortunately it's it is not that simple.

In an ideal world, if I could have avoided the Docker Dind it would have been even better, but I want to use BigTesty from a Docker image

wide finch Dec 13, 2023, 3:57 PM

#

I'm not following exactly what the issue is. I think both Cloud Build and Gitlab CI support running docker so Dagger can be used directly from there

#

If I want to include a Dagger app to Cloud Build or Gitlab CI, I am in a Docker in Docker situation and need to deal with Docker Dind.

you're not in a Docker-in-docker situation. You're using the docker engine already present in those environments to start the Dagger engine

#

what's the specific issue you currently have?

unkempt fiber Dec 13, 2023, 4:01 PM

#

Are you sure for the Dagger in Docker ?

Please check this link that shows an example to integrate Dagger in Gitlab CI :

https://docs.dagger.io/759201/gitlab-google-cloud/

.docker:
  image: golang:alpine
  services:
    - docker:${DOCKER_VERSION}-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_VERIFY: '1'
    DOCKER_TLS_CERTDIR: '/certs'
    DOCKER_CERT_PATH: '/certs/client'
    DOCKER_DRIVER: overlay2
    DOCKER_VERSION: '20.10.16'
.dagger:
  extends: [.docker]
  before_script:
    - apk add docker-cli curl
    - cd /usr/local && { curl -L https://dl.dagger.io/dagger/install.sh | sh; cd -; }
    - cat $GOOGLE_APPLICATION_CREDENTIALS | docker login -u _json_key --password-stdin https://gcr.io
build-deploy:
  extends: [.dagger]
  script:
    - dagger run go run ci/main.go

Use Dagger with GitLab CI/CD and Google Cloud | Dagger

Introduction

#

Moreover in my case, my Dagger app is packaged on a Dockerfile

https://github.com/tosun-si/bigtesty/blob/main/Dockerfile

Then in my container, Dagger depends in Docker.

Maybe I am wrong, but in my case it doesn't works without a privileged mode.

GitHub

bigtesty/Dockerfile at main · tosun-si/bigtesty

BigTesty is a framework that allows to create Integration Tests with BigQuery on a real and short lived Infrastructure. - tosun-si/bigtesty

wide finch Dec 13, 2023, 4:05 PM

#

Maybe I am wrong, but in my case it doesn't works without a privileged mode.

yes, you need privileged mode, but that's independent of Docker-in-docker

unkempt fiber Dec 13, 2023, 4:06 PM

#

How I build and launch my app :

Build :

docker build -t bigtesty  .

Run :

 docker run -it \
    --platform linux/amd64 \
    --privileged \
    -e PROJECT_ID=$PROJECT_ID \
    -e SA_EMAIL=$SA_EMAIL \
    -e LOCATION=$LOCATION \
    -e IAC_BACKEND_URL=$IAC_BACKEND_URL \
    -e ROOT_TEST_FOLDER=$ROOT_TEST_FOLDER \
    -v $(pwd)/tests:/app/tests \
    -v $(pwd)/tests/tables:/app/infra/bigquery/tables \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v $HOME/.config/gcloud:/root/.config/gcloud \
    bigtesty

wide finch Dec 13, 2023, 4:06 PM

#

if you're running the Dagger engine in a container, you need privileged mode

unkempt fiber Dec 13, 2023, 4:06 PM

#

Yes it's the case

wide finch Dec 13, 2023, 4:06 PM

#

unkempt fiber How I build and launch my app : Build : ```docker docker build -t bigtesty ...

if you're mounting the docker socket you shouldn't need privileged in this case

unkempt fiber Dec 13, 2023, 4:06 PM

#

Maybe I am not in the right direction

Thanks again for your help

wide finch Dec 13, 2023, 4:07 PM

#

the only thing that you need is to add the docker-cli to your application image

unkempt fiber Dec 13, 2023, 4:07 PM

#

wide finch if you're mounting the docker socket you shouldn't need `privileged` in this cas...

Yes locally, but from Gitlab or Cloud Build, I need the privileged mode

wide finch Dec 13, 2023, 4:07 PM

#

unkempt fiber Yes locally, but from Gitlab or Cloud Build, I need the privileged mode

no you don't since you can also use Gitlab's host engine

unkempt fiber Dec 13, 2023, 4:07 PM

#

A ok

unkempt fiber Dec 13, 2023, 4:08 PM

#

wide finch the only thing that you need is to add the `docker-cli` to your application imag...

A added it

#

The content of my Dockerfile :

FROM golang:1.20-alpine as builder

WORKDIR /app

COPY . .

RUN GOOS=linux GOARCH=amd64 go build -ldflags="-w -s" -o bin/bigtestyapp

FROM alpine:latest

WORKDIR /app

COPY . .

RUN apk add docker-cli curl

COPY --from=builder /app/bin/bigtestyapp .

ENTRYPOINT ["/app/bigtestyapp"]

wide finch Dec 13, 2023, 4:09 PM

#

perfect

#

in that case the reason why it doesn't work in gitlab is because gitalb doesn't give you access to /var/run/docker.sock

unkempt fiber Dec 13, 2023, 4:09 PM

#

Yes

wide finch Dec 13, 2023, 4:09 PM

#

you need to set the DOCKER_HOST and DOCKER_TLS variables accordingly

#

but again, you don't need privileged mode when you have access to the host engine

unkempt fiber Dec 13, 2023, 4:11 PM

#

Yes, in Gitlab, it works in my case, but in Cloud Build, I don't find the way to use the Docker Dind service + the needed variables

#

There is no example in this case

#

I also tried to do the runtime with Podman for my Dagger app (rootless), but it didn't work

wide finch Dec 13, 2023, 4:12 PM

#

do you have access to /var/run/docker.sock in cloud build?

unkempt fiber Dec 13, 2023, 4:12 PM

#

wide finch do you have access to `/var/run/docker.sock` in cloud build?

I am not sure 😢

#

I tested it with a Docker Composer in Cloud Build :

steps:
  - id: start-dind
    name: docker/compose
    args: [ '-f', 'docker-compose.yaml', 'up', 'dind-service' ]
  - id: 'Check service is listening'
    name: alpine:latest
    args:
      - '-eEuo'
      - 'pipefail'
      - '-c'
      - |-
        apk add curl
        until $(curl dind-service:2376); do
          echo '##################Waiting################'
          sleep 5
        done
        curl dind-service:2376
    entrypoint: "sh"
    #    args: [ "dind-service:2375" ]
    #   waitFor: [ start-dind ]
    waitFor: [ '-' ]
#  - name: 'europe-west1-docker.pkg.dev/gb-poc-373711/internal-images/bigtesty:latest'
#    args: [ './scripts/run_tests_bigtesty.sh' ]
#    env:
#      - 'DOCKER_HOST=tcp://dind-service:2375'
#      - 'PROJECT_ID=$PROJECT_ID'
#      - 'LOCATION=$LOCATION'
#      - 'TF_VAR_project_id=$PROJECT_ID'
#      - 'TF_STATE_BUCKET=$_TF_STATE_BUCKET'
#      - 'TF_STATE_PREFIX=$_TF_STATE_PREFIX'
#      - 'GOOGLE_PROVIDER_VERSION=$_GOOGLE_PROVIDER_VERSION'
#      - 'ROOT_TEST_FOLDER=$_ROOT_TEST_FOLDER'

wide finch Dec 13, 2023, 4:21 PM

#

yo shouldn't need a dind-service

unkempt fiber Dec 13, 2023, 4:22 PM

#

docker-composer.yaml

version: '3'
services:
  dind-service:
    image: docker:24.0.6-dind
    privileged: true
    ports:
      - "127.0.0.1:2375:2375"
      - "127.0.0.1:2376:2376"
networks:
  default:
    external:
      name: cloudbuild

wide finch Dec 13, 2023, 4:22 PM

#

if you can already run docker commands

#

cloud build already has a docker daemon running

#

can you run ls -la /var/run/docker.sock as a step so see if that file is there in Cloud build?

#

I don't have access to a GCP accont now 😦

unkempt fiber Dec 13, 2023, 4:23 PM

#

A ok, I will test it

#

Thanks

wide finch Dec 13, 2023, 4:24 PM

#

if you can also run env to see what env variables you have avaiable that could also help

unkempt fiber Dec 13, 2023, 4:25 PM

#

Ok, I am going to try it tonight

I keep you posted, thanks again

wide finch Dec 13, 2023, 4:25 PM

#

👍 but you get the idea

#

you don't need dind

unkempt fiber Dec 13, 2023, 4:25 PM

#

What do you think about running the app with Podman and rootless

#

?

wide finch Dec 13, 2023, 4:26 PM

#

since you can pass your bigtesty container the host docker connection info and that should be enough

wide finch Dec 13, 2023, 4:26 PM

#

unkempt fiber What do you think about running the app with Podman and rootless

Dagger doens't work with rootless containers

unkempt fiber Dec 13, 2023, 4:26 PM

#

A ok
I checked from this link

#

https://docs.dagger.io/541047/alternative-runtimes/

Use Dagger with Alternative OCI Runtimes | Dagger

Introduction

#

Runtimes
Podman
Requirements
This guide assumes that you have Podman installed and running on the host system. If not, install Podman.

Configuration
Podman is CLI-compatible with Docker and therefore can be used by creating a symbolic link to the Podman executable in your system path and naming it docker:

sudo ln -s $(which podman) /usr/local/bin/docker

wide finch Dec 13, 2023, 4:40 PM

#

yes, podman works but it is still used with --privileged mode by Dagger

unkempt fiber Dec 13, 2023, 5:05 PM

#

You are right, sorry, I passed the --privileged mode with Dagger and Podman, but I had an issue

failed to list containers: exit status 125
Error: failed to run container: Error: failed to get new shm lock manager: failed to create 2048 locks in /libpod_lock: function not implemented
: exit status 125
panic: EOF: failed to list containers: exit status 125
Error: failed to run container: Error: failed to get new shm lock manager: failed to create 2048 locks in /libpod_lock: function not implemented
: exit status 125

The advantage to use Podman is I don't need in this case, to mount the docker sock as volume

#

I have a dedicated branch for my test with Dagger and Podman :
https://github.com/tosun-si/bigtesty/tree/feature/add_runtime_with_podman

My Dockerfile :

FROM golang:1.20-alpine as builder

WORKDIR /app

COPY . .

RUN GOOS=linux GOARCH=amd64 go build -ldflags="-w -s" -o bin/bigtestyapp

FROM mgoltzsche/podman:rootless

WORKDIR /app

RUN ln -s $(which podman) /usr/local/bin/docker

COPY . .

COPY --from=builder /app/bin/bigtestyapp .

ENTRYPOINT ["/app/bigtestyapp"]

GitHub

GitHub - tosun-si/bigtesty at feature/add_runtime_with_podman

BigTesty is a framework that allows to create Integration Tests with BigQuery on a real and short lived Infrastructure. - GitHub - tosun-si/bigtesty at feature/add_runtime_with_podman

#

My Docker run command :

docker run -it \
    --privileged \
    -e PROJECT_ID=$PROJECT_ID \
    -e SA_EMAIL=$SA_EMAIL \
    -e LOCATION=$LOCATION \
    -e IAC_BACKEND_URL=$IAC_BACKEND_URL \
    -e ROOT_TEST_FOLDER=$ROOT_TEST_FOLDER \
    -v $(pwd)/tests:/app/tests \
    -v $(pwd)/tests/tables:/app/infra/bigquery/tables \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v $HOME/.config/gcloud:/root/.config/gcloud \
    bigtesty

#

The issue :

failed to list containers: exit status 125
Error: failed to run container: Error: failed to get new shm lock manager: failed to create 2048 locks in /libpod_lock: function not implemented
: exit status 125
panic: EOF: failed to list containers: exit status 125
Error: failed to run container: Error: failed to get new shm lock manager: failed to create 2048 locks in /libpod_lock: function not implemented
: exit status 125

wide finch Dec 13, 2023, 5:10 PM

#

is that with podman?

#

well.. I assume that /var/run/docker.sock is not a thing when running Podman?

#

so it doesn't really apply in this scenario

#

I think podman is /var/run/podman.sock?

#

so you need to change your docker run to mount /var/run/podman.sock:/var/run/docker.sock

#

so the docker-cli inside your bigtesty container actually finds that

unkempt fiber Dec 13, 2023, 5:24 PM

#

Sorry I did a mistake in my copy paste, the command line is :

 docker run -it \
    --privileged \
    -e PROJECT_ID=$PROJECT_ID \
    -e SA_EMAIL=$SA_EMAIL \
    -e LOCATION=$LOCATION \
    -e IAC_BACKEND_URL=$IAC_BACKEND_URL \
    -e ROOT_TEST_FOLDER=$ROOT_TEST_FOLDER \
    -v $(pwd)/tests:/app/tests \
    -v $(pwd)/tests/tables:/app/infra/bigquery/tables \
    -v $HOME/.config/gcloud:/root/.config/gcloud \
    bigtesty

Without the Docker sock.

I thought there is no need to pass the podman sock, but maybe I am wrong

#

If I need to pass the Podman sock, there is no advantage in my case to use a runtime with Podman

Might as well stay with native Docker.

wide finch Dec 13, 2023, 5:59 PM

#

unkempt fiber Sorry I did a mistake in my copy paste, the command line is : ```text docker...

but this is not going to work because bigtesty will not be able to start the the Dagger engine

#

as there's no docker compatible daemon to connect to

unkempt fiber Dec 13, 2023, 6:08 PM

#

Ok I understand

wide finch Dec 13, 2023, 6:27 PM

#

LMK if you need more information about how the Dagger engine gets started

#

maybe this will give you better ideas on how to package and distribute this

unkempt fiber Dec 13, 2023, 7:40 PM

#

wide finch LMK if you need more information about how the Dagger engine gets started

Thanks again for you help, I really appreciate it 🙏🏻

wide finch Dec 13, 2023, 7:44 PM

#

unkempt fiber Thanks again for you help, I really appreciate it 🙏🏻

sure, anytime! Happy to help

unkempt fiber Dec 14, 2023, 6:03 PM

#

Hello @wide finch

I tested to pass the sock from the Cloud Build context to BigTesty while running the container, but from now it doesn't work..

This script is launched from the Cloud Build job and my yaml file :

docker run -it \
  -e PROJECT_ID="$PROJECT_ID" \
  -e LOCATION="$LOCATION" \
  -e SA_EMAIL="$SA_EMAIL" \
  -e IAC_BACKEND_URL="$IAC_BACKEND_URL" \
  -e ROOT_TEST_FOLDER="$ROOT_TEST_FOLDER" \
  -v tests:/app/tests \
  -v tests/tables:/app/infra/bigquery/tables \
  -v /var/run/docker.sock:/var/run/docker.sock \
  bigtesty

steps:
  - name: 'europe-west1-docker.pkg.dev/gb-poc-373711/internal-images/bigtesty:latest'
    args: [ './scripts/run_tests_bigtesty.sh' ]
    env:
      - 'PROJECT_ID=$PROJECT_ID'
      - 'LOCATION=$LOCATION'
      - 'SA_EMAIL=$_SA_EMAIL'
      - 'IAC_BACKEND_URL=$_IAC_BACKEND_URL'
      - 'ROOT_TEST_FOLDER=$_ROOT_TEST_FOLDER'
logsBucket: 'gs://cloud-build-ci-cd-groupbees'
options:
  logging: GCS_ONLY

Unfortunately, having to mount the "sock" in volume adds complexity for me 😭

wide finch Dec 14, 2023, 6:38 PM

#

unkempt fiber Hello <@336241811179962368> I tested to pass the sock from the Cloud Build con...

any ideas why it didn't work?

unkempt fiber Dec 14, 2023, 7:04 PM

#

I am not sure the syntax is correct, to pass the sock from Cloud Build.

I will continue to explore it.

unkempt fiber Dec 14, 2023, 9:39 PM

#

failed to list containers: fork/exec /usr/bin/docker: exec format error
Error: failed to run container: : fork/exec /usr/bin/docker: exec format error
panic: EOF: failed to list containers: fork/exec /usr/bin/docker: exec format error
Error: failed to run container: : fork/exec /usr/bin/docker: exec format error

unkempt fiber Dec 14, 2023, 10:27 PM

#

This first issue was due to an archi problem when I built my image from my M1 Macbook

#

I solved it and I advance 😁

wide finch Dec 15, 2023, 2:30 AM

#

nice that makes sense! is it still not working?

unkempt fiber Dec 15, 2023, 5:13 PM

#

Hi @wide finch

I publish the BigTesty Docker image to Artifact Registry (private repo on GCP)

I execute a Docker from my local machine and it works :

europe-west1-docker.pkg.dev/gb-poc-373711/internal-images/bigtesty:latest

docker run -it \
    --platform linux/amd64 \
    -e PROJECT_ID=$PROJECT_ID \
    -e SA_EMAIL=$SA_EMAIL \
    -e LOCATION=$LOCATION \
    -e IAC_BACKEND_URL=$IAC_BACKEND_URL \
    -e ROOT_TEST_FOLDER=$ROOT_TEST_FOLDER \
    -v $(pwd)/tests:/app/tests \
    -v $(pwd)/tests/tables:/app/bigtesty/infra/resource/tables \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v $HOME/.config/gcloud:/root/.config/gcloud \
    europe-west1-docker.pkg.dev/gb-poc-373711/internal-images/bigtesty:latest

But when I pull this image from Cloud Build, I have a not clear issue

Dagger process returns this error

e.withEnvVariable.withEnvVariable.withEnvVariable.withExec.withExec.directory process "echo --------------------------- Creating the ephemeral infra ---------------------------" did not complete successfully: exit code: 1
Stdout:

Stderr:

#

Cloud Build that pull the BigTesty with Dagger returns a standard error (exit code 2) :

BUILD FAILURE: Build step failure: build step 0 "europe-west1-docker.pkg.dev/gb-poc-373711/internal-images/bigtesty:latest" failed: step exited with non-zero status: 2
DEBUG: (gcloud.builds.submit) build 3a6d3d3d-42b5-457b-b528-9c9fbcc43ab8 completed with status "FAILURE"
Traceback (most recent call last):
  File "/Users/mazlum/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 987, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/Users/mazlum/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 807, in Run
    resources = command_instance.Run(args)
  File "/Users/mazlum/google-cloud-sdk/lib/surface/builds/submit.py", line 252, in Run
    build, _ = submit_util.Build(
  File "/Users/mazlum/google-cloud-sdk/lib/googlecloudsdk/command_lib/builds/submit_util.py", line 971, in Build
    raise FailedBuildException(build)
googlecloudsdk.command_lib.builds.submit_util.FailedBuildException: build 3a6d3d3d-42b5-457b-b528-9c9fbcc43ab8 completed with status "FAILURE"
ERROR: (gcloud.builds.submit) build 3a6d3d3d-42b5-457b-b528-9c9fbcc43ab8 completed with status "FAILURE"

#

The Cloud Build code is :

steps:
  - name: 'europe-west1-docker.pkg.dev/gb-poc-373711/internal-images/bigtesty:latest'
    args: [ './scripts/run_tests_bigtesty.sh' ]
    env:
      - 'PROJECT_ID=$PROJECT_ID'
      - 'LOCATION=$LOCATION'
      - 'SA_EMAIL=$_SA_EMAIL'
      - 'IAC_BACKEND_URL=$_IAC_BACKEND_URL'
      - 'ROOT_TEST_FOLDER=$_ROOT_TEST_FOLDER'
logsBucket: 'gs://cloud-build-ci-cd-groupbees'
options:
  logging: GCS_ONLY

#

The BigTesty Docker image is :

FROM golang:1.20-alpine as builder

WORKDIR /app

COPY . .

RUN GOOS=linux GOARCH=amd64 go build -ldflags="-w -s" -o bin/bigtestyapp

FROM alpine:latest

WORKDIR /app

COPY . .

RUN apk add docker-cli curl

COPY --from=builder /app/bin/bigtestyapp .

ENTRYPOINT ["/app/bigtestyapp"]

#

In the Dagger code, I also use a Docker image from an the TTL ephemeral repo :

bigTestyEnvImageName := "ttl.sh/bigtesty-env:8h"

wide finch Dec 15, 2023, 7:18 PM

#

Dagger process returns this error

e.withEnvVariable.withEnvVariable.withEnvVariable.withExec.withExec.directory process "echo --------------------------- Creating the ephemeral infra ---------------------------" did not complete successfully: exit code: 1
Stdout:

Stderr:

do you have any more logs here?

wide finch Dec 15, 2023, 8:48 PM

#

Dagger should print more logs when the clients connects to the engine and starts running the pipeline

unkempt fiber Dec 16, 2023, 10:28 AM

#

I don't understand why, but I don't have more logs 😦

Maybe I can set a debug or verbose mode when I launch my Go binary and app from my Docker image

wide finch Dec 16, 2023, 6:32 PM

#

@unkempt fiber just did a test and I was able to run Dagger successfully in Cloud build

#

here's my code:

steps:
  - name: "gcr.io/cloud-builders/docker"
    script: |
      docker run --rm -v /var/run/docker.sock:/var/run/docker.sock ttl.sh/dagger_echo:1d

and this is the the code for that docker image

package main

import (
    "context"
    "fmt"
    "os"

    "dagger.io/dagger"
)

func main() {
    if err := build(context.Background()); err != nil {
        fmt.Println(err)
    }
}

func build(ctx context.Context) error {
    fmt.Println("Building with Dagger")

    // initialize Dagger client
    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
    if err != nil {
        return err
    }

    defer client.Close()

    client.Container().From("alpine").WithExec([]string{"echo", "hello"}).Sync(ctx)
    return nil
}

this is the Dockerfile I've used to build it:

FROM alpine:latest

WORKDIR /app

COPY dagger_echo .

RUN apk add docker-cli

ENTRYPOINT ["/app/dagger_echo"]

#

📎 message.txt

unkempt fiber Dec 16, 2023, 6:53 PM

#

@wide finch thank you so much 🙏🏻
I will test it soon and keep you posted !!

wide finch Dec 17, 2023, 3:06 PM

#

cc @bronze zinc. Idea to add a Cloud Build snippet to our docs

bronze zinc Dec 17, 2023, 3:07 PM

#

wide finch cc <@1013436980249505882>. Idea to add a Cloud Build snippet to our docs

Current or zenith docs?

#

Can you please create an issue for it and assign it to me?

unkempt fiber Dec 17, 2023, 11:34 PM

#

@wide finch when I execute your Dagger code (minimal program) in Cloud Build it works.

But if I execute the BigTesty Dagger logic, I have the same issue than when I wanted to directly pull the BigTesty image from Cloud Build

#

#5 ERROR: process "echo --------------------------- Creating the ephemeral infra ---------------------------" did not complete successfully: exit code: 1

#6 from ttl.sh/bigtesty-env:8h
panic: input:1: container.from.withWorkdir.withMountedDirectory.withDirectory.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withExec.withExec.directory process "echo --------------------------- Creating the ephemeral infra ---------------------------" did not complete successfully: exit code: 1
Stdout:

Stderr:


Please visit https://dagger.io/help#go for troubleshooting guidance.

#

goroutine 1 [running]:
main.main()
    /app/main.go:155 +0x21a6
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 2
DEBUG: Starting new HTTPS connection (1): cloudbuild.googleapis.com:443
DEBUG: https://cloudbuild.googleapis.com:443 "GET /v1/projects/gb-poc-373711/locations/europe-west1/builds/3ffe79d3-3a38-43f5-bd08-b64cdf8b6f84?alt=json HTTP/1.1" 200 None
DEBUG: Starting new HTTPS connection (1): storage.googleapis.com:443
DEBUG: https://storage.googleapis.com:443 "GET /975119474255.cloudbuild-logs.googleusercontent.com/log-3ffe79d3-3a38-43f5-bd08-b64cdf8b6f84.txt HTTP/1.1" 416 168
DEBUG: Reading GCS logfile: 416 (no new content; keep polling)
DEBUG: Starting new HTTPS connection (1): storage.googleapis.com:443
DEBUG: https://storage.googleapis.com:443 "GET /975119474255.cloudbuild-logs.googleusercontent.com/log-3ffe79d3-3a38-43f5-bd08-b64cdf8b6f84.txt HTTP/1.1" 416 168
DEBUG: Reading GCS logfile: 416 (no new content; keep polling)
----------------------------------------------------------------------------------------------------------------------------------

#

BUILD FAILURE: Build step failure: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 2
DEBUG: (gcloud.builds.submit) build 3ffe79d3-3a38-43f5-bd08-b64cdf8b6f84 completed with status "FAILURE"
Traceback (most recent call last):
  File "/Users/mazlum/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 987, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/Users/mazlum/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 807, in Run
    resources = command_instance.Run(args)
  File "/Users/mazlum/google-cloud-sdk/lib/surface/builds/submit.py", line 252, in Run
    build, _ = submit_util.Build(
  File "/Users/mazlum/google-cloud-sdk/lib/googlecloudsdk/command_lib/builds/submit_util.py", line 971, in Build
    raise FailedBuildException(build)
googlecloudsdk.command_lib.builds.submit_util.FailedBuildException: build 3ffe79d3-3a38-43f5-bd08-b64cdf8b6f84 completed with status "FAILURE"
ERROR: (gcloud.builds.submit) build 3ffe79d3-3a38-43f5-bd08-b64cdf8b6f84 completed with status "FAILURE"
❯

#

I think the problem occurs when I invoke a Python script from the Dagger code :

WithExec([]string{
            "python",
            "-m",
            "bigtesty.infra.main",
        }).

#

I don't understand why I don't have more info on the errors

#

The BigTesty logic is not so easy :

Dagger

Invoke a Pulumi Python script to create the ephemeral infra
Invoke a Python script with the GCP BigQuery client to ingest test data in the ephemeral infra
Invoke a Pulumi Python script to destroy the ephemeral infra and apply tests assertions

#

I also need to deal with authentication on Google Cloud from the Dagger code.

#

From the local machine, I use Applications Default Credentials

I share my local gcloud config with the gcloud in the container (dagger), then the Dagger code is automatically authenticated in GCP

#

From the CI context (Cloud Build), normally I don't need to share the gcloud config, because ADC (App Default Credentials) can deduce the credentials from the authenticated SA (metadata in the runner VMs)

#

I will continues some tests and I keep you posted !!

Maybe I don't correctly set the SA in Cloud Build.

I will test it with the traditional and not secure apprach and a SA token key (long-lived), to be sure the error is not due to the authent part

#

I was just expecting to have more elements for errors...

#

I will also reduce my Dagger logic to only one action on GCP, to test parts separately

wide finch Dec 19, 2023, 2:58 AM

#

👍 let us know if you're getting stuck and we can troubleshoot it together 💪

unkempt fiber Dec 19, 2023, 10:38 PM

#

Thanks so much, we're not giving up, we're going to do it !! 🦾

I think my issue is due the invocation of my Python script (Python module and logic separated in multi files) from a Dagger step :

WithExec([]string{
            "python",
            "-m",
            "bigtesty.infra.main",
        }).

but the mystery is why I don't have any error logs 😥

Stout and Stderr are empties

panic: input:1: container.from.withMountedDirectory.withDirectory.withEnvVariable.withEnvVariable.withExec.withExec.directory process "echo --------------------------- Inserting Test data to Tables ---------------------------" did not complete successfully: exit code: 1
Stdout:

Stderr:

unkempt fiber Dec 19, 2023, 11:37 PM

#

I replaced the Python command by a classical gsutil command and I have the same issue :

WithExec([]string{
            "gsutil",
            "ls",
            "gs://mazlum_dev/dataflow/",
        }).

wide finch Dec 21, 2023, 9:33 PM

#

hey @unkempt fiber ! wouldn't like this to go cold. Need any help here?

unkempt fiber Dec 21, 2023, 9:45 PM

#

@wide finch to be honest, I'm a little stuck and can't solve the problem 😢

#

I don’t understand why I don’t have error logs

#

Not easy in this case to have more element to understand the issue.

It works well locally, but from Cloud Build, when I invoke an exec command (Python script or a gcloud command), I have this issue

#

Maybe we can organize in little video call, and I can explain you more the steps and the code

#

Tonight, I will try again.

The code was reduced to only one step with a gsutil command

#

I also pass some folders as volume

Tonight, I will remote all these elements

#

To have the smallest possible code

wide finch Dec 22, 2023, 3:47 AM

#

What timezone are you in? Let's try to talk tomorrow

wide finch Dec 22, 2023, 6:32 PM

#

👀

unkempt fiber Dec 23, 2023, 11:28 AM

#

I am in Paris timezone, are you available the next Tuesday ?

#

You are really great 😀, I think together with a video call, we can solve this issue 🙂

wide finch Dec 23, 2023, 2:11 PM

#

Yes, I'm in GMT -3. Lets talk on Tuesday

unkempt fiber Dec 27, 2023, 2:56 PM

#

@wide finch are you available tomorrow at 11 AM at your GMT (GMT-3) ?

wide finch Dec 27, 2023, 3:39 PM

#

yep, I'll be here

unkempt fiber Dec 27, 2023, 4:41 PM

#

Do you prefer a Google Meet invitation ?

wide finch Dec 27, 2023, 4:46 PM

#

yes, that works

#

you can send it to marcos@dagger.io

unkempt fiber Dec 27, 2023, 6:48 PM

#

Ok thanks 🙂

unkempt fiber Dec 28, 2023, 1:31 PM

#

Hi @wide finch

#

Sorry I have an personal problem

#

Can we postpone our meeting until tomorrow ?

#

Sorry again

wide finch Dec 28, 2023, 1:55 PM

#

hey! sure, no problem

unkempt fiber Dec 29, 2023, 4:33 PM

#

@wide finch thank you so much for your time and your help

You are great 🤩

wide finch Dec 29, 2023, 4:33 PM

#

we fixed this 🙌 . It was a mix of pushing the correct docker image platform and some missing steps in the pipeline. 💪

happy to unblock you @unkempt fiber and happy new year!

#

I'm closing this thread.

#BigTesty project with Dagger and packaged in Docker : issue with Docker Dind and Cloud Build