Error: Dagger Functions in GitLab CI —> Wrong socket | Dagger | Page 1

flat surge Mar 14, 2024, 3:38 PM

#

Hey 👋 ,
I need your help. I want to run my GitLab Pipeline with a Dagger function.
I get the attached error.
My GitLab yml looks like this:

.docker:
  image: alpine:latest
  services:
    - docker:${DOCKER_VERSION}-dind
  variables:
    DOCKER_HOST: tcp://tcp:2376
    DOCKER_TLS_VERIFY: '1'
    DOCKER_TLS_CERTDIR: '/certs'
    DOCKER_CERT_PATH: '/certs/client'
    DOCKER_DRIVER: overlay2
    DOCKER_VERSION: '20.10.16'
    DAGGER_CLOUD_TOKEN: $DAGGER_CLOUD_TOKEN
.dagger:
  extends: [.docker]
  before_script:
    - apk add docker-cli curl
    - curl -L https://dl.dagger.io/dagger/install.sh | BIN_DIR=/usr/local/bin sh
build:
  tags:
    - dind
  extends: [.dagger]
  script:
    - dagger install github.com/shykes/daggerverse/hello@3338120927f8e291c4780de691ef63a7c9d825c0
    - dagger -m github.com/shykes/daggerverse/hello@3338120927f8e291c4780de691ef63a7c9d825c0 call hello

As I can see the first step installing the dagger module is ok.
After executing Dagger with the module I get the error.

If I add a variable _EXPERIMENTAL_DAGGER_RUNNER_HOST: tcp://dagger-engine:1234
I get another error: connect ERROR: new client: context deadline exceeded

build:
  tags:
    - dind
  extends: [.dagger]
  variables:
    _EXPERIMENTAL_DAGGER_RUNNER_HOST: tcp://dagger-engine:1234
  script:
    - dagger install github.com/shykes/daggerverse/hello@3338120927f8e291c4780de691ef63a7c9d825c0
    - dagger -m github.com/shykes/daggerverse/hello@3338120927f8e291c4780de691ef63a7c9d825c0 call hello

Bildschirmfoto_2024-03-14_um_13.41.58.png

#

Error: Dagger Functions in GitLab CI —> Wrong socket

digital sable Mar 14, 2024, 4:01 PM

#

Hey @flat surge!! I cant currently test it myself and I'm not Gitlab CI expert. Comparing your setup to https://docs.dagger.io/user-guide/ci/734201-gitlab I see that the DOCKER_HOST is set to tcp://tcp:2376, while on the guide it says tcp://docker:2376, not sure exactly how things are started up but maybe the other host does not like it? It seems to be related to not having the docker daemon properly configured. Can you try with a setup that is like the one on the guide? Happy to debug with you if it does not work!

GitLab CI | Dagger

The following code sample demonstrates how to integrate Dagger with GitLab CI.

flat surge Mar 14, 2024, 4:06 PM

#

digital sable Hey <@960764294851608606>!! I cant currently test it myself and I'm not Gitlab C...

Updatet to:

.docker:
  image: alpine:latest
  services:
    - docker:${DOCKER_VERSION}-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_VERIFY: '1'
    DOCKER_TLS_CERTDIR: '/certs'
    DOCKER_CERT_PATH: '/certs/client'
    DOCKER_DRIVER: overlay2
    DOCKER_VERSION: '20.10.16'
    # assumes the Dagger Cloud token is
    # in a masked/protected variable named DAGGER_CLOUD_TOKEN
    # set via the GitLab UI
    DAGGER_CLOUD_TOKEN: $DAGGER_CLOUD_TOKEN
    
.dagger:
  extends: [.docker]
  before_script:
    - apk add docker-cli curl
    - curl -L https://dl.dagger.io/dagger/install.sh | BIN_DIR=/usr/local/bin sh
build:
  extends: [.dagger]
  script:
     - dagger install github.com/shykes/daggerverse/hello@3338120927f8e291c4780de691ef63a7c9d825c0
     - dagger -m github.com/shykes/daggerverse/hello@3338120927f8e291c4780de691ef63a7c9d825c0 call hello

#

Bildschirmfoto_2024-03-14_um_17.06.39.png

#

@digital sable I created a repo here: https://gitlab.com/doe0003p/test-dagger

#

I don't know how I can fix this.

#

And if I set this:
_EXPERIMENTAL_DAGGER_RUNNER_HOST: tcp://docker:2376 this let to connect ERROR: new client: context deadline exceeded

round marten Mar 14, 2024, 4:57 PM

#

flat surge And if I set this: ` _EXPERIMENTAL_DAGGER_RUNNER_HOST: tcp://docker:2376` this ...

Hey Patrick! By any chance so you have a DAGGER_CLOUD_TOKEN set in your gitlab CI config?

#

Oh, I see you do. Can you check really quick if commenting that helps to temporarily resolve the issue?

flat surge Mar 14, 2024, 4:59 PM

#

round marten Hey Patrick! By any chance so you have a DAGGER_CLOUD_TOKEN set in your gitlab C...

Give me a second

flat surge Mar 14, 2024, 5:01 PM

#

round marten Hey Patrick! By any chance so you have a DAGGER_CLOUD_TOKEN set in your gitlab C...

https://gitlab.com/doe0003p/test-dagger/-/jobs/6396499156

#

Gives me the same error:

1: connect ERROR: new client: failed to run container: docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.
: exit status 125

round marten Mar 14, 2024, 5:09 PM

#

flat surge Gives me the same error: ``` 1: connect ERROR: new client: failed to run contai...

Oh, wait. You need to set the _EXPERIMENTAL_DAGGER_RUNNER_HOST

flat surge Mar 14, 2024, 5:23 PM

#

round marten Oh, wait. You need to set the _EXPERIMENTAL_DAGGER_RUNNER_HOST

Only for clarification: I will set the _EXPERIMENTAL variable same as the docker host, right?

#

I will try it when my kids are sleeping

round marten Mar 14, 2024, 5:25 PM

#

flat surge Only for clarification: I will set the _EXPERIMENTAL variable same as the docker...

Yes, that should be the case

#

Wait, sorry I got confused

#

I have a gitlab repo where this works. Gimme a sec

round marten Mar 14, 2024, 5:47 PM

#

Can we try something @flat surge. could you set DOCKER_HOST right before calling dagger call?

#

i.e: DOCKER_HOST=tcp://docker:2376 dagger call....

round marten Mar 14, 2024, 6:04 PM

#

This seemed to work https://gitlab.com/marcosnils/dagger-ci-test/-/jobs/6396988776

GitLab

build (#6396988776) · Jobs · Marcos Lilljedahl / Dagger Ci Test · G...

GitLab.com

#

https://gitlab.com/marcosnils/dagger-ci-test/-/blob/main/.gitlab-ci.yml?ref_type=heads

GitLab

.gitlab-ci.yml · main · Marcos Lilljedahl / Dagger Ci Test · GitLab

GitLab.com

#

I'm using the python image in this case since previously I was running some python stuff. Not sure if that changes anything..

flat surge Mar 14, 2024, 7:00 PM

#

round marten Can we try something <@960764294851608606>. could you set DOCKER_HOST right befo...

This is also not working.
https://gitlab.com/doe0003p/test-dagger/-/jobs/6396499156

flat surge Mar 14, 2024, 7:02 PM

#

round marten This seemed to work https://gitlab.com/marcosnils/dagger-ci-test/-/jobs/63969887...

I try it like your example now...

#

Ok that is crazy.
https://gitlab.com/doe0003p/test-dagger/-/blob/main/.gitlab-ci.yml

#

Failed!.

#

The same gitlab.yml file is not working.

round marten Mar 14, 2024, 7:05 PM

#

flat surge The same gitlab.yml file is not working.

Lol. Doesn't make any sense

flat surge Mar 14, 2024, 7:06 PM

#

https://tenor.com/view/haha-funny-lol-laughing-out-loud-giggle-gif-15515505

Tenor

#

Nope absolutely not.

#

This is now my file:

.docker:
  image: python:3.11-alpine
  services:
    - docker:${DOCKER_VERSION}-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_VERIFY: '1'
    DOCKER_TLS_CERTDIR: '/certs'
    DOCKER_CERT_PATH: '/certs/client'
    DOCKER_DRIVER: overlay2
    DOCKER_VERSION: '20.10.16'
    # assumes the Dagger Cloud token is
    # in a masked/protected variable named DAGGER_CLOUD_TOKEN
    # set via the GitLab UI
    #DAGGER_CLOUD_TOKEN: $DAGGER_CLOUD_TOKEN
    #_EXPERIMENTAL_DAGGER_RUNNER_HOST: tcp://docker:2376
.dagger:
  extends: [.docker]
  before_script:
    - apk add docker-cli curl
    #- curl -L https://dl.dagger.io/dagger/install.sh | BIN_DIR=/usr/local/bin sh
    - cd /usr/local && { curl -L https://dl.dagger.io/dagger/install.sh | sh; cd -; }
build:
  extends: [.dagger]
  script:
     #- dagger install github.com/shykes/daggerverse/hello@3338120927f8e291c4780de691ef63a7c9d825c0
     - dagger -m github.com/shykes/daggerverse/hello@3338120927f8e291c4780de691ef63a7c9d825c0 call hello

#

And the result: https://gitlab.com/doe0003p/test-dagger/-/jobs/6397380006

1: [7.52s] registry.dagger.io/engine:v0.10.2
1: pulling registry.dagger.io/engine:v0.10.2 [7.24s]
1: starting engine 
1: starting engine [0.02s]
1: connect ERROR: new client: failed to run container: docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.
: exit status 125
Error: new client: failed to run container: docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.

#

@round marten Now it is running!

#

And that is really weird

round marten Mar 14, 2024, 7:10 PM

#

:Wat:

#

Lol

flat surge Mar 14, 2024, 7:11 PM

#

I deleted the environment Variable in the CI settings...

#

Here

#

Bildschirmfoto_2024-03-14_um_20.15.16.png

#

If I set the variable here then it is failing again

round marten Mar 14, 2024, 7:18 PM

#

What variable are you setting there?

flat surge Mar 14, 2024, 7:19 PM

#

It was the DAGGER_CLOUD_TOKEN

round marten Mar 14, 2024, 7:28 PM

#

Ok, that's very strange. The same thing happened to another user. I'll investigate tomorrow 🙌

flat surge Mar 14, 2024, 7:29 PM

#

Thank you for your support.

flat surge Mar 18, 2024, 9:38 AM

#

Hey @round marten,
any news on that? My pipeline is running now without the Dagger Cloud... 😭

jagged pumice Mar 30, 2024, 6:53 AM

#

I have this exact same issue. If I set the DAGGER_CLOUD_TOKEN you get the Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? error.

If you set the _EXPERIMENTAL_DAGGER_RUNNER_HOST environment variable, the GitLab job just hangs until it times out.

#

It will be interesting to see if this error goes away by setting up a self-hosted runner (which isn't ideal, as it adds a lot of extra setup for a user)

round marten Mar 30, 2024, 11:31 AM

#

Hey! Sorry I couldn't get back to this. I'm on vacation until Apr 7th. cc @thin stump might be able to take a look before I get back 🙏

sudden hemlock Mar 30, 2024, 2:09 PM

#

We were running into the same thing originally.
I'm not sure where we got this value from, but we have

variables:
    _EXPERIMENTAL_DAGGER_RUNNER_HOST: 'unix:///var/run/buildkit/buildkitd.sock'

and our traces are showing up in cloud

jagged pumice Mar 30, 2024, 3:50 PM

#

round marten Hey! Sorry I couldn't get back to this. I'm on vacation until Apr 7th. cc <@1356...

Have a nice vacation 😄

jagged pumice Mar 30, 2024, 4:07 PM

#

sudden hemlock We were running into the same thing originally. I'm not sure where we got this v...

Do you have an example job definition that you are running in GitLab? Still hangs for me! 😦

#

Are you using a custom runner?

sudden hemlock Mar 30, 2024, 6:08 PM

#

Ya, we are using a custom runner,, which is probably why that worked for me

thin stump Apr 1, 2024, 2:52 PM

#

jagged pumice It will be interesting to see if this error goes away by setting up a self-hoste...

You're using gitlab hosted runners, correct?

urban oxide Apr 2, 2024, 5:33 AM

#

jagged pumice I have this exact same issue. If I set the `DAGGER_CLOUD_TOKEN` you get the `Can...

I ran into the same issue just trying to connect to a DOCKER_HOST, its very strange to me that DAGGER_CLOUD_TOKEN has anything to do with this

I filed an issue here: https://github.com/dagger/dagger/issues/6989

GitHub

🐞 DOCKER_HOST is not respected when DAGGER_CLOUD_TOKEN is set · Is...

What is the issue? I am running docker on a remote VM and connecting to it using the DOCKER_HOST env var. I can run docker commands without any issues on the remote host and when I try to call a da...

jagged pumice Apr 2, 2024, 11:40 AM

#

thin stump You're using gitlab hosted runners, correct?

Yes. I am just using the default runner on gitlab.com

thin stump Apr 2, 2024, 10:21 PM

#

jagged pumice Yes. I am just using the default runner on `gitlab.com`

Thanks! I haven't tested with the default gitlab runners before, but it sounds like Lev's issue probably has the same root cause

flat surge Apr 3, 2024, 8:06 AM

#

sudden hemlock We were running into the same thing originally. I'm not sure where we got this v...

That is also not working for me.

thin stump Apr 4, 2024, 7:42 PM

#

@jagged pumice @flat surge are you able to try again with v0.11.0 just released?

flat surge Apr 4, 2024, 7:43 PM

#

thin stump <@1220745895021252674> <@960764294851608606> are you able to try again with v0.1...

Of course I will try it tomorrow .

flat surge Apr 4, 2024, 9:22 PM

#

@thin stump what was the problem?

thin stump Apr 4, 2024, 9:26 PM

#

flat surge <@135620352201064448> what was the problem?

A regression that Erik was able to track down very quickly! 🚀 https://github.com/dagger/dagger/pull/7006

GitHub

engine client: fix env propagation to docker subcommand by sipsma ·...

Fixes #6989, confirmed with a (particularly gnarly) integ test the integ test didn't work only in the test workflow on our EKS engine so I had to remove it but still confident this fixes it, se...

jagged pumice Apr 5, 2024, 5:35 AM

#

@thin stump I can confirm that this fix has prevented the GitLab Shared Runner from reporting the connection error, and the pipeline has succeeded. However, I do not see anything in Dagger Cloud regarding the run on gitlab.com.

If I deliberately type in an incorrect DAGGER_CLOUD_TOKEN, I get the following errors, which does show me that my pipeline is talking to it:

Connected to engine 6fc07f48d450 (version v0.11.0)
05:27:06 ERR OpenTelemetry error error="traces export: failed to send to https://api.dagger.cloud/v1/traces: 401 Unauthorized"
05:27:06 ERR OpenTelemetry error error="traces export: failed to send to https://api.dagger.cloud/v1/traces: 401 Unauthorized"
05:27:06 ERR OpenTelemetry error error="failed to send to https://api.dagger.cloud/v1/logs: 401 Unauthorized"
05:27:06 ERR OpenTelemetry error error="failed to send to https://api.dagger.cloud/v1/logs: 401 Unauthorized"

flat surge Apr 5, 2024, 5:59 AM

#

@thin stump I can also confirm that the fix is working. But the Dagger Cloud has some problems.
As @jagged pumice already mentioned I do not see anything in Dagger Cloud regarding the last run. Here I only see the run before 2 days.
Last if under All Changes when I click on the first entry (also 2 Days before) I get another error.

Finally what I want to mention is, that it feels pretty slow.

Connected to engine 94fbf7f92735 (version v0.11.0)
✔ connect 21.0s
✔ initialize 1m37.9s
✔ Hello.hello: String! 0.5s
hello, world!

thin stump Apr 5, 2024, 3:17 PM

#

Hrm ok that's not great. @glossy sparrow do you know of that's still related to your fix or something different?

gloomy temple Apr 5, 2024, 10:06 PM

#

I stopped seeing pipeline executions at /runs as well, but they're available at /traces

sudden hemlock Apr 6, 2024, 5:27 PM

#

Where it shows up depends on the engine version.

>= 0.11.0 - /traces
<  0.11.0 - /runs

jagged pumice Apr 7, 2024, 5:43 AM

#

@thin stump I can confirm that I see my traces within Dagger Cloud (from GitLab). I am using the new v2 (beta) version of the UI daggerfire

#Error: Dagger Functions in GitLab CI —> Wrong socket