Unable to run dagger in gitlab CI | Dagger | Page 1

celest birch Feb 3, 2023, 9:23 AM

#

HI, im trying to run a dagger command in Gitlab CI, but I get the following error: "error": "Post \"http://dagger/query\": EOF\nPlease visit https://dagger.io/help#go for troubleshooting guidance."

Any ideas how I can resolve this?

steep mica Feb 3, 2023, 3:18 PM

#

👋 dupe of: https://discord.com/channels/707636530424053791/1071076724458131610

I'm checking what's happening there

#

@celest birch which Dagger version are you using?

celest birch Feb 3, 2023, 3:21 PM

#

v0.4.2

#

Thats from the go.mod, is there anything else I should check?

steep mica Feb 3, 2023, 3:38 PM

#

mind bumping to latest / v0.4.4 and trying again?

celest birch Feb 3, 2023, 3:38 PM

#

Sure let me give that a try

#

OK yeah that fixed the problem for me, thanks

steep mica Feb 3, 2023, 3:45 PM

#

sure! np!

#

we're still trying to investigate here what happens since we can't repro

celest birch Feb 6, 2023, 2:56 PM

#

Hi, i'm running into this problem again, on 0.4.4

steep mica Feb 6, 2023, 2:59 PM

#

👋 does it happen all the time? or randomly?

#

did you change anything in your pipeline that triggers the issue?

steep mica Feb 6, 2023, 3:00 PM

#

celest birch Hi, i'm running into this problem again, on `0.4.4`

replying here so you get a gentle ping

celest birch Feb 6, 2023, 3:01 PM

#

All of the time

#

Its now failing on a different command I think; its failing on the export function

steep mica Feb 6, 2023, 3:11 PM

#

awesome. Do you mind running a very simple export example to see if that triggers it please? I'm doing the same here

#

simple export example seems to work here.

package main

import (
    "context"
    "fmt"
    "log"
    "os"
    "path/filepath"

    "dagger.io/dagger"
)

func main() {
    ctx := context.Background()

    client, err := dagger.Connect(ctx)
    if err != nil {
        log.Println(err)
        return
    }
    defer client.Close()

    _, err = client.Container().From("alpine:latest").
        WithWorkdir("/tmp").
        WithExec([]string{"wget", "https://dagger.io"}).
        Directory(".").
        Export(ctx, ".")
    if err != nil {
        log.Println(err)
        return
    }
    contents, err := os.ReadFile(filepath.Join(".", "index.html"))
    if err != nil {
        log.Println(err)
        return
    }
    fmt.Println(string(contents))
}

#

I'm wondering @celest birch if you might be trying to export something that's particularly very big

steep mica Feb 6, 2023, 3:20 PM

#

steep mica I'm wondering <@171010552020074497> if you might be trying to export something t...

doesn't seem to be the case. Just exported a very big file without issues

#

@celest birch does it also happen in your local machine? or just CI?

celest birch Feb 6, 2023, 3:21 PM

#

Only CI it seems

steep mica Feb 6, 2023, 3:21 PM

#

go 1.20 I'd assume as well?

celest birch Feb 6, 2023, 3:21 PM

#

Using golang:latest yes

#

But i've just bumped it down to 1.19 to test whether that resolves it

#

        From("pulumi/pulumi").
        WithWorkdir("/app").
        WithEnvVariable("PULUMI_CONFIG_PASSPHRASE", "local").
        WithExec([]string{"login", "--local"}).
        WithExec([]string{"new", "kubernetes-typescript", "--name", name, "-s", "dev", "--yes"}).
        Directory("/app").
        WithoutFile("Pulumi.dev.yaml").
        Export(context.Background(), "dir/to/output")```

#

This is what we were using

#

Okay, error does not seem to occur in 1.19

steep mica Feb 6, 2023, 3:24 PM

#

yeah.. same error @hollow birch was seeing. I'll try to repro locally from that snippet to troubleshoot 🙏

celest birch Feb 6, 2023, 3:24 PM

#

Okay cool, thanks

#

If it helps, we're running this on Gitlab Runner via Kubernetes

steep mica Feb 6, 2023, 3:25 PM

#

are you caching to your go dependencies in your gitlab pipeline by any chance?

celest birch Feb 6, 2023, 3:25 PM

#

We're storing no artifacts in our pipeline

steep mica Feb 6, 2023, 3:26 PM

#

so go deps get pulled every time? Just to make sure

#

basically, no cache https://docs.gitlab.com/ee/ci/caching/#cache directive?

Caching in GitLab CI/CD | GitLab

celest birch Feb 6, 2023, 3:26 PM

#

  image: golang:1.19
  stage: test
  services:
    - docker:dind
  variables:
    DOCKER_HOST: tcp://localhost:2375
    DOCKER_TLS_CERTDIR: ""
  before_script:
    - apt-get update && apt-get install -y docker.io
    - curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash
    - source ~/.bashrc
    - nvm install v18.14.0
    - npm install -g pnpm
    - curl -sSL "https://github.com/bufbuild/buf/releases/download/v1.13.1/buf-$(uname -s)-$(uname -m)" -o /usr/local/bin/buf
    - chmod +x /usr/local/bin/buf
    - go mod download
    - go build -ldflags="-X 'gitlab.com/jobilla/devops/ekto/pkg/commands/version.Version=${CI_COMMIT_TAG}' -X 'gitlab.com/jobilla/devops/ekto/pkg/commands/version.GitCommitSHA=${CI_COMMIT_SHORT_SHA}'" -o /usr/local/bin/ekto
    - git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/jobilla/backend-services/omnia.git /omnia
  script:
    - cd /omnia
    - /usr/local/bin/ekto --debug create omnia-test-service```

#

Thats it all as it stands now, ignore the 1.19 since thats just changed, but everything else is exactly as our job setup

#

The ekto --debug create ... is the command that has the Dagger stuff triggered from within it

#

Alot of the other dependencies are unrelated also, like pnpm etc

steep mica Feb 6, 2023, 3:33 PM

#

cool, thx

hollow birch Feb 6, 2023, 3:36 PM

#

hmm, interesting.

#

I just double check our pipeline and didn't able to reproduce same error with latest version

steep mica Feb 6, 2023, 3:38 PM

#

maybe it's some sort of race condition. 🏎️

hollow birch Feb 6, 2023, 3:38 PM

#

our error was consistent as well but after I clean cache saw some flake in behavior

steep mica Feb 6, 2023, 3:39 PM

#

hollow birch our error was consistent as well but after I clean cache saw some flake in behav...

dagger's or golang's?

hollow birch Feb 6, 2023, 3:40 PM

#

but for this scenario we had sigs.k8s.io/controller-runtime/tools/setup-envtest as dependency. maybe it's related with some depenedency

hollow birch Feb 6, 2023, 3:40 PM

#

steep mica dagger's or golang's?

golang, it failed with same error in later steps

steep mica Feb 6, 2023, 4:00 PM

#

@celest birch @hollow birch just to double check, can you validate that client.Close() is not being called before your pipeline finishes? That could cause an EOF error

hollow birch Feb 6, 2023, 4:03 PM

#

no we're not. But to be sure I'll double check it

celest birch Feb 6, 2023, 4:34 PM

#

Erm, I don't think we're calling client.close at all

#

    opts := make([]dagger.ClientOpt, 0)

    if log.DebugMode {
        opts = append(opts, dagger.WithLogOutput(os.Stdout))
    }

    client, err := dagger.Connect(context.Background(), opts...)

    if err != nil {
        log.SLogger.Errorw("failed to connect to Dagger", "error", err)
        return err
    }

    if step["pipeline"] == "pulumi-project" {
        p, _ := pterm.DefaultSpinner.Start("Creating deployment configuration...")
        defer p.Info()
        log.SLogger.Debug("generating pulumi project")
        dir := dagger2.CreatePulumiProject(client, ektoSvc.Name)
        _, err := dir.Export(context.Background(), filepath.Join(ektoSvc.GetFullPath(), step["dest"].(string)))

        if err != nil {
            log.SLogger.Errorw("failed to create directory", "error", err)
            return err
        }

        terminal.CheckPrinter.Println("Successfully created Pulumi project")
    }

    return nil
}```

#

For context here, CreatePulumiProject is the method that has the been pasted above, and the dir.Export is the export step. Thats all we do with Dagger, no deferred close or anything

steep mica Feb 6, 2023, 5:12 PM

#

@celest birch @hollow birch if any of you can repro this locally I'd like like if you could help me test something 🙏

celest birch Feb 6, 2023, 5:14 PM

#

I don't have a repro locally, but I might be able to repro it again in CI

#

if i change back to 1.20

#

What do you wantto test?

steep mica Feb 6, 2023, 5:17 PM

#

it's hard to test in CI since it requires spawning the dagger session binary manually and then setting up dagger to use that 😕

hollow birch Feb 6, 2023, 5:26 PM

#

Sure, can do in a few hours, just send me what to do

steep mica Feb 6, 2023, 5:34 PM

#

you should have a binary in your ~/.cache/dagger or $XDG_CACHE_HOME/dagger called dagger-$VERSION
run the binary described above as: dagger-$VERSION session. This will start a process and print a JSON output with port and session_token fields.
run your pipeline with the values obtained above as DAGGER_SESSION_PORT=$port DAGGER_SESSION_TOKEN=$token go run main.go .
What I'm curious to see is if the dagger-$version binary exists before the pipeline finishes. This would cause an EOF error.

#

in "normal" situations, the dagger-$version binary should not exit

steep mica Feb 6, 2023, 6:17 PM

#

👋 opened an issue for this: https://github.com/dagger/dagger/issues/4538

GitHub

🐞 Sometimes pipelines fail with EOF randomly · Issue #4538 · dagger...

What is the issue? Some users have reported that seems like after bumping go to 1.20, some pipelines have randomly started to fail with an EOF error: https://discord.com/channels/707636530424053791...

#

cc @queen mango in case you have any other inputs

hollow birch Feb 7, 2023, 12:11 PM

#

I didn't had time yesterday but today will try to execute those steps

steep mica Feb 7, 2023, 2:31 PM

#

🙏 thx

#Unable to run dagger in gitlab CI