[SOLVED] Not Seeing Consistent Log Output in Gitlab Runners From Helm Chart

[toxic comet]

Hey All,

We're currently seeing some issues with running dagger call ... not always returning the output to standard out. In fact, we're not seeing it more than we are.

We've deployed the Helm Chart from oci://registry.dagger.io

We're using v 0.18.8 on a three node K8s clusters.

We've tried using --progress plain but that doesn't seem to have an impact on the output.

This is the type of output were are seeing most of the time:

/builds/my-proj # dagger -d --progress tty call --netrc-secret="file://$HOME/.netrc" tidy

15:32:58 DBG subscribing to telemetry client=z23f9e5e2fh1ixtlir3oz1v81 remote=unix:///var/run/dagger/engine.sock
15:32:59 DBG executing query query="query{myProj(netrcSecret:\"<REDACTED>\"){tidy}}"


Setup tracing at https://dagger.cloud/traces/setup. To hide set DAGGER_NO_NAG=1

Anyway, could use some help with this one. This is a bit of show stopper for us from adopting dagger within our CI/CD pipelines at the moment.

@mental monolith / @gentle zealot maybe one of you can point us in the right direction. This is related to this question: #gitlab message

Thanks for any tips!

cc @topaz fox @stark prawn

[mental monolith]

@toxic comet is the output you're expecting to see part of the main function output? O it's from one specific step within the pipeline execution?

It's also interesting that this always works locally. If that's the case, I'm sure we'll figure out what's happening here

is there any chance you could provide a simplified example of what your module is doing with k3s?

starting from there I can setup a kind cluster locally and see if I can repro

[toxic comet]

is the output you're expecting to see part of the main function output? O it's from one specific step within the pipeline execution?
It's anything. Normally I would see lots of text flying by or some kind progress in the TUI. In this case, I don't see anything. Locally, I've never had a problem with the output--I always see the output regardless of the output type.
I don't think it has anything to do with my module, because I see the same issues with something like dagger -c 'container | from alpine | with-exec cat /etc/os-release | stdout'

Here is some example output for when I do see something: (first image)

And here is some output when I don't see any of the steps: (second image)

Although I do see stdout from the os-release file.

In one of my modules I do something like this:

// Run the golangci lint config
func (m *MyProj) Lint(ctx context.Context) (string, error) {
    m.BuildGoModCache()

    return m.WorkContainer.
        WithExec([]string{"go", "tool", "golangci-lint", "run"}).Stdout(ctx)
}

And I don't see any of the output on success or failure in the pipeline. Locally, I always seem something.

Not sure when I'll have time yet to setup a k3s cluster with the issue. I was hoping we might discuss what possible issues might be? I'm using the standard dagger image to run in my Kubernetes cluster:

 image: registry.dagger.io/engine:v0.18.8

.dagger-config: &dagger-config
  tags:
    - k8s-runners
  image: registry.dagger.io/engine:v0.18.8
  variables:
    OTEL_SERVICE_NAME: dagger
    OTEL_EXPORTER_OTLP_ENDPOINT: http://gitlab-runner-collector:4318
    OTEL_TRACES_EXPORTER: otlp
    OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
    OTEL_TRACES_SAMPLER: traceidratio
    OTEL_TRACES_SAMPLER_ARG: 0.15
    _EXPERIMENTAL_DAGGER_RUNNER_HOST: unix:///var/run/dagger/engine.sock

Other vars that might help.

[mental monolith]

@toxic comet any chance you can quickly jump into #911305510882513037 and show me what you're seeing

?

even though I get the issue, I'd like to see if I spot something evident while you're testing it

[toxic comet]

@mental monolith Sure!

[toxic comet]

From the call:

• Try disabling OTEL
• Upgrade to latest Dagger

Don't want to speak too soon, but since commenting out the OTEL stuff, I'm seeing better output. Going to run a few more jobs 🤞 .

[mental monolith]

cc @silent spindle

[toxic comet]

@mental monolith @silent spindle . the OTEL was definitely the problem. I've run many more jobs today and haven't seen an issue. Will consult the engineer who originally set up the OTEL config and see what he wanted to do with it.

Thanks so much for the help!

cc @topaz fox

[silent spindle]

Great, glad y'all got to the bottom of it!