Custom OpenTelemetry endpoint overridden

[magic roost]

I'm currently trying to execute a command within a container but provide custom OTEL_... environment variables to it from the outside. Sadly, it looks like Dagger is somewhere overriding at least the following three:

• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
• OTEL_EXPORTER_OTLP_TRACES_PROTOCOL
• OTEL_TRACE_PARENT

Is there some way to disable this behaviour? I've already tried it with .WithoutEnvVariable but sadly to no avail 🙁

The main problem is, that Dagger (or buildkit) sets the endpoint to http:///dev/otel-grpc.sock and the protocol to grpc and I'd like to use http/protobuf instead.

Thank you 😄

[limpid cloud]

IIUC from reading the code it's not currently possible to override the current OTEL config since it's set by buildkit and the engine doesn't have an option to override it yet. cc @spiral coral @proud crystal

[magic roost]

Thank you for checking 🙂 For now I'm using a different set of environment variables to inject the settings and map them inside the application to the standard variables. Luckily, I have the application under control 🙂

[spiral coral]

Hey @magic roost — you mean you’re setting some env variables in your container and they don’t up there?

What do you mean “from the outside”?

[magic roost]

Yes, buildkit overrides them but only those 🙂

[spiral coral]

Do you mean using WithEnvVariable?

[spiral coral]

If I understand this correctly:

WithEnvVariable(“OTEL_ …”, “foo”).WithExec(“env”).Stdout() —> rather than getting back “foo”, buildkit is overriding that with a grpc sock?

[limpid cloud]

yes, this seems to be the case Andrea

looks like buildkit is overriding this.

[spiral coral]

Briefly looked into it — For once, I think this is a feature and not a bug 😀

Super interesting feature at that: apparently when otel tracing is enabled, buildkit automatically passes it along containers executed (using that grpc proxy):

https://github.com/open-telemetry/opentelemetry-specification/issues/740#issuecomment-924449891

I’m thinking this means that if you set the OTEL_ variables when running dagger, then all your executed containers will inherit from it

Q:

1. does it actually work? Apparently it works for docker build, but I don’t know if the docker CLI had to do some wiring to make it work

2. Is there a way to disable it if you want to pass your own OTEL_ variables

/cc @lavish hemlock

[merry gulch]

Is this still the case? I've been trying to export traces to a custom collector and it doesn't seem to work 😦

[limpid cloud]

if you set the OTEL_ variables when calling the dagger cli, that should be enough. Have you tried that?

[merry gulch]

I did try it out but it wasn't working

I'll give it another go and let you know later on

[limpid cloud]

let us know how it goes!