full laurel
Hello Team 🙂
I am using dagger engine 0.11.8, but my server is failing with below error
ERR failed to flush spans error="traces export: Post \"
I am looking for way to disable these traces as i don't have otel collector ready yet
can soneone help on this?
livid mauve
Hello Team 🙂
I am using dagger engine 0.11.8, but my server is failing with be...
👋 traces are sent if you're logged in Dagger cloud. If you run dagger logout, Dagger won't send any traces.
full laurel
i am not using dagger cloud, still i see it tries to send traces
Full error
@full laurel do you have any OTEL envs set?
full laurel
Not really
livid mauve
this is strange.. it shouldn't be sending any traces if you don't have an *OTEL* variable and you're not logged in to Dagger Cloud. cc @static carbon ?
static carbon
Looks like there's one set to me. Maybe for the engine container?
livid mauve
Looks like there's one set to me. Maybe for the engine container?
will the variable in the engine container have any effect? I thought only the client ones were relevant for telemetry
static carbon
will the variable in the engine container have any effect? I thought only the cl...
it shouldn't, but they mentioned the error is coming from a server, so might as well check it. The engine side is OTEL_* aware but it should only be for logs. But there's a lot of telemetry config code shared between the engine and the CLI
shut crane
I saw a mention of a DO_NOT_TRACK env var in the docs? Would that disable it?
static carbon
@shut crane that's more around analytics (like daggerverse function usage tracking) - it shouldn't disable trace exporting, since that's already opt-in
full laurel
For me, i see same error on client when using simple dagger query to remote dagger engine server as well as in engine logs
full laurel
morning guys 🙂
let me know if there 's easy fix for that
static carbon
Not really
it must be getting OTEL_ configuration from somewhere - you said "not really" before 😂 what does that mean? are there any set? what's in env?
does this error come from something running in Dagger? or the outer Dagger CLI?
static carbon
if you don't have OTEL_ vars set on the client side or the engine side I don't really know what it could be. 😕 It must be getting that IP_ADDRESS:4317 value from somewhere.
shut crane
Here is what I am seeing:
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 2s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 4s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 8s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 16s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 32s.
No OTEL env vars are set, my engine is running as a ds in k8s. Client is in a gitlab runner. No connection ot the internet. This is 0.11.6.
static carbon
No OTEL env vars are set, my engine is running as a ds in k8s. Client is in a gi...
hmm, where exactly are these logs appearing? Trying to figure out what component it's coming from, since with OTel it could be many different places. This looks like it might be coming from Python? (There's also a chance this will be fixed by upgrading, there's been a lot of work on OTel since 0.11.6, with more coming in 0.12.0)
shut crane
I see it in my gitlab runner output
I’ll upgrade and test. Unfortunately it isn’t consistent as to when they show up
full laurel
@static carbon I found that we have OTEL_EXPORTER_OTLP_ENDPOINT is set from where we call dagger cli
Actually, this machine is our k8s pod where we have company wide standard support for OTEL shipping, though endpoint is set but it is still not able to to send
questions:
