#otel traces, metric, log from pipeline to external backend?

1 messages · Page 1 of 1 (latest)

young tangle
#

Is it possible to send otel data for dagger pipeline execution to external backend like honeycomb etc?right now i believe it's being used by dagger cli TUI

exotic plank
#

you need to set those in the engine

young tangle
#

Hey @exotic plank 👋
i have enabled otel with these vars (we are using remote dagger engine not using cloud version)
and these env set at host machine where dagger client is installed
OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL
i can see span and traces are getting generated in honeycomb
but I cannot see logs and metrics in loki and mimir
Am i missing something here, could you please help?

young tangle
winged girder
#

👋
Metrics and logs OTEL APIs are not implemented yet, only traces
I remember supporting metrics being mentioned somewhere, not sure about logging
cc @ocean wedge

ocean wedge
#

logs are supported, metrics are not, but logs have to be explicitly enabled with a separate env var (OTEL_EXPORTER_OTLP_LOGS_ENDPOINT)

young tangle
#

Logs are now shipping to otel collector, but the console logs that I usually see from app to server stdout console, is not coming to otel. Seems engine logs coming without any app debug/info that i have used. Any idea how can we send those console logs to otel collector via dagger. i assume now it runs dagger session internally from sdk after it got upgrade, which i am not sure if taking care of application logs(non-dagger)
let me know

young tangle
#

@ocean wedge any idea on this issue 🙂 ?

ocean wedge
#

@young tangle not sure I'm following 100%, but make sure you're configuring OTEL_* vars on the client side (where you're running the dagger CLI), not on the engine side

young tangle
#

i can see logs, but those logs are either dagger engine errors or span traces logs but I cannot see any other logs generated by application(they are non dagger code), i thought running whole app like dagger run go run main.go can send console logs as telemetry too
I don't have much context on what logs will it shipped, just trying my luck

young tangle
#

FYI..I am able to fix that issue 🙂

ocean wedge
#

nice - what was it?

young tangle
#

my application actually generating logs and putting in files but not stdout in console. And i observed that only stdout logs can be captured as part of dagger otel logs which is taken care by dagger cli automatically
Earlier i was only able to see dagger specific logs and now i can see all application logs

Important Note - this is working well in my local(TTY is present) but not in gitlab console 😦

young tangle
#

but now i am blocked on this issue https://discord.com/channels/707636530424053791/1263141263524237363

basically this is working fine on my local mac machine. but it does not work in gitlab console bcz apparantly TTY is not suppored there(my best guess)
so few approaches i tried which did not work for me either

  1. I tried to use --progress=plain, but seeing huge logs on gitlab conole is not good for our customer ui experience
  2. I tried to use --silent feature, but also stopped application logs being shipped to grafana loki

so i am again kinda stuck 😦

@ocean wedge i would appreciate if you can guide me here 🙏

ocean wedge
#

--progress=plain is the solution there, but i think it's currently more verbose than intended. I'd just set that for now and wait for us to fix the bug

young tangle
#

so tty mode is only recommened for local development as of now?

#

also by any chance if you know is there a good way to add tty capability in gitlab console using any external utility tool?

young tangle