Dagger tracing?

[tepid inlet]

tl;dr - I dont seem to get traces from buildkit steps. I'd expect to see all the steps that are output to console, to be shown in jaeger traces.

I'm going pretty good w/ the golang sdk now, and as part of the case I'm building to use dagger, I'd like to do some benchmarks on step timings etc. Essentially I want to prove out how much better we can cache w/ dagger.

I've set up jaeger / dagger buildkit daemon as shown here

docker run -d -p6831:6831/udp -p16686:16686 -p14268:14268 jaegertracing/all-in-one:latest
docker run -d --name dagger-buildkitd -e JAEGER_TRACE=<MY_IP>:6831 --privileged moby/buildkit:v0.10.5

After running my pipeline, I only see traces for Dockerfile steps that result from a dagger.Container().Build(Dockerfile)

A good way to reproduce this is to run the above setup, just grab one of the examples (i.e. gradle) from the (awesome) examples repo and check jaeger output. I see no traces.

Jaeger screenshot - https://imgur.com/iwUU8Ei

Jaeger screenshot of a pipeline with container.Build(Dockerfile) - https://imgur.com/ANbbZCI

lmk if you want me to move this to GH issues or anything.

[stiff oyster]

@hybrid dome @wild flint Thoughts on this?

[hybrid dome]

@tepid inlet Let me try locally -- we haven't used tracing in a while, might be broken

(that's why it's not documented)

[hybrid dome]

@tepid inlet Ok so a few things:

• both dagger and buildkit support opentracing
• buildkit tracing is very low level, whereas dagger tracing is "closer" to the code you're running
• the opentracing API changed. To enable opentracing on buildkit, it's OTEL_EXPORTER_JAEGER_ENDPOINT=http://jaeger:14268/api/traces
• dagger opentracing support is unfortunately off -- I can show you a one-liner to change if you're interested in enabling it?

^^^ this is after enabling it

[tepid inlet]

ooooof. I think someone even told me already that env var was different, I just mistyped from bash history.

I'll give it a go w/ the env var, and yeah totally interested in the dagger tracing - I think moving up a layer to dagger could be helpful.

Essentially trying to show:

jenkins build step "foo" takes x seconds
dagger build step "foo" takes y seconds

[hybrid dome]

Interesting bit will be running dagger twice -- only things that changed will get recomputed, the rest is fetched from cache. Whereas jenkins will do the work all over again, every time

[tepid inlet]

yeah exactly 🙂

[hybrid dome]

If you're interested to turn on opentracing (warning: experimental) in dagger

• clone dagger/dagger
• change one line
• in your go.mod, change the dagger dependency to use your local dagger version
• run your code using the OTEL env variable

I can show how to do that if you're interested?

[tepid inlet]

yeah if you've got a link to the LoC - I can handle the rest

[hybrid dome]

diff --git a/engine/engine.go b/engine/engine.go
index e1a2bb74..f5910398 100644
--- a/engine/engine.go
+++ b/engine/engine.go
@@ -14,6 +14,7 @@ import (
        "github.com/dagger/dagger/project"
        "github.com/dagger/dagger/router"
        "github.com/dagger/dagger/secret"
+       "github.com/dagger/dagger/tracing"
        bkclient "github.com/moby/buildkit/client"
        bkgw "github.com/moby/buildkit/frontend/gateway/client"
        "github.com/moby/buildkit/session"
@@ -45,6 +46,7 @@ type Config struct {
 type StartCallback func(context.Context, *router.Router) error

 func Start(ctx context.Context, startOpts *Config, fn StartCallback) error {
+       tracing.Init()
        if startOpts == nil {
                startOpts = &Config{}
        }

[tepid inlet]

lol so simple, love it

[hybrid dome]

that's it -- tracing.Init() at the beginning of engine.Start()

[tepid inlet]

great I'll give it a shot

[hybrid dome]

need help with go.mod replacing?

You'll need these 2:

replace github.com/dagger/dagger => <path to dagger repo root>
replace dagger.io/dagger => <path to dagger repo root>/sdk/go

[tepid inlet]

Gonna be a bit before I get around to this though - dagger time is usually my late night fun haha

[hybrid dome]

Then finally:

export OTEL_EXPORTER_JAEGER_ENDPOINT="http://localhost:14268/api/traces"

from your shell when running code using the dagger SDK

Oh and I run jaeger with this compose-file:

version: "3.8"

services:
  jaeger:
    container_name: jaeger
    image: jaegertracing/all-in-one:1.33.0
    ports:
      - "6831:6831/udp"
      - "14268:14268"
      - "16686:16686"

I think that maps 1:1 with your docker run, as far as I can tell