seems like things are getting fairly | Dagger | Page 1

royal onyx Dec 12, 2024, 12:40 AM

#

@empty ore 👋 figured I'd thread

#

this is what shows in cloud - (new v3 ui very nice btw 🙂 )

#

each of these is a little more complicatedd under the the hood too, and it's hard for teammates to find the right spot in there

empty ore Dec 12, 2024, 12:45 AM

#

are you using modules or dagger run?

royal onyx Dec 12, 2024, 12:53 AM

#

no

empty ore Dec 12, 2024, 12:55 AM

#

ah, you're running via go run...?

royal onyx Dec 12, 2024, 12:55 AM

#

yeah - we compile it but yes

royal onyx Dec 12, 2024, 10:17 PM

#

so no way to do this? 😦

#

short of maybe custom otel integration?

empty ore Dec 13, 2024, 12:04 AM

#

@royal onyx lots of automatic labelling around modules (which you're not using), but there is also custom otel spans option that I haven't really worked with much. Maybe it could help?

royal onyx Dec 13, 2024, 12:06 AM

#

I'm not able to move to modules unfortunately, we've put considerable effort into the go sdk, and a prior attempted transition to modules failed.

#

Are there any docs on the otel options? I have to go digging in vito's bass project to find anything close to this, and it would be super helpful just to have a simple example of the recommended pattern

empty ore Dec 13, 2024, 12:09 AM

#

https://docs.dagger.io/features/visualization/#terminal-ui-tui:~:text=The TUI is driven,for monitoring and debugging.

Visualization | Dagger

Visualize pipelines in real-time

empty ore Dec 13, 2024, 12:09 AM

#

royal onyx I'm not able to move to modules unfortunately, we've put considerable effort int...

agreed cc @bronze epoch

#

I'll dig

royal onyx Dec 13, 2024, 12:11 AM

#

awesome thanks so much!

empty ore Dec 13, 2024, 12:16 AM

#

just tried this prompt with chatGPT
"can I have a dagger go module example with custom otel spans that get emitted?"

#

Got something that looks plausible. Trying from the airport wifi 🙂

royal onyx Dec 13, 2024, 12:16 AM

#

just for a little context, we init a set of pipelines like so

    buildEg, _ := errgroup.WithContext(runCtx)
    for _, builderTester := range builderTesters {
        loopBuilder := builderTester
        buildEg.Go(func() error {
            return loopBuilder.Build()
        })
    }
    err = buildEg.Wait()

and each of these builders may have some set of sub pipelines.

I'd hope that I could init traces at the top level, so that I might have a trace stack that looks something like

service1
  deployable1Build
     ..explicit dagger steps
  deployable1Deploy
     ..explicit dagger steps
  deployable2Build

empty ore Dec 13, 2024, 1:12 AM

#

@royal onyx got this working

https://github.com/jpadams/testing-dagger-otel-span-go

dagger login
go build .
dagger run ./main

GitHub

GitHub - jpadams/testing-dagger-otel-span-go

Contribute to jpadams/testing-dagger-otel-span-go development by creating an account on GitHub.

#

empty ore Dec 13, 2024, 1:31 AM

#

Cc @frail knoll

bronze epoch Dec 13, 2024, 4:24 AM

#

empty ore agreed cc <@1013436980249505882>

I've opened an internal issue to add this to the cookbook, although it will be using the modules approach

royal onyx Dec 13, 2024, 5:27 PM

#

would love to see a non modules approach!

#

ran the example from jeremy above, and not setting new groupings appear in the dagger UI

#

think jenkins - I want to group sets of steps into larger groups or parent spans. it appears in the example I lose groupings entirely and just get a blob of json

#

if this is just a use case the team doesn't want to support that's fine, just lmk. My dev team has lots of difficulty understanding what's wrong with their CI builds (via dagger UI) and I'd need to create some work around

empty ore Dec 13, 2024, 5:53 PM

#

royal onyx ran the example from jeremy above, and not setting new groupings appear in the d...

Hi Steven! That was a total hack based on some ChatGPT and thrashing to get something working right before boarding a flight...I haven't tried to annotate an actual small pipeline. Did you try with some of the otel functions above?

royal onyx Dec 13, 2024, 5:54 PM

#

yeah I ran the code exactly

#

I'm trying various other calls to things like dagger.Tracer() etc, but can't seem to get my spans to show similar to how an "exec" call might look (But with a name I specify)

empty ore Dec 13, 2024, 5:55 PM

#

Not much Dagger in there.

// Start a Container build process with Dagger
    container := client.Container().
        From("alpine:latest").
        WithExec([]string{"echo", "Hello from Dagger!"})

Thought to try with a little more.

royal onyx Dec 13, 2024, 5:55 PM

#

yeah that step doesn't even show in the example though 😅

empty ore Dec 13, 2024, 5:56 PM

#

right!?

royal onyx Dec 13, 2024, 5:56 PM

#

I only see an entry for connect

empty ore Dec 13, 2024, 5:58 PM

#

I wonder if the example is hijacking the root or something...not very savvy on this
// Start a root span

#

this seems relevant
#1308109601803079691 message

royal onyx Dec 13, 2024, 6:12 PM

#

ah yeah seems very similar

frail knoll Dec 13, 2024, 7:30 PM

#

I've experimented with this a bit but do not have a foolproof method yet. I agree we definitely need a guide in our docs.

I don't think there is a huge difference for this specific feature when it comes to modules vs non-modules

The best version of this that I know right now is in dagger/dagger

For example: this -> https://github.com/dagger/dagger/blob/4539614d43e748b89f5cab960ba34972c2b13ce4/.dagger/docs.go#L70

becomes this (image attached)

full trace url: https://dagger.cloud/dagger/traces/65f17230a1cf5f0184e5361ac05ebcd4

Screenshot_2024-12-13_at_11.29.50_AM.png

GitHub

dagger/.dagger/docs.go at 4539614d43e748b89f5cab960ba34972c2b13ce4 ...

An engine to run your pipelines in containers. Contribute to dagger/dagger development by creating an account on GitHub.

royal onyx Dec 13, 2024, 7:46 PM

#

Hmm yeah I'm using this code (very similar)

package main

import (
    "context"
    "fmt"
    "log"

    "dagger.io/dagger"
    "dagger.io/dagger/telemetry"
    "go.opentelemetry.io/otel/codes"
)

func main() {
    // Initialize the OpenTelemetry tracer
    //tracerProvider := InitializeTracer()

    ctx := context.Background()
    ctx = telemetry.Init(ctx, telemetry.Config{Detect: true})
    // Create a Dagger client
    client, err := dagger.Connect(ctx)
    if err != nil {
        log.Fatalf("failed to connect to Dagger: %v", err)
    }
    defer client.Close()
    tracer := dagger.Tracer()

    // Emit a custom span for Dagger pipeline execution
    ctx, span := tracer.Start(ctx, "dagger-pipeline")
    //stepSpan.SetAttributes(attribute.String("step", "Alpine Container build with Dagger!"))

    // Start a Container build process with Dagger
    container := client.Container().
        From("alpine:latest").
        WithExec([]string{"echo", "Hello from Dagger!"})

    // Execute the container process
    _, err = container.Sync(ctx)
    defer func() {
        if err != nil {
            span.SetStatus(codes.Error, err.Error())
        }
        span.End()
    }()

    if err != nil {
        log.Fatalf("failed to execute container: %v", err)
    }

    // Print a success message
    fmt.Println("Executed container successfully!")
}

and get no span named "dagger pipeline"

#

#

very interesting that it has the output of my println, but not the traces or output from my withexec

frail knoll Dec 13, 2024, 7:55 PM

#

Yeah so maybe I am just wrong about this 🙂

I don't think there is a huge difference for this specific feature when it comes to modules vs non-modules

Its possible that there is some extra init happening when its with a module.

In either case; yeah we should get some solid examples going.

if this is just a use case the team doesn't want to support that's fine, just lmk. My dev team has lots of difficulty understanding what's wrong with their CI builds (via dagger UI) and I'd need to create some work around

I'd love for @tame osprey to share his thoughts once he is back next week.

royal onyx Dec 18, 2024, 9:46 PM

#

bumping lest this be forgotten 🙂

empty ore Dec 19, 2024, 12:07 AM

#

royal onyx bumping lest this be forgotten 🙂

Saw this working today live!!!

https://github.com/shykes/x/blob/main/gpt/main.go

GitHub

x/gpt/main.go at main · shykes/x

Solomon's experimental Dagger modules. Contribute to shykes/x development by creating an account on GitHub.

#

Wonder if it works for you @royal onyx

#

Tracer stuff cc @frail knoll

royal onyx Dec 27, 2024, 6:47 PM

#

looks the same as my snippet for the most part

I think the difference here is that the module is injecting a valid otel context in the function param

func (m Gpt) WithPrompt(ctx context.Context, prompt string) Gpt {
    log := "🧑: " + prompt
    ctx, span := Tracer().Start(ctx, log)
    span.End()
    hist := m.loadHistory()
    hist = append(hist, openai.UserMessage(prompt))
    m.Log = append(m.Log, log)
    return m.saveHistory(hist)
}

vs my call in the snippet above where I use my own context.


    ctx := context.Background()
    ctx = telemetry.Init(ctx, telemetry.Config{Detect: true})
    client, err := dagger.Connect(ctx)
    if err != nil {
        log.Fatalf("failed to connect to Dagger: %v", err)
    }
    defer client.Close()
    tracer := dagger.Tracer()
    ctx, span := tracer.Start(ctx, "dagger-pipeline")

Do I need to init telemetry differently somehow?

royal onyx Dec 27, 2024, 8:42 PM

#

tbh it seems like I get less traces when I add telemetry calls.

frail knoll Dec 27, 2024, 8:54 PM

#

royal onyx tbh it seems like I get _less_ traces when I add telemetry calls.

Are you able to test some of this stuff out on latest version of dagger? Not sure that it will make a huge difference but would be great to rule out older versions as a starting point.

royal onyx Dec 27, 2024, 8:57 PM

#

yeah just ran with the latest 15.1 - no improvement

royal onyx Jan 13, 2025, 6:52 PM

#

still wishing one day this would be possible...

unborn sapphire Feb 21, 2025, 10:42 PM

#

royal onyx still wishing one day this would be possible...

I managed to get this working today (i'm on v0.16.1):

package main

import (
    "context"
    "fmt"
    "log"
    "os"
    "path/filepath"
    "sync"

    "dagger.io/dagger"
    "dagger.io/dagger/telemetry"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/trace"
)

var dag *dagger.Client
var clientMu sync.Mutex

func Tracer() trace.Tracer {
    return otel.Tracer("dagger.io/sdk.go")
}

func initClient() *dagger.Client {
    clientMu.Lock()
    defer clientMu.Unlock()

    if dag == nil {
        var err error
        ctx := context.Background()
        dag, err = dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
        if err != nil {
            panic(err)
        }
    }
    return dag
}

// Close the engine connection
func Close() error {
    clientMu.Lock()
    defer clientMu.Unlock()

    var err error
    if dag != nil {
        err = dag.Close()
        dag = nil
    }
    return err
}

func main() {
    _ = initClient()
    defer Close()

    ctx := context.Background()
    ctx = telemetry.Init(ctx, telemetry.Config{
        Detect: true,
    })
    defer telemetry.Close()

    err := build(ctx)
    if err != nil {
        log.Fatal(err)
    }
}

func build(ctx context.Context) error {
    ctx, span := Tracer().Start(ctx, "Build", telemetry.Encapsulate())
    defer span.End()
        // do stuff with `dag`
        
        return nil
}

tame osprey Feb 22, 2025, 3:08 PM

#

This should get easier once we bikeshed the API: https://github.com/dagger/dagger/pull/9327

GitHub

add OTel span API + Go/Python SDK support by vito · Pull Request #9...

Goals

High level API for OTel spans

e.g. async with dag.span("my span", internal=true)
Add withFoos for customizing the span

Ergonomic SDK APIs, without too much effort

Th...

unborn sapphire Feb 22, 2025, 4:34 PM

#

tame osprey This should get easier once we bikeshed the API: https://github.com/dagger/dagge...

looks interesting!

unborn sapphire Feb 22, 2025, 5:17 PM

#

any thoughts on that API/something else exposing the ability to override the root span? in my SDK application i'd love to have Release Workflow instead of dagger run go run . for example.

my understanding is the root span is ctx, span := Tracer().Start(ctx, spanName(os.Args)) in cmd/dagger/engine.go

royal onyx Feb 24, 2025, 6:16 PM

#

unborn sapphire I managed to get this working today (i'm on `v0.16.1`): ``` package main import...

   // do stuff with `dag`

This is the issue for me, not using modules 😦

unborn sapphire Feb 24, 2025, 6:19 PM

#

The code I shared isn’t using modules, that dag is a global variable

royal onyx Feb 24, 2025, 6:19 PM

#

ah whoops - misread trying it out

unborn sapphire Feb 24, 2025, 6:19 PM

#

You can initialize it however you want

royal onyx Feb 24, 2025, 6:22 PM

#

yeah still nothing - must use the dagger run call which isn't something we typically do

#

Like technically we could wrap the CI calls in this, but it breaks the write once, run anywhere stuff since we now rely on an additional binary. Devs at my company have reacted poorly to having to use/understand/maintain another tool - we've been running on just a compiled internal binary with dagger client. (equivalent to running go run . in this case)

#

I guess maybe I just need to figure out how to set up the dagger engine in the same manner as dagger run when I do my initClient call

unborn sapphire Feb 25, 2025, 2:17 AM

#

yep, what i shared was using dagger run. when we use go run, the otel dagger TUI and engine telemetry isn't setup and thus no exporters configured. i spent a few minutes on it and managed to get it working by initializing my own TUI and engine telemetry (https://github.com/dagger/dagger/blob/aa708a22a30d2fc994a62a222aadf277af22d919/cmd/dagger/engine.go#L56-L143). screenshot attached.

#

#

the beauty is that you get the same presentation and interactivity of dagger run go run . without having to have the dagger CLI installed. the tradeoff is you've got to copy and paste a bit of unexported code (lines i linked above) from the dagger source. i'll spend a few more minutes on it tomorrow and see if I can trim things down to eliminate copy and paste and provide a clean minimum working example.

tame osprey Feb 25, 2025, 2:32 AM

#

unborn sapphire any thoughts on that API/something else exposing the ability to override the roo...

hmm makes sense, will think about it some more, and try to keep the non-module use case in mind in general

#

if you have any more feedback/requests re: controlling the frontend/UI, here's a good spot: https://github.com/dagger/dagger/issues/9374

GitHub

Frontend API design · Issue #9374 · dagger/dagger

We've received numerous requests in just the past couple of weeks for more ability to control the UI. At a base level, we know we don't want developers to have to work directly with the Ope...

royal onyx Feb 25, 2025, 7:03 PM

#

@unborn sapphire - any chance you'd mind sharing that snippet? getting a little lost here, can't seem to init right

unborn sapphire Feb 26, 2025, 6:34 AM

#

@royal onyx sure, I haven't had time to do a second pass to clean things up, but here's a gist of my initial pass: https://gist.github.com/mcblair/a642c0febc609da221acce601afc32b0

other than having copied and pasted a fair amount of boilerplate from dagger source, the roughest edge is with the go.mod replace directives:

replace (
    github.com/dagger/dagger/engine/distconsts => ./distconsts

    github.com/moby/buildkit => github.com/dagger/buildkit v0.0.0-20250128235329-9c8ee9e867a5
)

#seems like things are getting fairly