Custom OTEL spans failing to send: Request timed out

[novel mesa]

Hi!
I'm trying to create custom spans in a TypeScript module and consistently get this error:

Here's the code:

@func()
async foo(): Promise<Container> {
  const tracer = getTracer()
  return await tracer.startActiveSpan('CUSTOM_SPAN', async (span) => {
    try {
      return dag.container().from(`debian:12-slim`)
    } finally {
      span.end()
    }
  })
}

For context, I'm using dagger on MacOS (Apple Silicon) with OrbStack for running docker

Here are the env vars as seen by console.log(process.env):

My dagger.json:

{
    "$schema": "https://docs.dagger.io/reference/dagger.schema.json",
    "name": "workers-monorepo",
    "engineVersion": "v0.18.3",
    "sdk": {
        "source": "typescript"
    },
    "source": ".dagger"
}

I also tried sending spans from within a script called from withExec() but got the same error

I wonder if this is an issue with bun somehow

okay yeah, I switched my package manager to pnpm and no longer get the issue 😦

It would be cool if dagger modules would copy over the patches directory so that I could patch the dependency

Huh, I guess it doesn't respect the package manager version specified in package.json - the screenshot above shows 1.1.38

[novel mesa]

Okay this is pretty hacky, but I managed to patch the file using a postinstall script:

"scripts": {
    "postinstall": "bun -e \"Bun.write('node_modules/@opentelemetry/otlp-exporter-base/build/src/transport/http-transport-utils.js', await fetch('https://gist.uuid.rocks/jh/a1a5ca98573f4506af0e4878f6ab2cd1/raw/HEAD/http-transport-utils.js'))\" && cat node_modules/@opentelemetry/otlp-exporter-base/build/src/transport/http-transport-utils.js"
},

[manic rose]

cc @tawny kraken

[novel mesa]

The thing I'm struggling with now is that it's sending a successful request to /v1/traces, but no spans show up in Dagger Cloud.

Here's the script inside a withExec that's exporting spans - I wonder if I need to do something special to get it to show up?

[tawny kraken]

Bun is having an issue with Otel, I remember we add a post about that few weeks ago

[tawny kraken]

Yeah the script needs to pass the trace parent, otherwise it's an orphan so it's not diplayed

[tawny kraken]

This issue: https://github.com/open-telemetry/opentelemetry-js/issues/5260

However! https://github.com/dagger/dagger/pull/10094 may fixes the problem because it will use the node version of otel, it's also fixing a timeout issue for long running operation

#1347311312975302718 message referenced here

[novel mesa]

This is what I was missing! Between that and my temporary patch for the otel lib, the tracing works!!

[tawny kraken]

How did you pass the trace parent? I'm soo curious

If that works fine! Feel free to open a PR with that fixes for Bun on our repo so everybody can benefit from that 😄

[novel mesa]

First I copied this getContext() function that I found in .dagger/sdk/src/telemetry/telemetry.ts

and then I passed it to the root span like this

[tawny kraken]

Very smart! So you did it all in the module's code side?

Just to know if by any change we could make it work for everybody, because it seem you're doing a post install + this to make it work

[novel mesa]

BUT my main script calls many other scripts using turbo, so I created this function to create a TRANSPARENT variable from a span, and then I pass that in the env when I call other functions

[novel mesa]

No, this is all within a CLI I wrote for wrapping commands like turbo in my monorepo

so the dagger module calls my cli, which is what implements all the spans

[tawny kraken]

Oh I see! That's smart! You should demo it in a community call, to show how to propagate traces from a dagger module to the actual execution!

[novel mesa]

Oh and here's the patch needed to make this work in either a dagger module or any script that runs with bun (this is a pnpm patch for my cli, not the dagger module):
file: @[e-mail]

diff --git a/build/src/transport/http-transport-utils.js b/build/src/transport/http-transport-utils.js
index 74ad3277d8538e5790df8716e3ca1db43ef275c6..e95a3c881c9e1e2b9f39ca3acfe09db5982f3180 100644
--- a/build/src/transport/http-transport-utils.js
+++ b/build/src/transport/http-transport-utils.js
@@ -48,6 +48,7 @@ function sendWithHttp(params, agent, data, onDone, timeoutMillis) {
         const responseData = [];
         res.on('data', chunk => responseData.push(chunk));
         res.on('end', () => {
+            console.log('STATUS', res.statusCode)
             if (res.statusCode && res.statusCode < 299) {
                 onDone({
                     status: 'success',
@@ -84,10 +85,10 @@ function sendWithHttp(params, agent, data, onDone, timeoutMillis) {
     });
     const reportTimeoutErrorEvent = nodeVersion >= 14 ? 'close' : 'abort';
     req.on(reportTimeoutErrorEvent, () => {
-        onDone({
-            status: 'failure',
-            error: new Error('Request timed out'),
-        });
+        // onDone({
+        //     status: 'failure',
+        //     error: new Error('Request timed out'),
+        // });
     });
     compressAndSend(req, params.compression, data, (error) => {
         onDone({

My postinstall script is only needed for spans I add within the module itself because dagger doesn't support patches when installing deps

[novel mesa]

I would but the problem seems to be within @opentelemetry/otlp-exporter-base, not dagger 😦

[tawny kraken]

Okay, that would be nice if this could be live on their side, it would fixe https://github.com/open-telemetry/opentelemetry-js/issues/5260

[novel mesa]

Thanks! Yeah not really a dagger problem.

That said, I think the ability to patch dependencies is useful in general and would be nice to support in dagger modules

[novel mesa]

when is the call?

[manic rose]

cc @fast knot

it's every other Thursday. Next one is on Thrusday 24 🙏

[novel mesa]

ok cool! Will see if I can make it

one thing I might be interested in PR'ing is updating bun and adding patches support in dagger

[tawny kraken]

We're not copying the node module inside the runtime, I don't have much experience with patches but if there's a way to apply them after installation consistently, then that's easily doable 😄

[novel mesa]

Yeah that's exactly how patches work! bun/pnpm/etc will apply the patch after installing

[tawny kraken]

Then yeah! I think it's a matter of adding an extra command in the runtimes if patches are found 😄

[novel mesa]

yep! sounds simple enough

[tawny kraken]

I have this PR (https://github.com/dagger/dagger/pull/10196), that make the runtime code much more readable, if you can wait until it lands, that would simplify your contribution 😄

It depends on https://github.com/dagger/dagger/pull/10094 so I need both to be merge 😄

[novel mesa]

ah cool! yeah I can wait - plenty busy all the time with work 😅

oh wait it's gonna generate a bundled version of the sdk dir? uh oh, I've been patching stuff in there too ahah

[tawny kraken]

Basically, there will be 2 way to configure your TS dagger module:

1. By using the SDK bundle (default), so it improves performances, dependencies management, less installation time, less code etc...
2. By copying the TS sdk directory locally, as a subpackage of the module (it's the current way things work but it's slow and sometime confuses users)

[novel mesa]

oh, anything to make things faster I'm all for 🙂

I just need to PR a typescript fix in the decorators

[tawny kraken]

That way, the SDK bundle is already inside the engine, the only thing we actually generate is the client because it's dynamic 😄 and we don't need to install any deps except Typescript so it makes things way faster than before.

• The PR that cleanup the runtime improve operations and reorder few things, that could in theory make this faster too

[tawny kraken]

There's an issue in our decorators?

[novel mesa]

Yeah - I prefer to set "strict": true, in tsconfig.json, which causes this error:

I fixed it by adding | undefined to propertyKey

[tawny kraken]

I'm confused why strict mode triggers this error there, the type is quite simple 😮

[novel mesa]

because propertyKey can be undefined I think?

and strict mode forces you to deal with things that may be undefined

[tawny kraken]

Hmm yeah I guess, well it's a simple fix, feel free to ping whenever you open a PR and I'll review it 😄

[novel mesa]

I'd also recommend adding strict to the generated tsconfig.json - it helps catch so many bugs

[tawny kraken]

As long as it doesn't create additional issues for users, I'm good with it

[novel mesa]

it generally helps prevent a lot of bugs so I think it'll be the opposite !

[modest stratus]

can anyone please help us here also, to get deno native otel tracing getting visualised in dagger cloud https://docs.deno.com/runtime/fundamentals/open_telemetry/
#1333326734506070066 message

Just to add in context
https://github.com/dagger/dagger/issues/10190
https://github.com/dagger/dagger/pull/9716

[manic rose]

@modest stratus reversing the question since we have a limited bandwidth over here. What would you need from our end so you can contribute that?

In a similar way Jacob is helping here

[modest stratus]

I understand everything bro and supporting dagger from solomon's first tweet about it...but we are stuck and just we need direction only...
not asking to solve for us anything...just guide us to get it solved...that's it...

[novel mesa]

PR to enable strict mode and fix that decorator bug: https://github.com/dagger/dagger/pull/10203

[tawny kraken]

You can already use traces with Dagger & deno, I don't understand the issue?

It's already working inside module

  @func()
  async foo(): Promise<string> {
    const tracer = getTracer()

    return await tracer.startActiveSpan("example", async () => {
      return await dag.container().from("alpine:latest").withExec(["echo", "hello"]).stdout()
    })
  }

So what exactly do you need? 😮

[tawny kraken]

Reviewed, left 1 request, otherwise lgtm for the current changes 😄

[novel mesa]

Thanks! Done ✅

[novel mesa]

Okay I need to put more time into updating tests - will work on it later!

[tawny kraken]

Np problem 😄

[novel mesa]

On another note, I've started making progress sending traces to both Dagger and Axiom

(it sends my CLI's traces to both, just haven't figured out how to send Dagger's builtin spans to both places yet)

[manic rose]

Nice! If there's any Cloud feedback you have for us, would be greatly appreciated

[novel mesa]

My workflow so far has been to run dagger in vs code and open the trace on my second monitor while it’s running.

The thing I’ve wanted the most is a persistent tab that always has the trace for the current dagger command I’m running.

It’s been annoying having to constantly close tons of dagger tabs (and have them often open in the wrong browser window)

Another thing - my primary use for Dagger is long running scripts with a lot of log output that I want to watch while it’s running, but I’ve found that logs do not reliably stream as it’s running. Logs often stop streaming, requiring me to refresh and navigate all the way back to the dagger span with the logs I want to see

[manic rose]

Oh! There's a hidden 🐰 that you can use for this which is using v3.dagger.cloud/$org/follow. That should always give you the latest trace

[novel mesa]

Oh what 🤯

[manic rose]

We will be adding a menu link today🙏

[novel mesa]

Yay! Looking forward to trying it next time I’m at my laptop

[manic rose]

Will look into this 🙏

cc @ember mauve

[ember mauve]

[novel mesa]

Thanks!!

[novel mesa]

Does this work for y'all? It only seems to show a trace from 4 days ago. I tried both local dagger and CI runs.

Looks like this:

There are definitely more recent traces

[ember mauve]

Hey Jacob! sorry for not sending an update here. I'm currently working on this feature as it was not ready yet, it was kinda hidden so we could test the UX before surfacing it

I'm guessing later today/tomorrow we should have it working properly

From the screenshots you shared though, it seems like the traces are being sent from a token instead of a user.
The follow feature only allows following users, not tokens.

When there's a user logged in and a DAGGER_CLOUD_TOKEN set in the env, the latter takes precedence.

Do you think following tokens is something you'll find valuable as well?

[novel mesa]

Ohh okay gotcha. Yes, following a token would be great because outside of developing the Dagger module, I mostly push to a branch and watch the CI job. So following a token would be nice! (Although I could see that being tricky when a single git push triggers multiple concurrent jobs)

What’s weird is I definitely ran some jobs after running “dagger login” and then running the jobs locally, so from your description I think it should have worked

[novel mesa]

Oops nvm, I forgot that my dagger wrapper uses a token. My bad!

[ember mauve]

No problem!

[ember mauve]

Although I could see that being tricky when a single git push triggers multiple concurrent jobs

Yep, this was my first thought.
I'm wondering, are you focusing the commit/ref by clicking on the CI traces to track the traces of a specific git push?

Or would you like to actually focus the traces themselves instead of the traces list for that commit?

[ember mauve]

Also, when you look at the logs, are you looking at them so you can identify an artifact/specific data, or is your goal to monitor when the trace has finished it's execution?

Something that's been in the backlog are notifications, so you can keep working on your thing while a pipeline or a specific trace runs. But unsure if that's the reason why you're foucusing on a trace while it runs.

[novel mesa]

Okay so after using this more today, I think I'd be pretty happy if each git push made it navigate to the trace list for the commit. E.g. https://v3.dagger.cloud/jhands/ci?branch=dagger-decorator&ref=a6ebe2db8dc6964ab57a4635dad89123a21d1553&repo=github.com%2Fuuid-rocks%2Fworkers-monorepo

Before, I was going to GitHub Actions logs to find the Dagger Cloud link, but now I'm finding myself avoiding GHA logs all together and only using the Dagger Cloud interface.

A CI follow feature would basically save me 2 clicks on each push (click Dagger logo -> click on my commit)

I think this could still be tied to the user because push events should have my GitHub username attached to it?

[novel mesa]

Now that you mention it, a lot of the time when I'm opening logs in Dagger, it's because the progress bars don't seem to work unless I expand logs.

If the progress bars / spans consistently updated throughout the build, I probably wouldn't look at logs that much. I'm primarily just interested in progress / result most of the time

Also I will say - once I'm done setting up CI with dagger, I will not be looking at running builds as much live like I am now. Normally I git push and keep coding while keep an eye on these buttons in vs code which show whether a build is running (and turns red if it failed).

[manic rose]

I think this could still be tied to the user because push events should have my GitHub username attached to it?

That is generally not the case since the engines that live in CI should have the org token and not the user one. I guess it might be reasonable to add an option so you can follow traces by that token as well. Also, adding the ability to follow git push events might become handy as well. Feedback taken 🙌

[novel mesa]

yeah following the git push was what I was thinking. Thanks!

[ember mauve]

a lot of the time when I'm opening logs in Dagger, it's because the progress bars don't seem to work unless I expand logs.

When you say opening the logs, are you referring to focusing and inspecting specific span logs in the trace or just opening the trace?

just interested in progress / result most of the time

So are you finding yourself returning to the tab constantly until the execution is done, whether it's an error or success? Or are you only visiting the page if it fails based on what VSCode displays?

Sorry for the interrogation, I'm trying to come up with the best DX for your case 😝

[ember mauve]

BTW the polished /follow feature is out, so the experience around that should be better.
Clicking on a user avatar will take you to follow user traces, and from there you can select others.
There are some details that I'll add during the next weeks!

[novel mesa]

viewing span logs - my CI jobs have only a few spans, with the main 2 of them having thousands of logs each

[novel mesa]

with my old setup, I'd typically only go to view the CI job if it failed. If vs code says it succeeded, I don't view it.

The confusing part of this convo (probably my fault), is that I'm at a very different part of the "journey" with Dagger where I'm setting up the CI pipeline itself and thus want to watch every single job after I make a tweak to see if it works

So right now basically every git push or local run, I'm going to watch it progress in the tab

[novel mesa]

nice!! I used it a bunch yesterday and it's reallyyyyy nice having it always view the latest trace

[ember mauve]

(probably my fault)

nope, I think what you're doing is how I'll do it too.

[ember mauve]

I'm going to take a look at this if I have some time this week cc @vale quail

[novel mesa]

@ember mauve One thing that would be really nice in Dagger Cloud traces is an "artifacts" section. I find it annoying that I have to scroll through a ton of spans just to get the URI's for all my published docker images.

What would make it even better is if we could tag arbitrary OTEL spans as "artifacts", so that I could add a custom artifact showing how many Cloudflare Workers got deployed

[ember mauve]

Interesting, do you think you'll like to mark specific functions in your pipeline as artifacts, maybe as a parameter of a function, so you can explicitly mark what should be surfaced in the UI?

If so, are you thinking about a portion of the trace UI that displays the artifacts for that trace?

Or were you thinking about a different UI for it?

[novel mesa]

Not totally sure what the API would look like, but I know I'd want to add specific artifacts from within a function so some kind of API would be prefferred. Maybe something like:

const con = dag.container() .... // do some stuff
const artifact = parseArtifact(await con.stdout()) // or maybe I'd get it from a file within the container
con.publishArtifact('Deployed Cloudflare Workers', {logs: artifact}) // something like this so that it links the spans correclty

As for where to surface it, I think I'd prefer it be on the trace page somewhere (maybe an expandable section?)

[manic rose]

cc @pliant marlin @bold breach

[ember mauve]

That's definitely interesting, will discuss it with the team to see what they think about it
I also went through the need of bubbling up specific outputs, given that the content was deeply nested in some logs given the nature of the pipeline

These are some great insights, many thanks for the feedback

[ember mauve]

I still haven't looked into this, but it's in the backlog for sure

[manic rose]

@novel mesa FWIW you can create custom OTEL spans currently. I understand it's not the same as having a top level artifacts feature but definitely helps when trying to improve the readability and browsing in Dagger Cloud

[novel mesa]

True! Maybe I could have a function in a submodule return the URI of the image so that my top-level module can create a top-level span so that I don’t have to dig into the trace to get docker image URIs

[novel mesa]

A great start here might be to show published docker images at the top level before getting into custom artifacts

Like some kind of “summary” span

[manic rose]

this is an example on how we create top level spans for our DaggerTests for example https://v3.dagger.cloud/dagger/traces/df6006185d3eb5a02c74bbb7d0ff7028