All of my traces are empty, both old ones and new ones
#All Dagger Cloud traces broken?
last hazel
V2 seems to still be working. Looks like it's a V3 problem
Okay so I updated to dagger v0.18.8 (from v0.18.6), and tracing is working again.
I guess Dagger Cloud is not compatible with older dagger versions
last hazel
jk, it's broken again...
last hazel
Okay I think it's just randomly breaking sometimes
raven lichen
Okay I think it's just randomly breaking sometimes
Thx for sharing. Do you get any warnings in the terminal when this happens by any chance?
raven lichen
V2 seems to still be working. Looks like it's a V3 problem
The same trace renders in v2?
raven lichen
Mind sharing a broken traceID please?
last hazel
Thx for sharing. Do you get any warnings in the terminal when this happens by an...
I didn't see any, now
last hazel
Mind sharing a broken traceID please?
f51e1f0508200f37289f64f76d75b24a
that trace renders fine in v2
raven lichen
that trace renders fine in v2
Yes I was about to mention thwt
Cool, will fix asap. Thx for reporting 🙏
unique magnet
resolving the issue is taking a bit more time than expected, will let you know when I have any updates
raven lichen
@last hazel mind checking if you're seeing the same problem please?
pushed a fix a few hours ago that was causing some traces to show incorrectly
I have high hopes it's related to the same issue you're seeing
last hazel
Still broken for the trace ID I sent 😦
raven lichen
k, thx. We keep investigating... cc @unique magnet
unique magnet
Made some progress when investigating during the weekend, I found the cause, trying to find the cause of the cause 🙂
unique magnet
Still broken for the trace ID I sent 😦
does this also happen with new traces as well?
unique magnet
sorry, need to hard ping 🙂
@last hazel if you send a new trace to Dagger Cloud, do you get the same error?
My theory is that on Friday we had an issue with 500s, so some traces were incomplete, and worse, some of them were partially complete, so the error comes from spans that have incomplete information, so the tree rendering broke. This means that the trace was generated, but some spans inside the trace did not have a link to the rest of the spans, and ended up being rendered as the main span of the trace.
also, before sending a new trace, can you make sure you clean up everything docker-wise before?
the thing is, we keep a local database of spans to be sent, so it's likely "turning on and off" that database and everything related to spans might do the trick
this error was not only present in your organization, other users had it too
unique magnet
tldr; There's different logic under the hood
v2 is purely React, whereas v3 is WASM and the logic is shared/similar to that of the CLI
While the behavior is indeed strange, it makes sense for them to be different mostly because v2 is only being mantained and has not received any updates for a while: what's being set as the main span differs in one implementation and the other
I was able to set the correct span with local changes in v3 and was able to make it work, but a working patch for handling missing data will need a bit of bikeshedding, and it's something we'll look into, but I want to be 100% certain that the cause of the issue was missing data in this case.
If so, the actual "fix" should be handling these edge cases where the missing data might impact unrelated traces.
raven lichen
Still broken for the trace ID I sent 😦
@last hazel I misread this.
LMK if by any chance you see it keeps happening with new traces in v3 please 🙏 given that the issue that we fixed could have corrupted some trace information 🙏
last hazel
So far so good! Tested a handful of traces both in CI and locally