eliminating OTel draining

[frozen bison]

Loose idea is to eliminate pub/sub, so we can save telemetry independently from consuming it, and then rather than draining waiting for spans/logs to complete, we just read from the telemetry store until the client goes away.

And for nesting, I'm hoping I can land on something simple like 'child clients just subscribe and re-export their telemetry to the outer one' - rather than the pub/sub trying to handle all that in one shot. Right now nested clients don't actually subscribe; tests that assert on errors from the CLI are relying on the fact that we shove stdout/stderr into error responses from GraphQL, which is the bit I'm trying to clean up.

[outer valley]

So "store telemetry" as in on disk in some db? Either way makes sense on a high-level to me.

Does telemetry ever need to cross from one session to another? i.e. a client in one session receiving telemetry from another client in a separate session? If not, then can just put the state in the session, which should make knowing when to GC it easy

[frozen bison]

@outer valley Yeah - I'm thinking just SQLite since I've had pleasant experiences with it in the past, but I'm not wedded to it. Might look for prior art in the OTel ecosystem too. I think persistence makes sense because 1) it can get quite large, and 2) @upper inlet and I were tossing around the idea of exposing telemetry through the API proper in the future.

Will be sure to avoid any complicated migration scenarios, now that persistence is involved. We can probably just make a new empty store on start tbh.

For GC, yeah I'm figuring something either session-based, client-based, or time-based, depending on which one is the most foolproof. If session/client just work, I probably don't need to bother with time, but if there are annoying races related to obtuse nesting scenarios I'll probably consider it.

The most important thing imo is to end up with boring problems (GC, migrations, what DB to use) instead of the current insanely complicated ones.

(Also, currently reviewing your socket PR, so far so good, don't want to leave that hanging before this)

Persistence could be helpful for a dagger history command too 🤔

[outer valley]

Yeah I like sqlite in general. The only caveat I know of is the "standard" sqlite go bindings require CGo, which makes compilation more annoying. But that's just last time I looked which was years ago and not a deal-breaker anyways. I don't love boltdb but it's what buildkit+containerd use (and thus us), so it's another option

[frozen bison]

I considered boltdb but iirc it's more of a flat key-value store, and for this I think I'd want to be able to store and fetch a bunch of ordered rows, so SQLite felt more appropriate. I remember the CGo shenanigans, will keep it in mind. I know there are slower pure Go implementations (https://datastation.multiprocess.io/blog/2022-05-12-sqlite-in-go-with-and-without-cgo.html), which might be fine

[gray rune]

fyi, it feels like this issue has the look+feel of an otel draining problem: #1254757484057464975 message
very much all for the idea of getting rid of it once and for all