Stream logs from job container?

[ornate aurora]

Hello, I'm exploring using dagger as a generalized task runner to be able to write tasks that can execute locally, on github actions, and to build a web UI around for remote calls that aren't dependent on github actions.

I was wondering if there is any way to stream the output of a container. I see that there is a way to stream the output of the engine as the type is a Writable, but for stdout and stderr, we're returned a string at the end of the container's execution. I would love to be able to get a stream of logs instead and separated by the containers I'm running so I can display logs in real time and possibly parallel with each other.

Thanks!

[lapis tree]

Hi Adrian! The Dagger API doesn’t support programmatic access to the container streams at the moment (you’re not the first to ask 🙂

HOWEVER Dagger ships with a pretty awesome terminal visualizer, just wrap your tool with dagger run as in, for example dagger run node myscript.js or dagger run go run mytaskrunner.go and you’ll see the difference right away 🙂

[ornate aurora]

Are there possibly plans to add support for container streams?

And follow up question, is there a way to wrap dagger run in my program? I'm planning to distribute this as an npm package to my developers and want to keep the program easy to use.

[lapis tree]

Here’s a blog post with interactive example: https://dagger.io/blog/dagger-0-6-0

[ornate aurora]

Also are there work arounds maybe to get the container streams that might not be native to dagger?

[lapis tree]

I believe the TUI visualizer might be an option already available in the Dagger Node SDK? double-checking with @pine junco @acoustic cliff @errant jacinth @vivid thicket ?

[ornate aurora]

This looks really nice, I'll definitely play around with this and see if it fits our immediate needs! Thanks 😃

[lapis tree]

If wrapping visualizer in the Node SDK is not possible now, it should be easy to add

(since all SDKs under the hood wrap the CLI anyway)

[ornate aurora]

Oh cool, if it's not currently possible I'd love to give a shot at opening a PR for it, then.

I am noticing that the experimental interactive tui is crashing on the db-service nodejs example in the example repo.

examples/nodejs/db-service on  main via  v18.17.1 
❯ _EXPERIMENTAL_DAGGER_INTERACTIVE_TUI=1 dagger run node ./build.js
Caught panic:

runtime error: invalid memory address or nil pointer dereference

Restoring terminal...

goroutine 23 [running]:
runtime/debug.Stack()
    runtime/debug/stack.go:24 +0x64
runtime/debug.PrintStack()
    runtime/debug/stack.go:16 +0x1c
github.com/charmbracelet/bubbletea.(*Program).Run.func1()
    github.com/charmbracelet/bubbletea@v0.24.2/tea.go:440 +0x8c
panic({0x105202ba0, 0x105c1a170})
    runtime/panic.go:890 +0x258
github.com/dagger/dagger/internal/tui.(*Details).SetItem(...)
    github.com/dagger/dagger/internal/tui/details.go:37
github.com/dagger/dagger/internal/tui.Model.Update({0x140007139c0, {0x1053ab7e0, 0x1400037e4b0}, 0x1400037e570, 0x1400037e5a0, 0x1400037e5d0, 0x14000151900, {{0x0, 0x0}, 0xd3, ...}, ...}, ...)
    github.com/dagger/dagger/internal/tui/model.go:134 +0xf4
github.com/charmbracelet/bubbletea.(*Program).eventLoop(0x1400072c180, {0x1053b3450?, 0x140000b4fc0?}, 0x0?)
    github.com/charmbracelet/bubbletea@v0.24.2/tea.go:373 +0x5d8
github.com/charmbracelet/bubbletea.(*Program).Run(0x1400072c180)
    github.com/charmbracelet/bubbletea@v0.24.2/tea.go:503 +0x74c
main.interactiveTUI.func1()
    github.com/dagger/dagger/cmd/dagger/engine.go:136 +0x28
created by main.interactiveTUI
    github.com/dagger/dagger/cmd/dagger/engine.go:135 +0x1c8

Haven't checked to see if there is already an open issue for that but if there not, I'll make one in the morning.

Or maybe it's something about my system because it's always crashing. I'll investigate more in the morning.

[errant jacinth]

I'll take a look in a few hours, I don't remember actually 🙏 given Adrian's test, doesn't seem like it 😇

[lapis tree]

@ornate aurora probably not the cause of the crash, but you can drop that experimental_… env variable, the feature is no longer experimental

[ornate aurora]

I was just looking at the releases and it looks like 0.8.5 fixes the issue, I'm on 0.8.4 right now. I'm updating my nix config right now to update to latest

Looks like nixpkgs is lagging behind a bit, so I'll test in the morning

[vivid thicket]

dagger run node <file> work, we have example in our doc for node that work with it https://docs.dagger.io/sdk/nodejs/783645/get-started

[lapis tree]

Yes but ideally, Adrien would like node <file> to have the same output, without requiring calling dagger run.

[vivid thicket]

If I try with the go sdk (go run main.go), it's not running in TUI mode either, do we actually have a SDK who can run in TUI without dagger run?

[lapis tree]

No I don't believe so

We're talking about the Node SDK because that's the one Adrian uses, but I believe the same thing is true for all SDKs

[errant jacinth]

Oh, so you're suggesting adding an option to SDK to support it 👌

[lapis tree]

Yes 🙂

[errant jacinth]

Totally doable, Solomon is right. Please create an issue and a draft PR if you have an idea, we could discuss on any implementation detail / blocker on there Adrian 🙏👼

[pine junco]

I don't think it's trivial because the TUI needs to own stdin or something like that, otherwise you get unexpected behavior if multiple processes need it. The CLI is centralizing that control but to invert that to the SDKs they need to manage this so we need a protocol to follow here. @acoustic cliff can explain what's needed.

[lapis tree]

My assumption is that, when you use that option, the wrapped dagger CLI (and therefore the Dagger SDK wrapping it) take over the entire terminal: stdin, stdout, stderr

(which is the same fd under the hood anyway if you're really in a terminal)

[pine junco]

To clarify, the SDKs run a subprocess for dagger session and the in/out pipes need to be correctly managed. We're using stdin to hold the session and close it when we want the dagger engine to finish.

We also use stdout from that subprocess to get the connection details for the client session (port and token). And only stderr for the engine logs.

[lapis tree]

Makes sense, I don't expect any of that to be a problem (but could be wrong). dagger session --tui could open the current session's terminal and hook up the TUI code to that, independently of its own stdin/stdout (which as you point out won't be a terminal). My unix is getting rusty but I think that (or something like that) should do the trick

[errant jacinth]

As a last resort, I wonder if "just" wrapping dagger run as a subprocess under a pseudoterminal could work. It's a bit more complex, but at least it should allow all possibilities? 🤔 (and if that can give the same control ?)

[pine junco]

Yeah, certainly. I was just saying I don't think it's a trivial change but it may be a small enough change. I defer to Alex for what's actually needed for the TUI/dagger session and therefore what change in "protocol" for the SDKs. I'm familiar with how SDKs use dagger session now but not with how the TUI works. I just remember some pieces of explanations when it came out.

[acoustic cliff]

There's also the option of just passing the usual log output config to the client, like dagger.WithLogOutput(os.Stderr) (that's Go, not sure what the NodeJS equivalent is) - I don't see that discussed above. It won't be the full blown fancy TUI but it's super easy to use and should make it easy to grok output of things that run in parallel.

Technically speaking it's probably possible for dagger session to attach to the TTY by passing the file descriptor along if stdin/stdout/stderr are all already in use, but one problem is it can't know if anything else is printing to the TTY, and if so you'll get jumbled output. So in general the recommendation is to only set up the TUI at the outermost layer, and for anything else just use the plain old console-format stream from the dagger.WithLogOutput equivalent for each SDK. (It still has pretty colors and such.)

I would like to support streaming stdout/stderr from containers but we'd have to figure out how that should work at the API level.

[thick wadi]

@acoustic cliff how about using the _EXPERIMENTAL_DAGGER_JOURNAL flag and hooking into that from a separate process? 🤔

can't recall if WithLogOutput gets passed the same stream is as we output from the JOURNAL or if the JOURNAL contains more metadata the logouput doesn't have

[acoustic cliff]

That should work yeah, provided you're willing to code against the Progrock protocol format. WithLogOutput will always print the console format iirc, the env var is in addition to it

[ornate aurora]

I feel like I might be out of my element for trying to make this change now. I think I'm still going to explore the code base because I think it's fascinating, and I did see a bit about what @pine junco and @acoustic cliff were talking about with dagger session and the way it gets attached as a subprocess. I haven't looked into any of the JOURNAL stuff yet that @thick wadi has talked about, so I'm not entirely sure how that solution would work. It does seem like the difficult part would be figuring out how to passthrough or take over the TTY when executing from the SDK.

I still think for me the ideal solution would be a way of dealing with a stream of data, even if that's in a different form as using something like .stdout() from the sdk. Maybe something like LogOutput but instead of the current stream of logs coming from what looks like BuildKit, a stream of events in a machine readible format that contains the container ids.

For today, I can use an outer dagger run wrap for my team's POC and maybe try to parse the output stream of LogOutput to use it as a bunch of events since each line does start with a sequential id. There is just some missing information like the exec that is running.

[acoustic cliff]

fwiw the JOURNAL stuff Marcos mentioned should accomplish the "stream of events in a machine readable format" - it'll be a JSON stream in this format: https://github.com/vito/progrock/blob/main/progress.proto

But it's a bit tricky, if you want live streaming you'd need to start a process that tails the file as it's written.

[ornate aurora]

Hmm, I might explore that avenue, but I also wonder if that is the case, how hard it would be to write that back to the SDKs along with LogOutput, or if it could be an option to either get the original format of LogOutput to stderr or instead get that JSON stream to stderr.

Progrock is so cool, though. Haha

[acoustic cliff]

Yep - I'm sure there's something we could figure out, but the tricky part is any change that needs to be done for every SDK needs to be done with a lot of caution and planning, so we've been kicking the can down the road here in favor of other priorities. For now I would recommend going down the experimental journal env var route and seeing if it's something you'll still want in the long run. I'll be curious to hear how it goes!

[lapis tree]

@ornate aurora may I ask about your use case? What are you trying to build?

[ornate aurora]

My team is looking to build a general doer of things CLI tool for our developers and to be used within GitHub Actions. One of the workflows we're looking to port to dagger as a pipeline is our ephemeral environment creation workflow, which runs many different jobs in parallel. We want to strike a good balance to ease of use here and the logging structure was one of the concerns. We want to supply enough logs to know that things are working, but not too many logs that they become noise to the user. Later one we want to use what we've built to create a web service that manages our ephemeral environments for our users, so being able to split out those logs becomes more important.

I've attached an image of what our dependency graph looks like today for creating an ephemeral environment, which is why we're trying to figure out a good way to output logs. I really like the idea of the interactive tui, but am worried about requiring developers to all run dagger run with the specific environment variable. Making it into a web app is more ideal for us for this specific pipeline, and we really like the idea of using dagger to orchestrate all the jobs that need to be ran.

The worry with stdout() right now is that a job could hang and we won't be able to see the logs for it until it has finished which has been a really awful experience as of recent with GitHub Actions. (They used to show logs real time-ish, but as of recent that functionality seems to be have been broken)

Also recently saw that you guys are developing a helm chart to deploy the dagger engine to kubernetes, will be taking a look at that as well

[lapis tree]

I see, thanks for the details!

A few thoughts:

• We have a web pipeline visualizer 🙂 It's part of Dagger Cloud, the optional control plane that complements the Dagger Engine. Still in early access, but we have production customers already. Let me know if you'd like to kick the tires

• I understand the appeal of wrapping everything in your own CLI instead of wrapping everything in dagger run. We can make sure you can get the equivalent of dagger run from your own CLI (as discussed above)

• Personally I think you should not pursue programmatic access to the streams and developing your own visualization. It's a lot of work, and based on your description, your goals are exactly the same goals as dagger run: strike a balance of simplicity and detail, sane defaults and control, etc. I would recommend piggybacking on Dagger's TUI (embedded or not) and helping us improve it by influencing its design - telling us where it breaks for you etc. I'm confident that this will get you to the right experience faster than if you start from scratch with programmatic access

[ornate aurora]

I think I agree with you all on your points. dagger run really feels to me like the ideal way to view these jobs from CLI, LogOutput I think feels good enough for GHA, but Dagger Cloud sounds like more of what we want. My only concern with Dagger Cloud right now is the early development of it right now. I think my team would be open to doing a POC in it, but I'm not sure the company as a whole will be okay with moving to something that's not super mature. I could be wrong though!

I'd like to learn more about Dagger Cloud though. From some searching around this discord it sounds like it's bring your own runners and there's not a whole lot of public documentation. Do you guys have a primer document about it at all? Something I can show my team and manager?

[lapis tree]

Re: Dagger Cloud. It's been live for several months, but we initially prioritized core functionality and hardening the service itself. Now that we have successful production users (including some of decent volume), we've been focusing on docs and self-service. Expect rapid movement on that part in the next days

@ornate aurora if that sounds good to you, @lavish musk (our head of solutions) can DM you and we can get on a zoom to show you Cloud, answer your questions live, and figure out together if it's a good match?