In a Dagger custom application (executed

[deft trellis]

There are two issues that I see here that I wanted to clear up.

TUI vs Cloud
What you see in the TUI and in Dagger Cloud should be the same, the output of cached steps is not shown in either place at the moment. If that is not your experience can you send me a link so I can investigate?

Logging cached steps
This is a bit tricky, but this is the way the system works today. You should not rely on the output of with_exec to always be there because it the step is cached its not going to log as you've seen. I suspect this also applies even if you are returning stdout or stderr

I think the best approach would be to either use print as youre suggesting, or consider making some custom spans with OTEL: https://docs.dagger.io/cookbook/#create-custom-spans

[mental matrix]

If that is not your experience can you send me a link so I can investigate?
Good point, looks like I was hallucinating. Ideally, it would be. 🙂

I think the best approach would be to either use print as youre suggesting, or consider making some custom spans with OTEL: https://docs.dagger.io/cookbook/#create-custom-spans
I might be mistaken but custom spans don't seem to be the appropriate tool if I want to log hundreds of lines of output?

Looks like doing something like ctr = ctr.with_exec(…); print(await ctr.stdout()) after every .with_exec() in question would indeed be the way to. Though it'd be cool to have a convenience method like .with_exec(…).show_stdout(). Maybe I'll monkey-patch Container for the time being.

Thinking about this some more, I don't think a simple print() does the trick if several async tasks (several Dagger containers) are being awaited in parallel. It would mess up the TUI.

[deft trellis]

@pale lantern or @noble apex do you have an opinion here? This topic has come up a bit in the past but I think its still one of the rough spots of our existing DX

[noble apex]

you can use the with helper function in the SDK's for that

edit: I'm referring to a stopgap way of printing the stdout after each with_exec.

[pale lantern]

I think it could make sense for cached steps to still print their logs from the cache hit. There's a pretty similar limitation that you hit with LLMs where they'll run a command, and if it's a cache hit, they just see no output

not sure if that's particularly difficult cc @split quiver @white rune

could be a matter of storing the combined output (another feature somewhere deep in our backlog, instead of the split stdout/stderr streams) and just re-printing it on hit

or maybe persisting telemetry logs pertaining to caches

[deft trellis]

I'd love to see cahe step print previous logs, @mental matrix do you mind opening a GH issue please?

[mental matrix]

@noble apex

you can use the with helper function in the SDK's for that
Ah, I didn't know / forgot about .with! Would that really work in the present case, though? I am not merely trying to apply a synchronous function Container -> Container without breaking the chain, I need to await container execution and want to continue with/return the executed container afterwards.

Either way, as I mentioned before, a simple print(), whether invoked by hand (ctr = ctr.with_exec(); print(await ctr.stdout())) or through a lambda passed to .with(), likely won't work as desired:

Thinking about this some more, I don't think a simple print() does the trick if several async tasks (several Dagger containers) are being awaited in parallel. It would mess up the TUI.

[mental matrix]

@deft trellis I have created https://github.com/dagger/dagger/issues/10783 – let me know if this captures what you had in mind!

I would still like to discuss my specific use case, though, and how I could print the .with_exec()'s output (assuming it is available) without breaking the TUI. Maybe by passing a new parameter show_in_tui=True or collapse_logs=False to with_exec() ? The reality is simply that some steps & outputs are much more relevant than others, but right now all commands I execute in my container are treated equally (i.e. in the TUI all their output is hidden; in Dagger Cloud, the corresponding steps are all collapsed).

[noble apex]

@mental matrix here's what I have in mind

func (m *Foo) Foo(ctx context.Context) *dagger.Container {
    return dag.Container().From("alpine").
        With(execPrint(ctx, []string{"echo", "hello"})).
        With(execPrint(ctx, []string{"echo", "bye"}))
}

func execPrint(ctx context.Context, args []string) func(*dagger.Container) *dagger.Container {
    return func(c *dagger.Container) *dagger.Container {
        c.WithExec(args).Stdout(ctx)
        return c
    }
}

as you can see using With and execPrint lets me do what you're trying to do without breaking the chain

even if the steps are cached, they'll still be printed in the TUI output:

│ ╰─▼ Container.from(
│   │ ┆ address: "docker.io/library/alpine:latest@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1"
│   │ ): Container! 0.0s CACHED
│   ╰─● resolving docker.io/library/alpine:latest@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 0.0s
├─$ .withExec(args: ["echo", "hello"]): Container! 0.0s CACHED
├─▼ .stdout: String! 0.0s
│ ┃ hello
│
├─$ Container.withExec(args: ["echo", "bye"]): Container! 0.0s CACHED
╰─▼ .stdout: String! 0.0s
  ┃ bye

[mental matrix]

@noble apex Weird, I just tried this: The output appears briefly on the command line and then disappears again once the pipeline terminates since at this point all pipeline steps get collapsed:

$ dagger run uv run ./main.py
▶ connect 0.5s

▶ version: String! 0.0s

▶ Prepare Pulumi project container 1.5s

● directory: Directory! 0.0s

[…]

▶ .withExec(args: ["pulumi", "login", "azblob://…"]): Container! 2.4s
● .withExec(args: ["pulumi", "stack", "select", "dev", "--non-interactive", "--create"]): Container! 2.0s
▶ .withExec(args: ["pulumi", "preview", "--suppress-progress"]): Container! 20.5s
▶ .stdout: String! 24.9s

Full trace at https://dagger.cloud/<my org>/traces/79d37c77e74a3f7ed15ef9d8147a8539

A new release of dagger is available: v0.18.12 → v0.18.14
To upgrade, see https://docs.dagger.io/install
https://github.com/dagger/dagger/releases/tag/v0.18.14

$

Meanwhile, the output shows up in Dagger Cloud just fine.

Note that I simplified your code and simply appended .stdout() to the very last command in my .with_exec() chain. Also, when .with_exec(…).stdout() gets executed inside a custom span, not even the ▶ .stdout: String! 24.9s part shows up on the command line.

[noble apex]

Weird, I just tried this: The output appears briefly on the command line and then disappears again once the pipeline terminates since at this point all pipeline steps get collapsed:

In my case I'm running with -v si it shows the collapsed spans.

Note that I simplified your code and simply appended .stdout() to the very last command in my .with_exec() chain.

that's fine, as long as the last command is what you want to see. The previous login and stack won't show any outputs if they're cached.

[mental matrix]

In my case I'm running with -v si it shows the collapsed spans.
Ah! Thanks, I'll give this a shot next week!

[mental matrix]

Works like a charm, thanks again!

Then again, once https://github.com/dagger/dagger/issues/10783 gets implemented, I suppose this would impact the behavior of stdout()? Namely, its presence/absence wouldn't really make a difference anymore, since the output of all commands would always be available and shown by dagger run -v. However, I only want to show the output of selected (important) commands.