Export directory when catching ExecError

[mighty barn]

Hi,
it's me again!
I'm trying to catch ExecErrors when running my pipeline (which builds and runs a bunch of dotnet things), and export a "snapshot" of the current state of the container filesystem at time of error for debugging purposes. But for some reason, I can't get it to work.
I've read https://github.com/dagger/dagger/issues/4706#issuecomment-1571677488, https://github.com/dagger/dagger/pull/5184 and everything related, but I might be doing something wrong

try:
    await test_pipeline.exit_code()
except dagger.ExecError as e:
    await test_pipeline.directory("/src/temp/").export(
        f"test/test_runs/foobar"
    )
    # you can choose what to do with the error
    # (e.g., abort, log, ignore...)
    # e.message
    # e.exit_code
    # e.stdout
    # e.stderr
    # e.cmd
                    ...

I can catch the error, but the export is not working. Is this even possible? Please let me know if you need further information, sadly I cannot simply copy-paste the output because of corporate stuff in there.
Thanks in advance!

[spiral badger]

Hi. What you're doing feels intuitive but we need to document it better. It's been discussed before, maybe we're missing a proper issue for it. The TLDR is that since test_pipeline failed, you can't run anything after it which will trigger the same error. A workaround for that is to run your command in a shell to always exit with 0 and save the exit code to analyze later.

This user has a similar use case to yours. His command generates a report that's needed weather the command succeeded or not: https://discord.com/channels/707636530424053791/1102535114937860117

This user already was doing this but wasn't aware that doing this caches the failure: #1111300802338881646 message

While this user was just getting a trimmed output on errors: https://github.com/dagger/dagger/issues/4979#issuecomment-1557046765
Look at the "Stop gap" section. If you don't mind caching the failure, that's what I'm suggesting here.

In your case, what you really want is https://github.com/dagger/dagger/issues/4463 (you can lobby your need for it there). But until then, maybe there's something else you can do to debug without caching the failure. Doesn't the logged output give you enough information?

[mighty barn]

Thanks @spiral badger , that's very helpful!
I'll take a look around and see if I can adapt one of the suggested alternative solutions.
The logged output is sufficient in like 99% of the pipeline failures we encounter, but since there a lot of dynamic file operations going on in the pipeline, it would have been easy to "just a look around" the filesystem in case of failure. It definitely seems like https://github.com/dagger/dagger/issues/4463 is what I'm looking for, you're right!

Maybe the logical thing to do for my use case would be to provide a wrapping function similar to the solution suggested in https://github.com/dagger/dagger/issues/4979#issuecomment-1557046765 and use it when debugging a specific failing exec locally!

[mighty barn]

I would like to provide a function which can be chained using .with_(function) which does the following (similar to the ideas above):

• take a command, open a new shell, run command, pipe output to new file /exit_code, and return
• export a directory in case content of exit_code file is !=0, and raise ExecError in this case
• return container otherwise, so the remaining pipeline can be finished.

But I'm failing again. Excuse my ignorance...
.with_(cb) expects a callable. What I usually do is one of the two variants suggested in here https://github.com/dagger/dagger/issues/5236).
but obviously, I need some modifications for it to work in my context..

async def export_snapshot(
    cmd: str,
    src_path: str,
    export_path: str,
) -> dagger.Container:
    def _export_snapshot(ctr: dagger.Container) -> dagger.Container:
        return ctr.with_exec(["sh", "-c", f"{cmd}; echo $? > /exit_code"])

    ctr = _export_snapshot
    exit_code = await ctr.file("/exit_code").contents()
    if exit_code != 0:
        await ctr.directory(src_path).export(export_path)
        raise dagger.ExecError
    return ctr 

Obviously I would need this function to be async, because I have some some coroutines to await.
This raises AttributeError: 'function' object has no attribute 'file', which makes sense since I'm just passing a reference with _export_snapshot, so I'm kinda lost..

[spiral badger]

Ah, that's unfortunately not possible right now using with_, because it expects a normal function and to do what you want you'd need to move all that logic inside _export_snapshot since that's where the dagger.Container from the pipeline comes from. However, you can just break the chain here:

async def export_snapshot(
    ctr: dagger.Container,
    cmd: str,
    src_path: str,
    export_path: str,
) -> dagger.Container:
    ctr = ctr.with_exec(["sh", "-c", f"{cmd}; echo $? > /exit_code"])
    exit_code = await ctr.file("/exit_code").contents()
    if exit_code != "0":
        await ctr.directory(src_path).export(export_path)
        msg = f"Command {cmd} failed with exit code {exit_code}"
        raise Exception(msg)
    return ctr

And in your pipeline:

ctr = client.container()...
ctr = await export_snaptshot(ctr, "my cmd", "src", "dest")
...

[mighty barn]

As usual, thank you very much @spiral badger !
Thats what I assumed, just wanted to make sure I'm not missing anything..
Does breaking the chain have any performance impacts btw.? Not that it matters, I'm just curious.

[spiral badger]

No, just DX.

The reason with_ doesn't support async is that it would break the chain, or at least make it pretty awkward. Let's say you want stdout in the end:

out = await (
  (
    await (
      client.container()
      .from_(...)
      .with_env_variable(...)
      .with_mount(...)
      .with_aync(export_snapshot("my cmd", ...))
    )
  )
  .with_exec(...)
  .stdout()
)

Might as well break:

ctr = await (
  client.container()
  .from_(...)
  .with_env_variable(...)
  .with_mount(...)
  .with_aync(export_snapshot("my cmd", ...))
)
out = await (
  ctr.with_exec(...)
  .stdout()
)

But if you're already breaking it, then there's not much point to with_.