Dagger, and work sessions

[charred tinsel]

zellij / tmux / screen are persistent (effectively nohup'd) so they'll stay running if the connection drops.. but.. i can't see how to reconnect to the session?

[rain thorn]

the thing is that once you detach the program from the tty, I think the dagger pipeline will continue executing and very likely will end

since terminal has been mostly designed for troubeshooting purposes and not intented for long unattended programs

it actually panics 😅

.withExec(args: ["apk", "add", "tmux"]): Container!

[detached (from session 0)]
exit 0
Caught panic:

runtime error: index out of range [115] with length 115

Restoring terminal...

goroutine 1 [running]:
runtime/debug.Stack()
        /usr/lib/go/src/runtime/debug/stack.go:26 +0x5e
runtime/debug.PrintStack()
        /usr/lib/go/src/runtime/debug/stack.go:18 +0x13
github.com/charmbracelet/bubbletea.(*Program).recoverFromPanic(0xc00074c000, {0x1aece00, 0xc000eac138})
        /go/pkg/mod/github.com/charmbracelet/bubbletea@v1.3.5/tea.go:810 +0xac
github.com/charmbracelet/bubbletea.(*Program).Run.func

cc @simple bridge

dagger -c 'container | from alpine | with-exec apk add tmux | terminal --cmd tmux '

^ and then detach from that session

[charred tinsel]

I don't think I'm thinking about this correctly. Maybe a better approach would be to expose ssh with AsService, and then ssh -> terminal mulitplexer

[rain thorn]

what are you trying to achieve?

getting into a working session that you can eventually resume?

the thing is that none of the things that you do in that working session will be permanently stored given the core primitives of Dagger

[charred tinsel]

the idea is to have a dedicated env for a given task. currently if the connection drops: the session is thrown away mid-task. At the end of the session I usually decide, throw this whole thing away, or selectively preserve portions (git commit / push) to used for future sessions

actually what does happen to the image when | terminal ends?

[charred tinsel]

so it's not so much resume, as, not give up on, without notice

[rain thorn]

I'm thinking how different this would be from running dagger in the background like nohup dagger -c 'whatever' & and then fetching the result from that later once it has finished

[charred tinsel]

currently my dagger client is local, and I'm running the engine on aws under docker: so there's lots of ways to lose my connection.

The client connects with ssh over "aws ssm".

I get this intermittently (sometimes after minutes, sometimes hours):

Full trace at https://dagger.cloud/cross-stream/traces/7901d4a49b59b5b5bd171b8d6fadf1e9
Error: Post "http://dagger/query": command [docker exec -i dagger-engine-v0.18.10 buildctl dial-stdio] has exited with exit status 1, make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=command [ssh -o ConnectTimeout=30 -T -l root -- i-0c399fef00acac0cb docker system dial-stdio] has exited with exit status 255, make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=mux_client_request_session: session request failed: Session open refused by peer
ControlSocket /Users/andy/.ssh/cm-root@i-0c399fef00acac0cb:22 already exists, disabling multiplexing
Bad packet length 565646221.
ssh_dispatch_run_fatal: Connection to UNKNOWN port 65535: Connection corrupted

[rain thorn]

yeah.. if the engine and the client have a connection error, the pipeline will be aborted. On the other hand, everything will be cached up to the point before the pipeline finished to it should be safe to re-run the pipeline in this case and it should resume from where it left

[charred tinsel]

Just confirming: it won't include the changes made during the interactive terminal session?

[rain thorn]

that's correct. terminal sessions steps are not part of the DAG

[charred tinsel]

I'm thinking about a flow, something like ... | terminal | publish registry:5000 <session-id> -- I could then use container | from <session-id> | terminal | publish registry:5000 <session-id> to resume

[rain thorn]

yes but whatever you do with the terminal, it won't be saved in the container that you end up pushing

[charred tinsel]

oh! so the container that's returned is the container that was piped intoterminal?

[rain thorn]

yes, exactly. Terminal will return the same container you had before calling it. Whatever change you do in there, it won't be persisted

unless you use a cache volume and persist the changes there but cache volumes are "best effort" ephemeral caching which could be garbage collected