Shell job control and variable assignments / value returns?

[pearl magnet]

Is there any way to actually reference the result of something done in the background (&) in the shell after awaiting it ?
E.g. something like this:

( hi=$( container | from alpine | with-exec echo hi) ) &
( bye=$( container | from alpine | with-exec echo bye) ) &
.wait
$hi | stdout
$bye | stdout

[graceful bone]

using functions and the power of the Dagger cache ™ you should be able to do something like this:

#!/usr/bin/env -S dagger shell --no-mod



build() {
    container | from alpine | with-exec echo build
}

lint() {
    container | from alpine | with-exec echo lint
}

build &
build_pid=$!

lint &
lint_pid=$!


.wait $build_pid $lint_pid

build | stdout
lint | stdout

[pearl magnet]

ah ok 😅

I happen to have explicit cache invalidation markers in some of my withExecs in the actual use-case here, to make sure they always run fresh (because they depend on external calls that might have new data), so this won't work for me unfortunately 🙁

[graceful bone]

you could parametrize those cache markers maybe? i.e:

#!/usr/bin/env -S dagger shell --no-mod

cache=$RANDOM

build() {
    container | from alpine | with-env-variable CACHE $cache | with-exec echo build

}

lint() {
    container | from alpine | with-env-variable CACHE $cache | with-exec echo lint
}

so if your module receives a --cache-key argument it can use that or otherwise generate a new one on each run

it's tricky to do this in some other way because bash doesn't have a native way to achieve this and the way people usually resort to is to use mkfifo or output things to files which is 😬

[pearl magnet]

yeah the cache key would work

output things to files
this is what I'm probably gonna do 😅

in my actual use-case the results are dagger.Directory so I think I'll just export that and then pass $(host | directory ...) in places where I would have used the variable (/ second function invocation)

[graceful bone]

well.. in this case it's tricky because you can't output the state of a pipeline to a file and "import" it afterwards to keep calling chainable functions

[graceful bone]

this works but it would have a performance penalty of exporting / importing directories on each step

[pearl magnet]

I'll test ofc, but do you expect this to be actually significant ? it's a tiny (20K maybe?) dir

...and wouldn't the cache hit technically also be a disk read?

[graceful bone]

if it's a small directory it should be fine. I can't exactly recall if exporting and re-importing files / directories within the shell actually works. I think it does for what I remember

[pearl magnet]

yeah all that works fine 🙂

thanks for the insight ! 🙏