final owl
c := client.Container().From("alpine")
new1 := c
new2 := c
c = c.WithExec([]string{"touch", "file"})
new1 = new1.WithExec([]string{"touch", "file1"})
new2 = new2.WithExec([]string{"touch", "file2"})
Will there be 3 different containers each having 1 file created on it?
I want to create a container will deps installed
then create 3 new ones based on this and run a build on each (different arch).
in parallel
Or do retries based on the "base container" on each retry
mossy spade
```go
c := client.Container().From("alpine")
new1 := c
new2 := c
c = c.WithExe...
Yes, c, new1, and new2 are each their own container with 1 file. Depending on how the dag is structured, they may be built in parallel. For example, this pipeline https://docs.dagger.io/cookbook/#perform-a-matrix-build each container is automatically built in parallel because the final result of the dag is a directory that depends on each container that contributes to it, so the engine can automatically parallelize those container builds. It's a bit complicated 😅
final owl
Mine is "simpler"
from a step and afterwards it just fan out.
No fan in again
I had to implement a retry decorator func. cause cloudflare api is sometimes unstable.
Instead of having to trigger the pipeline again I just retry 3 times.
I get a "snapshot" and reassign the snapshot to a new variable on each try.
I return early if it succeeds
otherwise I return the last attempted container (to gather error later on)
Example of a result of the current pipeline
And regarding speed/perf:
Current pipeline:
Dagger pipeline (currenlty in testing):
Eventuially those 4 nodes will become ubuntu/dagger runners
I dont expect less than 5min cause node is slow to build.
But will have more room for other pipelines to run.
I still need to tune GC on cache, cause I have small disks at the moment
300gb and are getting filled too soon.
mossy spade
Very cool!
The fan-in part of the example I linked is what makes dagger automatically build each container in parallel, but there's certainly other ways of parallelizing.
The pattern in your original snippet is pretty common in dagger pipelines actually because you might have some common part of a container that you'll use with some modification throughout your pipelines. So by defining it once you simplify your code but also improve the performance of all of the pipelines because they're reusing a common resource. For example, I might define a base container with a set of dependencies installed, and then refer to that base from my test, lint, and build pipelines where they may modify the container further and then run commands.
Thinking about what you're saying for retrying, depending on what the actual limitation is, you may not necessarily need a new variable. Like if you just need c to successfully build, it might just look something like
supple raft
c := dag.Container().From("alpine"). (all the extra business logic here)
success := false
while !success {
c, err := c.Sync(ctx)
if err == nil {
success = true
}
}
@hushed pebble will probably like to know more about this too!
final owl
Wait if I run sync again, it reruns the steps?! (or from the last succesfuly sync?)
I do sync after each step, so I can get output etc
mossy spade
It won't rerun the steps if they were successful, but if they failed for some reason and you need to retry, a sync will work