#Buildkit Parallelism with Dagger

1 messages · Page 1 of 1 (latest)

marble field
#

Hey yall, loving the progress on the go sdk. Just checked the demos from last weeks' community call, and of course I had to try it myself. Things look good! Loving the native experience.

I've got a question that I think relates to dagger/LLB.. Do I need to do anything special to execute parallel steps? I have a snippet in the golang sdk that looks like below, and the steps do not execute in parallel.

// initialize new container from image
golang := core.Container().From(golangImage)
golang2 := core.Container().From(golangImage)

// mount working directory to /src
golang = golang.WithMountedDirectory("/src", src).WithWorkdir("/src")

// execute command
cmd := golang.Exec(api.ContainerExecOpts{
  Args: []string{"/bin/bash", "-c", "sleep 10 && date"},
})
cmd2 := golang2.Exec(api.ContainerExecOpts{
  Args: []string{"date"},
})

I put sample code and a walkthrough here. It shows a similar setup in multi-stage docker that does execute in parallel

white lark
#

@marble field in order for things to happen in parallel you need to use your language's native concurrency / parallelization primitives. In the case of Go, you need to change your code to use goroutines for that to happen

glass tree
white lark
#

Yep, correct Jeremy. Waitgroups, Errorgroups and/or channels are all a good option

marble field
#

Yeah so I think it ended up working ok w/ goroutines. Output is ..slightly confusing (with the ordering of numbers, note that step 15 output appears before step 15)... but I think that's ok.

...
wg.Add(2)
go sleep("1", container2)
go sleep("11", container)
wg.Wait()
...

results in below


QUERY: query{container{from(address:"golang:latest"){exec(args:["/bin/bash","-c","sleep 1 && date"]){stdout{contents}}}}}
QUERY: query{container{from(address:"golang:latest"){exec(args:["/bin/bash","-c","sleep 11 && date"]){stdout{contents}}}}}
...
#14 /bin/bash -c sleep 11 && date
OUT After 1 Seconds: Mon Oct 17 21:52:40 UTC 2022

#14 ...

#15 /bin/bash -c sleep 1 && date
#0 1.084 Mon Oct 17 21:52:40 UTC 2022
#15 DONE 1.7s

#14 /bin/bash -c sleep 11 && date
#14 11.07 Mon Oct 17 21:52:50 UTC 2022
#14 DONE 11.1s
OUT After 11 Seconds: Mon Oct 17 21:52:50 UTC 2022

I think that without cache, this results in both queries pulling the image down, not sure if buildkit has some magic to resolve that. But again, that doesn't seem super important.

uneven rover
#

If I recall correctly, buildkit has this magic, and only the 1st one will pull the image, and the other one will be waiting on the result of this 1st one.