hey folks! trying dagger for the first | Dagger | Page 1

rapid sparrow Nov 12, 2023, 11:12 PM

#

is there a good pattern for "putting a synchronization barrier" when there is no real dependency between containers? e.g.

#

say I have a couple "stages" in my pipeline ("checks") which I'd like to running concurrently, but then a further one that I'd only like to see happening after those initial ones are done

#

  CHECKS                    BUILD


  (container1)
    vet        -----.
  (container2)      |       (container4)
    test            +-->  image build&push ---> ....
  (container3)      |
    lint       -----'

#

i can totally go with the Go way doing some errgroup on vet, test, and lint, calling a .Sync() on each, then after that block is done (.Wait() on the group) going forward with the container4 (.DockerBuild().Publish() ...), but, if I wanted to offload that flow control entirely to dagger, is there a good way of doing so?

#

my initial thought is that yes it's possible, but maybe not great by forcing container4 to copy a file from each one of those containers (pushing the flow control to the DAG solving), but I'm not entirely sure if there's a better primitive/way (or if this is actually not the proper way of thinking about the problem at all)

with caching in mind, not even sure what I'd write to those files 😅

#

thank you!

unreal moat Nov 12, 2023, 11:23 PM

#

hey hey! afaik (and I just landed from a 2 week vacation so maybe there's been some activity/discussion), the state of the art is captured in this issue: https://github.com/dagger/dagger/issues/5083

tl;dr current approach is to just use the language's native facilities as you mentioned, or otherwise yea force some kind of dependency between them to form a DAG that gets executed with the desired parallelism. but we are interested in figuring out how to make this first-class, since there's varying amounts of complexity/cludgery involved in each language. just hasn't made it to the top of the priority list lately 🙂

#

there might be overlap here with higher-level concourse-style artifact pipelines too, though I'm hesitant to jump to that conclusion for the sake of it

rapid sparrow Nov 12, 2023, 11:26 PM

#

oh sweeet, I'll take a look at the discussions there!

#

haha yeah I've been trying to not put too much concourse bias into it, but instead mostly thinking as if I was writing a multi-stage dockerfile, it's been quite fun approaching the ci problem with a different mindset

#

(fwiw leaving some notes here https://github.com/cirocosta/dagger-notes as I go, not structured at all but it's been interesting digging into what you all built, really fascinating)

GitHub

GitHub - cirocosta/dagger-notes: nothing to see here - wip

nothing to see here - wip. Contribute to cirocosta/dagger-notes development by creating an account on GitHub.

unreal moat Nov 12, 2023, 11:29 PM

#

oh nice, will check that out!

rapid sparrow Nov 12, 2023, 11:30 PM

#

👍
heading to dinner, will tinker more throughout the week (and be around here too haha), have a great one @unreal moat !

unreal moat Nov 12, 2023, 11:38 PM

#

You too! Good to run into you again 👋

still heath Nov 12, 2023, 11:50 PM

#

Hey @rapid sparrow ! Great to see you here. Your xfs docker volume plugin has helped me countless times 🤗

left onyx Nov 13, 2023, 2:22 AM

#

Hey @rapid sparrow, nice to see you spending time trying Dagger, thanks for the notes 🙂

#hey folks! trying dagger for the first