woven viper
I have to do a large overhaul of my pipelines. It seems like a great time to adopt Dagger. But every time I investigate, I hit the same concern...
My Github pipeline is optimized using parallel jobs, reusable workflows, and matrixes. I see how you can describe this behavior in Dagger... But how well does Dagger's parallelization perform compared to Github workflow's methods? When is it better to use each?
molten prawn
I don't think you would stop using github's features like that. Use Dagger (if it suits your needs) to run the pipeline logic, but use Github to trigger the (combinations of) pipelines you need. This way you keep the optimisation work you've already done, but access the features and benefits of Dagger
woven viper
Thanks. But Dagger looses a lot of it's shine if I can't use it—at minimum—to run concurrent jobs as efficiently as github does.
urban brook
This is sort of the conundrum I'm seeing too. I still don't get where automation of CI starts and where Dagger starts and where or when they both end.
deep mist
@woven viper our short answer generally is: paralellize as much as you can with Dagger and use your CI orchestrator paralellization features when you specifically need them like spawning a specific architecture VM or things like that.
The main reason for this is that Dagger is extremely well optimized for running things in a single host due to its caching and de-duping capabilities. In other CI systems like GHA you're kind of forced and "encouraged" to use multiple jobs since the ability to parallelize within a job is quite poor and also not economically "convenient" for them (https://github.com/orgs/community/discussions/8726).