#future of generated github workflows
lethal creek
So, I've been struggling with this recently.
I love the idea of generating .github using modules/gha, but we never really fully integrated it, there are still workflows that don't use it - additionally, we never really got any users on it, so we're actually using dagger in CI entirely differently than all our users, which feels like a missed opportunity.
One of the core ideas was to make it easier for everyone to hack on this part of dagger, but I think it's made it harder, because modules/gha is another thing to keep in mind.
Additionally, because we're using this approach, we're actually neglecting dagger-for-github because we're not dogfooding it. There are bugs in it, features missing (no shell support), etc.
Me and @empty edge have been working on how to integrate PARC into dagger/dagger - and this is kind of requiring some extensive changes so that we can do it. It feels like we're at a fork, where we either:
- Leave things as they are. Things are split, and integrating PARC changes takes twice as long going forward.
- Double down on the
modules/ghaapproach. Convert everything, iron out every problem with it, and try and get some users! Then use this to ingrate PARC. - Switch things back to having hand-written YAML files using our
dagger-for-githubaction - we can use composable actions to remove boilerplate, and dagger shell should help us write cleaner actions. Then integrate PARC here, simulating an experience that we hope our end-users have.
I'm really leaning towards the latter - it feels a lot easier, decreases our maintainance burden, and helps improve velocity by just having fewer moving parts. Additionally, it's just much easier to do (I'm guestimating a morning, instead of several days and bikeshedding over the modules/gha approach).
It's a really cool experiment - I think we should have checked our success earlier, but better late than never.
interested in feedback especially from @tulip swift who worked on this in the first place, but also from the rest of the team: would it make easier to hack on our ci if we moved more github-yaml, or more generated github-yaml?
exotic sedge
reading this, i can't help but think that native CI is a massive elephant in the room here... like if we want to run at building a hosted event-driven "continuous thing do-er" (✈️) option 3 feels even more obviously like the right thing to do -- reduce how much of the GHA surface area we encode with the intent of starting to replace it
lethal creek
hard agree on wanting to use our own "product" for this and eventually stop touching github actions at all - i think it's about finding the right path there
wary shard
I totally agree with @exotic sedge if we want to put the bricks for native CI, we should start from the current user experience and see how we can improve that experience while developing the native CI product so we replace the yaml with our product.
That way we can dogfood efficiently. Also our CI is pretty huge so if things works well with our repo, we can be confident for the future.
Doubling down on modules/gha doesn't loook like a good idea, what is the value people are getting from that? Would that lead to new customer?
IMO the right path is PARC, which can naturally become a product people can pay for. That said, the entrypoint to call PARC outside of local would be from people's CI script, which is were a company can get the most value from once we solve these cache & performances issue.