Starting off with Dagger - trouble defining structure and what 2 split-up

[tough drum]

So I'm trying to start off with dagger but 1 of the things that's quite unclear for me at the moment is how to structurize all of it.
I run alot of different CI's but dagger has my interest, my favorite for now is actually Tekton and the reusability of tasks.
I'll try 2 describe kind of the different steps I currently have for my buildprocess, and BTW I know the branching strategy could be better 🤫
What I mainly want to know is do I try to put the different scenarios into 1 large container each, do I spin up multiple containers 1 for each step (had alot of issues passing the information from 1 container to the next this way).
And also if you know good examples of dagger pipelines written in go (especially with multibranch - semantic versioning, ...)

Pipeline Flows

Main Branch

KIND: BUILD-DEPLOY
NOTE: Build & test code, publish a new tag, deploy the new tag to TST.

Pipeline:

• prepare
  • preflight:check:env:vars:build
  • preflight:check:env:vars:deploy
  • semantic-release:find:next:version
• build
  • build:image
• publish
  • semantic-release:publish
• test
  • lint:eslint
  • lint:styleline
  • scan:trivy
  • scan:php:cs
• infra-update
  • update:infra:repo (TST)
• infra-sync
  • deploy:argocd:sync (TST)

F / B / H / R Branches

KIND: BUILD

NOTE: Build & test code, publish a new tag, no deployment

Pipeline:

• prepare
  • preflight:check:env:vars:build
  • semantic-release:find:next:version
• build
  • build:image
• publish
  • semantic-release:publish
• test
  • lint:eslint
  • lint:styleline
  • scan:trivy
  • scan:php:cs

[gloomy galleon]

Hey @tough drum ! welcome to the community and thx for checking out Dagger.

One question before jumping into the suggestions, where are you planning to run the dagger pipelines? Would you still use Tekton as a CI runner as described here https://docs.dagger.io/ci/integrations/tekton/ ?

If you haven't checked this part of our docs yet (https://docs.dagger.io/ci/adopting/#phase-2-the-poc) I'd strongly advise you to. It basically covers most of the concerns that you're asking while providing an overall guidance on how to maximize your approach while adopting Dagger for you, your teams and your organization

[tough drum]

That's a bit why i was leaning towars dagger, the fact is that we currently have alot of ci systems.
jenkins being the main one, gitlab CI, github actions, tekton, azure pipelines, ...
We often get the question of customers to use their GIT of choice and also sometimes to use their CI system, ... So it's kind of maintenance hell at the moment.
yes i did read the docs, but still would appreciate some insights on how to best structure it aiming for a high reusability.
Just a matter of not having to re-structure everything after a couple of months., I guess you already know how much trouble it is getting time to implement changes and how much backlash it creates if you have to re-structure shortly after...

The scenario I showed is one of the simplest we have at the moment TBH and is somewhat the MVP to get it sold.

[gloomy galleon]

That's a bit why i was leaning towars dagger, the fact is that we currently have alot of ci systems.
jenkins being the main one, gitlab CI, github actions, tekton, azure pipelines, .

cool, perfect use case for Dagger.

but still would appreciate some insights on how to best structure it aiming for a high reusablity.
Just a matter of not having to re-structure everything after a couple of months., I guess you already know how much trouble it is getting time to implement changes and how much backlash it creates if you have to re-structure shortly after...

architecting Dagger pipelines is not so different to architecting a traditional app to be honest. When it comes to composability and scalability, you'll realize as you start writing your modules that you can refactor certain common logic in re-usable functions which you might want to be able to make public via dagger call directly.

Once piece of advise I'd personally recommend is to start principally with a single module in a repo and then after having a MVP of your pipeline, think about if it'd make sense to refactor some of that logic into separate modules that could be use somewhere else.

In effect, the way you architect your pipelines and your modules will also be very influenced by your organizational structure and how you're envisioning your teams to be able to consume them.

If you're looking for inspiration on how a more complex project uses Dagger, our main dagger/daggerpipelines (https://github.com/dagger/dagger/tree/main/.dagger) are quite comprehensive and have been througout a series of improvements and refactors which got is a to a pretty good spot nowadays

As a wrap up, my recommendation is to focus on that "hair on fire" problem our docs mention and start from there. You'll soon realize that as your pipelines get bigger, you'll find yourself having different realizations on how you could refactor them to improve their reusability.
In any case, if you have any particular questions about a specific approach, you can always reach out so we can work together to find the best solution to your use case

[tough drum]

okay so if I understand it correctly; try to keep your logic / flow in a single docker unless it makes sense to run it at the same time.
for the reusability part you're supposed to write modules (or use the build-in / daggerverse ones).

I think I was a bit stuck in my way of thinking to convert current steps to also seperate them and call them individually from the ci system. but I see now in the repo that you've send me that the entire logic is in 1 single go file.
Thanks you're explenation helped me alot. 😉 .

[gloomy galleon]

okay so if I understand it correctly; try to keep your logic / flow in a single docker unless it makes sense to run it at the same time.

what do you mean by a "single docker" specifically?

[gloomy galleon]

but I see now in the repo that you've send me that the entire logic is in 1 single go file.

the main Dagger pipeline entrypoing is within a single file, but the complete list of operations that you can perform is scattered across different files and modules within the same repo. For example, here's the list of re-usable modules that the main Dagger pipeline uses: https://github.com/dagger/dagger/tree/main/modules