pipeline definition for multiple projects/repositories

[covert plume]

Hi, I have just discovered Dagger and I'm having a small understanding problem.

what I do understand and tested so far:

• how to call modules with the cli (pass args, returns, containers ..)
• how to init a custom module with a custom functions + how to call/integrate modules from the daggerverse into my custom functions

what i sadly don't understand (based my reasearch):

• how to create an actual pipeline which is executing a set of tasks in order (stages) out of individual functions. or even in parallel like 1: lint + build + test 2: build image 3. scan image 4. publish .. (should this be done in a module or as a extra golang module or is this the wrong approach here).
• how to do put this modular pipeline definition in a central repository that e.g. 3 different golang based microservices can consume the same definition local and on a ci system like github.

thank you 🙂 i am really enjoying dagger so far 🚀

[hollow aurora]

Hey!

Check out my open source go project for an example of this in a few different CI systems

https://github.com/levlaz/snippetbox

this shows both how to make a pipeline programatically, and how to make one using the CI yaml config

how to do put this modular pipeline definition in a central repository that e.g. 3 different golang based microservices can consume the same definition local and on a ci system like github.

For something like this, assuming the pipeline does more or less the same thing for a given codebase I'd suggest creating a repo to hold your dagger module and then passing in the project repo as an argument

[covert plume]

hi @hollow aurora thanks! so what I understand from .dagger/main.go and the pipeline files (great collection!!) you are running dagger call ci, which is a function and your wrapping eveything in there. habe you executed dagger functions in parallel, is thsi even possible? thanky for your hint with the repo. so I do understand now everything is about modules and functions in dagger 🙂

[hollow aurora]

Yes its possible to do things in parallel using the built in features of go.

A good example is in dagger/dagger - the Lint function does 4 things in parallel using the built-in errgroup in go

https://github.com/dagger/dagger/blob/918cfde1e9a9d67dc90fc6d0257416f817dad581/.dagger/docs.go#L63

Sometimes the parallelism happens automatically too based on the shape of the DAG

For example, in this go matrix built all the items in the for loop are going to be built in parallel

https://docs.dagger.io/cookbook#perform-a-matrix-build