Child pipelines

[ruby hollow]

Is the "Dagger-way" for child pipelines to build an image with the child pipeline and then run that within the parent? Or is there some simpler way to do it. Do you think this kind of recursion makes sense and works? We have a use case where we need to have versioned pipelines for each environment but we also have some common jobs.

[cloud hinge]

Hi @ruby hollow 👋 IIRC you're using the Go SDK, right?

Could this repo offer some inspiration?

It has some different sub/child pipelines that could be could be invoked however you need as Go functions. Here's a function called Ci() that calls Lint(), Test(), and Build() functions, for example.
https://github.com/kpenfound/greetings-api/blob/main/ci/tasks/ci.go

Each of those funcs has a Dagger pipeline in it. Like Lint() for example: https://github.com/kpenfound/greetings-api/blob/main/ci/tasks/lint.go

👇 This one is even a bit easier to see. All in one file 😎
https://github.com/kpenfound/greetings-api/blob/d4gh_demo/ci/ci.go

It's using the new Pipeline construct to name the sub-pipelines.

[ruby hollow]

Hey, thanks for quick answer.

Yes we are using Go SDK. Our use case is bit more complex than that. We have one general pipeline that is ran for all the envs but then each environment has different kinds of dependencies and sometimes those dependencies might not be backwards compatible with each other or our pipeline code. This means that we would need to lock our environment specific to certain version of pipeline so we can't just call functions in our master-branch.

So our idea is that we have a sub-pipeline that is then built into a container, this way we can run version 1.2.3 of our pipeline for prod and version 2.0.0 for uat. Then from our main-pipeline we start these sub-containers with whatever version they are locked to. Also we are planning to run these sub-jobs concurrently.

So I think my main question is will Dagger-engine handle situations where we launch a container that is running Dagger-code from within a Dagger-pipeline?

Can you catch the idea from here?

[dim grail]

Yes, dagger-in-dagger is possible. You just need to enable the experimentalPrivilegedNesting flag in WithExec and that will allow you to trigger Dagger pipelines within other pipelines 🚀

does that help in your case @ruby hollow ?