Generic pipeline | Dagger | Page 1

glossy sleet Dec 15, 2025, 7:55 AM

#

Im trying to create generic pipeline for our company.

We have > 200 repos, and while most are simple and could use directly some module, there are some that need to do extra steps to build their product, like installing extra things on build image (not just standard apt-get), or doing some extra tests using some complicated docker setup, or even generating few extra docker images as a result.

All that extra steps canot be directly done "outside" our main pipeline without copy/pasting a lot of code.

Our current approach is to have generic pipeline, that allowed extra steps/configuration at most of the steps of pipeline..

Our steps looks more/less standard.

prepare build image
build using prepared image
test using prepared image
create run image
push run image

You could redefine each step totally, or just add your own code snippets at various points of the steps.

Any idea how to make it with dagger, to not force users to copy/paste too much while still allowing for doing most of the things with ease.

dusky prism Dec 15, 2025, 11:46 AM

#

Hey @glossy sleet!

Like you mentioned I think it depends on what kind of extra steps the teams would need to make. An approach we've been working on at our own CI in github.com/dagger/dagger is an experimental feature we call "toolchains". Toolchain modules essentially provide functions that, when installed on another module, belong to that module as if it were its own. If I have a toolchain that has a Lint, Test and Publish functions and I install it in my awesome-app module under the name generic then you can call those same functions with dagger call generic lint. This means that most teams could leverage the existing functions provided by the toolchain but still write their own (maybe even re-using some of the functions of the toolchain) when they need to.
This is an experimental feature we are still working on, which is why you'll find it as a hidden command under dagger toolchain.

Curious to see if something like this would work for you! Happy to explore other options if it doesn't sound like the right fit

glossy sleet Dec 15, 2025, 12:10 PM

#

Thanks @dusky prism.
I looked at it already, seems neat, but I dont think it scales enough..
Below I will be thinking loud, maybe I will be saying some stupid things (just started with dagger), so please point the places where im doing so..

For example, lets assume my team needs some extra tests, they would write own test target and thats fine, but then:

at CI level I have to somehow choose proper "test" module, (I could always check functions and if not provided use generic), which is doable, but users locally would need what target to choose when trying to run test
Its hard to parallelize it because they are different shell commands, you need to run one after each other.. and its hard to run tests and build at the same time (maybe doable on CI but tricky)
We have part of the tests that are "system" tests (that we want to run always), I could probably extract them as own tool

That was simple example that would work somehow..
But what if a team needs to do some extra magic to prepare base image for build/tests etc:

I could add option for all the function to get "input" container (already inited), but I dont think that works with "toolchains", so we would need to revert to "dependencies"

BTW how much slower will be if I will run ~ 5 dagger commands (for each step), than just 1 with full pipeline? With remote runner?
I feel that each dagger command have some "static" (not that small) delay..

#

Below is my idea (seems impossible for now with python at least):

Create generic module (with many interfaces for different purposes like the one provided)

@interface
class InitPreparer(typing.Protocol):
    @function
    async def prepare(self, cont: dagger.Container) -> dagger.Container: ...

@object_type
BasePipeline(DefaultBasePipeline) # I want to make it based on simple python interface so clients will have the same functions out of the box
  source: Annotated[
        dagger.Directory,
        Doc("The context for the workspace"),
        DefaultPath("/"),
    ]
   ...
    init_preparer: Annotated[
        InitPreparer | None,
        Doc("Optional provisioner to prepare the build environment"),
    ] = None

    @function
    async def build_all(
            self,
    ) -> dagger.Directory:
      ...

    @function
    async def prepare(
            self,
    ) -> dagger.Container:
       ...
       if self.init_preparer: # If we provided prepared just use it
         container = await self.init_preparer.prepare(container)
       ...

#

Now client side (using that module up):

@object_type
class AlertManagerPreparer:
    """Provisioner for alert-manager builds."""

    @function
    async def prepare(self, cont: dagger.Container) -> dagger.Container:
        """Prepare the base container with necessary tools and dependencies."""
        return (cont
                .with_exec(["apk", "add", "--no-cache", "git", "openssh-client", "ca-certificates"]))

@object_type
class AlertManager(DefaultBasePipeline):
  source: Annotated[
        dagger.Directory,
        Doc("The context for the workspace"),
        DefaultPath("/"),
    ]
   ...
   
    async def generate_build_pipeline(self):
        return await dag.base_pipeline(
            source=self.source,
            init_preparer="somehow pass AlertManagerPreparer" # seems impossible for now in python
        )

    @function
    async def build_all(
            self,
    ) -> dagger.Directory:
        return self.generate_build_pipeline().build_all()

outer nebula Dec 16, 2025, 7:59 AM

#

so you want to generate the code of your dagger functions?

glossy sleet Dec 16, 2025, 9:11 AM

#

No..
I want to have generic pipeline and just allow my users to overwrite what they need..

But yesterday I found out maybe i dont need "dagger" modules at all.
Just keep everything as plain python modules and just do all the magic at that level..
That would allow me to redefining all the methods, and add all the entrypoints I need easily..
Just need to confirm that everything will work as I need it..

quaint island Dec 16, 2025, 12:46 PM

#

@glossy sleet toolchains already provide a mechanism to achieve what you need. You can do something like:

dagger toolcain install github.com/myorg/base

^ once the base toolchain is installed, if you want your users to run any of the base functions they can just do dagger call base test for example.

On the other hand, if you need to customize/override these functions you have different options:

Use toolchains customizations (https://github.com/dagger/dagger/pull/11480)
Create an override function in your main module which programmatically calls the toolchain overriden function with whatever arguments are needed

#

LMK if this makes sense or if you'd like me to expand on any of these 🙏

glossy sleet Dec 16, 2025, 2:39 PM

#

@quaint island Could you link any example of Create an override function in your main module which programmatically calls the toolchain overriden function with whatever arguments are needed

#

Cause that sounds like something that could work..

outer nebula Dec 16, 2025, 2:42 PM

#

@glossy sleet here's an example from our own Dagger configuration: https://github.com/dagger/dagger/blob/main/toolchains/cli-dev/main.go#L81-L90

#

We have a custom toolchain cli-dev for developing the Dagger CLI. It has a function binary which builds & returns the CLI binary. Under the hood it wraps a generic go toolchain, with some additional arguments injected.

glossy sleet Dec 16, 2025, 2:43 PM

#

I see I should rewrite all my code to golang.. cause python is kind of "lost" in translation..

#

so you can call toolchains more/less as modules?

outer nebula Dec 16, 2025, 2:44 PM

#

Toolchains are modules, imported into another module in a particular way. A special kind of module dependency

#Generic pipeline