Toolchain DX

[green holly]

I think both of these methods would be valid and first class. What @prisma hatch wants with config is also the general idea of toolchains FWIU. Most things can be configurable without touching code

allowing the users, (who are devs right?)
You'll be surprised how many devs don't want to write code and want to maintain scripts/config instead. As you say it's a mindset change but it hurts adoption when promoting a new framework to ask them to change mindset. I agree with you 100%. Not the reality, at least where I work.

@prisma hatch , if I'm understanding what you are requesting, you want the user to be able to pass an arbitrary set of env vars to any given constructor?

Would it be feasible to accept a withEnvs []string and set KEY=VALUE,KEY2=VALUE2...? That can be set in their config and you parse the values in your constructor and set it.

[slim summit]

You'll be surprised how many devs don't want to write code and want to maintain scripts/config instead.

Sure, but script are code and learning to write "Dagger code" isn't that hard. But, once it is understood, then the devs will relish it. I'll bet money on it. 🙂

[shadow night]

I was thinking of another option which I think it fits nicely in the Dagger model which is the ability to run checks to any custom type. So for example you could have something like this:

type MyModule struct {
    cr *CheckRunner
}

func MyModule() *MyModule {
    return &Lala{
        cr: &CheckRunner{},
    }
}

func (m *MyModule) WithEnv() *CheckRunner {
    // add env to CR
    return m.cr
}

type CheckRunner struct{}

// +check
func (r *CheckRunner) RunCheck(ctx context.Context) {
    fmt.Println("running check")
}

and then be able to something like dagger call with-env foo,bar checks

I don't like the fact that it could potentially shadow the checks method. Another option could be doing that via a --run-checks flag or something. But having the ability to potentially run checks on custom types might seems appealing.

[green holly]

dagger call with-env foo,bar checks
call and checks kinda conflict here right? It's usually either dagger check or dagger call

[green holly]

I am not disagree'ing with you. I'd bet the same money. However, it makes it hard to promote a brand new thing by telling them they have to write more code. They don't consider yaml,toml to be code.

That's the carrot to get them in the door 🙂 Then show them how easy it is to customize.

[shadow night]

yes, mostly thinking out loud here about how a hybrid call / check model could look like since I do believe OPs observation about the check model to dynamically set checks properties being a bit more limited than dagger call

@green holly your comment about moving the withEnvs to the constructor works as the OP initially mentioned but it feels counter intuitive to me to promote module methods to constructor optional arguments mostly because of this limitation

[green holly]

I did bring up this concern with @raw halo in regards to checks and toolchains in general. The toolchain direction still is to move everything to the constructor isn't it?

[shadow night]

The toolchain direction still is to move everything to the constructor isn't it?

so far this has been the approach, yes. I'm probably missing some details so let's for Solomon or Kyle to chime-in.

[raw halo]

Yes move module-wide config to the constructor. BUT with dynamic workspace API, modules can also read any file anywhere in the repo. So modules can have their own custom config files, or piggy-back on existing config files. For app-specific settings this is often better for two reasons:

1. Devs will prefer a config file (or at least config file location) that they are familiar with, when available

2. It allows one instance of a module to handle multiple instances of an app/package/site/project, each with its own settings

let me share a concrete example shortly

[prisma hatch]

Sorry getting caught up on these comments.

@shadow night Just thinking out loud, one way I see that could handle this is allowing custom structs as args, and allowing them to be handled in dagger.json properly in json format or something:

type MyToolchain struct {
    IndexAuths []IndexAuth *dagger.Directory

}
type IndexAuth struct {
    Registry string
    Username string
    Password *dagger.Secret
}

Then in dagger.json/workspace you're moving to a user could give me something formatted that I can parse and use builder functions I have in my toolchain. WithEnvs would work too I think, assuming you could pass in secrets. Your hybdrid module sounds even better to me since htat seems more "dagger native" allowing a user to utilize chaining.

Just to clarify, picking on @slim summit comment for a second. The point of what I am trying to show is that the buck stops with checks, and you can no longer utilize wonderful dagger chaining without a user being forced to write what I view as essentially boilerplate. Your example showcases that. If you added +check to your linter_check(), then a user could install the toolchain, and with zero code, run dagger check and it would run linter_check() for them for free. However, if they need with_registry(), they have to restate .linter_check() in their own local module, and add in .with_registry() where needed. At that point, what benefit did a toolchain give them? They could've just installed a dependency on my toolchain module and done it anyway. Imagine you have 10 checks in your toolchain. That's a lot of code they are just restating/wrapping.

I could be wrong on the intent of a toolchain, but to me the intent is so that a user can run things without needing wrapper/boilerplate code where possible and ultimately include it in things like dagger check/generate/etc. I have no qualms telling a user who is doing something off-script to write code for it, I actually encourage it. My qualm is only when simple mechanics like adding auth cause a user to essentially rewrite my toolchain module again locally, which removes any advantage they get from the toolchain in the first place

[raw halo]

@prisma hatch can you summarize your use case (or use cases) so that we can use it as an acceptance test for the different best practices & design ideas being discussed?

<@&946480760016207902> for visibility

[prisma hatch]

Sure. Trying to keep simple. Toolchain module python:

type Python struct {
    // Base container (with cache mounts added)
    Base *dagger.Container

    // +private
    Source *dagger.Directory
}

func New(
    // top-level source code directory
    src *dagger.Directory,
    // base development container
    // +optional
    // +defaultAddress="ghcr.io/astral-sh/uv:debian"
    base *dagger.Container,
) *Python {
    return &Python{
        Base:     base,
        Source:   src,
    }
}

// base UV container with cache mounts and git-credential-helper config set
func (python *Python) base() *dagger.Container {
    return python.Base.
        WithWorkdir("/app").
        WithMountedCache("/root/.cache/uv", dag.CacheVolume("uv-cache")).
        WithMountedCache("/root/.local/share/uv", dag.CacheVolume("uv-home-cache"))
}


// Add creds for private python package index
func (python *Python) WithIndexAuth(ctx context.Context,
    //name of index in pyproject.toml
    name string,
    // username to authenticate with
    username string,
    // password to authenticate with
    password *dagger.Secret,
) *Python {

    name = strings.ToUpper(strings.ReplaceAll(name, "-", "_"))
    python.Base = python.Base.WithEnvVariable(fmt.Sprintf("UV_INDEX_%s_USERNAME", name), username).
        WithSecretVariable(fmt.Sprintf("UV_INDEX_%s_PASSWORD", name), password)

    return python
}

// +check
// Runs ruff check and returns a container that will fail on any errors.
func (python *Python) RuffLint(
    ctx context.Context,
    // extra arguments to pass to the ruff lint check
    // +optional
    extraArgs []string,
) *dagger.Container {

    return python.base.WithExec([]string{"ruff", "check"})
}

User dagger toolchain install python and User wants to call dagger check to run RuffLint but needs WithIndexAuth() for it. How can they do that without writing code?

[wet kraken]

And the main complication is that there's an undetermined number of registry auth configs needed, right?

[prisma hatch]

yep, exactly

[wet kraken]

If we could support a custom type in constructor defaults like you proposed here #1488060778035806330 message (which maybe we already can if it can be serialized to json), then maybe the workspace configs where we set values for that could work like HCL2 block configs where we add entries for each element of the list

[raw halo]

• Can the config be encoded as a list of strings? Or does it have to be a map? And if a map: what are the keys?
• Is this config module-wide, or per-app? (or per-package, per-project, whatever the native toolchain's concept it)

[shadow night]

The biggest concern for me with this model is that if you started your module without the toolchain/check mindset and you added some very useful chainable helper functions like WithIndexAuth and potentially many others, then you have to refactor your code by moving all these things to constructors and your dagger scripts to use the dagger check primitive.

My rough proposal idea above was to still leverage the chaining capabilities while keeping the module consistent and avoid polluting its constructor with things that might not be needed there

[raw halo]

It boils down to: devs don't want to write boilerplate code for each dagger function they need. So the chainable helper functions are not actually useful, from their point of view. Not worth jumping through hoops to preserve a UX that users don't actually want.

Devs that actually do want to write code, can write their own custom module, and call all the chainable helpers that they want.

But we can't have both 1) a great UX that doesn't require boilerplate code on day one, and 2) a hard requirement that the config model support calling chainable functions

The priority is to make modules themselves as powerful and flexible enough, so module authors (who actually want to write dagger code) have maximum control, and can meet end users where they are

[shadow night]

what I proposed doesn't require devs to write any code if the still need to use the chainable aspects of the module while also being able to run checks

the main difference is that you can invoke the checks execution on any custom type that's returned from a function. If that type has checks, then they'll get executed. Command specifics aside, basically being able to do something like dagger call --run-checks with-custom-config mycfg or something along those lines which still allows to keep the chainable UX if desired

It also allows module creators to define checks to different types within the same module if desired which might be interesting for performance / scoping reasons

[raw halo]

I guess I would have to look at an example of workspace config to make sure I understand

One way or the other it has to fit in the workspace config, so that I can run dagger check and not go back to having to know which arguments to pass

Right now the workpspace config model is very simple - key-values that map to constructor args. It's possible to make it more powerful (and less simple), we just need to discuss exactly how

[shadow night]

👍 I'll iterate it once more having the workspace UX and config aspects in mind 🙏

[slim summit]

This is what we are counting on. And on the point of "not writing boilerplate code", I get it, but I also see such code slightly differently. To me, code isn't boilerplate, if it gets something done. And in fact, we (the platform team) can offer a default module that would do the minimum expected work, removing the boilerplaty coding issue or rather making it clear, how the minimum setup should work. After that, instead of the topic being avoiding boiler plate, it turns into getting any specialized work done as the dev may need it, but in a fairly simple and more importantly, controlled and imperative way. Fairly simple, because of modules. Any dev who can't see the value in that should be shot. LOL! 😄

Think of it as moving from a Toolchain Library to a CI Framework. By defining a core Interface—requiring specific methods like test() or build()—we flip the script. The platform team orchestrates the high-level 'Golden Path' (handling mandatory security scans or registry auth behind the scenes), while the developer simply 'fulfills the contract' in their local module. If they want to use a pre-built community module to do the work, it's a one-liner. If they need a complex, one-off build step, they have the imperative power to just code it. It’s the ultimate Inversion of Control: we provide the guardrails, and they provide the engine.

[raw halo]

How do you implement this "CI framework" exactly? You have a "master module" that acts as the entrypoint for every project? How is that module installed and loaded?

[slim summit]

This is a rough (and AI assisted) explanation...

We solve this by completely abstracting the Dagger Engine away from the developer.

Instead of relying on a user to run a CLI command to load a remote "master module," we built a dedicated CI-Engine in Go that acts as the orchestrator. Here is the breakdown of the execution flow:

1. The Control Plane (Go + Temporal)
   Our orchestrator is a Go application that runs either as a sidecar to the developer's Remote Development Environment (RDE) or as the centralized CI runner. It natively manages the connection to the Dagger Engine. To ensure pipeline durability and handle the inevitable network blips or runner failures, we wrap the Dagger SDK calls inside Temporal activities. Temporal guarantees the workflow state, while Dagger handles the execution graph. The developer herself only uses our own CLI to either run the whole pipeline or just a part of it. We also have built in the capability to run bespoke CI tasks.

2. Interface Enforcement
   The developer never accesses the Dagger Engine or CLI directly. Instead, their repository contains a local Dagger module that simply fulfills a strict contract/interface we dictate (e.g., test() returning a Container, build() returning a Directory).

3. The Loading Mechanism
   When a build is triggered, the developer isn't "installing" our framework. Our Go application uses the Dagger Go SDK to dynamically target the developer's local directory, validates that their module implements our required interface, and then executes the graph.

In short, the "master module" isn't a library the user installs; it is the control plane itself. The user just provides the domain-specific implementation in their repo, and our Go/Temporal engine acts as the universal remote that securely loads it and presses "Play."