Design knot: Part 3 - Artifacts

[glacial kraken]

🧵

<@&946480760016207902>

feel free to comment here or in the gists

Part 3: artifacts

Dagger Design - Part 3 - Artifacts

cc <@&1278792484406821068>

[spark stag]

@night scarab this is the proposal for a better way to orchestrate artifacts between functions

[glacial kraken]

it still lacks details... rushed it out to get the convo going

[spark stag]

Is this where the contextual workspaces come into play? Like if I have /.dagger/config.toml, /cmd/foo/.dagger/config.toml, and I dagger check --path=cmd/foo, it gets the workspace from the nested config? Likewise if I have cmd/foo and cmd/bar each with their on configs, and I dagger check GoModule:test which applies to both foo and bar, they'll each run test with their own config?

[glacial kraken]

https://tenor.com/view/no-further-questions-gif-5672430

[glacial kraken]

honestly, I don't know, it's a good question. I had not thought about the interplay between artifact path filtering on the one hand, and configuration paths on the other.

[glacial kraken]

Design knot: Part 3 - Artifacts

@spark stag I think answer is no: which config is loaded would be determined strictly by client workdir (either implicit system workdir, or explicit with dagger --workdir).

Completely separate from that, artifacts can be filtered a bunch of different ways, including by source path (artifacts --path=...). So that would be orthogonal.

I get how this could be confusing though. wdyt?

[spark stag]

I think that could be fine, it kind of depends on how we solve the artifact discovery part of https://gist.github.com/shykes/aa852c54cf25c4da622f64189924de99#module-composition

for example in dagger/dagger we have cmd/dagger and cmd/codegen. I'd never cd into one of those directories to build them, so it makes sense that if workspace is determined by the place I call dagger from then it wouldn't load a workspace from cmd/dagger/.dagger/config.toml. But I may need to configure what cmd/dagger needs to test or build separately from what cmd/codegen needs.

[glacial kraken]

yeah my assumption is that, with dynamic workpace access & dynamic artifacts, you can have relatively large and statically configured workspaces. Basically the default would be a single workspace config at the repo root.

If you split in multiple workspaces it would be for organizational reasons, not stack breakdown

[spark stag]

yup sounds simple enough

[glacial kraken]

you'd still have project-specific modules for custom builds. They wouldn't be called "toolchains" but would work the same.

Question is: where do we put them 🙂

Updated gist with artifacts API

[spark stag]

In a world of toolchains==modules, I guess under /modules/? Given that we've decided scattering them across your project to be colocated with project components doesn't work

or if we already have a /.dagger/config.toml, maybe /.dagger/modules/

more aligned with the .github/ model

[glacial kraken]

would those be auto-installed?

[spark stag]

maybe they can be automatically installed when they're initialized, but I think its probably good to keep the toml explicit

[glacial kraken]

yeah. too much magic & ugly import path