Ship functions 🧵

[crimson narwhal]

@charred island just a place for lightweight discussion before escalating to the github thread 🙂

--> for reference https://github.com/dagger/dagger/discussions/11653

[crimson narwhal]

@charred island I have a general question on workflow composition... Separate from the specifics of which syntax, which file etc.

If we foresee workflows of arbitrary complexity between check functions, ship functions, and any other "special" functions that we may add (like perhaps "up" functions etc). Then, we have an arbitrary DAG. What are the exit nodes of that DAG? The could be ship functions, check functions, up functions, or a combination of those. So dagger check, dagger ship or dagger up are not sufficient to resolve all exit nodes. Then what is?

This feels like new territory, because so far we've only ever resolved a DAG with either:

1. Exactly one exit node (dagger call, dagger run)
2. Or now: multiple exit nodes sharing the same function type (dagger check '**')

How to deal with:

3. Any number of exit nodes, not sharing the same function type

[charred island]

I see - put another way, what command do you actually run to run everything (i.e. your 'ci') - 0..n checks, 0..n ships, etc.?

[crimson narwhal]

Yeah, seems like if we support this new, higher level of workflow, we need a new high-level function to go with it. Like dagger ci or something

or dagger workflow?

[charred island]

yeah, was thinking similar. we can at least hone it down to a commit as the input

[crimson narwhal]

Add it to the pile of cans of worms 🙂

[charred island]

dagger integrate, and then you run that continuously, voila, CI 😛

[crimson narwhal]

Yeah, my worry is that it's starting to feel a lot like a whole new platform

But would have to mock it up in practice

[charred island]

i'm also not sure this is something you'd ever run on your own from the CLI, outside of a parlor trick

[crimson narwhal]

It's the combo of 1) new declarative config over functions, with rich workflow semantics + 2) super tied to git commits & git metadata

[crimson narwhal]

Exactly, which is super different from everything else in dagger

[charred island]

reason being that to most people I'd venture being centralized is an important part of CI - there needs to be an authority that the team rallies around

[crimson narwhal]

But it feels very possible that users will take something that was previously possible and good in the CLI, and move it to the "other side" where nothing runs in the CLI

[crimson narwhal]

Yes but we already support that by letting you run the same things in different places, with different inputs, and trust the outputs differently

eg. the same dagger check runs locally and in our CI, but we trust the ones from CI more. You have that without making it impossible to run the checks locally - that would be a regression

I worry that we open a pandora's box that leads to this kind of regression being too eaasy, and gradually becoming the norm

After all it is the norm in literally every CI..

[charred island]

regression being that people write checks that somehow only pass in CI, or are tightly coupled to the CI environment?

[crimson narwhal]

Yes, or encode larger and larger parts of their CI/CD in "workflows" that only run in CI, instead of discrete dagger functions that run anywhere

It's a vague concern since this higher-level "worfklows" is still vaguely defined

What if there was an artifact-centric twist to these check & ship functions? Even on the check side, I still have an unresolved issue, that often there are multiple "entities" you want to check (ie. multiple go modules, or apps or whatever). We will have the same issue with ship. (there are multiple artifacts you actually want to ship, how to distinguish them?)

Thought experiment: what if +check and +ship didn't mean "this is a globally available check / ship function", but "this is a check / ship method on this artifact type"

In other words "here is a way to check or ship this particular artifact"

[ancient rivet]

I've felt the pull in this direction after a bunch of our conversations

[crimson narwhal]

(I realize I'm pulling in another unresolved design problem... the question of "dynamic content" in toolchains)

I guess that would overlap with point 1, "toolchain-centrix UX" and point 5 "test splitting" from this list 👆

[ancient rivet]

and covers the "build arch matrix" problem

[crimson narwhal]

https://tenor.com/view/mietse-grover-help-struggle-tangled-gif-13904542

[ancient rivet]

it really is a tangle

[charred island]

Makes sense to me - so dynamic checks could just be a function that returns objects that have a @check on them?

[crimson narwhal]

Right. We still wouldn't rely on that for test splitting/filtering - that would be too fine-grained for what users expect to be a check, as we discussed. But we would use it for eg. apps, go modules, js apps

The main drawback of this approach is that it's one more dependency on a as-of-yet non-existing feature 🙂