FYI I think it’s an anti pattern to

[vagrant brook]

Can you explain more about those two concerns? For 2, the vanilla graphql clients have the exact same power as the codegen client because the checks/artifacts/etc. all get stitched into the schema too

For 1, I'm not 100% sure I understand, just need clarity on the concern

[peak heart]

Yes but it’s a consistency problem. Two ways to do the same thing. If docs examples or more use the codegen version (why not?) then the best practice becomes “use the codegen when possible”. “codegen not possible” then becomes the special case, which over time makes it a second class citizen. Not to mention the overall complexity of having 2 ways to do something the whole time

There’s no benefit to codegen for those entrypoints since they’re designed precisely for the schema to be known in advance

It just kind of muddieds the waters

also makes it harder to say “it’s just regular graphql, easy to integrate with”

dynamic entrypoints: say I load a go extension and it scans my workdir and registers one check for each gomod it finds

or a docker extension scans my docker compose file and maps compose services to dagger services

those won’t be in codegen

[vagrant brook]

I mean the whole point of what we're doing is to have multiple ways of doing things, each with tradeoffs:

1. Raw graphql API - ultimate flexibility, but probably the least usable unless you already know + love graphql and don't mind constructing queries and reading responses dynamically
2. "Type-unsafe" environment api - e.g. c.Environment().Load(...).Check("someCheckName"), that lets you dynamically grab any check by name, but at the expense of type-safety
3. Type-safe codegen api - full type safety but not dynamic

There’s no benefit to codegen for those entrypoints since they’re designed precisely for the schema to be known in advance
There's a good example of this being very beneficial here in the demo environment's "publish-all" command, which calls out to the other commands from the dependency envs: https://github.com/sipsma/dagger/blob/ca5fc66308ed51b47d30eae655879eb0c787d6dc/universe/_demo/main.go#L22-L36

[vagrant brook]

also makes it harder to say “it’s just regular graphql, easy to integrate with”
I don't think that's true, doing this still results in a graphql schema. I would actually say that our current approach makes it even easier for raw graphql clients

[peak heart]

with the exception of functions, option 3 offers no additional type safety over 2

[vagrant brook]

Yes it does, you can see that the checkcommand accepts an argument of type string called version. You get type checking on that. In the type unsafe api, you don't get that. You have to provide the name of the checkcommand and the name of the flag a strings and hope you're right

[peak heart]

the check shouldn’t take an argument

[vagrant brook]

Sorry I meant command

[peak heart]

ah yeah command is a blurry in between I agree

[vagrant brook]

But also I disagree with that 🙂 probably a separate thread though

[peak heart]

if everything is going to be parametrizeable at will, in ways that require introspection and codegen , there’s no point in having specialized entrypoints, might as well make it all functions

example integration: a CI tool that loads all dagger environments it finds in the repo, and runs all checks

[vagrant brook]

An example of a place you would want to parameterize checks would be enabling race detection on go tests. E.g. func UnitTests(enableRaceDetector bool) *Check {...}

[peak heart]

the lack of parameters in checks is a must for this integration

[jolly dock]

this is a huge deal imo - I basically never use ./hack/make engine:test because the vast majority of the time, I want to pass it something like -run TestICareAbout

[vagrant brook]

I don't see how that's the case, it just means you need to check whether the checks you're calling are parameterized (confusing phrasing, but English is overloaded...)

[vagrant brook]

Yes that's exactly what I was thinking of

[peak heart]

isn’t that handled by subchecks?

[jolly dock]

Sure, if everything is structured the way we talked about in one of the demos, where it generates a report and also supports selectively re-running. But -run also supports regexes and partial matching, so we'd need to support that too (probably a reasonable thing). And there are all sorts of other flags I pass to go test too, like --count=100 (sussing out flakiness), -race, etc.

[peak heart]

I love the enableRaceDetector example (and the other go test arguments you talk about @jolly dock ) I agree we need a great way to support them. I worry that in doing so we make all entrypoints “almost functions” which I think is not desirable

(re: my CI integration example)

Could env settings help here?

[vagrant brook]

I think it's not quite that, the other entrypoints would have much tighter constraints on the input types, whereas functions are pretty much arbitrary (just need to be serializable objects), other entrypoints need to be invokable from the CLI

[vagrant brook]

I don't think I understand how being parameterized breaks that use case. That use case is inherently extremely dynamic, I don't understand why parameters make that any worse

[vagrant brook]

Also need clarity on what you mean here

[jolly dock]

CI/CD also pretty often supports parameterized builds (as much as I hated when folks asked for it in Concourse)

[peak heart]

So you would configure your go testing preferences (count=100, race detector on, etc) at the environment level. then those settings are applied globally to all go checks

I think it’s very powerful that for example any CI can take a webhook event from github, and populate a list of checks into github. Without prompting anyone for additional domain-specific parameters

[vagrant brook]

I think you'd start hitting corner cases around global vars that make that really hard to use. E.g.

func AllTheTests() *Check {
  ch := dag.Check()
  // run the unit tests, ALWAYS enable race detection
  ch = ch.WithSubcheck(dag.Go().Test("./pkg"), Opts{RaceDetector: true})
  // run the integ tests, don't enable race detection cause that is too slow
  ch = ch.WithSubcheck(dag.Go().Test("./integ"), Opts{})
}

If the only way to parameterize was global, something like that wouldn't be possible

[peak heart]

i get the power of parameterizing everything (and don’t have a great alternative to offer to some of the use cases) but over-parameterization is a real risk

[jolly dock]

If all we're talking about is optional params, don't you have the best of both worlds?

[peak heart]

Maybe yes

Depends on typing flexibility and codegen I think

which in turn impacts introspection

[jolly dock]

here are GitHub's parameterizable workflows btw: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#providing-inputs

[vagrant brook]

This would still work, it's just that you're not gonna see every single dynamic subcheck that relies on parameters. In the demo for example, you'd see UnitTest and IntegTest as the checks

[peak heart]

The “it’s all just data” pitch resonates. This could break it

query { here { checks { success } } } -> stuff like this

middleware I guess

[vagrant brook]

To @jolly dock's point, that would just result in the default parameter values being set.

But also, going back to your example of creating a web ui on top of all this, which parameters exist on each check (and command, etc.) are also part of the schema: query { here { checks { name flags {name type } } }

[peak heart]

I think if you’re there, you encode these as 2 custom checks

[jolly dock]

I think it still holds with parameterization; you're just "selecting" a different cell in a build matrix, so to speak

[vagrant brook]

So your web ui would just handle looking at the parameters, if any, and displaying them in the ui and allowing the user to enter non-default if desired

[peak heart]

but do I need domain knowledge to select

[jolly dock]

I think I see what you're getting at: that you would prefer to have all of the build matrix cells expressed as subchecks instead?

[peak heart]

right

not saying it’s better, just scared to lose unix-like encapsulation and the middleware possibilities that depend on it

[jolly dock]

I think the part that's causing friction is that a user at a CLI and a generic CI system visualizing your DAG are two very different things. A user has domain knowledge and wants higher fidelity, CI wants to be brainless

But right now we're trying to cover both with entrypoints

(caveat: they won't always have domain knowledge, but if they're running your env code, they probably will soon...)

[peak heart]

there’s also a user who doesn’t know the native tooling

ie. frontend dev who needs a backend instance. ML team wants to test their new model in a complete app

[vagrant brook]

That's what the documentation on the parameters are for, the ones you see when you run dagger <xyz> --help

(I realize that's just for humans, not an automated CI system)

[peak heart]

right, I’m just saying there’s a spectrum, like I would expect artifacts to be available without 50 required arguments I don’t understand

[vagrant brook]

Yeah totally, that would be bad design on the environment implementers part.

[peak heart]

or optional arguments I need to change but don’t know where to look

[vagrant brook]

And as much as I don't love handing out footguns willy nilly, I think not accepting parameters at all would hurt usability so much that I'd rather hand out the footgun than constrain users to that extent

[peak heart]

in the case of artifacts I definitely think pushing the matrix upfront as different artifacts with different labels js better than parametrizing artifacts

[peak heart]

right but there’s always functions. So the bar is that it has to be better than functions in some way

[vagrant brook]

It's better in that functions aren't callable from the cli. I can call dagger checks unit-tests --race true --count 100

[peak heart]

I think CLI/commands are the right place for that

[jolly dock]

Strawman mental model: everything is functions under the hood, and entrypoints are just special categories given to functions (oh god I sound like a Haskell programmer) that must not take any required params

[vagrant brook]

But then we lose all the TUI display niceties of using checks

[peak heart]

For that we can add a status API that any function can call

(we need that anyway)

[vagrant brook]

So users all have to make calls to display the checks they are running?

[peak heart]

I like the checks TUI etc it’s just the part where passing CLI arguments to checks is a must have where I’m skeptical

but I get the appeal

I still think env-wide settings + functions when you need a footgun is a viable option worth exploring further

[vagrant brook]

Yeah as long as the settings are per-env and not automatically propagated between them or something like that there could be something there

Honestly there might be a connection with dagger run we were talking about before too....

🤔

[peak heart]

I think it would be cool if you have really custom test config that requires eg. half of the tests to be with race detector and the other half not, etc. at that point you’re basically implementing custom checks so you should register them as such

[vagrant brook]

Agree, but I still want to be able to call that from the CLI

[jolly dock]

for me the race detector case is more of, "I just ran some sketchy concurrent code and want to run it with -race as a one-off," and less of a per-suite setting

[peak heart]

yes of course but you won’t need to pass 5 flags to it though 🙂

[vagrant brook]

Yes totally, and that's great, but the use cases that do call for passing some flags are still important too

[peak heart]

yeah I don’t have a good answer to that case

https://tenor.com/view/stitch-okay-gif-9555327

[jolly dock]

IMO a larger issue would be if we get people adopting dagger checks as their day-to-day workflow, passing params and such is going to be second nature, whether we want people to think that or not

[peak heart]

I’ll just be very sad if a generic Dagger-native CI tool isn’t possible because domain-agnostic checks API just isn’t good enough (because you need to know the correct incantation on an env-by-env basis)

likewise for a generic “artifact store” integration

[jolly dock]

I think the incentives are already aligned there though. Wouldn't you want to expose checks that don't require params, so you can see it in CI?

[vagrant brook]

I feel like there's still some possibility to get best of both... thinking out loud:

1. Checks+Artifacts don't accept parameters
2. But functions do
3. When you register a function, if it has a type signature we support in the CLI, then it is invokable (parameters and all) in the CLI under some generic command. dagger run, dagger do, dagger whatever, etc. TBD best name

[peak heart]

sadly all it takes is 1% of the ecosystem doing it wrong to ruin the fun for everyone

so we definitely can’t rely on best practices alone

[vagrant brook]

If we have that, the pattern would be something like:

func main() {
  dag.Env().WithCheck(UnitTests).WithFunction(GenericUnitTests)
}

func UnitTests() *Check {
  return GenericUnitTests(true, 100)
}

// TODO: better name
func GenericUnitTests(race bool, count int) *Check {
  return dag.Go().Test("...", Opts{Race: race, Count: count})
}

Then, dagger checks would only call UnitTests, but if you need to have some one off flexibility, you do dagger run generic-unit-tests --race=true --count=100

I think that could work? The biggest problem is that GenericUnitTests is an awful name

But if we could find a pattern along those lines we'd support both use cases

[peak heart]

I don’t think invoking functions from the CLI is something we should try right now, impedance mismatch on types would be strong

and with CLI/commands we already did the work to expose a CLI-friendly API, so env devs can do the mapping themselves

we can always add magic later

[vagrant brook]

I don't care about functions being invokable from the CLI per-se, I'm just saying that we would have a bucket for invoking checks (no params) and a bucket for "arbitrary thing which is invokable from the cli"

Functions doesn't have to be the second bucket

I guess we could use commands there too, but we'd need to have the dagger do implementation see "oh you returned a check! I'm gonna use my fancy tui to display that!"

Which actually is extremely possible

We would be opening up the meaning of command quite a bit, but in a way I feel comfortable with since it would allow us to support all these use cases

[jolly dock]

I don't quite follow this - wouldn't the only user impacted be yourself? What would cause it to affect everyone?

[peak heart]

What I mean is that if 1% of envs out there have required arguments, then all CI tools have to handle that case (or maybe they don’t get written because it’s not worth it)

[jolly dock]

oh, yeah I'm suggesting that anything that has required arguments wouldn't qualify as a check in the first place

[peak heart]

then Hashicorp (or anyone making a very valuable/popular extension) decides optional parameters are an anti pattern

[jolly dock]

I totally agree that required args for checks/artifacts/etc. aren't something we want

[peak heart]

right. we’re in agreement

[vagrant brook]

I agree too for the record 🙂

[jolly dock]

(brb dinner)

[vagrant brook]

And if we expand the meaning of Command to include "anything that accepts zero or more parameters that are passable via a cli and returns either a primitive/check/artifact/other entrypoint type", then we can just make dagger do (or dagger run, whatever) that generic bucket for cli users to run their one off things with whatever parameters they want

I think the important thing for me is just that it's the end client (i.e. dagger cli or the web ui) which handles the result type and displays accordingly, it shouldn't be the user env code that has to setup displaying a check specifically

[peak heart]

dagger function run ?

[vagrant brook]

I can't see why not just dagger run, but if it came down to it I wouldn't protest against dagger function run that hard (though I'd probably setup a short alias for it in my bashrc very quickly 🙂 )

[peak heart]

Yeah sorry I think I have several parallel design ideas mixed up

all related by subtly different

[vagrant brook]

dagger run as I remember the conversation the other day was already gonna have to be pretty polymorphic since it could be used for commands or for specifically defined shells, or the default shell for the env that is the "interactive manifestation of it", etc.

[vagrant brook]

no worries my brain is a fractal of chaos rn

im barely holding onto reality

[peak heart]

Right but it was all unambiguously CLI-specific stufdf

Let's not forget that @lone forge started this whole thread 😛

[vagrant brook]

True, but invoking a check with a bunch of parameters feels close enough to a CLI like activity that it makes sense in the dagger run bucket too, at least to me at this exact moment

I guess the important thing is that Checks would be unparameterized and reserved for "serious business" like "I need to run all the checks before I open my PR" or "CI needs to run all the checks". dagger run would be for manual one off tasks locally, like a shell or running customize parameterized checks, etc.

[peak heart]

Yeah but it is a big UX jump to go from "there is an entrypoint called CLI, and you can call it with dagger run", to "ok so dagger run is a bit special, hear me out..."

[vagrant brook]

Yeah I agree, I'm not settled on dagger run being the 100% correct bucket. But I do like the idea of there being some sort of bucket like that

[peak heart]

I think we could get our cake and eat it too:

• dagger run is CLI only (command entrypoint)
• however commands can somehow run checks return check-like status / results
• So eg. the go CLI could be "enhanced" such that good old go test CLI gets the TUI superpowers when you run it

I think this has other cool UX benefits, because now you're just running go test

[vagrant brook]

If we wanted to get really fancy we could even do something like:

1. I just ran dagger run unit-tests --race true --count=100 and loved it so much I want it to become a bonafide check now
2. So now I run dagger env create-check super-unit-tests "dagger run unit-tests --race true --count=100", and my env code is magically updated to have that as a check now too
   (just continuing a previous thought, looking at new messages now)

[vagrant brook]

Yeah I think I see the idea. You could literally just add withCheck to the Command api, which would result in the check executing when you invoke the command 🤔

I can't think of why that would be a bad idea yet. Possibly sort of confusing. But not that confusing?

At least something along those lines

I guess it shifts the mental model a little bit. A command is like a parameterized vertex in the DAG that can automatically fan out to execution of other vertexes just by attaching them to the command. Or something like that

The important bit being that even though commands are parameterizable, checks are not. So if you attach a check to a command then then that means that any parameters the command accepts have to be used to determine the check to attach. Once attached, they are static

We can also just support a command returning the Check type too of course, I'm just thinking if there's any generalizations. Maybe the generalizations are too confusing though

[peak heart]

The alternative approach is that we just allow checks to have optional arguments (but not required!), and I take a leap of faith that the co-existence of codegen and non-codegen will turn out OK 🙂

[vagrant brook]

Yeah of course, and the fact that parameters are part of the api too at least helps a bit, but I do very much get the appeal of there being no parameters at all and just dagger checks being sufficient for a zero-knowledge invoker to run everything

(in all cases, no exceptions)

Having support for totally static fixtures in the DAG is appealing from a safety perspective too I suppose. Your environment only needs to expose more potential footguns if it wants to

[peak heart]

I think it would help if I (or an imaginary frontend engineer with free cycles 😛 ) tried writing a simple web UI, to get practical experience on how introspection would work, how easy/hard it would be to pass parameters in practice etc

I find myself reaching for the "simple web UI" use case a lot, but have zero practical experience with it which is a problem

[vagrant brook]

My only frontend experience was with react, which I found surprisingly pleasant because frameworks like this make it really close to just assembling lego bricks: https://ant.design/components/overview It's just that you attach the bricks w/ typescript, which isn't too bad.

But yeah, I agree it's kind of hard to say either way yet until we try. I do think starting with unparameterized checks/artifacts + a dagger cli command for invoking arbitrary stuff (whether commands w/ checks attached, functions, etc.) is a worthy starting place until proven wrong.

One last thought before I go back to what I was originally working on, in a case where we support parameterized checks and the user creates checks like this:

func TestA(enableFoo bool) *Check {
   // do something with enableFoo
}

func TestB(enableFoo string) *Check {
   // do something else entirely with enableFoo
}

If you ran dagger checks --enable-foo=true, what would that do? Set enableFoo to a bool in TestA and a string in TestB? That feels extremely murky... Probably have to make that an error, unless we want to have javascript levels of insane type coercion.

So I think if we do end up supporting parameterized checks, we may at least need to force them to only be settable by an end user when a single check is being invoked. Which is possibly reasonable, but this definitely throws a decent sized wrench of complication both into our implementation and how we explain this to users.

dagger run wouldn't have that problem since it's different from checks in that you always just invoke a single entrypoint (even if it internally fans out to more in its implementation), whereas dagger checks can end up invoking a bunch of check entrypoints at once in parallel.

[jolly dock]

Yeah that's a good point. Also when you think of dagger checks as "get me all the *Checks and/or *CheckResults available in my environment" I can see how mixing parameters into the equation makes it feel more confusing, even aside from conflicts like that.

from that POV your idea around making everything runnable with dagger run makes more sense to me. dagger checks would be your "just before pushing" or sane-default way of running for simple changes, and for everything else you'd use a more specifically crafted dagger run command, but ideally the transition from dagger checks to dagger run feels completely fluid, so you don't have to try to remember what you called things/etc.

[jolly dock]

also seems to align with this thought

[vagrant brook]

Yeah making that transition smooth is what I trying to think about here: #1141488051017764905 message

Which in its current form is probably a bit too insane but maybe something along those lines

[jolly dock]

ah yeah even for that I just meant the transition from running dagger checks and seeing that one test failed, and then knowing exactly what dagger run incantation to run that will run just that test, and with whatever params you want to pass

[vagrant brook]

Oh sorry yeah I misread the order of the words in your message 😵‍💫 But yes I agree, I wouldn't want this to turn into useless boilerplate

I guess there's a risk that no one actually cares about any of this and every just uses the generic parameterizable command bucket and never even codifies checks; both locally and in CI they just call dagger run. I think there's enough value in dagger checks to incentive usage though...

There's always at least a small tinge of anxiety before sending out a PR. If I just had to mindlessly type dagger checks rather than try to remember every relevant dagger run command, that'd feel great and much more reassuring that I'm catching anything before opening the PR.

And allowing my CI job definitions to be as simple as that too is always nice for the same reasons.

[jolly dock]

Yeah agreed, also the value in being able to accrue more checks over time as your team finds more things that should gate CI / local pushes, and have them all propagate to every team member by virtue of everyone just running dagger checks

[peak heart]

one more thought, in the context of inner dev loop you’re also likely to have some of those advanced one-off go checks just be outside of dagger

[jolly dock]

That's true, unless you're dependent on Dagger and want to be able to just trust it with your entire toolchain. That's my current recourse (./hack/dev go test ... instead of ./hack/make engine:test), but there are limitations; some of our tests depend on services that we run in Dagger for example

IMO it's worth trying to have it encompass every aspect of the dev workflow so you only have to think about one, so to me that has to mean supporting these one-off calls too

(...and making sure it's easy to make everything Wicked Fast to compete with the native alternative, which mostly means knowing what exactly is busting your caches all the time)

[peak heart]

yeah you're right

then there's a gradual step up in sophistication, from one-offs in dagger shell, to a custom dagger run, to eventually a more formalized check

1. dagger shell -c 'go test -count 100 -run foo'
2. dagger run gotest --count 100 --run foo
3. dagger run goAlexSpecialTests foo
4. dagger checks --include go/alexspecial

[vagrant brook]

And some automated way of converting an arbitrary dagger run command to a check in your actual code would make the whole thing if we can nail it. The fact that we already do codegen makes it feel more tenable than it normally would feel.

I wonder if we could record the gql requests made during dagger run and use that to generate code for a given sdk that implements the same request...