Quick demo / UX notes

[spring mural]

• Cache volumes in that shell are an easy win, speed has a big influence on how that feature will be perceived

(will continue after next demo)

[stoic hollow]

those were added already

[spring mural]

Ah cool 🙂

[spring mural]

• Very verbose logs when preparing the shell. Wondering if we can make it less scary, either with TUI, or just hiding some of it?

• I think we should remove the word "universe" from the demo. From past experience, if we start using the word now, it will stick, and the presence of the word will then influence design based on branding associations of the word. Basically it's a "cattle before the horses" problem.

• A thought that came to me when discussing the "spectrum" of power users with @deep roost . Is there a need for a middle ground between "I use Dagger but never scripted it at all" and "I created an environment from scratch". What if we had a "mod" feature, where instead of creating an environment from scratch, you wrote an overlauy on top of an existing one. Kind of mods for a game. That would allow for eg. an app dev to add custom functions / artifacts / checks etc. without the learning curve of a potentially large and complex base environment.

• dagger extension subcommand. I think since it's a flagship feature, and a lot of the questions focus on the extensions feature, it's worth spending a little time implementing a basic dagger extension add dagger extension sync (the former being how you run codegen on top of a manually edited dagger.json)

• we should discuss the concept of "catalog" with a namespace of extensions

• Replace the codegen'ed DaggerClient with <EnvName>

• root? didn't get it

• need to talk about standalone vs. embedded envs

• "how does this work in a monorepo exactly?"

• showing GraphqQL queries in the demo will clarify lots of things about the underlying data model 🙂 (re: @tardy inlet 's question "what about individual unit tests")

• checks have sub-checks, but results also have sub-results? 🤔

• for future bikeshedding: I don't think we need sub-results

[crude bobcat]

This is why I'm down a terminal emulation rabbit hole. 😛 Slowly getting less janky... https://asciinema.org/a/AFZIQrthV4kCWJF6Nn3bQArIv (vim and htop mostly work now!)

[stoic hollow]

It's by default just a mirror of the subchecks you setup. So if you have CheckA that depends on CheckB and CheckC, then when you get the result for CheckA the returned result will have CheckB's result and CheckC's result as subresults.

I added it only for convenience originally. I think there are potential arguments for keeping it long term in terms of flexibility (i.e. programmatically constructing results without having to wrap them in a check), but I'd need to work through that more to be sure.

All else being equal then less api is better, so as long as we don't lose meaningful use cases then we can remove it for sure.

[stoic hollow]

Yeah, I think the "multiverse" question proves it needs some more workshopping. For now, we can just call them "builtin environments"? That's still a term, but it's at least more generic, doesn't sound like a trademark or whatever

[stoic hollow]

Yeah I don't even think this would be hard to implement at all. It'd almost work today by just writing e.g.

func main() {
   dagger.Environment().Load(dagger.Git(...)).WithCommand(MyExtraCommand)
}

Pretty sure there's one easy-to-fix problem in the current implementation, but yeah then we could make this pattern even easier with various cli commands. Being able to init an environment from a template (e.g. dagger environment init --template hello-nodejs-app), which has come up before, is not necessarily the same feature, but feels highly related. Maybe they could be the same.

[stoic hollow]

Yeah I'll do this quick. We already have commands for reading/writing dagger.json, so it's low overhead. This also allows us to avoid calling cat on dagger.json and thus not confuse anyone with the "root" field we currently need

[stoic hollow]

I'll try out some variations here. I would like there to at minimum be consistency between the default client (dagger.DefaultClient()) and the generated one (DaggerClient) somehow or another. I was playing around with various stuff when I added the custom codegen last week and there's a surprising number of annoyances to workaround in the current state that make it hard, which why I gave up for sake of time and left it in the current state. Worth another visit again now though.

Having the dependency envs not attached to the same object in particular has a downside though, which is that in an IDE they are less "discoverable". When they are all on the same object I just type DaggerClient(). and I see autocomplete for everything available, including those environments, which I don't get if they are top-level funcs. So I'll see what feels nice.

[spring mural]

How about we reserve something nice and simple like Env and that’s always the name used by codegen? Env().Container().From()…

Also I understand the consistency argument but why do we need dagger.DefaultClient to be so prominently used, couldn’t it be completely hidden?

I guess it would require always having codegen

but that’s also a consistency question right?

[stoic hollow]

Yes dagger.DefaultClient is used when you aren't using codegen. It's new to the Zenith branch and just saves environment developers from having to litter their code with c, err := dagger.Connect(...) calls (which is the defacto state today).

The way to get rid of it would be to change our default codegen such that even dagger.DefaultClient isn't needed and instead you can just do e.g. dagger.Container(), dagger.Directory(), etc. This was @wind cipher's suggestion from a while back.

I'm actually attempting that right now as part of the changes, but it's proving to be quite tricky due to conflicts between func names and struct names and such. I may have to cut it and just find something else as a better-but-still-intermediate solution

[spring mural]

Another possible approach is to always use DaggerClient (or whatever we end up calling it). In other words always do env-specific codegen

or maybe that’s silly? can’t tell for sure

would be consistent

[stoic hollow]

Hm yeah, I was trying to avoid that before since it seemed like a lot of overhead, but I suppose in concert with changes to dagger environment commands that just trigger codegen when you initialize a new environment it actually wouldn't be that bad

it would at least make all environment code consistent

[spring mural]

and a clean slate to codegen what we want without conflicts since it’s not in the dagger namespace

part of me thinks “0% env codegen, or 100%”

[stoic hollow]

Yeah that makes sense, and the only "new" requirement is that codegen is totally seamless, which is what we need to do anyways... I'll try this out. There's other limitations I need to work around in terms of how the go sdk does the magic reflection/ast parsing, so we'll see how it goes, but I'll give it a shot

[spring mural]

As for the name, maybe DAG instead of DaggerClient? 🙂 Borrowing from @crude bobcat ‘s trick

[stoic hollow]

Haha, I was thinking along the same lines (ignore the fields inside it)

[spring mural]

obviously fields have to be D, A and G

[stoic hollow]

This whole task has proven to be quite mind melting, but I'm getting pretty close now and I think I'm gonna include the stretch goal I had in mind where the environment codegen is for all of the dagger API, including core types. Which I realized unlocks the very cool possibility of allowing user code to extend any types from the DAG. So they could write

func (c *Container) MyCoolContainerAPI(...) (*WhateverReturnTypeIWant, error)

But then that would just be stitched into the graphql api under Container and be available to any other env codegen client and in the raw graphql api

Which is something we've talked about in the past but I never knew how to get it working in Go, but I think this may get us that?

[stoic hollow]

Okay finally got this working, I like the way it looks. Improvements so far:

1. dagger env init ... command now also does initial codegen for you
2. The stupid _ suffix hack on the env builder methods are all gone, works seamlessly now.
3. All environments use codegen, and codegen includes both core types and dependencies, so it's all totally seamless now
   • Demo Server env (no deps, just core types): https://github.com/sipsma/dagger/blob/zenith-codegen-refactor/universe/_demo/server/main.go#L9
   • Demo top-level CI env (with deps): https://github.com/sipsma/dagger/blob/4e513ba9fe7b21cc9925d2273e068e14ea60adab/universe/_demo/main.go#L10-L15
4. Including the core types in the custom codegen was initially a compromise to deal with how go reflection checks for equivalent types, but I do like that it opens up the possibility of extending core types as I mentioned above: #1141098832202043462 message
   • However, that doesn't actually work yet and if we don't want to go down that road, we can remove the need for the core types to be there (with more effort)

The biggest downside is that getting this to work was a true nightmare. Among many other things, the worst part is that now all of the go sdk methods for building the environment are actually just go templates alongside the existing go templates we're using for the rest of codegen. This makes it even harder to work with; generating code that is itself doing reflection/ast parsing is interesting to say the least...

I can go into the "whys" behind all that if interested (it's a long chain of cause+effect, not a single thing), but as a consolation I think with more time+refactoring it can all be cleaned up. But at the moment I apologize if anyone needs to edit that code 😬

Also, didn't push to the Zenith branch yet because I don't want to break everything before the team meeting demo tomorrow. I'm gonna add dagger env extend ... quick (saved the easy part for last) and then do more testing

cc @spring mural @crude bobcat let me know if you like the way the code I linked to above looks when you have some time

[crude bobcat]

@stoic hollow That's pretty wild, still wrapping my head around it. 🤯 (just looking at dagger.gen.go) Basically everyone gets their own personal dagger.io/dagger at that point. There are some interesting engine version compatibility questions there though.

Is there any way to use a type alias instead? like type Container = dagger.Container?

I've never actually had a reason to use those, mostly curious 😄

[stoic hollow]

Agreed, and like I said, that was initially just a compromise to make the implementation easier (surprisingly) and is also something we can back out of. But the ability to extend any API is quite intriguing.

For the version compatibility question, I agree but I also think it's actually the same set of problems we'll face w/ env version compatibility in general. Basically, I think our long term goal should be to treat the core api the same way we treat env apis. Eventually, we could even implement it that way (turn the the buildkit.Client we pass around the schema now into an actual API and run the core api as an env container with the buildkit.Client api available to it)

But while I like the theory behind that, there's obviously a lot of practical considerations in the meantime that might overrule it

[crude bobcat]

I'm not sure how I feel about extending core types. How do you imagine it working? Do the extended methods get "exported" somehow, i.e. non-namespaced extensions? Or is it purely sugar for local code?

[stoic hollow]

I'm not sure yet in Go what the "env builder syntax" would be. But as a total strawman you could maybe imagine a "netlify" extension doing:

func main() {
  dag.Environment().WithExtension(Directory{})
}

func (c *Directory) DeployToNetlify(<params>) error {
   ...
}

And then anyone depending on that netlify extension now has the magic ability to do:

func Publish() error {
   return dag.Host().Directory(".").DeployToNetlify(...)
}

I am thinking of netlify in particular just because back in the early cloak days I'm pretty sure we had something like that working.

Ultimately, this is total sugar, you could just write a function that accepts a directory instead of putting a method on Directory, but in terms of chainability it can create some really nice possibilities. Like today the With(func(*Container) *Container) method on Container (and same forDirectory, File, etc.) are nice but have a huge limitation in that they all can only map one type to itself.

Being able to extend existing types instead unlocks the ability to have everything be a single chain.

Is that worth it? I don't know, I'm not sure yet what the downside really are. I agree about being suspicious of versioning but I can't tell yet if that's really making the versioning problems better/worse yet.

[crude bobcat]

Oh sorry, by versioning I just meant knowing what engine version to use now that you're not using the dagger.Client, but I realize that's actually orthogonal since environment code is built in the engine anyway. Current state:

[stoic hollow]

I was attempting that at one point, tbh I forget the exact problems that caused me to back out because this all went through like 10 generations, but I think it was a combination of:

1. It causes the codegen between the current client and the env client to diverge more. Which is totally fine in the long run, but dealing with the stupid go templates is such a nightmare that I was trying to minimize template programming for my own sanity right now.
2. It's sort of confusing for the end user because now it's valid in environment code to refer to both dagger.Container and Container. Which is okay by itself but I would be sort of concerned with a big ecosystem of envs being out there where all of them have different approaches and it's not clear anymore if one is right and one is wrong. Not a huge deal at the end of the day, but a small wart

I'm pretty sure there was something else too... but like I said there were many generations of attempts here and it's highly possible the other problems were specific to a previous iteration and no longer apply

[crude bobcat]

But yeah, ok that actually makes me like it more, I thought you meant just arbitrary local functions, which my argument would have been "that's confusing because you don't know what's local sugar vs. what's actually in the DAG", but you're basically saying it's all in the DAG, via magical schema extending

[stoic hollow]

Overall, I wouldn't take the whole aspect of including core types as a hard requirement or permanent feature, it just sort of landed here and I thought it was interesting enough to leave in at the moment

[crude bobcat]

There are obvious dangers there with monkey-patching and compatibility, but hopefully those aren't such a huge concern in this context, and it seems like something we can detect and give nice errors for

[stoic hollow]

Yeah I wouldn't want there to be the ability to override or anything like that for sure, that would be an error if we do end up going down this route

[stoic hollow]

Oh this also reminded me of another tiny improvement, which is that previously it was actually possible for env code to close the dagger client, which is never what you want. Not possible anymore. Totally orthogonal to everything else though

[crude bobcat]

Yeah, and the likelihood of you importing two extensions with different APIs for DeployToNetlify seems low. But I think the real nightmare scenario would come if/when an ecosystem forms with people building competing environments with general-purpose API extensions, sort of like underscore.js or Rails. Seems like that could quickly result in conflicts

[stoic hollow]

Agree, that problem is pre-existing to an extent due to conflicts in environment name, but it does expand in scope if we allow arbitrary api extensions

[crude bobcat]

At one point we talked about aliasing envs too, I think? Sort of like withServiceBinding

[stoic hollow]

Another small argument in favor of arbitrary extension points is that raw graphql queries can become wayyy cleaner. If it's setup correct, no need to deal with IDs being passed between different queries, it's just one chain.

[stoic hollow]

That's right, which I suppose could in theory also apply to methods extending existing objects (though how users configure that, not totally sure)

[stoic hollow]

IIRC that's why back in early cloak days we were excited about this, in the graphql playground we could just type { host { directory(".") { deployToNetlify } } } rather than copy paste enormous IDs between multiple queries.

[spring mural]

one nice side-effect of “100% codegen” is that it puts the CLI more firmly in control of SDK versioning

you can ask the tool to update your env to a new version of the SDK, and it can generate new bindings; update your version of the client library; it feels natural too