Generated clients

[rotund kite]

What I mean by decoupling generating clients:

• A CLI subcommand like dagger client ... specifically to manage generated clients
• A concept of "client generators" that you can develop and distributed separately from a complete SDK.
• Unbundling of the client generators present in SDKs today, so that they can be used standalone (eg. to add generated Dagger clients to your external Go, Python, Typescript or Java codebase)
• An ecosystem of client generators beyond the ones currently bundled in the SDKs. For example, you could have generators for standalone CLIs; shell autocompletions configs; CI configs; makefiles; dockerfiles; React components; etc. etc.
• Support for configuring clients in dagger.json, independently of SDKs

Conversation starter:

dagger client install --generator=go ./cmd/mytool/internal/dagger
dagger client install --generator=github.com/someoneawesome/react-dagger-client ./frontend/src/components/dagger
dagger client generate --all

{
  "name": "mymodule",
  "clients": [
    {"path": "./cmd/mytool/internal/dagger", "generator": "go", "config": {}},
    {"path": "./frontend/src/components/dagger", "generator": "github.com/someoneawesome/react-dagger-client", "config": {}}
   ]
}

I plan on opening an issue but wanted to run it by you first, since it's relevant (I think) to your ongoing conversation about vendoring and core client vs. generated client

[real ridge]

what's the generator? i'm guessing it's essentially the same "SDK modules" we've already got?

I don't disagree on any point - very on board with this, but IMO, this is really entrenched in a lot of other things around the SDK interface - from a technical perspective. It doesn't feel like a particularly "easy" win, definitely some restructuring required (and breaking changes in some of the external SDKs)

I think the first "technical" step would be to split out the concepts of introspection + typedef registration (which is currently done at runtime), from the concept of codegen.
this would probably have some performance benefits as well, and once it's done, frees us up to have apis for doing codegen neatly

[rotund kite]

No, it's only one piece of it

[real ridge]

yeah, i mean the ideal "unbundled sdk" that there's an issue for somewhere

https://github.com/dagger/dagger/issues/7707

[rotund kite]

I think the first step before refactoring SDKs, would be to separately add the new client system, and allow users to install client generators without a SDK. This will support the "external clients" use case which we are really going to need soon

[rotund kite]

ah yes - correct

[real ridge]

i feel like (especially from the go sdk perspective) we have quite a bit of tech debt in our codegen - i think having client generators would be okay as a bolt-on thing, as long as we're not adding on vast amounts of complexity that then is even harder to clean up later

[rotund kite]

So far we've approached it as a big refactor that must be shipped all at once, or not at all. But actually I think it can be much more incremental: a new feature (client generators) whicih doesn't require touching SDKs much at all, but paves the way to more refactoring when we're ready

[real ridge]

i think this as a first step makes sense

[rotund kite]

my guess is that a clean implementation of client generators, without all the bagage, might actually give us a pivot point for cleaning up the rest of the mess more easily later

[upbeat iris]

Yes, that's totally where I want to get to eventually. 👍

[real ridge]

yeah you're not wrong

i think the eventual unbundled state is that an sdk module just happens to be a module that implements various interfacts: e.g. "codegen", "introspection + typedefs", etc

this is just another interface

i do think there's overlap with how the codegen for modules works with how we'd want this to work though - i think it's worth thinking through what we'd want that to look like

[upbeat iris]

Yeah, this ties very well into something I've wanted to do for the longest time, which is to create a centralized abstraction layer for codegen/introspection, which would simplify a lot all SDK codegen. Then they would no longer come off from the lower level introspection, and you could use it for anything really.

[rotund kite]

IMO there are client generators, and server generators. Our current SDKs bundle one of each.

But I think the words "client" and "server" will need to be more visible in our UX, for all this to make sense to users

[real ridge]

(aside - this does feel like a proper project to me)

[rotund kite]

"typedefs", "introspection" and generic use of "codegen" should not 🙂

[upbeat iris]

Yes, but only Go (and now Java), do actual server generation for modules support. The other SDKs currently do "server generation" via vendoring the client library with generated client bindings, and apply default template for a new module. Otherwise, no actual code is generated for modules, it's all handled at runtime in the client library. What I want to get at is that for most SDKs (currently) it's a trivial split.

[real ridge]

ha was trying to work out how to say this 😄 for all those other sdks, there's no server, and you can actually replace that entire "typedef registration" step with a neatly cacheable introspection step

go (and i guess java as well) does need something else, which is sad

[upbeat iris]

I don't think it will be that hard though. With the introspection abstraction layer.

[real ridge]

yeah, there's still some crappiness though around needing to do things like generate custom MarshalJSON functions on user-defined types
and interfaces, similarly, but that's getting into the weeds, and I can't remember the detail

[rotund kite]

so there is codegen done inside the runtime container, once it starts?

[upbeat iris]

I'd love to be able to prioritize this introspection abstraction layer, maybe it now can in the larger context of unbundling the sdks. It can be a smaller project, as a stepping stone to the other.

[upbeat iris]

No, in dynamic languages like Python, the language gives me the introspection (I mean Python types, not API) I need at runtime without any effort. So modules support is dynamic, not pre-generated.

[rotund kite]

To clarify what I mean by server: I mean everything needed to extend the Dagger API with new types and functions. Regardless of how much codegen is involved.

[upbeat iris]

Yes, we're on the same page 🙂

[rotund kite]

Ah right, this is different from the client generation - I get easily confused when discussing SDK internals. A good example of the need to decouple. It's hard for non-experts to understand what's going on

I think splitting it up will help the cognitive load for users and contributors

[upbeat iris]

Yeah, there's tons of benefits.

[rotund kite]

Generated clients

[sullen wigeon]

Is there any action item that I could pick up next week to start the work on that?
This is a technical blocker for a clean implementation of the MagicSDK, specially to handle more complex framework because I would love to call external modules from the dagger library and this could make that possible 🙂

For example, I had to copy past the proxy module for docker-compose support, because I couldn’t call it directly

[rotund kite]

Ha I see!

[rotund kite]

This is my action item 🙂 Prototyping this

You can design the "client generator" interface such that existing SDKs already implement it (if possible)

[sullen wigeon]

I’m not sure I understand what would be the end result of this prototype, a usable dagger client to call the module?

[rotund kite]

No, a usable CLI feature to generate a client lib for your module's dependencies

What I want is for the generation of client bindings usable by any software project, to be a first-class feature of Dagger

[sullen wigeon]

So for example, if have a module X, I would do dagger client install —generator=X ./output/path?

[rotund kite]

I didn't realize you needed client generation support for MagicSDK, but if you do - that would be one way to implement what you need

[sullen wigeon]

Yeah whatever the method, generating client to call module is useful for MagicSDK, that’s why I started to PoC my thing with the go lib

[rotund kite]

No, you would do:

$ cd my-external-python-project
$ dagger init
$ dagger install ../X
$ dagger install github.com/something/else
$ dagger client add --generator=python ./the/path/where/I/want/my/client/lib

Then in my python project I can:

from ./the/path/where/I/want/my/client/lib import dag

dag.X().foo().bar()
dag.else().bla()

[sullen wigeon]

Very interesting, so you can generate a client for each dependency

[rotund kite]

yes that way it will match exactly the way our codegen already works

both easier to implement for us (I think) and easier to understand for users

the only constraint is that you have to create a dagger module for your external project, even if it doesn't have a "server" implementation

[sullen wigeon]

That means we need to refactor how the codegen work, to be able to generate single module binding, instead of all the API + module

[rotund kite]

but I think it's cleaner that way

[rotund kite]

Well no, I don't think so? You want the same codegen

Look I updated my example to have 2 dependencies

[sullen wigeon]

That means you don’t install the dagger lib anymore with go get for example or npm install

That dagger client add CLI would be the new way to install the dagger client in your project, right?

One point that is weird is that we have multiple ways to codegen depending on the language, I don’t know if that makes things more complex

For go/ts, it’s the dagger/cmd/codegen binary, but for python it’s a Python script, idk about other SDKs

That was one of the problem I met with my poc, how to consolidate all codegen entry point

[sullen wigeon]

I’m not sure, for Go you generate both bindings and module support, but in your case we would ignore the module support, it’s just a matter of bindings and loading the dep so they can be called on the GQL server

We just want the client lib basically, nothing else related to module execution

(I mean the client.gen.go generates at the root source dir when I’m talking about module support, while internal/dagger/client.gen.go are the client bindings)

[rotund kite]

sure, but "ignoring module execution support" is not the same as "refactor how we do codegen"

[rotund kite]

btw it's really confusing that those two have the same filename.

[sullen wigeon]

I agree ^^

[sullen wigeon]

For Go it generates everything at the same time, but we could decouple that easily I think

[rotund kite]

I see, ok so maybe light refactoring 🙂

Ideally the generated client is standalone: once generated, it can be copied anywhere and will still work

[sullen wigeon]

Yup, exactly what I was trying to achieve

[rotund kite]

I think you might need light refactoring there too, since "open a dagger session with your dependencies loaded" and "load your own module" are currently tightly coupled I guess

[sullen wigeon]

Yeah, loading an external module from the client library is a bit weird, you have to do it manually so we would need to hide it somewhere

Compared to inside a module where it’s automatic

[rotund kite]

That's where the client-side dagger.json is important

[sullen wigeon]

In go we can hide it in the init func, I’m not sure how we can do it with typescript or python though

[rotund kite]

Because the loader will look for that I guess

[sullen wigeon]

Yes, and hardcode the loading in the init func of the client.gen.go

[rotund kite]

What does that init func do?

[sullen wigeon]

Serving the module to the GQL api

[rotund kite]

The module's own server implementation?

[sullen wigeon]

// Load proxy module
_, err := dag.
Git("https://github.com/kpenfound/dagger-modules.git").
Branch("main").Tree().Directory("proxy").AsModule().Initialize().Serve(ctx)
if err != nil {
panic(err)
}

This for example

(Sorry I’m on my phone)

[rotund kite]

Oh I see. Is that how it works today?

[sullen wigeon]

No, it’s done implicitly by the engine when it loads the module and its dep

[rotund kite]

I thought the engine looked at dagger.json and loaded your dependencies today

[rotund kite]

ah right - so keep that!

There's a dagger.json anyway

Oh but then library is not standalone - that's what you meant

[sullen wigeon]

But if we do some client binding generation, we would not this auto loading.
Or maybe we could add it in the dagger run

[sullen wigeon]

Yeah :/

if you do a go run, it wouldn’t load automatically except if you add the thing I shared

[rotund kite]

Maybe we should talk live this week 🙂

Basically I'm OK telling users "you can get auto-generated bindings to call any dagger modules from your external code. You just need to have a dagger.json + dependencies in your external code - making it a 'top-level module'"

So I'm ok sayng "this will not work if you don't have a dagger.json in your project"

[sullen wigeon]

If you wrap the exec in a dagger run, that’s a different story, it’s not loading anything right now but that’s fairly easy to add, so we just focus on generating client bindings

[sullen wigeon]

Dagger.json + dagger run is the simplest case

[rotund kite]

But why do you need dagger run? the SDK will execute dagger session which will take care of it no?

so even raw go run should work

[sullen wigeon]

Hmm actually you’re right! I got confused by my poc because it generated a module library that I had to adapt to make it work, but if we generate a standalone client, then yeah that would work without dagger run

As long as we load modules in the server at some point before executing any operation

[rotund kite]

But then if you go build and execute the binary later, it will fail because it looks for dagger.json ? So maybe you'll need to embed the dagger.json in the binary somehow

[sullen wigeon]

That’s a very good question

But not necessarily, if we generate a client based on the dagger.json, the path to load would be hardcoded so it should be fine

[rotund kite]

So yeah I think you're right that init() function would have to be generated from the contents of your dagger.json and/or generate a //+embed dagger.json with the right path depending on the location of the generated library

[sullen wigeon]

However it will be complex to load local modules

[sullen wigeon]

Yep!

[sullen wigeon]

Both solution works, except for local module again, but I think that’s a fair limitation

[rotund kite]

yeah at least for mvp this would be amazing

[sullen wigeon]

Or maybe we can embed the local module tooo

Well, I’ll work on that, I’ll have to update a bunch of thing in the codegen but it should be okay.
I’ll try to have something that work and we’ll see how to clean the implementation after

@upbeat iris I’m pinging you so you can catch up Monday and share your thoughts, I’m free for a call anytime if needed 🙂

[rotund kite]

Thanks @sullen wigeon ! It's a happy accident that you need this for magic sdk, because it's badly needed for other use cases also!

[livid pier]

one thing I would like to do is distribute a Go CLI that builds on Dagger, and in user projects, they could specify Dagger modules to write custom workflows or steps. (think something like BuildPacks with customization)

[sullen wigeon]

That should be possible with this

@tiny rock @mortal jetty @real ridge @upbeat iris Would you be against simplifing the go codegen binary by splitting the generations steps?
What I'm meaning is having subcommand to generate each files independently, specially because I also would like to bring some of the Ts related configuration inside the codegen (for tsconfig & init template)

We would end up with:

codegen module --language=xxx [options]
codegen client --language=xxx [options]

So this way it's much easier to know what we expect to be generated, because with some of my modification it becomes a bit ( a lot) messy to understand what's happening

Some part might be a bit duplicated and we can certainly refactor that after.

The idea is just to split what's generated instead of doing everything in a batch, this would also simplify the runtime at some point and make the split of the SDK interface simpler

Why?

• Because sometime we want to use the published library, sometime not (for both module or standalone client)
• Because the go codegen is doing way to much things IMO (module support, client, go mod tidy, copy internal needed package)

In module we could have a --init flag, to specify it's an initialization, that would let us implements https://github.com/dagger/dagger/pull/8201
With some differences but it could work

[real ridge]

Yeah, so for modules, you'd run the module gen command? For clients you'd just run the client gen command?
And each of those would generate all the needed files?

If so, yup, I like it

Seems like a much needed simplification

There's already some subcommands? Which is confusing, so sounds good to clean it up

[real ridge]

--init would also be good to be part of this

[sullen wigeon]

I was thinking about something different: module would only generate module support and client would only generate client.
A current dagger module is a combination of both (for go because we have 2 dagger.gen.go), for TS it's weird because the codegen is split between the runtime & the codegen

[real ridge]

One warning though, do be sure to make sure tests keep passing and such, our coverage might not be perfect either? So as much as possible, we should make sure that we don't break existing modules

[real ridge]

Oh I see

Hm, if that's possible to do cleanly, then suuuure

[sullen wigeon]

It shouldn't break anything

[real ridge]

It'll cache nicely, since we can have the client cache only on the dependencies

So yeah, okay, happy to go for that I think

(btw feel free to rename either of the dagger.gen.go - but note, it's hard coded specially in the go codegen, so you'd need to fix those references)

[sullen wigeon]

It's also because at some point, I would like to implement Helder's idea to just use the published library (specially for typescript)

[real ridge]

Mmm, yeah we discussed this kind of idea right? (Over discord last week)

[sullen wigeon]

Yeaah, I have a looot of work

[sullen wigeon]

Yup! But as we discussed, for our integration tests we need to use the local library

So the idea is that we handle both, but it's only for our integrations tests, users should only use the published library

[real ridge]

Okay cool

[sullen wigeon]

Also that means that the SDK could now correctly expose 2 functions, one for the runtime and the other for the client codegen
And they would call one codegen function or the others

[real ridge]

Since there's a lot here, does it seem feasible to split any of this work up easily?

I mean it might not be possible

But if so, since it sounds like a lot, it would make it a lot easier to understand and review

[sullen wigeon]

Yeah, all that I have mentionned above I'll do it in a follow up

[real ridge]

Yeah I like that

[sullen wigeon]

But that mean my first implementation of the client generation will be strange, because I used if statement to avoid any collision so follow up PRs will clean that up by implementing the ideas above

[real ridge]

I might do the work sometime to add another "introspection" step to the sdk interface (I think I mentioned this before too)

[sullen wigeon]

And honestly it's harder for me to consolidate everything than just do a clean implementation

[sullen wigeon]

Yeah that would be very nice!

[real ridge]

Okay cool 😎 sounds like we're all on the same page

[sullen wigeon]

Yeah, I'm almost done with the Go & TS support for client generation, it's not super clean but I think it's okay enough to be merge when tests are written

[real ridge]

Eyyy 🎉 lemme know if/when you want a review, happy to read something whenever even if it's not fully ready

[sullen wigeon]

I have only one weird issue that I couldn't fix yet around external implementation of the client gen, for a reason that I can't understand, it doesn't pick up the external implem module name so it fails to work, but that's not my priority to fix for now (it targets a user that might want to creates its own client generation module so 0 for now)

[sullen wigeon]

Tomorrow, once I pass integrations tests 😄

[real ridge]

Yeaaaaaa I think as long as you pass the integration tests, all should be good 😉 all the insane edge cases I can think of that would be hard to handle have tests for them

One thing I might remove though - the --merge codegen flag for go

I don't think we need it anymore, we haven't used it in ages (cc @dull coral)

I already suggested Erik removed it in his PR

[sullen wigeon]

Nice!

[real ridge]

That might remove a bit of complexity there

[tiny rock]

I will be doing this btw, I was attempting to see if I could keep it just to reduce the already large blast radius of that PR but it’s not actually possible to support without profound hacks

All the above sgtm too. I don’t think in the long term we probably even need that logic in a binary at all and it could instead just be in the api, but can’t really do that yet since we have to wrap it in an execop to get caching from buildkit

But I think the work above won’t go to waste in that hypothetical future anyways

[real ridge]

awesome 😄 yeah, won't be sad to see that go, sounds like with both your PR and @sullen wigeon's upcoming work, we should be simplifying a lot of the codegen 🙂

[real ridge]

👍 👍 yup! ideally, we'll just wrap this with the upcoming CustomOp and have it built directly into the engine
might still need to expose something, since typescript also uses that code, but all the other sdks use their entirely own thing

[upbeat iris]

but all the other sdks use their entirely own thing

Hopefully not for long. I want to centralize codegen in all SDKs through the API. At minimum to return a json result that is a dagger specific higher level abstraction, based on type defs. Then SDKs could choose to use their own thing, but just for presentation. The source of the introspection would still be the API. So I see a future where all of this converges.

[real ridge]

yup, if it's api-ified, then it's even easier, we can just use DagOp as a middleware

literally a one-line code change to add buildkit-style exec-op caching

[upbeat iris]

That simple? 🙂 Can you point me in the right direction to implement that? Basically, instead of returning an introspection.json like we do to SDKs, for example, I want to expose an API field that returns a json result based on the current type defs (like currentTypeDefs).

[real ridge]

we'd just do this: https://github.com/dagger/dagger/pull/9395/files#diff-89b0b6c38f29bb6615327b6126c2479d29b032ef974017a9358c0c0721fa17f9R571

then it gets automatically run as a buildkit op

[sullen wigeon]

@real ridge @upbeat iris https://github.com/dagger/dagger/pull/9531 PR is up and ready for review, I wrote a bit of integrations tests and I'll add more tomorrow (for self call, source overriding etc...) 😄

The codegen part isn't very clean but it will be improved, I also got some CLI improvement to handle but will do in follow up PRs
The feature works and that's the target for this PR 😛

Still need to add Python support, you can do it in a follow up PR if you prefer Helder

@tender dirge This also impacts the Java SDK if you want to integrate the feature 😄
Let me know if you want to have a chat about it

@naive summit you may also be interested in that one for PHP 😄

And finally @steep steppe for Elixir 😄

[real ridge]

left a high level review!

hopefully keeps this moving 😄

[sullen wigeon]

Thanks! Will take care of them

[sullen wigeon]

@real ridge I see all my jobs failing because of error: parse selections: parse field "secret": Query.secret has no such argument: "uri"
Is there something I'm missing? It doesn't seem related to my PR

[real ridge]

it's passing on main 😛

so i imagine it is something subtle in the pr

[sullen wigeon]

I'm checking locally, because it's a strange behaviour, it shouldn't fail because dagger call isn't impacted by my changes

Yup, locally it works lol

I'll dig to understand the reason

[real ridge]

🤔

somethings trying to use the new v0.15.4 api that was just released

but the engine in this pr is v0.15.3

[sullen wigeon]

Yeah that's what I'm understanding so far, but why would it work on main and not in the PR?

The engine in this PR is also 0.15.3

Maybe I should update my dagger cli and run dagger develop?

I'm still in 0.15.3

[real ridge]

triggering another run on main to see if it happens again

https://github.com/dagger/dagger/actions/runs/13289937745

welp it breaks on main 😛

so guess i go fix that

lol this is wrong https://github.com/dagger/dagger/blob/c604bc2c2e59d52b275bddb7d0d1c7544a841a98/modules/gha/steps.go#L50-L50

[sullen wigeon]

Sorry haha

[real ridge]

working on it now 🙂

[real ridge]

okay fix in https://github.com/dagger/dagger/pull/9570 - if you can review, we can merge and unbreak ci 😱

funny how we never actually found this till now

[sullen wigeon]

Tests are green for https://github.com/dagger/dagger/pull/9531, waiting for additional reviews 😄

I'll also add more documentation in code + tests so it's more understandable

[steep steppe]

Change dagger run to load a module if one is defined and serving the module with its dependencies.
It means that we can compile a module into a single binary and run it?

[sullen wigeon]

Yeah that's doable, but a standalone module, I don't know if that is something very useful but it's possible:

✗ dagger client add --generator=go --local-sdk        
✗ go build -o test .                          
✗ dagger run ./test

result: hello, world!

And technically it was already possible with just the client library, now it's possible to also call dependencies from a simple go file (not in the context of an module implementation)

[sullen wigeon]

@real ridge running outside dagger run now works: https://github.com/dagger/dagger/pull/9531#discussion_r1955159466 😄

[sullen wigeon]

However, it seems that it breaks some tests, I'm digging to understand what's happening
Right now I changed it with an extra parameter, to see if it solves the issue, will clean that tomorrow

[rotund kite]

@sullen wigeon I left some comments on the PR

Mostly I'm wondering about the client path in dagger client add

and how it could be used as an identifier for remove, update etc

on the one hand, making the target path explicit gives us an easy identifier., and it makes it more clear what's going on.

On the other hand, the minimum command is longer. In some cases maybe the generator can infer a good default path, for example internal/dagger/ ?

Perhaps it's enough to say "provide a target directory path. Then the generator will write client files there. The client must not erase any files already present"

or, if we think it's better for the user if the generator can provide a default path (relative to the module root? to the repo root?) then we could have the generator return the path, so the path could still be used as an identifier in the dagger.json.

For example:

$ dagger client add --help
Usage: dagger client add GENERATOR [PATH]

Configure a generated client, using the specified generator, and write it
at the specified target path.

$ dagger client add go
Adding Go client: ./internal/dagger
Generating Go client: ./internal/dagger

$ dagger client add go
Adding Go client: ./internal/dagger
Error: client already configured at path ./internal/dagger

$ dagger client add go ./pkg/foo/bar/my-dagger-client
Adding Go client: ./pkg/foo/bar/my-dagger-client
Generating Go client: ./internal/dagger

$ dagger client rm ./internal/dagger
Removing Go client: ./internal/dagger

$ dagger client list
PATH    GENERATOR
./pkg/foo/bar/my-dagger-client    Go

$ dagger client generate ./pkg/foo/bar/my-dagger-client
Generating Go client: ./pkg/foo/bar/my-dagger-client

I personally CANNOT WAIT for:

$ dagger client add github-actions
Adding Github Actions client: ../../.github/actions
Generating Go client: ../../.github/actions

By the way I think we will need dagger client configure to pass generator-specific configuration... Just connecting the dots with your ongoing work #1338841380960866345 @dull coral @real ridge @radiant copper

[rotund kite]

FYI https://github.com/dagger/dagger/issues/9582

[real ridge]

i like that yup

[steep steppe]

Have a quick PoC for Elixir here https://github.com/wingyplus/elixir-generator. I wonder how the localSDK argument works (currently, I ignoring it) and where should we put a function to (runtime, or a separate module).

[sullen wigeon]

Great!
LocalSDK is for integrations tests! To do so we need a local copy of the SDK sources in order to tests latest changes.
For example: imagine you introduce a change of parameter in the client connection, you will not be able to tests it in the dagger tests suites without it (otherwise we need to fetch the branch version of the package which is a possible solution too but was longer to implement + it's not working locally if you didn't pushed your latests changes)
However, I'm not sure you Elixir is integrated in our pipeline so you can just ignore it 🙂

[steep steppe]

Thanks for the information. It is clearly see that we can ignore it for now.

And one more question, we should put it to the runtime module right?

[sullen wigeon]

Yes exactly! 🙂
Later I'll work on a split of the SDK interface which would allow you to provide a module that just implement that specific function but it's not done yet (and will be done in a different PR indeed).
And for the language based SDK like yours or Go, TS, Python, PHP etc... I think it's much simpler to always have 1 module that implements the complete SDK interface

[upbeat iris]

Hopefully the "runtime" module will be called the "sdk module" eventually. So the sources dir in sdk/elixir/runtime could be renamed to sdk/elixir/.dagger.

[rotund kite]

@sullen wigeon could you add to this issue the proposed generator module interface, so we can bikeshed that API? I saw @steep steppe above wasn't sure about the meaning of one of the arguments for example

[steep steppe]

Agree. This would be an easy because it just rename a directory. 👍

[sullen wigeon]

https://github.com/dagger/dagger/pull/9531 is pretty much ready, CI failure are known flakes unrelated to the PR changes.
I'm waiting for some reviews, and then I'll keep going on the UX implementation (https://github.com/dagger/dagger/issues/9582)

[rotund kite]

Thanks. Added some more comments on the interface in my issue. I don't think the interface is finished.

[sullen wigeon]

https://github.com/dagger/dagger/pull/9531 is merged, now I'll focus on the UX and persistency 😄
I'll try to split the work into small PR so we can iterate super fast 😉

[rotund kite]

I'm happy about the feature but not happy that my concerns about the generator interface being too complicated have not been addressed

I worry that it is too tied to the internal details of our existing SDKs, and that the experience of writing new generators will be bad

[real ridge]

@sullen wigeon can you make an issue for this concern, and add it to the next milestone

Until it's discussed and resolved it shouldn't go out in the release imo, I hadn't realized this was still unaddrressed

Since setting the interface and having other non core SDKs implement it is very annoying to undo

[sullen wigeon]

The issue already exists: https://github.com/dagger/dagger/issues/9582

[sullen wigeon]

I'm open for comments/suggestions, I know the pain point is around localSDK, but it's important for development, however I'm open to any suggestion that could solve that problem in a better way

[rotund kite]

I guess I just don't understand it, so it's hard for me to have a clean opinion on the design.

@tiny rock @radiant copper @tender dirge does that localSdk flag make sense to you, and if so can you explain it to me? 🙏 😇

The other one is passing the introspectionJSON as a string argument. I dont understand why the generator module can't be given a Module type and introspect that? What's the point of having a full introspection API if we tell clients to not use it?

[tiny rock]

Asked @gray dragon about it here: https://github.com/dagger/dagger/pull/9531#discussion_r1965337538 I was hoping/figuring we could find a way to avoid it in future updates but it enabled testing for now so seemed okay for a first iteration (and just make it hidden if it ends up in a release)

[rotund kite]

I guess I just don't understand what the "local SDK" is

all the explanations seem to consider it to be self-evident

[tiny rock]

Believe it is just referring to the local code implementing SDKs (as opposed to SDK from official release), but Tom can confirm

[rotund kite]

Also @sullen wigeon when you say "it's merged" does that include dagger client subcommand? If so, just double-checking that it's hidden for now?

[rotund kite]

I see, so the core client library I guess the opposite of the core client library - like a development version of the core client library?

[tiny rock]

Yes that's my understanding

[sullen wigeon]

Not yet, this will come in follow up PR

[rotund kite]

update: just discussed this live with @sullen wigeon , it's much more clear now 🙏

[sullen wigeon]

Things that will change:

• IntrospectionJSON will not be passed as an argument but instead fetched using an API call on the given ModuleSource, so generator that doesn't use this file doesn't need it as a parameter.
• LocalSDK: there are 2 possible solutions:
  1. Renaming it to dev, so in dev mode, the behaviour of the generator may vary (that's the case for our SDKs)
  2. Adding an API call to get the current engine version from the client generator func (maybe it's already available @tiny rock ??), so it's up to the generator to know if that's a dev engine and adapt his behaviour
     I personaly prefer the solution 2, since it's much more scalable and can be use for any use case, not just the client gen 🙂

So the final interface will be:

"A client generator for Dagger modules"
interface ClientGenerator {
  "Generate a client usable by the given module"
  generateClient(module: ModuleID!) DirectoryID!
}

Like @rotund kite suggested in https://github.com/dagger/dagger/issues/9582#issuecomment-2664187269

Next steps for me are:

• Applying these changes (I'll do 2 different PRs)
• Add the persistency in dagger.json
• Handle deps bundling so a go bin can be executed anywhere and not just in the module root directory (not a priority though because we could leave that to the user responsibility?)
• Add the others CLI commands

[rotund kite]

Nice thank you!

Separately from that, I have a new side quest: convincing official generator to bundle EVERYTHING (module bindings; core client library; dependencies of core client library; everything) in a single flat file with zero dependencies 🙂

[sullen wigeon]

No dependencies is impossible IMO, that means you would need to use only the standard library :?
Even for the Go client, it uses lib like"github.com/Khan/genqlient/graphql" which has dependencies too, so you would need to vendor everything

Actually, vendoring everything is possible, specially in Go but for Typescript that almost impossible, you would end up with a 400MB directory

[rotund kite]

The SDK has 400MB worth of dependencies?

that doesn't seem like a good idea

[sullen wigeon]

The TS SDK yeah, you can ask @spiral plank 😄

[rotund kite]

Anyway there's a reason I called it a "quest" 🙂

[sullen wigeon]

welcome in Node world 😮

[spiral plank]

If it makes you feel any better, they compress to 25mb iirc XD

[radiant copper]

@Erik Sipsma @Rajat Jindal @Yves does that localSdk flag make sense to you, and if so can you explain it to me? 🙏 😇
The other one is passing the introspectionJSON as a string argument. I dont understand why the generator module can't be given a Module type and introspect that? What's the point of having a full introspection API if we tell clients to not use it?

(noticed that you had discussion with Tom regarding the localsdk variable).

[sullen wigeon]

@rotund kite I'm currently working on the CLI and I noticied that dagger client generate is almost the same as dagger develop, so do we really need that command?
If a user wants to regenerate its clients, he can just run dagger develop and it will work 😄

[rotund kite]

I think dagger develop needs to be decoupled in the same way that SDKs do

develop becomes "client generate + other things" just like SDKs become "client generator + other things"

[sullen wigeon]

So if I have a module with clients + module code, I would need to run both command to update everything?

[rotund kite]

I don't know. What would the other command do exactly?

You mean client & server code?

[sullen wigeon]

Today dagger develop generates both clients (generated + module client) and server codes
In the issue: https://github.com/dagger/dagger/issues/9582 you said that dagger client generate should only generate client, but not server code, which is doable but why? Since I can just run dagger develop that will already do it