Generate module client binding for library usage

[stoic spire]

Hey 😄

I'm trying to do some experimentation to see if it's doable to do some client binding generation of a module outside of it.
My idea is quite simple, I retrieve the introspectionJSON of a module and then run it through the codegen + do some post process to make it usable.
It's a utility tools to let me call external module from a simple client library without having to manually parse/serialize the input/output of my GQL request.

However, I'm having an issue getting the codegen binary, it seems that it's not available in the engine container https://github.com/dagger/dagger/blob/f4232c607140a03795ddeb9b5fa6698456d8cc02/.dagger/build/builder.go#L241

Is there a good reason for that not to expose the builder to the module or the codegen binary with a method?

/cc @obsidian sonnet @harsh pumice 😄

[stoic spire]

I opened a PR with the changes https://github.com/dagger/dagger/pull/9485
so we can continue the discussion from there

[unique narwhal]

@stoic spire trying to understand why you're requiring the codegen binary to be in the engine container

i.e: the dagger shell example you posted in the issue will only work as long as you're running it in the dagger/dagger repo

and seems like your PR is building the codegen binary for the produciton engine as well

which doesn't seem very useful

[stoic spire]

No, you can use the module from outside the repo

[stoic spire]

I’m trying to fetch the codegen binary from the dagger/dagger module so I can use it in another module, is there another way to get it?

It’s useful to use the codegen bin outside of dagger/dagger

[unique narwhal]

haven't checked but don't we have a function in the dagger/dagger repo that returns the codegen binary?

[stoic spire]

Nope, that’s my problem

And I thought there was a reason for that so I did it implictely, but if you tell me that I can add a function, I’ll be happy to

[unique narwhal]

I think exposing the function might be a better alternative than putting the codegen binary in the engine

[stoic spire]

I agree, I’ll do that then

I’ll update my Pr with some changes

[unique narwhal]

seems like a better approach to me 🙏

not sure where the codegen function should go 😄

[stoic spire]

Same as engine IMO,

[unique narwhal]

SGTM, we can try there

[stoic spire]

Actually I have a better idea, I should just add a new introspectionFile argument to all our generate function, so it can generate any provided module.
This would have much more value I think and simplify the usage

And indeed, it would be an optional argument 😄

So by default it generates the classic SDK codegen

Oh actually it might not work for every sdk

[stoic spire]

@unique narwhal @silver bane @deep jungle @chilly moth Hey 😄

I would like to bring your attention on this PR: https://github.com/dagger/dagger/pull/9485

It's a simple experimentation but this can be useful for MagicSDK and also improve the DX for people who want to use modules from the dagger lib.
There's a lot of work to do to make it work but it's doable.

Let me know your thought and if that's worth continuing

Also @obsidian sonnet 😄

Generate module client binding for library usage

[chilly moth]

For reference we are also discussing this here: #1334452944740814931

@stoic spire I am very very interested

[stoic spire]

Oh I didn’t saw the ping, my discord is weird…

[chilly moth]

No problem 🙂

[harsh pumice]

Looks very neat 🙂 kinda curious how it would work for python that doesn't use the codegen library

Some ideas:

• would be good to support multiple module sources, like a couple of dependency modules
• maybe have one function that combines both functionalities? then it's even easier to use

Kinda generically curious about how we would ship this though, it's good as a PoC, but ideally users would just have this through the dagger CLI instead of needing a separate module

We already have some introspection functionality we could expand, but for the codegen maybe we just need a special engine codegen object or similar

But yes it looks super neat 🙂 if we can get it working with the other SDKs, we should merge it (since it's just in the dagger/dagger module, which doesn't really have any of the same stability guarantees of the engine)

[unique narwhal]

Looks very neat 🙂 kinda curious how it would work for python that doesn't use the codegen library

had the same thought while reading this. I guess a python function could be also added to the codegen type that somehow runs whatever python code we have today to generate the client, correct?

would be good to support multiple module sources, like a couple of dependency modules

sounds interesting and I'm trying to think what the use-case would be in this scenario. Tom mostly needed this as he needs to codegen a module with its dependencies. In which case do you see the need to potentially codegen multiple independent modules?

Kinda generically curious about how we would ship this though, it's good as a PoC, but ideally users would just have this through the dagger CLI instead of needing a separate module

yes, I'm curious about this also