#dagger client install - any way to customise serveModuleDependencies?

1 messages · Page 1 of 1 (latest)

novel vapor
#

Hey, have been playing around with dagger client install go. I see that the dagger.gen.go exports a Connect function which in turn calls serveModuleDependencies, but this is hard coded to assume the moduleSource is ".". For our use case we want to use the generated client in our go cli so we get the nice client feel with the type safety and whatnot (previously we were just doing graphql), but we want to load the module from elsewhere on disk.

The main workaround I see from our perspective is we can initialise Client ourselves and mimic Connect with our own loading logic. This would be fine, but Client fields are private so I we'd need to codegen a NewClient function or something next to the dagger.gen.go. I tried this out and got it working, but obv feels suboptimal and wondering if you have any thoughts or if I'm missing something?

plain aurora
#

In the case of remote modules, you'll see that modSrc := client.ModuleSource(".") is pointing to . because the module full path is already present in the source path in dagger.json

novel vapor
#

the setup is:

/mymodule/dagger.json (sdk source=go)
/mymodule/main.go (my dagger funcs here)
/mymodule/dagger/<generated-client-here>

/mycli/... (i want to use the generated client here)

possible that i'm not adhering to the intent or scope of this client generate idea, but i'm unsure. i was hoping it would create self-contained "client bindings" basically that i could just copy over, but i notice that it isn't entirely self-contained because of the (a) implicit module loading and (b) the dag/dag.gen.go imports dagger via the package one level up by inferring the package name which also seems a bit tricky

actually now that i'm running through it with you... is this the right idea?

/mycli/dagger.json -> has dependency mymodule

and then i do a client generate in mycli instead?

plain aurora
#

when it comes to local modules, those always need to be referenced within the same monorepo of your project. This is mostly a security constraint so modules can't really be imported anywhere locally from disk.

#

i was hoping it would create self-contained "client bindings" basically that i could just copy over, but i notice that it isn't entirely self-contained because of the (a) implicit module loading and (b) the dag/dag.gen.go imports dagger via the package one level up by inferring the package name which also seems a bit tricky

the bindings that it generates are not entirely self contained because whenever you run your CLI in the future, it will need to have access to the module source so it can be pre-loaded in the engine before the generated bindings can effectively call the module functions

#

The best way to solve this is to version your module in a git repo and reference it by using the remote ref so that way your CLI doesn't depend on the local module's source to run anywhere

novel vapor
#

it will need to have access to the module source so it can be pre-loaded in the engine before the generated bindings can effectively call the module functions
yeah this is what i was hoping we could take control over (e.g. custom serveModuleDependencies, or non-private Client fields to achieve the same). i.e. import the client purely for the bindings and we are responsible for loading the module into the engine prior to invoking any calls via the bindings. it seems achievable through measures like replace in go.mod for the dagger import + exposing a NewClient function to workaround the unexported fields problem, but those are not so clean.

version your module in a git repo and reference it by using the remote ref so that way your CLI doesn't depend on the local module's source to run anywhere
to be fair this could be nice and clean. is it possible to remote ref a private git repo? this is a company-internal CLI tool, although i think we can safely trust our internal users have access to our private git repo

but overall we would prefer the former option as right now we go:embed our module into the binary, extract it at runtime, and load it into the engine. although our approach is probably a bit unusual we find to be an easier pattern than having to version our cli and module separately and keep track of that plus it keeps it quite self-contained

#

the flow i was planning to do is roughly (sketch):

func serveModule(ctx context.Context, client *ourdagger.Client, srcRef string) error {
    modSrc := client.ModuleSource(srcRef)
    configExist, err := modSrc.ConfigExists(ctx)
    if err != nil {
        return err
    }

    if configExist {
        if err := modSrc.AsModule().Serve(ctx, ourdagger.ModuleServeOpts{
            IncludeDependencies: true,
        }); err != nil {
            return err
        }
    }

    return nil
}

func main() {
    dag, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
    client := ourdagger.NewClient(dag, dag.QueryBuilder(), dag.GraphQLClient()) // <- this isn't a real func, Client fields are private
    extractOurModule()
    if err := serveModule(ctx, client, "/path/to/our/module"); err != nil {
        panic(err)
    }
    client.OurModule().OurFunc()
}
plain aurora
#

but overall we would prefer the former option as right now we go:embed our module into the binary, extract it at runtime, and load it into the engine. although our approach is probably a bit unusual we find to be an easier pattern than having to version our cli and module separately and keep track of that plus it keeps it quite self-contained

I was about to suggest this next 😛

#

yeah I see where you're coming from. TBH it doesn't seem ( @delicate void would know better) something extremely complicated to do. At the same time, as you're pointing out, it seems to be something quite specific to your use-case. This is the first time I recall seeing this type of request. Mostly bringing this up as I'm unsure the level of prioritiy this issue / feature requires might have 🙏

delicate void
delicate void
plain aurora
#

what @novel vapor is expecting is to have a way to generate the client bindings without all the serveModuleDependencies calls

#

they'll take care of pre-loading the modules in the engine

delicate void
#

I see

plain aurora
#

they're just looking for a way where the client bindings only contain the bindings

#

and not the module loading logic

delicate void
#

That's tricky because in 99% of the case, you want to rely on the auto loading

plain aurora
#

not sure how coupled the code is

delicate void
#

A flag or a manual config in the dagger.json clients field maybe

#

Because it's a very specific thing, that we cannot test easily btw

#

@novel vapor How do you load the module in the server if you cannot connect the client?

#

Because to do dag.Host.Directory(".").AsModule... you need to be connected

#

So you would need to 1: use the default library, load your module and then reconnect with the generated bindings

#

If that make sense

#

Or connect the generated client but be careful because you can only use the core until you manually Load/Serve the dep

novel vapor
#

would there be appetite for adding a NewClient func in the generated code? tbh that seems to be the only thing missing. although there's still the issue of the import paths in dag.gen.go but perhaps that can be resolved via the dagger.json in mycli instead of copying over the generated code

delicate void
#

I have an idea

novel vapor
#

like this:

    dag, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
    client := ourdagger.NewClient(dag, dag.QueryBuilder(), dag.GraphQLClient())
#

(yes, using dagger default lib)

delicate void
#

what if I add a flag to the generated client to disable module serve

#

So you could do `dagger.Connect(ctx, dagger.ServeModule(false))

#

Or something like that

#

And then I could make the internal serveModuleDependencies public, so you can call it later, or do your own thing

#

The only problem is that you cannot rely on the global dag package (which honestly I would love to not generate because it adds a looot of complexity)

novel vapor
delicate void
#

hmm

#

I'm looking at the code and NewClient may be a better option

#

So you can also override the global dag

#

dag.WithClient(c)

#

Adding a dagger.ServeModule option to Connect is tricky because that option would be part of the published lib... But the only use case for generated client.
On the other side, exposing primitives to connect your own client, that may work

novel vapor
#

qq: what's the reason the generated go client is not a go module? rn it just emits go code into the current module but that makes it difficult to import elsewhere. not sure i understand the correct approach to importing it into my cli package or the intended usage

delicate void
#

It creates a go module if no go.mod is found

#

And that's because dagger/dag requires a go.mod to generate the correct import path (because dagger/dag depends on dagger/dag.gen.go)

delicate void
novel vapor
#

It creates a go module if no go.mod is found
ah i see, but how can i achieve that? the command seems to fail if i try to emit the code up a level (e.g. outside of the current module):

$ dagger client install go ../client
Error: failed to export client: input: moduleSource.withClient.generatedContextDirectory failed to generate client go: failed to generate clients: failed to get modified source directory for go module sdk codegen: select: failed to stat file /:

but it succeeds otherwise:

$ dagger client install go ./dagger
delicate void
#

Are you in a git repo?

#

Dagger isn't allow to generate code outside of its module

#

So ideally you should put your dagger.json at the common parent

#

Like

dagger.json
my-cli/
my-module/
#

then dagger client install go ./dagger-client for example

novel vapor
#

ah nice thank you, i'll try that

delicate void
#

Np! Let me know if I can help you more or if there's another problem

#

Meanwhile, I'll think about that optional serve, I need to find the right DX