Registering functions?

[brave shuttle]

Where could I go ti understand the plumbing for registering a regular function? ie WithFunction

In particular I’m curious to understand how type checking works there

[neat quail]

It varies per-sdk since the parsing is obviously extremely language specific.

The code for the Go SDK implementation in the Zenith branch specifically is a big ol mess since we were just going for a prototype quality implementation. You can see the implementation here for example though if you really want to: https://github.com/sipsma/dagger/blob/4e0ad448ad3041fb5ad18550399a07e8242d96cd/universe/nix/dagger.gen.go#L992-L992

For python, it looks much better: https://github.com/dagger/dagger/blob/4e0ad448ad3041fb5ad18550399a07e8242d96cd/sdk/python/src/dagger/server/_functions.py#L14

Basically, the overall idea is that Functions only have one limitation, which is that the types in the arguments+return must be JSON serializable (same inherent limitation of graphql). Other than that, everything should work.

[brave shuttle]

graphql plumbing is environment_function, withArg ?

[neat quail]

Yes, but I will make a lot of cleanups to the API once we have a Zenith Functions PR we actually want to merge. But the general idea there would carry over

Cleanups would be to use enums and such, IIRC there's a lot of "typing-by-string" in the current schema in Zenith branch

[brave shuttle]

is the idea to seralize little GraphQL snippets in a string?

ah ok not in a string, but a “graphql type” type

🙂

[neat quail]

No, that's actually closer to what we used to do; we used to have the SDKs generate a literal entire graphql schema file. But we realized that added a ton of overhead which wasn't necessary. You have to do all the work to figure out the types of the functions and then figure out how to convert that into a graphql schema file. Now, all they have to do is figure out the types of the functions and then make API calls that are extremely obvious. Construction of graphql schema happens server-side

[neat quail]

Yeah more or less. And I'd want to clean that up with Enums for all the different types rather than strings. But same idea essentially

[brave shuttle]

there’s a risk of a rabbit hole if you want to support arbitrary combinations of scalars, arrays and structs though right?

is it like “scalars only for now, arrays of scalars soon, the rest later”?

[neat quail]

There will have to be some limits somewhere, but honestly the very nature of graphql helps us out here. E.g. Struct().WithField("foo", Struct().WithField("bar")).WithField("blah", String())

[neat quail]

We do support structs already actually

[brave shuttle]

ok I see

what about custom types? or extending core types for chaining like you mentioned recently I think?

funny it’s like a continuation of the query builder pattern: the schema builder

[neat quail]

We don't currently support chaining of custom types and don't support extending core types. Those would both be possible in this model, just hasn't been fully drawn out or implemented yet.

[neat quail]

It's kind of crazy, I've really come to appreciate graphql more and more, it has a very nice balance of simplicity with just enough complication to make otherwise complicated stuff like this fall into place easily. Once you grok it

[brave shuttle]

so generally speaking, a big advantage of this model is less work for SDKs across the board right?

[neat quail]

Yes that's the whole point. Do as much server-side as possible. SDKs should only be responsible for their language specific details and hand off to the engine from there

[brave shuttle]

mmmm how do I support custom types here? Isn’t there a chicken and egg problem with codegen?

[neat quail]

If you wanted to return a custom struct, the SDK just needs to know the name of the struct (easy, or at least no harder than what it does today) and all the fields in it. Then, I guess the API would hypothetically look something like this (hypothetical because today we only support structs as arguments and we also "unpack" them into separate fields, but that was just to save time rather than inherent limitations):

dag.Environment().WithObject("CoolStruct", dag.Object().WithField("foo", "String").WithField("bar", "Integer"))

All names and specifics chosen above are strawmen 🙂 Just something like that

There shouldn't be any chicken-eggs because we don't codegen self-bindings for an environment

Unless I'm missing something you're thinking of

[brave shuttle]

but how do I register a function that takes that type as argument, or returns it?

and how would a downstream environment importing my type create their own function using my type?

wouldn’t that require FunctionArgType itself to be codegened?

[neat quail]

The same way that codegen already works. So e.g. if your environment, named myEnvironment, has something like this:

func main() {
  dag.Environment().WithFunction(MyFunc)
}

type MyInput struct {
  Foo string
  Bar int
}

type MyOutput struct {
  Omg string
  Wtf int
}

func MyFunc(in *MyInput) *MyOutput {
   ...
}

That gets parsed by the sdk and via the api we were talking about above. Serve-side, a graphql schema like this gets generated (ignoring namespacing atm):

input MyInput {
  foo: String!
  bar: Integer!
}

type MyOutput {
  omg: String!
  wtf: Integer!
}

type MyEnvironment {
  myFunc(in: MyInput): MyOutput!
}

From there, our existing codegen kicks in, it will operate the same way as it does today on that schema