#@wompfox can I ask you some weird

1 messages · Page 1 of 1 (latest)

regal sage
#

I remember that we chatted in person about SDK generation

#

can you point me to who in the Dagger team is doing some of that magic?

plain mist
#

several people in the Dagger team have different context about that. Given your TZ Mauricio, I'd say @cloud pewter and @brisk yoke might be the best Dagger people to ping.

brisk yoke
#

Hey, you can ask your questions here, I'm online 😄

regal sage
#

@brisk yoke thanks for reaching out

#

I am curious about how the SDK are being created

brisk yoke
# regal sage I am curious about how the SDK are being created

@formal juniper Feel free to complete if I misses any details since you built an SDK last month 😄

So a SDK is a module that is able to load user's code that will extends Dagger capabilities and be callable from the CLI.
All modules created with Dagger uses a SDK, for example you can use the Go SDK to write a module using the Go programming language (see: https://docs.dagger.io/api/custom-functions for more details)

It's composed of multiple things:

  • A client: Used by the end user to use the Dagger API in his original language. This client is generated from a GraphQL schema that defines all the type definitions (typedef) exposed by Dagger and its differents extenstions.
  • A introspector: This will parse the user code and transform it into compatible type definition and then register it to the Dagger API, that's how we dynamically extends the GraphQL schema 🙂
    For example in Go:
type Example struct {}

func (e Example) Hello(ctx context.Context, myArg string) string { ... }

This will register a typedef on Dagger of type Object, named Example, that has a function Hello that returns a STRING and accept an argument named myArg of type STRING.
It's quite standard, basically you convert user code into readable typedef, all of this is done dynamically with API call that you need to make 😉

  • A runtime: This will create a container to run the users module and call the given function dagger call hello --my-arg foo will execute the function Hello for example.

All theses features are exposed with a Dagger module that needs to implements 2 functions:

  • ModuleRuntime: Create the runtime and call the user's module, either to register typedefinition or execute a function.
  • Codegen: Generate the client and any necessary files needed to execute the module (some statically generate the user's code execution like Go, other does it dynamically like Typescript but copy the library locally, you're free on that side)

I think that's pretty much what you need to create a SDK, the best way to learn is to look at how other SDK are made (Typescript & Python) are the easiest to understand in my opinion 🙂

regal sage
#

hmm.. not sure this is what I was looking for..

#

@formal juniper so all the Dagger SDKs are handcrafted today?

#

I thought at least some bits of the SDKs were automatically generated based on graphql

formal juniper
#

As en example, I pushed a fresh module with the generated sources, so you can have an idea of what's generated vs what's not: https://github.com/eunomie/dagger-java-example

#

So basically I'd say there's three parts in what we can call a SDK:

  • the runtime: this is a specific module in which everything will happen, both code generation and execution. You can see that in some way as "the container in which the code will run".
  • the codegen tool: it can be in any language, as we saw it's in Go for typescript, but it's in Java for Java SDK https://github.com/dagger/dagger/tree/main/sdk/java/dagger-codegen-maven-plugin
    • this tool will be called by the codegen function of the runtime above
    • this tool will generate all the dynamic types based on the list of types returned by the Dagger engine (types will be in a "introspection json" file) (the output depends on the languages)
  • the language specific SDK code (I don't have a nice name, as SDK conflicts, in some way that's also runtime but it conflicts): basically the code written by hand to be the glue between the Dagger platform and the generated types + user code
    • this is the very specific part to any SDK. A very short description could be this is a simple GraphQL client, but it's a bit more, there's code about serialization (that is a bit more complex than what we can think at first look)
    • here is the one from java https://github.com/dagger/dagger/tree/main/sdk/java/dagger-java-sdk (this matches roughly the target/generated-sources/dagger-io in the message above)

In some way I'd say there's one more part, but it really depends on the language: the entrypoint.
The entrypoint is the code that will be called by the ModuleRuntime part of the SDK runtime. In multiple SDKs this is a dynamic code, so written only once, that will find the objects to register and call them.
In Java this code is also generated during the packaging. It's an extra phase, but as said, it depends on the languages and choices. In Java this is done by this code https://github.com/dagger/dagger/tree/main/sdk/java/dagger-java-annotation-processor

#

I don't know if I introduces more complexity or if that helps to understand better 😅
I have the idea to create a full doc of building a new SDK, but it still not done. Maybe I'll convert all that, that could be great

regal sage
#

yes.. I would love to help you review that doc.. that will help me to understand the details

formal juniper
#

which language would you like to write an SDK for?

regal sage
#

I am more interested in the mechanism than writing an SDK in a new language

#

because the mechanism should promote consistency across SDKs