Python DX

[tranquil lodge]

@calm pier, I started working on Python DX but there’s fundamental differences with Go.

1. I can’t add a custom .with_check() method to the generated Environment class like in Go.
2. It’s more natural for pythonistas to use decorators here, but it needs to be lazy (not a problem in itself, unless for the need to have alternative versions of generated types).

So the way I’m trying to solve this is by creating a dagger.server.Environment (≠ dagger.Environment), that’s used to lazily declare the environment and only on “serve”, make the API calls and save the envid or execute the resolver:

import dagger
from dagger.server import Environment

env = Environment()

@env.check
async def lint(client: dagger.Client) -> str:
    ...

if __name__ == "__main__":
    # equivalent to what happens in main() for Go,
    # i.e., makes the actual client.environment().with_check(client.environment_check().with_name("lint"))
    # call and handles saving envid or running the resolver
    env()

I want to make the decorators chainable as well in case it helps more advanced use cases:

(
    Environment()
    .with_check(lint)
    .with_check(all_, name="all")
    .serve()
)

I’m not sure about client. With this DX the client instance is managed automatically, only available in resolver functions. Not sure what the use case would be to need a custom one. For example, in Go the API follows a normal API chain from client:

func main {
    ...
    client.
        Environment().
        WithCheck(Lint).
        Serve(ctx)
}

I can accept a custom instance in .serve(client) to use instead of the default. Or is it desirable to create environments with client.environment()? It should be possible, although I prefer the DX above:

import dagger
from dagger.server import check, run

@check
async def lint(client: dagger.Client) -> str:
    ...

async def main(client: dagger.Client):
    await (
        client
            .environment()
            .with_(lint)  # or .with_(check(lint))
            .serve()
    )

if __name__ == "__main__":
    run(main)

Things get more complicated with subchecks. Same with Environment, I’d need a custom Check to build it lazily, but I can manage that internally:

import dagger
from dagger.server import Environment, Check

env = Environment()

@env.check
async def unit(...) -> bool:
    ...

lint = env.check(name="lint") # just a container for subchecks

@lint.check
async def engine_lint(...) -> bool:
    ...

@lint.check
async def sdk(...) -> Check:
    # Alternative, without decorator, for advanced control.
    # In this case it could be replaced with:
    #
    #     sdk = lint.check(name="sdk")
    #
    #     @sdk.check
    #     async def python_lint(...):
    #         ...
    return Check().with_subcheck(python_lint)

async def python_lint(...) -> bool:
    ...

if __name__ == "__main__":
    env()

Again, it comes down to the decorator based DX being more natural, but I don’t know if the client based chain is preferable to unlock some use cases I’m not aware of and if passing around the server variations of Environment, Check, Command, etc, will be too confusing to developers. They allow better sugar though.

[calm pier]

Hey Helder, just catching up on threads now, what you described creating a dagger.server.Environment (≠ dagger.Environment), that’s used to lazily declare the environment and only on "serve", make the API calls and save the envid or execute the resolver is pretty much exactly what I was imagining we'd do in Python. Honestly the only reason I didn't do that in Go is that there just aren't decorators 😁

I don't think we need to present the chainability to the end users at all. We definitely don't need to hide the lower-level "direct" API calls either, it's just that they would be reserved for power-users, with the decorator approach covering 99% of cases.

I think what you described in terms of your preferences and what's handled internally look great

There is still an open question in my mind around how to minimize the friction between keeping the codegen api and the "higher-level" apis in sync, that same problem applies to Go since obviously the graphql WithCheck and similar don't really accept go functions as args.

My thinking on that for Go was that I might customize the codegen a bit so that the generated code for e.g. WithCheck internally calls out to a private withCheck method that accepts the go-specific types like func. Then withCheck needs to be implemented manually, but if we forget to do it or something gets out of synced the SDK will fail to compile, rather than just going unnoticed until runtime.

I don't think that exact same idea would apply to Python + decorators, but maybe there's something conceptually similar. That's okay too since obviously each SDK has it's own codegen for a reason.

[tranquil lodge]

Yeah I was thinking on something similar like marking with a directive in the Api the fields like withCheck that will map to a resolver. Maybe not even needing the add the resolver field to Query or the Environment implementation.