Best way to extend or fork python sdk

[sterile marsh]

I am trying to fork the python. Basically I copied the sdk/python from dagger into a separate directory, and then creates a dagger.json that references the runtime. I was hoping that would just work, but it didn't. I'll continue to plug away but thought I would post to see if my initial approach is right.. there are comments in the code that indicate extensibility was thought out. Such as:

     //The following functions were built to be composable in a granular way,
// to allow a custom SDK to depend on this one and hook into before or

Haven't been successful with the fork so asking to see what the right approach is. How best should I extend or fork the sdk... I would prefer to extend I think....

[hoary scarab]

normally if you point the sdk field at the full repo adress of the fork, eg. github.com/chad/dagger/sdk/python or whatever, it should work as expected

any dagger module that implements the expected functions, can be used as a SDK, no other special magic is required

[sterile marsh]

But there is no dagger,json in that location. I had to create one to point at the runtime.

Since I need both the base code of the module and the go runtime . It looked like it was working then errored out on mounting a codegen dir

[hoary scarab]

I guess to isolate the problem you should start by getting it to work against the upstream. then transpose to the fork

maybe add /runtime to the address I gave you?

sorry not at a computer right now

[sterile marsh]

@hoary scarab So that didn’t work. Looking at this https://github.com/dagger/dagger/blob/main/core/schema/sdk.go#L266 it looks like it loads a built in container that has the sdk loaded and assumes a runtime directory. The python sdk module assumes this had been loaded which is why it can’t find the codegen directory. I ca start trying to mount those in in my fork but was wondering if there is an easier direction. So a little bit of magic 🪄

[hoary scarab]

@sterile marsh OK I tried also, and couldn't get it to work either. So now I'm really curious how to do it also 🙂

maybe @rancid veldt knows

[rancid veldt]

Hm yeah I'd expect a fork url or relative path to just work, too. We did it in the past to work around a typescript sdk bug iirc

I thought the special builtin containers path was just glue without any tight coupling between that and the module

What is the error?

[hoary scarab]

For reference, here's what I tried:

1. dagger init --sdk=python
2. Edit dagger.json to set the sdk field to "github.com/dagger/dagger/sdk/python/runtime"
3. Call dagger functions
4. I get this error:

✔ connect 1.2s
✘ initialize 1.4s
! input: moduleSource.withContextDirectory.asModule resolve: failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to call sdk module codegen: select: call function "Codegen": process "/runtime" did not complete successfully: exit code: 2
  ✔ resolving module ref 0.1s
  ✘ installing module 1.3s
  ! input: moduleSource.withContextDirectory.asModule resolve: failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to call sdk module codegen: select: call function "Codegen": process "/runtime" did not complete successfully: exit code: 2
    ✔ ModuleSource.resolveFromCaller: ModuleSource! 1.1s
    ✘ ModuleSource.asModule: Module! 0.2s
    ! failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to call sdk module codegen: select: call function "Codegen": process "/runtime" did not complete successfully: exit code: 2

Error: input: moduleSource.withContextDirectory.asModule resolve: failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to call sdk module codegen: select: call function "Codegen": process "/runtime" did not complete successfully: exit code: 2

Stdout:
marshal: json: error calling MarshalJSON for type *dagger.GeneratedCode: input: directory.withoutDirectory.directory resolve: lstat /codegen: no such file or directory

5. Confirm that the SDK address is in fact a valid module:

$ dagger -m github.com/dagger/dagger/sdk/python/runtime functions
Full trace at https://dagger.cloud/dagger/traces/18a7ed7dbc1b6cfd5eab2e217571cf33

✔ connect 1.2s
✔ initialize 1.2s

Name                  Description
base-image            Image reference for the base container
codegen               Generated code for the Python module
common                Common steps for the ModuleRuntime and Codegen functions
container             Resulting container after each composing step
load                  Get all the needed information from the module's metadata and source files
module-runtime        Container for executing the Python module runtime
required-paths        List of patterns to always include when loading Python modules
sdk-source-dir        Directory with the Python SDK source code
use-uv                Check whether to use uv or not
uv-image              Image reference where uv is fetched from
uv-version            Version to use for uv
with-base             Initialize the base Python container
with-base-image       Override the base container's image
with-container        Replace the underlying container
with-install          Install the module's package and dependencies
with-sdk              Add the SDK package to the source directory
with-source           Add the module's source code
with-template         Add the template files to skaffold a new module
with-uv               Enable the use of uv
with-uv-version       Override the uv version
without-user-config   Disable the discovery of custom configuration
without-uv            Disable the use of uv

[sterile marsh]

Just to confirm that is the exactly same error I get.

I also tried moving the dagger.json to the root of the sdk and ref the source runtime and got the same error… like the elixir sdk does

[hoary scarab]

@sterile marsh what about sdks other than python? same error?

[sterile marsh]

Hmm. Got stuck on python but I am now curious and will try elixir and typescript

My guess is that elixir will work since the dagger.json is at the root and it is including the code but just a guess

[sterile marsh]

So back on this now that I have had a chance to look more. Elixir worked, but typescript didn't. Slightly different error, but directory not found again. I am reasonably confident that the typescript and python sdks assume that the builtin container has been loaded and all their dirs. It doesn't load in the actual sdk code like elixir does and mount it. My current tack is to make it like elixir and mount those dirs. @rancid veldt how did you guys fork the typescript sdk? Is my approach the right one?

[copper cedar]

Hey @sterile marsh, there’s a simpler way to extend the Python sdk, using the granularity that you quoted originally (without having to fork the sdk), but depends on what you’re trying to do. What’s your use case?

Basic idea is that you create a new runtime module that depends on the official python runtime module. There’s an example in our test suite.

[sterile marsh]

So I am trying to run in an air gapped env. Got pretty far but the uv lock file made it impossible (I think) so really I just want to no include that, or delete the lock file.

When I ran with interactive mode and deleted the uv.lock file, I got past the error I was having...

[copper cedar]

I’ll need more specifics. You don’t need to fork the official module, just extend it.

[sterile marsh]

Okay. Figured I might be going about this wrong how do I extend it. I need to replace or remove this file https://github.com/dagger/dagger/blob/main/sdk/python/codegen/uv.lock.

Peeling the onion but that is first step

[hoary scarab]

@copper cedar is it possible to do what we were trying to do? eg. use the Python SDK as a regular module?

[copper cedar]

Here's an example of an extended runtime module, which adds a git OS dependency: https://github.com/dagger/dagger/blob/c1f0afa63aa7965da66f3b10183a258fe2234062/core/integration/testdata/modules/python/extended/src/main/__init__.py

[copper cedar]

You can use it with dagger call directly, but I suppose you mean using the git remote ref in --sdk=, then not yet. None of the built-in SDKs support it, see https://github.com/dagger/dagger/issues/7585. In Python it's easier than the others because it doesn't need the codegen CLI, just need to get the SDK's library in the constructor when not provided. I could do the same as Elixir and PHP but I figured I'd wait for context directories instead for that.

An extended runtime module like #1275215230623813805 message can work around that though.