Python runtime

[slender nacelle]

@trail owl Where is the sdk files being added to the sdk runtime container when you access them with WithMountedDirectory("/sdk", dag.Host().Directory(root()))?

[slender nacelle]

I can see the files I want (filtered) in the built dev engine container after ./hack/dev, but it doesn't reflect on that dag.Host().

[trail owl]

Following the chain from the beginning:

1. The python sdk is a "builtin" sdk, so we include everything needed for the sdk/python/runtime module in the engine image as part of our CI: https://github.com/sipsma/dagger/blob/4a672b4bab8b03878118f5c16552848da72b5939/internal/mage/util/engine.go#L225-L225
2. When the python SDK needs to be loaded in order to serve as sdk for some module, it's loaded from the engine container and turned into a module: https://github.com/sipsma/dagger/blob/4a672b4bab8b03878118f5c16552848da72b5939/core/schema/sdk.go#L231-L231
3. Then, because the python SDK itself is using Go as its SDK, the logic for mounting the directories is here: https://github.com/sipsma/dagger/blob/4a672b4bab8b03878118f5c16552848da72b5939/core/schema/sdk.go#L354-L354

[trail owl]

Could it have something to do with the fact that there is both the "actual" SDK (i.e. ./sdk/python/src) and the "codegen" one that gets included in the module dir? Are we somehow using one when we meant to use the other?

[slender nacelle]

Oh, could be. The local one inside the module should be ignored, i.e., not sent as part of the module's files.

[trail owl]

Is that what the behavior was before? I know there's the vcs ignored files, but I thought they were exclusively used to create a .gitignore, rather than used to include/exclude files as part of module loading

There's existing dagger.json fields include and exclude for configuring what should and shouldn't be loaded as part of the module source dir

Maybe we want dagger mod init --sdk python ... to automatically set those values?

[slender nacelle]

So I guess in Go modules the locally generated files are always sent as part of the modSrc?

[trail owl]

I believe they are currently, yes. We in practice ignore them because we just re-run codegen anyways (to make sure it's all up to date), but they do get loaded

Slightly inefficient I guess but not to the point of being a bottleneck (afaik)