Removing `uv.lock` file in dagger module errors python SDK install for v0.18.4

[hollow pine]

Synopsys:
With python dagger v0.18.4 removing the uv.lock file from a dagger module breaks the module and python SDK install step.

How to reproduce it:

mkdir test
cd test
dagger init --sdk=python
# don't fail
dagger call container-echo --string-arg=teststr
mv uv.lock _uv.lock
# fails
dagger call container-echo --string-arg=teststr

The last shell command returns the error: The package testrequiresdagger-io==0.18.4, but 0.0.0 is installed

From what I've analyzed the following changes in this PR are the reason for this behavior.

As far as I understand this is what's going on:

1. The dagger python SDK identifies that there is a pyproject.toml file and marks it as initialized. Later it modifies the pyproject.tomlby changing dagger-io for dagger-io ==0.18.4
2. The dagger python SDK installs the python dagger-io module and the project dependencies with uv pip install -e ./sdk -e .. The dagger-io sdk is installed from a directory so it's installed with the fallback version 0.0.0
3. The dagger python SDK runs uv pip check and fails because the pyproject.toml states that needs dagger-io ==0.18.4, but it has dagger ==0.0.0 installed

Removing uv.lock file in dagger errors python SDK install for v0.18.4

Removing uv.lock file in dagger module errors python SDK install for v0.18.4

[dapper cradle]

Thanks for reporting this @hollow pine - this is a known issue and will be resolved in 0.18.5

there's a fix and mitigation instructions in https://github.com/dagger/dagger/pull/10252

[hollow pine]

Thank you for the fast response @dapper cradle !
I have another mitigation that I find quite useful to avoid being hit by this issue. The solution is playing with python package normalization (https://packaging.python.org/en/latest/specifications/name-normalization/)
I renamed the dagger-io dependency to dagger_io. By doing this the replacement is not being triggered and no conflict arises.
Maybe it is useful for others too.

[undone siren]

I’m curious. What’s your use case for removing uv.lock?

[hollow pine]

Hi @undone siren !
I'm in the development phase of an SDK for multiple languages (starting with Python) as a dagger.io module. I decided not to use a lock file and just fix things when dependencies break the implementation.
Regarding the SDK implementation I choose not to use uv.lock as the dependency record solution; I would like to chose PEP 751 instead that will be in the next uv release (uv PEP 751) . Considering that the SDK is developed with itself, for this two reasons I removed the lock file.
Yes, I could have adopted the uv.lock solution until PEP 751 is ready 😬 .
I know that it might sound confusing. If this isn't clear enough or you have any further questions, please don't hesitate to ask 🙂

[hollow pine]

@undone siren I don't know how hard it is, but it would be nice if the python dagger-io SDK is installed inside the dagger engine with its proper version instead of falling back to version 0.0.0. I think this happens, because git is not installed in the dagger engine container or maybe the SDK is not a git clone. I understand that this might be a decision to make the base engine image as slim as possible or not have the SDK cloned unnecessarily.
If the SDK is installed with it's proper version, that would allow the module developers to have correct dagger-io version limitations inside their pyproject.toml.
An idea is to perform the version replacement for the key fallback-version in the pyproject.toml of the dagger-io SDK. I imagine something like this might solve it: strings.Replace(sdkCfg, "fallback-version = \"0.0.0\"", "fallback-version =+m.EngineVersion+", 1). I don't do go development, but I suppose something like that might work.
Another solution could be to set the version field and remove the dynamic version solution, but that might involve more changes in the pyproject.toml file.
By doing this the python SDK will be installed with its proper version when doing uv pip install -e ./sdk

[undone siren]

If you remove the uv.lock file you’re basically opting out of the higher level project management interface with uv. The Python SDK will support pylock.toml natively but that’s basically a better requirements.txt, which will revert to the lower level pip interface.

I expect that users will want to opt out of uv.lock in order to support some other installer they use, like Poetry (in the host). What I find surprising is opting out of uv.lock but still wanting to use uv locally. This is why I asked because I want to understand your use case but I’m not yet entirely sure which is which in your case.

[undone siren]

That shouldn’t really matter. The intent is for you to either have a local copy to install the SDK client library, or a precisely pinned version to install from PyPI (that always matches the version of the engine you're running). Not both.

With uv.lock the command to install is just uv sync. In that case, if you have a section in pyproject.toml like this:

[tool.uv.sources]
dagger-io = { path = "./sdk" }

Then it overrides the version constraint:

dependencies = ["dagger-io==0.18.5"]

So there’s no conflict there.

You just hit a regression with the pip based workflow in v0.18.4, which is expected not to include the current package and the SDK, that’s why it’s added manually through CLI args:

pip install -e ./sdk -e . -r requirements.lock

The regression made the SDK think you didn’t want to vendor everywhere except the pip install command which assumes vendoring. That’s where the conflict came from for you, but it should be fixed in v0.18.5.

[hollow pine]

Hi @undone siren !
First of all, I know your time is very valuable so take into account that for me everything is working and I love the support provided the Dagger team.
Furthermore, this is a personal work in progress project and you might have more important topics than this 😄 .

I'll show what I'm doing adn try to provide a better picture of the proposal I made.

This is the project I'm working on: https://github.com/fvonbergen/shiryu/tree/feat-add-python-sdk
The branch feat-add-python-sdk has the heavy development and it's not released yet.

I'm trying to build a framework for developing SDK for multiple languages, I started with Python.

Currently I might be too purist, but I don't want my SDK to be attached to any specific installer. I don't want to force users to use a tool over another, besides PEP standards.

Considering that I'm using Dagger to develop my SDK; for this reason I removed the uv.lock, set the dependencies field in the pyproject.toml file with dependencies = ["dagger-io==0.18.5"] and removed dagger-io = { path = "./sdk" }. I encountered that doing this breaks local module testing and the CI/CD because engine installs the local dagger SDK with version 0.0.0 when it runs pip install -e ./sdk -e . -r requirements.lock.
The steps to reproduce it in a clean project are:

mkdir test
cd test
dagger init --sdk=python
sed -i 's/dependencies = \["dagger-io"\]/dependencies = ["dagger-io == 0.18.5"]/' pyproject.toml
mv uv.lock _uv.lock 
dagger call container-echo --string-arg=scream stdout

[undone siren]

Yeah, doing it like that is not supported. If you want to opt out of uv you should uv export into requirements.lock before removing uv.lock (assuming you still want to pin the dependencies), but the dagger-io client library will still be vendored and patched (re-generated client bindings) in sdk/ subdir, so it should still be a local editable install, not from PyPI.

As for opting out of uv, I see your point but isn't there a difference between developing your Python module and calling its functions to bootstrap a Python project? Maybe I don't understand how your software is meant to be used? To clarify, you only need to install uv locally if you want LSP/IDE support while changing the source code of the module.

It's entirely different from what you put in the module's pipelines. For example, you could have a Go module bootstrap a Python project. Similarly, a Dagger module written in Python using uv.lock doesn't affect in any way having a function that bootstraps a Python project using pylock.toml, Poetry, or anything else. Unless what your module intends to bootstrap is other Dagger modules written in Python.

You mention in your readme that users need to clone the Shiryu repo in order to call its functions, but if they're not meant to edit the Shiryu source code they don't need to clone. You can dagger -m github.com/fvonbergen/shiryu call python init --project-name=<project-name> export --path=<output_directory_path>. Dagger can get the module from a remote git URL, doesn't have to be local. Again, sorry if I misunderstand.

Currently I might be too purist, but I don't want my SDK to be attached to any specific installer. I don't want to force users to use a tool over another, besides PEP standards.

Hopefully you can see that users don't need any tool other than the dagger CLI in order to call functions from your "SDK" module. No need for uv or even Python. Of course if the goal is to init a Python project according to your templates, they'll need some tooling, but it's independent from Dagger, even if your SDK module is written in Python. That's because Dagger modules run in their own container automatically, not from the host.

[hollow pine]

Hi @undone siren !
Yeah, I assumed that was the decision taken by the Dagger team. Is there any place where I can state the Dagger versions my module supports?

My decision was to develop the shiryu SDK Dagger module the same as any other python project. So the answer is that I decided to arrange the Dagger Python module to be concise with the development of any other python project. What is happening is that I have to deal with the Dagger team decisions, but is something reasonable and expected. I know that if it gets too hard I can move out of the Dagger CLI. Although, I love the way Dagger resolved arguments and security concerns ;). That's why I'm forcing myself to look for alternatives.
I understand that I can use venv and uv, but I want developers to run exactly the same commands that run in the CI/CD. I want the IDE configuration to be picked from the configuration files and folders.

Keep in mind that this SDK is not thought as an SDK for Dagger modules, the idea is to define the structure and provide common CI/CD functions for python applications and libraries; and for other software languages in the future.

Regarding the git clone comment in the README.md, it might be confusing (I updated it). What I'm trying to explain is that the shiryu module can be executed from a cloned repository or directly installed from GitHub. The usefulness of cloning the repository is that it eases the shiryu SDK development by modifying it and testing it locally.

Yes, I see that users don't need any tool other than the Dagger CLI!!!! That's why I picked Dagger 😄 . To give you more context, in the past I built a rudimentary package that mimicked Dagger in the sense of calling everything inside containers. After a friend pointed me to Dagger I realized that what Dagger made was a better replacement of what I did.

Maybe some examples will clarify how I expect this project to be used (remember that it's in WIP).

Try shiryu with itself:

git clone https://github.com/fvonbergen/shiryu.git
cd shiryu
DO_NOT_TRACK=1 dagger call python linter lint
DO_NOT_TRACK=1 dagger call python checker check
DO_NOT_TRACK=1 dagger call python documenter document export --path="docs"

Try shiryu with a new project:

mkdir new
cd new
DO_NOT_TRACK=1 dagger --mod=https://github.com/fvonbergen/shiryu.git@feat-add-python-sdk call python init --project-name=new export --path=.
touch src/new/__init__.py
DO_NOT_TRACK=1 dagger --mod=https://github.com/fvonbergen/shiryu.git@feat-add-python-sdk call python linter lint --project=.
echo "\"\"\"new package.\"\"\"" > src/new/__init__.py
DO_NOT_TRACK=1 dagger --mod=https://github.com/fvonbergen/shiryu.git@feat-add-python-sdk call python linter lint --project=.

[undone siren]

My decision was to develop the shiryu SDK Dagger module the same as any other python project. So the answer is that I decided to arrange the Dagger Python module to be concise with the development of any other python project.

Yes, I understand that, and it’s totally fine, but to be fair, surely you don’t mean “any other project” because the Python SDK for Dagger is already structured like a normal Python project, it’s just that there’s so much flexibility in Python that it can mean a lot of things and still be true. So I’m approaching my feedback assuming you mean that you want your Dagger module to work the same way as a project that your functions create or bootstrap for its users.

Dagger uses uv by default to manage dependencies, and we recommend it as the easiest workflow locally if you want to author Dagger functions with it but you can use other installers like Poetry or Hatch, however, in your case it seems like you just want pip. It’s a lower level tool but it does work with it.

You can actually change a lot in your module’s pyproject.toml so it matches what your module produces for users more closely if that’s your goal. You can use any build backend you want, for example. There’s little that’s actually required for Dagger to work. It just needs your Dagger module’s Python package to be installable/importable, and it needs to install dagger-io from a local path. You can learn more about that in our docs:

• https://docs.dagger.io/api/module-structure/
• https://docs.dagger.io/api/ide-integration/

I want to make sure this is clear so forgive me if you’re well aware of all of this. The way to opt out of uv.lock is to use a requirements.lock file instead of uv.lock. Nothing special about requirements.lock except that it’s usually named requirements.txt, I just wanted to make a clear distinction on what it’s for because requirements files can be used in many different ways. Some people may use pip-tools to help compile the exact versions to pin on this file but it’s an external tool and since you want to avoid that you can still pip freeze > requirements.lock.

Again, this is all In order to be compatible locally with what happens with dagger call. If Dagger finds a requirements.lock file instead of uv.lock then instead of uv sync it will essentially do pip install -r requirements.lock -e sdk -e .. So it’s not expected for requirements.lock to include the current project (the Dagger module in Python), nor the dagger-io library (which is vendored and patched with dagger develop).

So you can use any tool locally that can work with a requirements.txt format, which is basically all of them, the difference is how easy said tool can make the development workflow. For example if your choice were to use Poetry, since Dagger doesn’t support the poetry.lock file, you’d have to poetry export into requirements.lock everytime you update the pins in poetry.lock so it’s kept in sync with what Dagger installs for dagger call. But that’s not your case since you want to stick to pip, I'm just saying that it's still doable.

That brings me to pylock.toml. We’ll support it natively as the recommended alternative to requirements.lock. When pylock.toml becomes readily available I hope to deprecate requirements.lock in favor of it. But we’ll support it natively soon, however it’s still just a better requirements.txt when it's used for pinning all dependencies. It can’t replace all the higher level things that uv.lock allows.

Let me know if there’s anything in particular that you’d like to be able to do but not sure how or if possible. Good luck with your project! 🙂