#why layer them to begin with - my
left wyvern
My approach uses explicit pins rather than implicit pins (i.e. the lower layer requirement files are used as constraint files for the upper layers, so you'll get locking failures rather than installation failures if there's a conflict). The reason for the layering complexity is to allow multiple application layers to share massive framework layers like torch without having to install everything into one big environment.
But yeah, one of the restrictions is that the intended use case is for one publisher to compose layered venv stacks that their consuming application puts back together in the target deployment environment.
grizzled creek
so instead of sharing storage on package level we get a new kind of dependncy tree ?
are you aware that uv already does a specific cache of unpacked wheels that can be added to a virtualenv using a copy on write copy of a file tree
left wyvern
You definitely wouldn't use this for dev environments (the uv approach is much better for that). But yes, I'm aware of that aspect of the way uv works.
runic harbor
uv actually does layer virtual environments
Like when you do uv run --with some-extra-dep in a project
grizzled creek
how does the uv layering work
left wyvern
It's worth noting that .pth files couldn't reliably handle non-ASCII characters until Python 3.12.4: https://github.com/python/cpython/issues/77102
But yeah, they're a good option when you can't rely on having exclusive control of sitecustomize.py
left wyvern
For anyone curious, the "why" announcement is out (integrating Python AI components stacks with LMStudio's existing optional download model): https://lmstudio.ai/blog/lmstudio-v0.3.4 🙂
The full details on the "how" (i.e. actually publishing the venvstacks repo rather than just the broad description in the blog post) is still a bit away as I work through the list of pre-publication cleanup tasks.
The earliest iterations of this idea actually tried to use conda environments for the foundational runtime layer (with conda-pack to convert them into portable archives), but that approach failed its Unicode path handling tests due to https://github.com/ContinuumIO/anaconda-issues/issues/10595 (and given that issue has been open for 5+ years, switching the base layer over to python-build-standalone was a more straightforward solution than attempting to solve that Unicode issue).
Supporting alternative runtime layer providers could still be an interesting future feature idea, though.