virtualenv

[fathom stump]

does anyone here know why? https://github.com/pypa/hatch/issues/372#issuecomment-1197764771

[mortal sigil]

Sounds like Debian? They hard-code local in the scheme so using sysconfig outside a virtual environment with a custom {base} etc. is totally broken
https://salsa.debian.org/cpython-team/python3/-/blob/master/debian/patches/sysconfig-debian-schemes.diff#L22

[fathom stump]

thank you! wow. how have I never ran into this? 🤔

[fathom stump]

fixed

[crimson leaf]

I'm probably missing something obvious, but does the latest virtualenv support 3.11rc1? I just tried creating a virtual environment using it (it's installed via pipx on CPython 3.8.5 if that's relevant) from a 3.11rc1 in-source-tree build and it's failing to start up 😦

❯ python
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/usr/local/lib/python3.11'
  sys._base_executable = '/home/ichard26/Downloads/Python-3.11.0rc1/debug/python'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = '/home/ichard26/programming/testing/black/3.11venv/bin/python'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python311.zip',
    '/usr/local/lib/python3.11',
    '/usr/local/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f241654f4c0 (most recent call first):
  <no Python frame>

Using 3.11's venv module, things just work.

[radiant crypt]

hmm, seems to work for me

[crimson leaf]

hmm.

[crimson leaf]

I could try installing the source tree build and see if that fixes the weird sys.path shenanigans

I don't even understand how I'm supposed to install this build lol, make install is doing things that look weird

[radiant crypt]

what does it do

[crimson leaf]

it's compiling things which IIRC make install (after running make) does not do with 3.10

[radiant crypt]

that does sound weird

[crimson leaf]

I might just recompile this entire build

[radiant crypt]

not that I compiled much of Python

[radiant crypt]

oh, you're changing prefix?

yeah, that just triggers a new build probs

[crimson leaf]

yeah, I'm not overwriting my user install python for debugging

[radiant crypt]

prefix is part of the build

if I recall correctly

[crimson leaf]

huh. I thought it was mostly just file generation and copying

to be fair, the other times I set prefix to something different was via ./configure --prefix=something

[radiant crypt]

I mean, it probably is but it still has to be available as a constant

[crimson leaf]

I just didn't want to rebuild the whole project

[radiant crypt]

maybe I'm wrong

[crimson leaf]

It's not rebuilding everything though so things seem fine (tm) it's using the object files from when I first built it

[crimson leaf]

makes sense 👍

yeah OK after installing the build virtualenv works

[radiant crypt]

based on what this says, I'm guesing the prefix was the issue

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>

there's probably something you need to for in build tree builds

[crimson leaf]

probably, just interesting that it works without any special treatment for in tree builds with 3.10

(I still don't have a 3.10 installation available, just a build tree)

[crimson leaf]

you're correct, comparing it with an in-tree build of 3.10, sys.prefix is different

this is rapidly exceeding my knowledge of Python environment setup though so I can't really tell what's really wrong.

[radiant crypt]

what was the original configure command you used

[crimson leaf]

./configure --with-pydebug --with-lto for both 3.11 and 3.10 IIRC

[radiant crypt]

did you use python in the root directory of the CPython source for this same as in the 3.10 one above? The logs says debug/python but maybe the build just symlinks these two, idk

[crimson leaf]

yup, the debug directory is simply me just thinking ahead since I presume I'd build a non-debug version later on

I extracted the CPython tarball into debug/ directly

[radiant crypt]

ohhh

okay lol

I thought it's like a subdirectory

./extracted_CPython/
./extracted_CPython/debug/

😄

yeah, I have no clue then

[crimson leaf]

yeah no, not that haha

I could maybe try reproducing it in a docker container? Although that sounds like it'd take ages.

Thanks for trying, I appreciate it!

[radiant crypt]

I'm running into the same issue with a docker container

[crimson leaf]

oh lol, I'm happy you did since I have almost zero experience with docker and was feeling a bit iffy writing a Dockerfile for the first time haha

[radiant crypt]

I didn't write a Dockerfile

You can just run a shell

[crimson leaf]

ah exec -it /usr/bin/bash in a pre-existing container?

[radiant crypt]

docker run -it --rm ubuntu:22.04

this is the whole command

except I use podman instead of docker because I like it more 🙃

I ran these commands to repro:

cd /tmp
apt update
apt install -y wget
wget https://www.python.org/ftp/python/3.11.0/Python-3.11.0rc1.tgz
tar -xvf Python-3.11.0rc1.tgz
cd Python-3.11.0rc1
apt install -y build-essential gdb lcov pkg-config libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev lzma lzma-dev tk-dev uuid-dev zlib1g-dev
./configure --with-pydebug
make -j
apt install -y pipx
pipx install virtualenv
cd /tmp
/root/.local/bin/virtualenv 3.11venv --python=/tmp/Python-3.11.0rc1/python
3.11venv/bin/python

[crimson leaf]

well I wanted to write a Dockerfile for ease of reproduction, but it's good to hear it's not just me who can reproduce it

[radiant crypt]

I don't know Docker enough to write Dockerfile either 😄

I just use it for lightweight reliable repros

or anything else I want to test not directly in my system

[crimson leaf]

(or bad, with my maintainer hat on, would've been nice to just file this away as PEBCAK on my end 😄 )

[fathom stump]

I ❤️ Docker

[radiant crypt]

lol, I just noticed that I totally named the directory 3.10venv instead of 3.11venv while taking your screenshots as a base

[quartz lichen]

containers are great

docker is a nightmare

I may have PTSD from having to look too closely at docker's internals

[radiant crypt]

I'm just innocently using them instead of making VMs for testing things out

I don't like that docker runs as root and requires a daemon which is why I use Podman tho

[fathom stump]

having to look too closely at docker's internals

try writing a k8s plugin https://github.com/ofek/csi-gcs

'twas rough lol

[quartz lichen]

lol

I've got one or two of those laying around

a k8s controller was what I ended using to write my first bit of go

in hindsight, probably not the best learn a language project

[fathom stump]

oof that's like learning assembly as your first language

[radiant crypt]

you ran it inside a working venv here though

okay, not exactly what I wanted to say

yes, it seems like the prefix doesn't get replaced inside venv on that 3.11 in-tree build

but another thing is that all prefixes actually point to /usr/local which seems kinda weird for running an in-tree build when you consider that this is what devguide says should be used for development of CPython

[crimson leaf]

yup.

[radiant crypt]

and that's true for both 3.10 and 3.11

[crimson leaf]

wait what, 3.10 also has sys.prefix set to /usr/local... from an in-tree build ? how the heck is it working then?

[radiant crypt]

outside venv I mean

inside venv, it gets overridden on 3.10

[crimson leaf]

I'm just going to pretend I understand all of this (I don't)

[radiant crypt]

it seems weird to me because this means that /usr/local/lib/python310.zip gets included in sys.path even though you're in an in-tree build

so like, it should be isolated if you want to develop CPython with it I would think

[crimson leaf]

isn't there like a module you can run to get python environment information? or am I misremembering that for pip debug?

[mortal sigil]

python -m sysconfig?

[crimson leaf]

ah that's it, thank you!

[crimson leaf]

OK, so this is probably specific to virtualenv then? I mean, it's possible either (or both) CPython changed something and virtualenv hasn't updated to support 3.11.

I'm currently building a docker image for 3.11 to see if I can reproduce the issue

unfortunately my internet connection is awfully slow so it will take a while

[radiant crypt]

https://github.com/python/cpython/commit/99fcf1505218464c489d419d4500f126b6d6dc28

this is the moment virtualenv stopped working with in-tree builds

[crimson leaf]

lol of course

FROM ubuntu:22.04
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Etc/UTC

RUN apt-get update && apt-get install -y wget

RUN wget https://www.python.org/ftp/python/3.11.0/Python-3.11.0rc1.tgz \
    && tar -xvf Python-3.11.0rc1.tgz \
    && cd Python-3.11.0rc1

RUN apt-get install -y --no-install-recommends build-essential gdb lcov pkg-config \
    libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev \
    libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev lzma lzma-dev \
    tk-dev uuid-dev zlib1g-dev

# Optimization level three (the default) can cause the build to take forever
# on my potato laptop.
ENV CFLAGS="-O0"
WORKDIR /Python-3.11.0rc1/
RUN ./configure --with-pydebug
RUN make -j

RUN apt-get install -y --no-install-recommends pipx && pipx install virtualenv
RUN /root/.local/bin/virtualenv 3.11venv --python=/Python-3.11.0rc1/python

CMD ["3.11venv/bin/python"]

I'm still building this image

[radiant crypt]

lol, optimizations

oh, the default is 3

I misread that

thought you enabled them which caused the slowdown 🙃

[crimson leaf]

sounds we might need report this to cpython?

[radiant crypt]

I don't know if it is a CPython problem

[crimson leaf]

right

[radiant crypt]

the documentation was updated in that commit so

it's possible that this is affects it in some way

I imagine Bernát is more likely to know if it's virtualenv issue or CPython issue 😄

because I have no idea how all of this works

[crimson leaf]

I think I might actually have built my first docker image successfully 👀

[radiant crypt]

I guess based on what getpath does it's more likely CPython issue

[crimson leaf]

sudo docker run -it virtualenv-test-3.11:v1
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '3.11venv/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/usr/local/lib/python3.11'
  sys._base_executable = '/Python-3.11.0rc1/python'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = '/Python-3.11.0rc1/3.11venv/bin/python'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python311.zip',
    '/usr/local/lib/python3.11',
    '/usr/local/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f25dca82740 (most recent call first):
  <no Python frame>

heck yeah 🎉

it only took like 80 minutes lol

[radiant crypt]

I found this issue which might be related
https://github.com/python/cpython/issues/90186

[crimson leaf]

it's around 11PM over here so I won't be much help anymore tonight fyi

[radiant crypt]

Well, it's probably enough for @stuck merlin to go off from tbh

[crimson leaf]

I'm working on revision two of my dockerfile, got a three revision ready to go after I verify v2 works llol

I've been sucked into docker 👀

[crimson leaf]

OK here's my final Dockerfile to reproduce this crash.

FROM ubuntu:22.04 as builder
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Etc/UTC

RUN apt-get update && apt-get install -y wget
RUN apt-get install -y --no-install-recommends build-essential pkg-config \
      libbz2-dev libffi-dev libreadline6-dev libssl-dev uuid-dev zlib1g-dev
RUN apt-get install -y --no-install-recommends pipx && pipx install virtualenv

RUN wget https://www.python.org/ftp/python/3.11.0/Python-3.11.0rc1.tgz \
      && tar -xvf Python-3.11.0rc1.tgz
WORKDIR /Python-3.11.0rc1/
# Optimization level three (the default) can cause the build to take forever
# on my potato laptop.
ENV CFLAGS="-O0 -g0"
RUN ./configure --with-pydebug
RUN make -j
RUN /root/.local/bin/virtualenv 3.11venv --python=/Python-3.11.0rc1/python

FROM ubuntu:22.04
COPY --from=builder /Python-3.11.0rc1/ /Python-3.11.0rc1/
CMD ["/Python-3.11.0rc1/3.11venv/bin/python"]

Now time to call it a day 💤

[stuck merlin]

We do have it in the CI https://github.com/pypa/virtualenv/actions/runs/2851257100

Though not the RC

[stuck merlin]

I wish would work better on lower resource systems...

[radiant crypt]

Yeah so the problem is with virtualenv not working with a build that can be run in-place specifically (https://devguide.python.org/getting-started/setup-building/#unix). Whether that's an issue with CPython or virtualenv, that I don't know.

[crimson leaf]

can confirm 3.10.6 in tree builds work fine with virtualenv in docker

[radiant crypt]

yeah, the issue started in 3.11.0a3

[crimson leaf]

virtual environment created from 3.11rc1 in-tree build crashes on startup

[heady girder]

Is there any way to add behaviour to the virtualenv creation with a plugin? (context is needing to add gc root on nixos to prevent breakage)

[stuck merlin]

A plugin?

[radiant crypt]

Speaking of virtualenv's plugin system (not trying to switch topics, it just reminded me of an idea I had), does it allow to easily add more packages to install (in addition to the default pip, setuptools, wheel)?

[stuck merlin]

Depends on your definition of easily 🙂

You can swap out the seeder and add more packages though 👌

[surreal scaffold]

While we are here, are there any plans to refactor virtualenv into lib with CLI interface being an addition and not the only way to interact with the package?

[stuck merlin]

That's already the case

https://virtualenv.pypa.io/en/latest/user_guide.html#programmatic-api

[radiant crypt]

hmm, would that work with the upgrade functionality or is that only for the default seed packages

[stuck merlin]

Depends on what exactly you're looking for

[radiant crypt]

I was wondering about installing ipython in all my venvs by default so I was thinking that fstring the FromAppdata functionality on creation would be nice + keeping the "cached" wheels up-to-date with --upgrade-seed-packages

[stuck merlin]

It's doable-ish but you'll likely need to do quite a bit of work.

You can extend FromAppData and run additional operations after the original layer runs

But we don't currently support extending upgrade seed packages as no one requested it yet 🙂

Another major problem is that FromAppData only handles pure python packages. So if you have other packages in the ipython dependency chain you'll need to extend our bundled installer 🙂

[radiant crypt]

I can hope the whole dep chain is pure Python but I honestly have no idea 😄

[stuck merlin]

Would be a fun side project to try to make it work. I'm open in accepting PRs that would facilitate a plugin enabling this functionality.

But not adding this feature to the core per as 😅

[radiant crypt]

I've been thinking about it every once in a while but my todo only grows so maybe one day but probably not any time soon unless it suddenly becomes incredibly important to me to have this lol

[radiant crypt]

And yeah, that totally makes sense

Somethimg that would make it possible more easily but not add the actual functionality seems absolutely fine

But yeah, if I ever start playing with it, I'll know that you're open to what you said

[stuck merlin]

💪😎

[stuck merlin]

20.16.5 is out https://virtualenv.pypa.io/en/latest/changelog.html#v20-16-5-2022-09-07, if you use batch on Windows this might just interest you!

[high rune]

Hi,

I have created a virtualenv using this command on Windows: virtualenv venv
and I activated it: venv\Scripts\activate.bat

The full path of the virtualenv is this: C:\Users\omerg\Desktop\Ömer\venv

But when I try to use pip, the system uses global pip installed on C:\Python310\lib\site-packages\pip instead of C:\Users\omerg\Desktop\Ömer\venv\Scripts\pip

I am investigating, what I found is that the PATH environment variable is not set correctly.

My venv\Scripts\activate.bat is something like this:

@set "VIRTUAL_ENV=C:\Users\omerg\Desktop\Ömer\venv"
...
@set "PATH=%VIRTUAL_ENV%\Scripts;%PATH%"

But when I try to echo the VIRTUAL_ENV or PATH environment variable

I am seeing this

But when I try to set VIRTUAL_ENV env var manually, there is no problem

[mortal sigil]

I suspect activate.bat is written in an incorrect encoding

Could you share the file? (Don’t copy-paste it, upload the entire file directly instead)

[mortal sigil]

Yeah, the file is in UTF-8, but bat files need to use the same code page as the console. It’s a pain to do non-ASCII in Batch in general. Not sure if adding a chcp in front of the file would work (probably also need to change it back… also a pain), or maybe the script need to use the short path instead.

[high rune]

yeah, adding a chcp 65001 in front of the file works

but, I think, there should be a PR about this

[mortal sigil]

Not sure if adding a chcp would cause undesired side effects

[high rune]

maybe changing it back at the end of the file prevents side effects?

actually I found something

this is created using python -m venv venv

not virtualenv

it uses chcp

and when I use original venv, there is no problem

I think it is because the batch file sets the codepage as utf-8 and back the original

[mortal sigil]

It’s added in https://github.com/python/cpython/pull/5757
A PR would probably be accepted rather quickly if you add those references

[high rune]

Yeah, I created a PR for this!

https://github.com/pypa/virtualenv/pull/2416

[neat forge]

hi! on an m1 mac, i found that venv creation hangs for a couple minutes while virtualenv creation works fine (py 3.8 and newer ones I tried). anyone know off the top of their head what could be the cause?

[fathom stump]

non-answer: b/c virtualenv is better/well maintained

never use stdlib venv if you can

[surreal scaffold]

Interesting approach

[fathom stump]

I thought that was known 🤷‍♂️

[radiant crypt]

venv creates a subprocess to setup pip while virtualenv can skip this entirely due to caching and stuff

[surreal scaffold]

I probably never used virtualenv directly. For testing stuff I use venv and for bigger projects, Poetry manages my envs (so indirect usage of virtualenv)

[radiant crypt]

If you don't need pip, both are fast:

• python -m virtualenv --without-pip .venv
• python -m venv --without-pip .venv

If you need pip and choose to use subprocess-based pip addition, both are slow:

• python -m virtualenv --seeder pip .venv
• python -m venv .venv

And if you use virtualenv's default method, it's quite fast:

• python -m virtualenv .venv

I use virtualenv for all local work because of its speed.
For any production-like setup, I use venv because it's part of stdlib and is probably just already installed so saves me a step.

obviously virtualenv is absolutely fine for production as well but I just don't see the need to bother when a new venv is not going to be created as often.

[fathom stump]

being part of stdlib is not a positive signal in my mind

[radiant crypt]

it just means that it isn't rapidly developed

[stuck merlin]

https://virtualenv.pypa.io/en/latest/user_guide.html#seeders

[radiant crypt]

venvs are (imo) one of the core functionalities of Python and it's essential to have a tool to make them, it doesn't need to be as feature-rich as 3rd-party tooling

[stuck merlin]

Virtualenv also allows you to create a virtual environment where where? Pip is the latest version, potentially saving you an extra step of upgrading pip after you create your environment

This is described here https://virtualenv.pypa.io/en/latest/user_guide.html#wheels

[radiant crypt]

oh yeah, I incorporated virtualenv --upgrade-embed-wheels into my system update routine

very convenient

[thin jetty]

???

[stuck merlin]

Read it as not updated often for bugs and improvement

[wicked orbit]

I'm getting weird behavior when I create a virtual environment in nushell everything appears to work just fine until I activate the environment the path is shown as <Closure 13> as shown in the screenshot. I'm using Windows11. Should I make a bug report? Or does it have to do with nushell 0.72.1?

[stuck merlin]

that's a bug to nushell

ask there first

[surreal scaffold]

from what I have seen so far, nushell is constantly causing trouble for virtualenv

[obsidian pendant]

Well it is v0.x and actively changing syntax and concepts, so that’s simply something users have to live with. I’m happy that virtualenv supports it!

[wicked orbit]

But doesn't virtualenv create the activate.nu file? It appears that if I use overlay for something else I can create the overlay and there is no issue. It seems in how virtualenv is creating the path in the activate.nu. But I can't find where exactly that Closure 13 shows up. If virtualenv doesn't create the activate.nu how does it get created? via nushell?

[heady girder]

im looking for possible recommended tools to ensure cache-able python virtualenvs based on given python requirements (hard or soft locks)

im trying to set up local and remote envionments with given package sets, possibly by pushing a number of wheels to a remote

[smoky sentinel]

I've an issue with virtualenv that doesn't seem to be found with a simple google search or scan through the open issues on github. When attempting to create a new venv with python -m virtualenv my_venv, I get the error StopIteration

python version is 3.10.9
virtualenv version is 20.17.1

Any ideas on what I could try?

when running with the --with-traceback argument

[smoky sentinel]

I've attempted to pip install --force-install the latest version as well as a few prior ones in the chance that this was an issue with dependencies

[mortal sigil]

If I understand this (linked) issue correctly, this is caused by some package set a virtualenv.discovery entry point incorrectly. You need to review what are installed in python (wherever that points to) and maybe try to uninstall some of those to find the culprit(s)
https://github.com/pypa/virtualenv/issues/2176

(Installing stuffs into the global python is almost always a bad idea because you never know how those packages might interact)

[smoky sentinel]

yeah, interesting. I missed that issue

it very much does appear to be failing in the discovery process

ok, that gives me a plan of attack at least. I'll try pip uninstalling all the packages and then add them back until I find the one that's causing the issue

ok, that was indeed the issue. Removing all pip installed packages and reinstalling virtualenv does allow me to construct a virtual env without issue. Now to find which of the (many) packages is causing the issue

cheers

[heady girder]

@stuck merlin is there any way to link discovery and creation, i want to enable virtualenv on nixos to auto-add a gc root to any virtualenv thats created outside of the nix store, so that its garbage collector wont kill python

[stuck merlin]

Not today

Could add PR

[celest remnant]

I've just noticed something very weird (Windows, virtualenv 20.21.0, python 3.11) - if I do virtualenv --upgrade-embed-wheels, 53 instances of the rustc compiler get started on my PC. The -vv output from virtualenv says nothing, it's all "upgraded X for python X.Y". Some wheels get installed, but that's all. Running rust sounds like installing something from source, but nothing is mentioned. If I remove rustc.exe, everything still works just fine.

I don't think there's anything wrong here, as such, and there's no way it is going to be a virtualenv issue, but it just seems weird. Can anyone suggest how I could work out what's going on here?

[unreal notch]

Have you tried looking at the process tree while that happens?

[celest remnant]

I can't see the process tree, unfortunately, the rustc command runs so briefly that by the time I check, it's gone again 😦

If you know of a Windows tool that can be triggered automatically when rustc runs and capture the tree for later perusal, that would be ideal. I don't know of anything like that, though.

[unreal notch]

You can use Process Monitor to log "process create" events. That way, you'll at least be able to see the command line.

[celest remnant]

Ah, I did that but hadn't realised it had the command line data. That's a good start, I'll give it a go.

(Side issue, docker desktop fires off a huge number of wsl processes. And that's when I'm not using it, it's just sitting there...)

[celest remnant]

Weird. All it's doing is rustc --version.

Found it! Process Monitor also has a "show process tree" which got what I needed. And ironically, it's pip that was doing it - we add the rustc version to the User-Agent string when pip downloads files. I even commented on the PR when it was submitted 🙄

Thanks @unreal notch for the pointer here.

[stuck merlin]

😁

[quartz lichen]

is something starting pip 53 times?

that seems wrong

[surreal scaffold]

I'd go a step further and say that with installer and build, starting pip seems not really needed (but it might be a different usecase)

[mortal sigil]

I was about to say I know pip does it but 53 times sounds way too many

[stuck merlin]

This is us trying to download the latest embedded wheels, that is one invocation per wheel per python version

[mortal sigil]

Why one per wheel (cross every Python version) instead of one per Python version?

[simple gyro]

@stuck merlin gz on the release!

[obsidian pendant]

ah yiss nushell support is restored!

[simple gyro]

is there a way to configure altinstall with virtualenv?

oh I guess run with --no-pip then install with ensurepip --altinstall

[stuck merlin]

altinstall is not a virtualenv feature 🙂 @simple gyro

[simple gyro]

Could it be?

[stuck merlin]

I don't think so 🙂

[heady girder]

is there any way to have some type of extra pass-over between discoverer and creator - with the current visible plugin interfaces it seems impossible to add a nixos gc root in the created virtualenv to prevent gc of the used python (similar effect to what happens with a dist-upgrade on more traditional distros - but it happens more easily on nix

[stuck merlin]

I'm not sure I understand entirely what you're trying to achieve

I have no knowledge of the Nixos system 😅

[heady girder]

@stuck merlin nix has pytohn interpreters in /nix/store/some-crypto-hash/.. - they can get garbage collected unless one adds a gc root when using them imperatively -

so when making a virtualenv on nixos with a python, unless a gc root for that python is added to the virtualenv, the underlying python may get garbage collected

so in essence there is need for some create time hook that ensures the gc root is added on that system,

[heady girder]

as far as i can tell from the current hooks, that isn't planned/intended

neither discoverers not creators nor a combination match the use-case atm, and im a bit at a loss how to correctly enable it

[heady girder]

i'll open a feature request

[stuck merlin]

what is gc root?

isn't this just run this step always post creation? 😄 I don't see how discovery is involved

IMHO you can just create your own cpython3-posix-nixos that inherits from cpython3-posix and be done? 🙂

I must admit I don't see the apeal in nixos 😊 and never understood it's purpose really

[heady girder]

Reproducible consistent system configuration by using patterns from functional programming for declaration of the intended system

And thanks to being reproducible, its always specified from source, but most packages are in public caches so one can download instead of building

Unfortunately those things do have numerous friction points when doing imperative things

[stuck merlin]

I find is more pain than the gain 😂 anyway, back to the point, I don't see anything here for virtualenv to do 😄

[heady girder]

Well, it's removed most of the accumulation crust issues for me,epic win compared to traditional systems

If I need a creator mixin,then how do i ensure virtualenv uses mine instead of the normal ones

[stuck merlin]

we don't do mixins so not sure what you're refering to

is pure subclassing what virtualenv recomends

[heady girder]

I need the added behavior for posix, Darwin, and for the related pypy as well

[stuck merlin]

I'm pretty sure that the custom plugins are tried first, so if they are accepted the built-in will not trigger so you shouldn't need to do anything

[stuck merlin]

you would define 3 creators 🙂 register all 3

how you share the code can be a simple method call that you invoke from within the subclasses

isn't nixos Linux only? 😄 does macOS too? 🤔

you never answered what a gc root is 😂

[heady girder]

A gc root is a symlink that tells the nix gc which packages to keep in the nix store

For normal declarative installs that is relatively straightforward, but whenever you make a virtualenv,then update everything off the declarative install, then nothing gave nix a note that a old python may still be in use outside of the full install

[stuck merlin]

I see

so a gc root is essentially a ref count bump for the nix os

[heady girder]

Yup

It's more akin to a ffi handle

[stuck merlin]

this is still the solution, and you can make it today

[foggy glacier]

nixos has come a long way and it's nice for reproducibility, but there are still some really annoying edge cases that many people would say outwiegh the benefits since there are other ways to get reproducibility (outside of considering binary reproducibility, which has other means of getting "we consider this to be tolerable security/trust involved"),

that said, would it make sense for someone involved with nixos to create a virtualenv template for nixos that could be documented in virtualenv given the growing popularity of nixos?

[stuck merlin]

Is there anything else needed than this gc root (ref count) link?

for that you can just create a plugin @heady girder described and install it by default

[heady girder]

Adding the plugin to any virtualenv

[foggy glacier]

I think that's all that's specific to nixos, so even just a note that you may need that somewhere in docs for discoverability could go a long way., I just don't know if that's an appropriate os specific specia case to document in virtualenv

[stuck merlin]

I guess for full reproducibility you likely want to make the default for the seed packages to be embeded 🙂 but also doable in a plugin.

Adding the plugin to any virtualenv.
This is done by just registering the new nixos custom creators, and making sure the plugin is installed when virtualenv is installed.

The virtualenv plugin system is by design not built to allow pluggy hooks as you would expect @heady girder that are enabled by default 🙂

[heady girder]

hmm - i never expected pluggy style hooks ?!

[stuck merlin]

So there's no run this code no matter what. You must put your logic in either a discovery, a creator or a seeder. And use inheritence to extend a base functionality.

Because if you want run creator x and then create this symlink, that is not creator x anymore but rather creator y 🙂

[heady girder]

i shold probably clarify terminology, but bascialy - what i need to ensure is that "my" discoverers/creators/seeders run before the default ones

[stuck merlin]

that should be the case by default

plugins take priority over default ones

[rigid bobcat]

Hi all 👋 I am not sure if this has been raised and if I am late to the party but here goes. I spent an hour pondering why my virtual env would not activate and then I came to the realisation that my path in the activate file was messing with the file activate on execution. If I have a path name that contains a single quote the code fails to execute as intended. As I am not yet familiar with the code and where the VIRTUAL_ENV is set when the virtualenv command is executed, is it possible to instead have the outer string be double quote to take single quotes into account or is there reason why this is not being taken into account?

[unreal notch]

My guess is that it would be tricky to do, because virtualenv supports multiple shells, all with their own escaping rules.

[stuck merlin]

Exactly the point from above 👍

If you can make it work for all please put in a pull request

[heady girder]

@stuck merlin any opinion on allowing the expansion of the seed packages, i'd really like to get truststore seeded into all virtualenvs i make as i need extra certs from the system and overwriting the file of certify is a horrible hack

[stuck merlin]

You can write a new seeder that inhrerits from current

[heady girder]

@stuck merlin as far as i'm understanding, that one would have to always added as argument?

[stuck merlin]

I think you can set it as a default in the plugin

[heady girder]

@stuck merlin for that i'd have to override the seederselector as far as i can tell (its get_default method returns app-data)

[stuck merlin]

Yeah something like it

[heady girder]

as far as i can tell this practically means i'd have to create a own virtualenv command to make it work, at first glance not feasible for my goal

[stuck merlin]

I don't think that's the case

As far as I can remember, you can set the defaults from plugins

I would need to have another look at it but if not pull request to allow that for plugins would be accepted.

[heady girder]

https://github.com/pypa/virtualenv/blob/main/src/virtualenv/run/plugin/seeders.py#L28 indicates a strong default

[stuck merlin]

I think that that is the built-in default

But you can override where is called if I remember correctly

Hence why that method is private

[heady girder]

its used right above, i don't see a override other than cli args

[stuck merlin]

Oh yeah! You're right. I think the only existing solution would be to monkey patch the class from your plugin and overwrite this get default. I would accept the pull request with the more elegant solution.

[heady girder]

@stuck merlin would you be amicable to include truststore in a similar manner to pip/setuptools/wheel - its a small pure python package and being able to opt into it easily would be quite a help

[stuck merlin]

Setuptools and wheel is no longer included for 3.12 or later

So the only package included is pip

What would be your explanation? Why it should be a score as that excluding convenience?

[heady girder]

Truststore enables to directly use the system truststore instead of certify that massively eases setup environments that requires additional internal CA be added

Else one needs to mess around with the requests env vars to get pip working against internal indexes behind ssl

[scenic obsidian]

Surely things will start depending on truststore?

[stuck merlin]

Any reason pip doesn't bundle it?

[surreal scaffold]

I think it does

https://github.com/pypa/pip/tree/main/src/pip/_vendor/truststore

it does

[stuck merlin]

So why do we need it in virtualenv?

[surreal scaffold]

¯_(ツ)_/¯

[heady girder]

hmm, i completely missed that its now bundled 🤦‍♂️

[valid moon]

~~Not sure if this matters at all, but why is the powershell activate scripts once using / and then using __PATH_SEP__? https://github.com/pypa/virtualenv/blob/fa283474fd199e3836f8b2c99510190c6b77e2bc/src/virtualenv/activation/powershell/activate.ps1#L49~~ PATH_SEP is ;, not \!

[dusty pecan]

that one always gets me as well.

[hot warren]

i feel like i am going bonkers. i have installed virtualenv on my mac, but when i run --help and look at the args i dont see --always-copy and when i try to run with --always-copy i get an error that the option doesnt exist (virtualenv: error: unrecognized arguments: --always-copy) and verified on virtualenv 20.25.1. before i open a bug, i wanted to ask if i am just doing something wrong.

[stuck merlin]

which -a virtualenv?

[hot warren]

/opt/homebrew/bin/virtualenv

[stuck merlin]

That doesn't seem right, did you install it with pip? Into a virtual environment?

Otherwise I'm afraid you need to reach out to the brew package maintenance

[hot warren]

i tried both ways, i can switch back to using the pipx install virtualenv method

doing that i then get /Users/itewksbu/.local/bin/virtualenv

[stuck merlin]

👍

[hot warren]

but i get the same problem

as i described

virtualenv: error: unrecognized arguments: --always-copy

[stuck merlin]

What's the virtualenv --help showing?

[stuck merlin]

It seems your platform does not support the copy mode, per https://github.com/pypa/virtualenv/blob/main/src%2Fvirtualenv%2Fcreate%2Fvia_global_ref%2Fbuiltin%2Fref.py#L66

This is expected

[hot warren]

okay

thanks

i figured it was something like that, but i couldnt find anything in the docs about it

[stuck merlin]

A pulley request to improve the documentation is welcome.

[low raven]

I have a similar problem to @hot warren above - I seem to be unable to create 'copy' virtualenvs via the virtualenv module. They create just fine with the venv module (with --copies). Any idea why this would be? MacOS.

setting VIRTUALENV_ALWAYS_COPY=1 VIRTUALENV_COPIES=1 makes no difference, and there's no --always-copy parameter.
This breaks every virtualenv on MacOS every time Homebrew decides to remove Python and install a newer version, so it's pretty annoying.

[rocky path]

Are there plans to upgrade to PIP 24 in the near future?

[inner thistle]

Any plans to drop 3.7? I think there are only two active PyPA projects that still support running on 3.7, and the other one is pyproject-metadata (😳). Pip 24.0 does still support 3.7, btw, but 24.1 (free-threading support!) will not.

[stuck merlin]

18 months post eol python version

[stuck merlin]

That's fine we can degrade to older pip where needed

[fathom stump]

@stuck merlin looks like a bad bug apparently just a heads up https://github.com/pypa/virtualenv/issues/2735

[stuck merlin]

Already yanked

[rose depot]

Hello All: I'm having a very strange issue with virtualenv using poetry to build inside a container:

ImportError

cannot import name 'fs_path_id' from 'virtualenv.info' (/opt/poetry-venv/lib/python3.10/site-packages/virtualenv/info.py)

at /opt/poetry-venv/lib/python3.10/site-packages/virtualenv/discovery/builtin.py:10 in <module>
6│ from contextlib import suppress
7│ from pathlib import Path
8│ from typing import TYPE_CHECKING, Callable
9│
→ 10│ from virtualenv.info import IS_WIN, fs_path_id
11│
12│ from .discover import Discover
13│ from .py_info import PythonInfo
14│ from .py_spec import PythonSpec

I've tried building with no cache, and get the same results. I have verified that fs_path_id is a function defined in virtualenv.info. This looks to be an issue inside virtualenv (it's the builtin.py file that is throwing the error).

Any suggestions welcome

[rich forge]

Hi experts - while trying to use a script from a Msys2/ucrt64 shell in Windows 11, I hit this error:

$ python -m venv C:/Users/F.Fumi/msys64/home/F.Fumi/MBedCE/mbed-ce-hello-world/mbed-os/venv
Error: Refusing to create a venv in C:/Users/F.Fumi/msys64/home/F.Fumi/MBedCE/mbed-ce-hello-world/mbed-os/venv because it contains the PATH separator :.

which is I guess a consequence of having "C:" in the path.
Is this expected?

[surreal scaffold]

it's complaining about . in the path, try quoting the path disregard

[inner thistle]

. is a path separator? How do extensions work then? : can be a path separator, though. I think it's saying : is the seperator and . is just the period at the end of the sentence. I think there's a unix-like way to spell out C:\ in Windows.

ChatGPT claims /mnt/c/ for WSL and /c/ for cygwin or git bash. Can't confirm ATM.

[surreal scaffold]

oh, yeah, you are right, been staring at code for too long

although that message should be improve, should have quotes on the symbol or something

[inner thistle]

Quotes would help

But it's in the stdlib, so an improved message would have to be a future Python release. 🙂

[surreal scaffold]

yeah, still, might be my first cpython contribution 🤣

[inner thistle]

A relative path should work, too

[hasty ibex]

It's also worth noting that the case of PATH (vs path) in that error message is significant:

$ echo $PATH
/home/acoghlan/.local/bin:/home/acoghlan/.cargo/bin:/home/acoghlan/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0/:/mnt/c/WINDOWS/System32/OpenSSH/:/mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/Program Files/Git/cmd:/mnt/c/Users/Alyssa/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/Alyssa/AppData/Local/Programs/Microsoft VS Code/bin

It's referring to the entry separator in the PATH environment variable, not to the directory separator in filesystem paths (Windows uses ;, Linux et al use :). (And yeah, it's /mnt/c for WSL, as the above example shows)

[coral shuttle]

Just an idea: Would be there any interest on having something like npm's isolated dependency trees with virtualenvs? For instance, I'd like to "inject" some additional plugin packages at runtime with their own dependencies isolated from the main program's original dependencies in the venv.

I've been thinking about utilizing subinterpreters, but currently there are many undefined edges -- like using separate venvs for different subinterpreters under a single Python process.

I'd like to "safely communicate with" arbitrary 3rd-party packages with a defined protocol, while isolating the dependency trees as the update cycle of the main program and plugins may differ.

AFAIK the current standard way to implement a plugin subsystem is to use the package entrypoints enumerable within the venv, assuming that the dependency trees of the main program package and plugins packages are compatible.

I'm not sure this could be implemented solely on either the CPython core or the virtualenv project. It would require both sides.

[stuck merlin]

That would require change on both sides, but primarily first on the core interpreter side.

[coral shuttle]

Hm.. it might be better to discuss in discuss.python.org. Thanks!

[coral shuttle]

https://discuss.python.org/t/separating-virtualenvs-for-subinterpreters-within-a-single-python-process/86043

[simple gyro]

You should have a chat to Armin Ronacher

[obsidian pendant]

It would be possible, it would solve some problems, and it would create many exciting new problems!

[compact flax]

Coverage.py's test suite is stuck between needing two different fixes in virtualenv, anyone have any clues to a path forward? https://github.com/pypa/virtualenv/pull/2878 is needed to work on 3.14b1, but https://github.com/pypa/virtualenv/issues/2829 is unfixed and affects me but not others I guess?

[half cave]

we're going to fix it in CPython for beta 2 https://github.com/python/cpython/issues/133653

does Coverage.py test nightlies, not tags? if so, it'll be fixed soonish

[compact flax]

Thanks, but i'd still like to be able to unpin virtualenv.

[civic smelt]

i've created an issue at pre-commit but they said it's being cause by virtualenv, is there anything i could do to fix it?
https://github.com/pre-commit/pre-commit/issues/3505
virtualenv: 20.33.0

[stuck merlin]

PR to fix it welcomed 😎

[civic smelt]

fair

i'll take a look at it after i wake up

[willow tundra]

I guess this is more about the venv PEP than the third-party project, but is include-system-site-packages in pyvenv.cfg actually necessary? Why not just use a .pth file to the system site packages?

[stuck merlin]

I guess it's more of a theoretical approach to it as in we should have all information about the environment in a single place and technically pth files are not really a thing as in they were never stand their diet are kind of an implementation detail of cpython

[hasty ibex]

It's plausible the idea just never came up, but even if it had, making venv support require arbitrary .pth file support wouldn't have been a popular idea (it was well after that point that we finally managed to narrow the calls for complete removal of pth files to just discussing removal of the arbitrary code execution aspect)

[willow tundra]

ah, presumably I missed something about the history of the .pth file standard.

[stuck merlin]

Tldr: Pth files are controversial and non-standardized.

[hasty ibex]

And if you do want to read, https://github.com/python/cpython/issues/78125 has the intermittently updated discussion (all 100+ messages of it)

[compact flax]

I'm trying to understand why the coverage.py test suite is failing on the tip of CPython main: https://github.com/nedbat/coveragepy/actions/runs/18708731899/job/53351874892#step:9:55

The failure seems to involve virtualenv's vendored copy of pip 25.0.1, but virtualenv also vendors 25.2. Any clues?

[half cave]

I think this is something that should be fixed when pip is next released, due on Friday. let me find the issue

[compact flax]

(of course hugo is typing...)

but there's already a 25.2 in virtualenv, why is it using 25.0.1?

[half cave]

it's the same error as https://github.com/python/cpython/pull/133602#issuecomment-3427148698

[half cave]

good question, when I create a new venv with 3.15, it has 25.2 in it

(brb, off to the Dutch embassy to vote 🇳🇱 🗳️)

[compact flax]

so international!

[compact flax]

new pip doesn't fix the problem. I don't understand what's happening here: https://github.com/nedbat/coveragepy/actions/runs/18817804320/job/53688688813#step:9:54

[crimson leaf]

It looks like virtualenv is using its own version of pip to then download an up to date pip?

huh

[crimson leaf]

Yeah, it uses the embedded pip to download other packages https://github.com/pypa/virtualenv/blob/0a644b0b6e0c68e89584fbce8339b366cb4cdf91/src/virtualenv/seed/wheels/acquire.py#L123-L127

Ah, and virtualenv doesn't know of Python 3.15 either so it defaults to using its older pip to do that https://github.com/pypa/virtualenv/blob/0a644b0b6e0c68e89584fbce8339b366cb4cdf91/src/virtualenv/seed/wheels/embed/__init__.py#L8

[compact flax]

nice find!

[crimson leaf]

Sounds like virtualenv needs to be updated for 3.15 with a bump to pip 25.3 included.

[compact flax]

Looking at the last time a new Python version was added, this doesn't seem like something I should attempt: https://github.com/pypa/virtualenv/pull/2741.

https://github.com/pypa/virtualenv/issues/2987

[stuck merlin]

https://github.com/pypa/virtualenv/pull/2989/files @compact flax

[compact flax]

Thanks! I'll jump on the next version

[stuck merlin]

ah, new pip breaking the CI will need to check

[obsidian pendant]

@stuck merlin just saw you fixed the nushell deprecations in virtualenv this summer, thank you!

Interesting that this only percolated onto my system now

[past raven]

Hi everyone, new here so apologies in advance for the dumb question. Is this a good space to discuss new features to extend the virtualenv project ?

[mystic jacinth]

Probably, but I don't know how activate the community or development is

[stuck merlin]

Yes, as long as is something that's within scope and you're open to do the
Work.

[past raven]

Sounds good, let me try. I'm a security engineer and currently working on software supply chain security. We've seen many attacks targeting the python packages ecosystem as discussed for instance here https://discuss.python.org/t/typosquatting-dependency-confusion-supply-chain-attack-call-it-as-you-wish/52615 and https://thehackernews.com/2025/06/malicious-pypi-npm-and-ruby-packages.html . These attacks are straightforward -- attackers create a malicious package on Pypi and eventually that'll get shipped to certain users through different channels (other dependencies? phishing? typosquatting? ..). I've been working on a sandbox-based solution to mitigate similar attacks -- in a nutshell, my sandbox creates an isolated environment somehow similar to docker so attacks that modify the system won't go through. Now people can obviously run it manually, but I was thinking that in several scenarios it would be cool to be able to enable the sandbox environment-wide inside a python virtual environment. This would give the possibility to create a "sandboxed" virtual environment where untrusted dependencies wouldn't be able to affect the rest of the system. LMK if that's kinda interesting for you and I can add few more practical details. Also, yes we're open to work if you believe there's interest. Thanks

[stuck merlin]

I'm aware about all this, but how do these isolated environments actually work? depending on your answer, might require to invole multiple people from different components, or maybe you just want to try out first to ship it as a plugin to virtualenv.

[past raven]

isolated environments' functioning depends on the implementation. In our implementation https://github.com/qualcomm/mini-sandbox we offer an execution mode that creates a "copy" of the filesystem and then sets up fine-grained permission levels. For instance all the system folders like /etc, /sys, /lib etc become read-only , we take special care of the home folder (so you can't access things like private keys , .bashrc, etc.) and we mark only a subset of folders as writable , mostly the current working dir and /tmp . Network access can be configured too , and all of this without any root access. Of course, we can sit and discuss what policies we would like to enable inside the isolated environment but conceptually I see this more as a feature for virtualenv -- ideally once created the virtual && isolated environment one should still be able to run pip install xxx and python xxx but if any of this creates harm it'd end up corrupting a fake environment or not having access to certain sensitive folders. A plugin to virtualenv would be a good starting point I think even though I'm not sure how that works

does this help ?

[stuck merlin]

Yes, it does help. One thing I will say that this project in particular tries to be cross-platform, and from what you're telling, comes to me like for focusing purely on Unix slash Linux.

Also feels to me like you're trying to do a lot more than just Python. So I would be interested to hear how you imagine more in detail that this would work with the Python ecosystem.

[past raven]

Agree on the platform aspect. We can still support Windows Subsystem for Linux (WSL) but the reality is that when it comes to pure Windows there's no an official Microsoft set of APIs to do anything similar (which is why all similar "container" based approaches work only on linux, see Docker as well for instance).

Looking at the "python" aspect of the thing , assuming we could design this thing inside the virtualenv, this is how I was envisioning it: 1) AFAIK during the creation of the virtualenv, a few symlinks get created in the virtualenv folder so the "key" binaries like python in the virtualenv are actually symlinks to the system binaries 2) let's say a flag enables the "secure" environment with isolation etc. we only have to add an indirection level between the virtualenv's python and the real python. So that'd be $HOME/.virtualenvs/venv/bin/python -> $HOME/.virtualenvs/venv/bin/sandboxed_wrapper_of_python -> /usr/bin/python 3) when someone calls python or pipthis will first start the sandbox, then the real python and finally the script and all its set of dependencies that will run in the sandbox.

This is a way. Another way would be to just start the sandbox in the activate when starting the virtualenv and let all the commands (not only python but also bash) run inside it. Security-wide better for sure but a little less "python" oriented 🙂 even though I'll argue that for those who want to use this feature a holistic approach should be a plus .

[untold breach]

Hey, the release yesterday broke hatch. There is a PR with a fix though it does undo some of the change that was desired in the 21 release

[stuck merlin]

I'm aware. That's why it was a major release.

[untold breach]

fair, but it puts a lot of people into a bit of a bind

[stuck merlin]

I helped with that PR. I don't think it undoes anything really.

[untold breach]

I'm struggling a little to see where the fix that was accepted is now. In virtualenv or in hatch?

[stuck merlin]

We added a pull request to the new discovery library to allow Hqlatch to do this in a less hacky way.

I did see that that pull request was opened, didn't check if they released it or not.

[undone pasture]

@stuck merlin To clarify https://github.com/pypa/virtualenv/issues/2087, I'm going to go with the approach of having the builtin classes picked first. I was going to hardcode the default to be "builtin" here (https://github.com/pypa/virtualenv/blob/0262fa6d9a0f8b45c80e6b2f82109ab1250ec336/src/virtualenv/run/plugin/discovery.py#L19). It's not great but I can't think of a better way of solving the problem.

[stuck merlin]

yes

I think that's the best option here 🙂

[slim garden]

🥳 source: https://www.jetbrains.com/lp/python-developers-survey-2020/

[stuck merlin]

Really should be split between venv/virtualenv there :-D or maybe we should stop referring to it as virtualenv, and just say virtual environment 🤔

[mortal sigil]

Can someone send a time machine to tell Ian to name the project something else? attrs sounds like a good name. 😉

None: 22% makes me sad

[digital flare]

For the eyebleed: v(irtual)env

[ashen sigil]

Hi! I need some help regarding the transition from virtualenv 16.7.10 to 20.*
I’m unpacking the virtualenv wheel using python’s zipfile module, and then for 16.7.10 I’m running python3 ./virtualenv.py.
20.2.2 has a different structure, so I’m running python3 ./virtualenv.
The former (16.7.10) worked anywhere, but the latter (20.2.2) fails on Ubuntu with

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "./virtualenv/__main__.py", line 77, in <module>
    run_with_catch()  # pragma: no cov
  File "./virtualenv/__main__.py", line 58, in run_with_catch
    from virtualenv.config.cli.parser import VirtualEnvOptions
ModuleNotFoundError: No module named 'virtualenv'

There is all it needs in the directory structure, but it acts like there’s no virtualenv directory there. What am I doing wrong here?

[stuck merlin]

that mode is not supported

virtualenv 20 wheel must be installed into a python environment for use.

if you want a download and run mode you should be using the zipapp variant of it https://virtualenv.pypa.io/en/latest/installation.html#via-zipapp

[ashen sigil]

Thank you for answering. Am I right that if I want to have the latest pip and setuptools installed to the virtualenv created via zipapp, I'll have to trigger it with --upgrade-embed-wheels before creating virtualenv? Will it cache the bundled wheels?

[stuck merlin]

it will

if you want to always download the latest just pass the --download flag or the VIRTUALENV_DOWNLOAD=1 env-var

[ashen sigil]

Hi again, thank you for your help. I tried the --download flag, but I'm not sure if I understand how it’s supposed to work. It looks like it keeps installing the bundled wheels unless it’s triggered before with —upgrade-embed-wheels. What does —download flag do? If I have not-latest zipapp, it creates an environment with not-latest pip & setuptools.

I'm sorry for the trouble, but I want to finally figure it out. 🙂

[stuck merlin]

Oh missed this ping me tomorrow and can help

First though read https://virtualenv.pypa.io/en/latest/user_guide.html

[ashen sigil]

I’ve read the user guide again and it says: “If … or download option is set <virtualenv> will use pip download to load the latest version available from the index server”. But if I have an old (downloaded several months ago) virtualenv zipapp, calling it with —download flag results with old (bundled) pip:20.2.4 & setuptools:50.3.2.
Also, according to the guide, if called without the —download flag, it should trigger a background process to update the embedded wheels, and it doesn’t seem like it does. Can I check that the process was triggered? top doesn’t show anything. Or would it update only after it saw the newer versions and waited 28 days?

[stuck merlin]

Sorry, this is too late for me 😬 Europe times

You should though check the contents of the state file within the app cache folder

Is it the same if you delete the application data folder

[ashen sigil]

It’s okay, I didn’t expect an answer till tomorrow 🙂
I’ll try your suggestions in the morning, thanks!

[ashen sigil]

Could you clarify what do you mean by state file, app cache folder and app data folder?
I’m on Ubuntu now, there’s some json files for pip, setuptools and wheel in the ~/.local/share/virtualenv/wheel/3.8/embed/1
The pip.json there looks like this:

{
  "completed": null,
  "periodic": true,
  "started": "2021-05-19T14:05:15.913174Z",
  "versions": []
}

If I delete virtualenv folder from ~/.local/share and then create new virtualenv from the same zipapp the process’s taking a second longer, and there’s the same structure in ~/.local/share/virtualenv after that, pip.json looks like this (only the timestamp changed):

{
  "completed": null,
  "periodic": true,
  "started": "2021-05-19T14:19:41.009741Z",
  "versions": []
}

Am I looking into the right file? When will it trigger the background process for updating the embedded wheels? How can I check it?

[stuck merlin]

Yes, that's the file 🙂

that there shows that it started an update of the embedded wheels, however seems to not finish

hence why completed is null

can you delete the folder and then run virtualenv --upgrade-embed-wheels ?

that should do the ugprade inline

what version of virtualenv are you using?

[ashen sigil]

20.1.0 I guess, the folder in ~/.local/share/virtualenv/unzip says that

[ashen sigil]

It works, but will it do the periodic update thing after that automatically? I'm not sure how to check that

[stuck merlin]

it should, 🙂

the only difference between the manual invocation and the automatic is that one shows its output on the stdout

while other is in the background silent

[ashen sigil]

The thing is (again) that I’m trying to make virtualenv creation in PyCharm fast and install the latest pip&setuptools at the same time. The previous approach (create venv with bundled pip&setuptools and then trigger an update using pip in the background) was a bad idea, because usually right after the packaging tools installation we want to install some packages and it interferes with the update. So I thought that using virtualenv could help to make the process faster and also will install the latest packaging tools automatically. But the thing is we don’t want to update the bundled virtualenv zipapp often (maybe almost never?..)
So sorry for the trouble and thank you so much. I guess that's not my last question though 😅

[stuck merlin]

but otherwise follows the same codepath

virtualenv should always create with latest bundled 🙂 so if you run virtualenv --upgrade-embed-wheels often enough in background you should be mostly up to date

the alternative is to force virtualenv to always reach out to PyPI for latest versions, via the download option, but that will come at a performance cost

[ashen sigil]

That doesn't work for me on Ubuntu for some reason, python3 ~/virtualenv.pyz --download dest does the same thing as the one without the --download option (the only difference in output is FromAppData(download=True part, but otherwise it finishes fast and doesn't download the latest packages)

[stuck merlin]

that sounds like a plugin

• like a bug

can you fill an issue for it?

also what's the virtualenv.pyz version you're running?

[ashen sigil]

the same one, 20.1.0
I'll file an issue

[ashen sigil]

Maybe this is a dumb question, but: how do I download some old version of virtualenv? I assembled a link according to documentation: https://github.com/pypa/get-virtualenv/blob/20.1.0/public/3.8/virtualenv.pyz?raw=true, which shows me the string "../virtualenv.pyz" and nothing happens.
I wanted to check the --download option on macos on the version that is newer than 20.1.0, but not the latest one.

[stuck merlin]

can you try with 20.4.6, just to check if an upgrade fixed it?

I don't think the py version part will work

https://raw.githubusercontent.com/pypa/get-virtualenv/20.1.0/public/virtualenv.pyz should though

you can file an issue about the https://github.com/pypa/get-virtualenv/blob/20.1.0/public/3.8/virtualenv.pyz to. https://github.com/pypa/get-virtualenv though

[ashen sigil]

I filed an issue about --download and dug a little into the code, it looks like this flag is ignored (but I'm not sure how to fix this right now).
If I pass --seeder=pip instead of it to zipapp version, it fails, so I filed another issue: https://github.com/pypa/virtualenv/issues/2121

[stuck merlin]

Thanks

Will look into them, but no promises when so if you have time to debug yourself that would speed up things🙏

[ashen sigil]

I’ll try😅

[stuck merlin]

That's all we can ask 👍if you get stuck and have specific question don't hesitate to ask

[ashen sigil]

Hi, I have another question:
If I download and run virtualenv==20.4.7 on a machine with existing virtualenv app data folder, it uses older bundled pip & setuptools (21.0.1 and 56.0.0, respectively). Is it intended?
I guess it’s because in app data folder there’s a json for each bundled wheel which says that the last periodic update was on the 2021-05-14 and at that time pip versions 21.1 and 21.1.1 were released less than 28 days ago. And it doesn't even see the new bundled 21.1.2.
So virtualenv zipapp update doesn’t help to get newer packaging tools if there’s an existing app data folder?

[stuck merlin]

You can remove the app data folder as a workaround, but this might be a bug.

[elfin zinc]

I was pointed here for poetry questions, so I hope this is the right channel

poetry update
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
The currently activated Python version 3.6.9 is not supported by the project (^3.7).
Trying to find and use a compatible version.
Using python3.8 (3.8.0)
Updating dependencies
Resolving dependencies... (37.7s)

  AssertionError



  at ~/.local/lib/python3.6/site-packages/poetry/mixology/incompatibility.py:60 in __init__
       56│                     # intersection, they're mutually exclusive, making this incompatibility
       57│                     # irrelevant, since we already know that mutually exclusive version
       58│                     # ranges are incompatible. We should never derive an irrelevant
       59│                     # incompatibility.
    →  60│                     assert by_ref[ref] is not None
       61│                 else:
       62│                     by_ref[ref] = term
       63│
       64│             new_terms = []
``` I have a very confusing assertionError that I can't really make sense of trying to install https://github.com/greatscottgadgets/luna

[clever cipher]

You are probably looking for https://discord.gg/sQpj2hDY :)

[elfin zinc]

aha thank you

[native falcon]

Hey @stuck merlin, could you cut a release of the current legacy virtualenv branch (specifically with #2109)?

[thin jetty]

@clever cipher how do I get a new invite?

[clever cipher]

https://discord.gg/yGxFmgkF

[thin jetty]

@clever cipher Thank you!

I believe I found sth buggy with the package_data={...} in setup.py, bc I got the FileNotFoundError using it -- substituting it with include_package_data=True made the package find the data files.

[stuck merlin]

eh, sure, why? 😄

probably better to raise an issue for it because this chat might get things lost 😄

[thin jetty]

here: https://github.com/pypa/setuptools ?

done. thx. bye

[robust herald]

I'm getting the following error when I run python -m virtualenv on Python 3.9.6:

AttributeError: 'dict' object has no attribute 'select'

Any ideas what could be causing that?

Line causing the error seems to be #14 in src/virtualenv/run/plugin/base.py

return OrderedDict((e.name, e.load()) for e in cls.entry_points().select(group=key))

[slim garden]

You should consider creating an issue at virtualenv's GutHub project. Make sure to also include the version of virtualenv.

[stuck merlin]

This seems like something introduced by @hard flume 🤔 what version you have and how was it installed? List of all dependencies in your env and their versions? What platform?

[stuck merlin]

@native falcon https://pypi.org/project/virtualenv/16.7.11/

[hard flume]

It sounds like backports.entry_points_selectable isn't working as designed. What do you get for python -c "from backports.entry_points_selectable import entry_points; print(entry_points())?

I note that python -m virtualenv works just fine for me: https://gist.github.com/685206cd96128bb9df37dd44f0aa6116

[hard flume]

(That's with Python 3.9.5)

[native falcon]

For running pip's tests against 3.10. :)

We realllly need to update pip's test suite to work with modern virtualenv, but that just hasn't happened yet.

[bleak hull]

i have a question

how does virtualenv find the root folder to put /virtualenv

because i use Termux and the virtualenv folder is in /data/data/com.termux/files/virtualenv which is next to the usr/ folder

[clever cipher]

you must provide the folder as an argument to virtualenv, if there's a virtualenv there it was created by some other program invoking virtualenv (or venv)

unless that's not a virtualenv but the virtualenv configuration folder

see https://github.com/platformdirs/platformdirs/blob/0a61690d341cdbdf319e21056a814bc98c89088b/src/platformdirs/android.py, virtualenv uses the user_data_dir

[bleak hull]

config

[bleak hull]

how do you tell if it's android

[clever cipher]

https://github.com/platformdirs/platformdirs/blob/0a61690d341cdbdf319e21056a814bc98c89088b/src/platformdirs/__init__.py#L19

[bleak hull]

alr ty

[stuck merlin]

@bleak hull any specific reason you ask?

[bleak hull]

im just wondering how i could implement that in a different python project

[stuck merlin]

Probably just use platformdirs ☺️

[bleak hull]

i just need to find the tmp dir on linux, android and mac

[stuck merlin]

That's the whole point of that project ☺️ to hide away how to get that

[bleak hull]

huh

ill look into it ty

[heady girder]

@stuck merlin aware of a nice tool when wanting to create/cache a number of python virtualenvs with specific properties

i want to regression test setuptools_scm against old setuptools and dont want to have to go docker to run things

those tests are end2end tests where i provide find links binaries/sdists to setuptools and verify specific features for setup.py install and pip install

[stuck merlin]

specific properties such as?

[heady girder]

setuptools/pip versions

python versions

pydistutils config files/pip config files

[stuck merlin]

virtualenv --download --setuptools 40 --wheel 0.29.0 venv --clear -p 2.7

created virtual environment CPython2.7.18.final.0-64 in 1108ms
  creator CPython2Posix(dest=/home/bernat/git/attrs-strict/venv, clear=True, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=True, pip=bundle, setuptools=40, wheel=0.29.0, via=copy, app_data_dir=/home/bernat/.local/share/virtualenv)
    added seed packages: pip==20.3.4, setuptools==40.0.0, wheel==0.29.0
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator

and then create the config files

you can pass in pip too there 🙂

I generally just write a pyest fixture for it, and use https://github.com/pypa/virtualenv/blob/main/src/virtualenv/run/__init__.py#L19 to invoke things

you don't need the download flag if you have a wheel cache of the old versions and pass it along via https://virtualenv.pypa.io/en/latest/cli_interface.html#extra-search-dir

[heady girder]

@stuck merlin lovely, i think this has all i need, i'll start with download and sort out he rest later

what minimal version of setuptools should i use, and is there a helper that tells met the avaliable python versions ?

whops s/setuptools/virtualenv/

[stuck merlin]

in a test suite, just use latest xD though any 20.x should do (ignoring past bugs)

there's no way to find out what's available, without trying it 🙂

https://github.com/pypa/virtualenv/blob/main/src/virtualenv/discovery/builtin.py#L60

you'll want to pass in the right key/spec and virtualenv will try to find it

[heady girder]

hmmm, i guess i'll go with python36 as a evil starting point

and add modern ones as i go

[stuck merlin]

e.g. is 3.9.2 vs 3.9.3 different Python version? 😄 or is 3.9.3-32 different than 3.9.3-64; that likely depends on what you're looking for

hence there's no way to know without you trying to discover it and see if something sticks

for your purpose you'll likely want something like:

available = [get_interpreter(f'3.{i}', []) for i in range(6,10)]

[heady girder]

@stuck merlin wrt logging, i just call the setup for it once then run the session with it set to false?

[heady girder]

@stuck merlin am i missing a easy helper to get the python executable when given a virtualenv path

[heady girder]

@stuck merlin it seems that importing virtualenv triggers warnings in setuptools plugins

[stuck merlin]

I generally set False when calling it programatically

you only need logging if you're running as an app

[stuck merlin]

you want to define a session https://github.com/pypa/virtualenv/blob/main/src/virtualenv/run/__init__.py#L36, and then session has a creator property (https://github.com/pypa/virtualenv/blob/main/src/virtualenv/run/session.py#L31) that contains that information

namely https://github.com/pypa/virtualenv/blob/c52fe3ea099e3d07c7fcf8982353eaede25e2e3f/src/virtualenv/create/describe.py#L80

[stuck merlin]

what kind of

[heady girder]

@stuck merlin when it fetches the distutils metadata it triggers the finalize_options entry-point hook, which then triggers a setuptools_scm warning

@stuck merlin when i try to pass a veresion of 9 for pip in the options, i still get pip 21

will try using the cli args now instead of a option instance

[stuck merlin]

which options

the options argument for session_via_cli is only useful to acquire the outcome of the CLI flags; but you cannot use it to pass in options; the only interface supported is the args in it:-)

[stuck merlin]

Hmm, not sure how we could avoid that 🤔

[heady girder]

i'll add a setuptools scm test to avoid it for virtualenv ^^

[stuck merlin]

🙏

[heady girder]

@stuck merlin the error/arning from the setuptools_scm side is gone

[stuck merlin]

Cool

[heady girder]

@stuck merlin are there any known issues on 20.7 about pip being missing in some virtualenvs (having issues with the setuptools scm pr and the tests against old pip) (unfortunately i cant paste a link as discord crashes when i try to paste a link)

(on python 3.6)

i'll defer debugging this one for after a release

[heady girder]

release is out

[stuck merlin]

No known issues

If you run into anything it's a bug

[half cave]

FYI the filelock project (used by tox and virtualenv) is "kinda dead": https://github.com/benediktschmitt/py-filelock/issues/84#issuecomment-896611059

the maintainer said in August:

I don't know who I can hand it over. I thought about asking the venv people, because it is used there. I'll think about what to do with this project in the next days.

[keen fossil]

Something broke virtualenv -p python3 on Debian 10 (buster) within the last few hours.

# virtualenv -p python3 /tmp/v
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /tmp/v/bin/python3
Also creating executable in /tmp/v/bin/python
Installing setuptools, pkg_resources, pip, wheel...
  Complete output from command /tmp/v/bin/python3 - setuptools pkg_resources pip wheel:
  Looking in links: /usr/lib/python3/dist-packages, /usr/share/python-wheels/
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/4e/2e/f8e006dbaaa46ed1e762c287585b92476deb8d3ccb79b720ed3b86bc6113/setuptools-58.1.0-py3-none-any.whl (816kB)
Collecting pkg_resources
Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://pypi.org/simple/pkg-resources/

----------------------------------------
...Installing setuptools, pkg_resources, pip, wheel...done.
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 11, in <module>
    load_entry_point('virtualenv==15.1.0', 'console_scripts', 'virtualenv')()
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 724, in main
    symlink=options.symlink)
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 996, in create_environment
    download=download,
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 926, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/usr/lib/python3/dist-packages/virtualenv.py", line 817, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /tmp/v/bin/python3 - setuptools pkg_resources pip wheel failed with error code 1

[scenic obsidian]

That seems Debian-specific, I'll have a look

[keen fossil]

Yes, pkg_resources is added by this Debian patch: https://sources.debian.org/src/python-virtualenv/15.1.0+ds-2/debian/patches/use-wheels.patch/

But that package hasn’t changed since 2018. Something must have changed very recently to break it.

Was there perhaps a server-side redirect from pkg_resources to setuptools to work around this Debian weirdness that was recently removed?

[scenic obsidian]

Yeah, that's what I was wondering, too

FWIW, --no-download works as expected.

[keen fossil]

Yeah, I’ll probably go with that for now. I also tried --no-setuptools, and that works on Debian 10 but breaks on Debian 11.

# virtualenv --no-setuptools -p python3 /tmp/v
KeyError: 'setuptools'

(That’s all I get, no traceback.)

Ah, --with-traceback:

# virtualenv --with-traceback --no-setuptools -p python3 /tmp/v
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 33, in <module>
    sys.exit(load_entry_point('virtualenv==20.4.0+ds', 'console_scripts', 'virtualenv')())
  File "/usr/lib/python3/dist-packages/virtualenv/__main__.py", line 65, in run_with_catch
    run(args, options, env)
  File "/usr/lib/python3/dist-packages/virtualenv/__main__.py", line 18, in run
    session = cli_run(args, options, env)
  File "/usr/lib/python3/dist-packages/virtualenv/run/__init__.py", line 32, in cli_run
    of_session.run()
  File "/usr/lib/python3/dist-packages/virtualenv/run/session.py", line 47, in run
    self._seed()
  File "/usr/lib/python3/dist-packages/virtualenv/run/session.py", line 60, in _seed
    self.seeder.run(self.creator)
  File "/usr/lib/python3/dist-packages/virtualenv/seed/embed/via_app_data/via_app_data.py", line 43, in run
    with self._get_seed_wheels(creator) as name_to_whl:
  File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "/usr/lib/python3/dist-packages/virtualenv/seed/embed/via_app_data/via_app_data.py", line 130, in _get_seed_wheels
    if name_to_whl['setuptools'].path.is_relative_to(BUNDLE_FOLDER):
KeyError: 'setuptools'

Which is also in a Debian patch: https://sources.debian.org/src/python-virtualenv/20.4.0+ds-2/debian/patches/include-pkg_resources.patch/

I guess I’ll file two Debian bugs. But if there was previously a server-side workaround for the first, it’d be nice if that were restored.

[scenic obsidian]

I already filed them

[keen fossil]

Ah, the Debian maintainer (is that you?) filed https://bugs.debian.org/994952 and https://bugs.debian.org/994953.

Cool, thanks.

(I submitted one too before seeing yours but it hasn’t shown up yet.)

The buster version fails with Python 2 too, in the same way.

[scenic obsidian]

I'll merge the duplicate bugs.

[fathom stump]

@stuck merlin hey! will there be a new version soon that ships https://pip.pypa.io/en/stable/news/#v21-3

[stuck merlin]

usually we wait for the first patch release 🙂

the first release is always too unstable 🙂

@fathom stump so yeah, there will be in a few weeks

[native falcon]

@stuck merlin now's the time to do that. :P

[stuck merlin]

Indeed

I'll do over the weekend

[stuck merlin]

virtualenv 20.9.0 is out https://virtualenv.pypa.io/en/latest/changelog.html#v20-9-0-2021-10-23 (new pip bundled, support for prompt=. and PyPy 3.8 support)

[fathom stump]

when I upgraded to ^ it was still using the old pip until using --pip embed once. should that always be used then when depending on the specific shipped version of pip?

[stuck merlin]

that must be some bug in our wheel cache logic

[oblique helm]

Using pinned versions that are not bundled or in the distribution embed JSON file always triggers a download even if it's already been downloaded once. I'd really like to get the speed improvements seen when installing with bundled / embed wheels for pinned versions as well. For now, the download (which is nonetheless cached by pip) adds 2.2 seconds (macOS, CPython 3.9) on the environment creation.
Is this something that's worth pursuing in virtualenv issue tracker / PR ? (I'm thinking to update the embed JSON file whenever download_wheel is called or something like that)

[stuck merlin]

Yeah please go ahead

[fathom stump]

@stuck merlin would you be open to no longer using __file__? https://github.com/indygreg/PyOxidizer/issues/69

[fathom stump]

https://pyoxidizer.readthedocs.io/en/latest/oxidized_importer_resource_files.html#porting-code-to-modern-resources-apis

[stuck merlin]

Where?

[fathom stump]

virtualenv

I want to build a binary but ^ prevents it atm

[stuck merlin]

Can you link to virtualenv code that does it

[fathom stump]

warning: virtualenv.activation.python.activate_this contains __file__
warning: virtualenv.create.creator contains __file__
warning: virtualenv.create.via_global_ref._virtualenv contains __file__
warning: virtualenv.create.via_global_ref.api contains __file__
warning: virtualenv.create.via_global_ref.builtin.python2.python2 contains __file__
warning: virtualenv.create.via_global_ref.builtin.python2.site contains __file__
warning: virtualenv.discovery.cached_py_info contains __file__
warning: virtualenv.info contains __file__
warning: virtualenv.run contains __file__
warning: virtualenv.seed.wheels.embed contains __file__
warning: virtualenv.util.zipapp contains __file__

https://github.com/pypa/virtualenv/blob/20.13.0/src/virtualenv/seed/wheels/embed/__init__.py#L6

[worldly pulsar]

Howdy, everyone. Getting a weird error:

 ericfletcher@Erics-MBP  ~  mkvirtualenv -a ~/mopidy-dev --python $(which python3.7) \
  --system-site-packages mopidy
RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.7'

Any idea @stuck merlin ?

[stuck merlin]

Can you replicate with virtualenv -v, without the mkvirtualenv thing

[fathom stump]

@stuck merlin I just had a thought: would you ever consider adding Hatchling to embedded wheels as an option?

[stuck merlin]

Embedded wheels?

[fathom stump]

like how it ships pip, setuptools, & wheel

[stuck merlin]

Oh to virtualenv? 😅

I want to remove setuptools and wheel to if anything 😅 people should use build-requires and not the embedded wheel functionality

[fathom stump]

hmm 🤔

what's faster: installing build-system.requires or virtualenv's copying of embedded wheels?

[stuck merlin]

Invalid question 😅

Under what situation?

[clever cipher]

well, it's highly unlikely you'll be building without isolation (either via pip or build, ultimately pip), so what purpose would provisioning hatchling serve? virtualenv bundles setuptools and wheel for legacy installs

[fathom stump]

let's say build-system.requires has setuptools & wheel. would a frontend using virtualenv + pip installing them be faster if they were already there so pip is no-op?

[stuck merlin]

pip doesn't do that, always creates fresh isolated build environment

Just because the right version of setuptools and wheels is installed that doesn't mean it would be a valid environment, e.g. user could have installed additional setuptools-magic package though that would end up in a different package. Pep -518 doesn't just require at least these package, but more like exactly these packages

[fathom stump]

ok very sorry allow me to rephrase: is pip's cache faster than virtualenv's embedded wheel cache/install?

[clever cipher]

starting a pip (sub)process is several magnitudes slower than reading the hatchling wheel from cache. Besides, pip doesn't use virtualenv for isolation

[fathom stump]

I talking from the POV of build or hatch where we make a venv and within that do pip install <build-system.requires>

[clever cipher]

yeah, build isn't clever enough to skip invoking pip if all the build deps are installed so you're still incurring the subp cost

I guess you could shave a couple of seconds off a build if you didn't have to run pip to install hatchling

[stuck merlin]

no 🙂

but, no PEP-518 ever should use virtualenv embedded wheels

what you're trying to achieve, solve here? or what's the problem you're trying to solve?

[fathom stump]

just wondering about potential time-to-build decrease after you remove the wheels

yeah plus Hatchling has deps whereas setuptools doesn't so still extra steps, nvm then. thanks!

[stuck merlin]

should not change at all 🙂 unless you're not using build isolation

e.g. if hatch uses the build project to build the wheel then likely should switch to a cached isolated build env (something tox also does) to not have to pay the isolated build env creation every time

[heady girder]

@fathom stump wouldn't reuse of compatible preexisting environments buy way more speed

[fathom stump]

yeah but wouldn't be "isolated"

[heady girder]

@fathom stump but why would it matter if multiple builds of different packages used the same immutable venv

[fathom stump]

wouldn't be immutable since build reqs may differ

[heady girder]

Different req would use different envs

[fathom stump]

yeah I might do it, not high priority

[slim garden]

Congratulations, @stuck merlin

[stuck merlin]

That's a bit misleading though because 99% of people think venv == virtualenv 😅

[slim garden]

shhhh 😄

[mortal sigil]

Is virtualenv’s Python interface stable now? The docs mention Q3 2020 which has passed for quite a while https://virtualenv.pypa.io/en/stable/extend.html

[stuck merlin]

I'd say yes, can you add a PR to change it? 😂

[mortal sigil]

Just delete the callout?

[stuck merlin]

Indeed