#pip | PyPA | Page 2

hoary mist Jul 19, 2023, 7:22 PM

#

(also jsut the fact that pip install reports are pip specific :V)

shy echo Jul 19, 2023, 7:22 PM

#

Oh, I agree. :)

#

It's... Well, institutions want it for compliance reasons so that governments don't fine them about it. And, IMO volunteers shouldn't be responsible for too much stuff around them and moving "costs" of that on those institutions (which ideally handle this by supporting the upstream foundational infra). 😅

mint patrol Jul 19, 2023, 7:30 PM

#

I agree with that! I'm mostly poking around if there's already been thinking about anything related wrt installers (or whether the project could become something that's under PyPA, or completely separate). I'm kinda taking pip-audit as the closest comparison, but noting that indeed SBOMs are likely to be of less utility to non-institutional open source users than vulnerabilities. I think there's still value in having a tool that does this work be close to the body that creates the standards for packaging.

shy echo Jul 19, 2023, 7:30 PM

#

100%

#

One of my hopes is that the whole packaging governance story settles with some reasonable body for people to throw money toward these PyPA projects.

mint patrol Jul 19, 2023, 7:32 PM

#

That would be an excellent outcome indeed! Has anyone talked to PSF about that?

shy echo Jul 19, 2023, 7:32 PM

#

In what form?

mint patrol Jul 19, 2023, 7:33 PM

#

Just about ways to help make that a reality. I know PSF does fiscal hosting for some OSS projects already.

shy echo Jul 19, 2023, 7:33 PM

#

PyPA is a fiscal sponsoree already.

#

The issue is the lack of unified direction (part of the motivation for a unified packaging vision).

mint patrol Jul 19, 2023, 7:34 PM

#

Gotcha

shy echo Jul 19, 2023, 7:35 PM

#

And, there's also a Packaging-WG

#

who handles the funds IIUC.

#

Not sure I fully understand the relationship between the PyPA pool and the packaging-WG pool. OTOH, I don't really care enough to find out the details about it.

sharp pebble Jul 19, 2023, 7:53 PM

#

I remember reading a discuss thread about a Packaging Council awhile ago. pithink Is that the body that would ultimately be responsible for this?

shy echo Jul 19, 2023, 8:00 PM

#

That's my hope.

hoary mist Jul 19, 2023, 8:12 PM

#

There's no answer on what the packaging council would be ultimately responsible for atm, because it's still just an idea so it has to get officially proposed and decided on... but I think the general consensus of what I've seen is that body would ultimately be responsible for these kind of ecosystem wide vision / scope decisions

cyan zephyr Jul 25, 2023, 7:50 AM

#

$ pip install django
Collecting django
  Obtaining dependency information for django from https://files.pythonhosted.org/packages/d4/83/227ebf197e413f3599cea96dddc7d6b8ff220310cc5b40dd0f1a15e5a9d1/Django-4.2.3-py3-none-any.whl.metadata

Just noticed the metadata file for the first time. OMG, it's cool!

keen summit Aug 15, 2023, 1:29 AM

#

does anyone know why pip's resolver would error out with

  ERROR: Could not find a version that satisfies the requirement jax[tpu]==0.4.14 (from redacted) (from versions: 0.0, 0.1, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.1.10, 0.1.11, 0.1.12, 0.1.13, 0.1.14, 0.1.15, 0.1.16, 0.1.18, 0.1.19, 0.1.20, 0.1.21, 0.1.22, 0.1.23, 0.1.24, 0.1.25, 0.1.26, 0.1.27, 0.1.28, 0.1.29, 0.1.30, 0.1.31, 0.1.32, 0.1.33, 0.1.34, 0.1.35, 0.1.36, 0.1.37, 0.1.38, 0.1.39, 0.1.40, 0.1.41, 0.1.42, 0.1.43, 0.1.44, 0.1.45, 0.1.46, 0.1.47, 0.1.48, 0.1.49, 0.1.50, 0.1.51, 0.1.52, 0.1.53, 0.1.54, 0.1.55, 0.1.56, 0.1.57, 0.1.58, 0.1.59, 0.1.60, 0.1.61, 0.1.62, 0.1.63, 0.1.64, 0.1.65, 0.1.66, 0.1.67, 0.1.68, 0.1.69, 0.1.70, 0.1.71, 0.1.72, 0.1.73, 0.1.74, 0.1.75, 0.1.76, 0.1.77, 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.2.5, 0.2.6, 0.2.7, 0.2.8, 0.2.9, 0.2.10, 0.2.11, 0.2.12, 0.2.13, 0.2.14, 0.2.15, 0.2.16, 0.2.17, 0.2.18, 0.2.19, 0.2.20, 0.2.21, 0.2.22, 0.2.23, 0.2.24, 0.2.25, 0.2.26, 0.2.27, 0.2.28, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.3.5, 0.3.6, 0.3.7, 0.3.8, 0.3.9, 0.3.10, 0.3.11, 0.3.12, 0.3.13, 0.3.14, 0.3.15, 0.3.16, 0.3.17, 0.3.18, 0.3.19, 0.3.20, 0.3.21, 0.3.22, 0.3.23, 0.3.24, 0.3.25, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.4.5, 0.4.6, 0.4.7, 0.4.8, 0.4.9, 0.4.10, 0.4.11, 0.4.12, 0.4.13, 0.4.14)
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 316, in _backjump
    name, candidate = broken_state.mapping.popitem()
KeyError: 'dictionary is empty'

for bonus fun, this only happens when running pip-compile; pip install with the exact same requirement works fine 😕

#

(@shy echo I guess)

cyan zephyr Aug 15, 2023, 8:51 AM

#

keen summit does anyone know why pip's resolver would error out with ``` ERROR: Could not...

I've tested on macOS/py310 with latest pip-tools, "works on my machine":

$ echo jax[tpu]==0.4.14 | pip-compile -qo- \
    --no-header --no-annotate --no-emit-options --no-strip-extras \
    --find-links https://storage.googleapis.com/jax-releases/libtpu_releases.html
jax[tpu]==0.4.14
jaxlib==0.4.14
libtpu-nightly==0.1.dev20230727
ml-dtypes==0.2.0
numpy==1.25.2
opt-einsum==3.3.0
scipy==1.11.1

Do you have a reproducible example? It would be nice if you could report a bug on the pip-tools' bug tracker.

keen summit Aug 15, 2023, 8:56 AM

#

cyan zephyr I've tested on macOS/py310 with latest pip-tools, "works on my machine": ```cons...

unfortunately my only reproducer right now requires running pip-compile on not only our entire monorepo, but also using our current set of pins as the base versions 😦 I've got hypothesis running to try to trim that down, but no luck so far

cyan zephyr Aug 15, 2023, 9:14 AM

#

keen summit unfortunately my only reproducer right now requires running `pip-compile` on not...

Interesting... Shouldn't try-except (IndexError, KeyError) catch KeyError?

Traceback (most recent call last):
    name, candidate = broken_state.mapping.popitem()
KeyError: 'dictionary is empty'

https://github.com/pypa/pip/blob/0778c1c153da7da457b56df55fb77cbba08dfb0c/src/pip/_vendor/resolvelib/resolvers.py#L316-L317

keen summit Aug 15, 2023, 9:19 AM

#

yeah, the traceback continues as

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 316, in _backjump
    name, candidate = broken_state.mapping.popitem()
KeyError: 'dictionary is empty'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 434, in resolve
    success = self._backjump(causes)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 318, in _backjump
    raise ResolutionImpossible(causes)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('jax[tpu]==0.4.14'), parent=EditableCandidate('file:///root/code/redacted'))]

The above exception was the direct cause of the following exception:

#


Traceback (most recent call last):
  File "/root/.pyenv/versions/3.10.11/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/piptools/scripts/compile.py", line 592, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/piptools/resolver.py", line 593, in resolve
    is_resolved = self._do_resolve(
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/piptools/resolver.py", line 625, in _do_resolve
    resolver.resolve(
  File "/root/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 101, in resolve
    raise error from e
pip._internal.exceptions.DistributionNotFound: No matching distribution found for jax[tpu]==0.4.14

#

I don't think there's any additional information content in that part though

lunar gyro Aug 15, 2023, 9:38 AM

#

The KeyError is an implementation detail in resolvelib that basically means it has exhausted all possibilities. That exception is caught inside resolvelib to reraise ResolutionImpossible to indicate the direct cause and users should catch that to display relevant messages. From the above I think piptools fails to catch and handle an exception raised along the stack

shadow shuttle Aug 15, 2023, 9:01 PM

#

i made the keyerror not be reported in https://github.com/sarugaku/resolvelib/pull/133

cyan zephyr Aug 15, 2023, 9:21 PM

#

However, it seems odd that it fails to find the package, even though version 0.4.14 is available according to the log:

ERROR: Could not find a version that satisfies the requirement jax[tpu]==0.4.14 ... (from versions: ..,, 0.4.14)

shy echo Aug 15, 2023, 9:37 PM

#

I wouldn't be surprised if there's some sort of state corruption affecting the resolve here.

lunar gyro Aug 16, 2023, 12:38 AM

#

I mean even the latter two exceptions should not have been shown either (even if the package should not be found), and combining with the fact that pip does find a solution (indicating the package is obviously found to satisfy it) means this is really 100% on pip-tools and nowhere else

keen summit Aug 16, 2023, 9:24 AM

#

eventually my workaround was to... go in and manually copy the new requirements I was passing to pip-compile into the old pins file, and that turned out to be a valid resolution. total diff -3 +3.

#

no idea how pip-compile was screwing that up

shy echo Aug 16, 2023, 9:50 AM

#

If you can dump state and write the output somewhere with PIP_RESOLVER_DEBUG=1 (making appropriate redactions), that could be useful diagnosis logs.

foggy forum Aug 16, 2023, 10:39 AM

#

@keen summit what did the pins look like when you run pip-compile and it failed?

foggy forum Aug 21, 2023, 7:59 PM

#

with the requirement of <5% installs on EOL python versions, isn't that going to be quite difficult to achieve because newer non-EOL python versions bundle latest pip?

shy echo Aug 21, 2023, 10:30 PM

#

I'm not sure I follow your thinking here.

hidden flame Aug 22, 2023, 12:04 AM

#

I think the line of thinking is that since non-EOL Python comes with the latest pip, those environments won't need to install pip as frequently as one based on an EOL Python version, causing the EOL versions to be overrepresented in the install statistics.

#

I don't buy that. pip is upgraded constantly regardless of the Python version

foggy forum Aug 22, 2023, 10:22 AM

#

hidden flame I think the line of thinking is that since non-EOL Python comes with the latest ...

Yeah that's what I'm thinking

stuck girder Aug 22, 2023, 10:34 AM

#

the 3.7 proportion for pip isn't much different compared to other packages
pip:

┌──────────┬─────────┬─────────────┐
│ category │ percent │   downloads │
├──────────┼─────────┼─────────────┤
│ 3.7      │  24.17% │  39,846,601 │
│ 3.9      │  19.77% │  32,590,855 │
│ 3.8      │  19.19% │  31,647,764 │
│ 3.10     │  13.60% │  22,416,844 │
│ 2.7      │   8.90% │  14,682,014 │
│ 3.6      │   5.25% │   8,649,690 │
│ 3.11     │   5.20% │   8,579,509 │
│ null     │   3.53% │   5,817,611 │
│ 3.5      │   0.26% │     420,770 │
│ 3.4      │   0.09% │     155,353 │
│ 3.12     │   0.05% │      75,603 │
│ 3.13     │   0.00% │         998 │
│ 3.3      │   0.00% │         822 │
│ 3.2      │   0.00% │         355 │
│ 2.6      │   0.00% │           2 │
│ 3.1      │   0.00% │           2 │
│ 4.0      │   0.00% │           1 │
│ Total    │         │ 164,884,794 │
└──────────┴─────────┴─────────────┘

setuptools:

┌──────────┬─────────┬─────────────┐
│ category │ percent │   downloads │
├──────────┼─────────┼─────────────┤
│ 3.7      │  19.78% │  53,635,167 │
│ 3.8      │  19.66% │  53,311,659 │
│ 2.7      │  15.75% │  42,709,012 │
│ 3.9      │  13.87% │  37,612,549 │
│ 3.10     │  13.52% │  36,680,128 │
│ 3.11     │   6.40% │  17,367,188 │
│ null     │   5.77% │  15,661,435 │
│ 3.6      │   5.02% │  13,605,348 │
│ 3.5      │   0.17% │     470,967 │
│ 3.12     │   0.05% │     131,666 │
│ 3.4      │   0.01% │      34,022 │
│ 3.13     │   0.00% │       3,650 │
│ 3.3      │   0.00% │       1,125 │
│ 3.2      │   0.00% │         283 │
│ 2.6      │   0.00% │          81 │
│ 3.1      │   0.00% │           1 │
│ Total    │         │ 271,224,281 │
└──────────┴─────────┴─────────────┘

#

urllib3:

┌──────────┬─────────┬─────────────┐
│ category │ percent │   downloads │
├──────────┼─────────┼─────────────┤
│ 3.7      │  31.16% │ 110,540,942 │
│ 3.8      │  19.04% │  67,558,918 │
│ 3.9      │  15.23% │  54,025,932 │
│ 3.10     │  11.69% │  41,458,651 │
│ 3.6      │   7.58% │  26,891,708 │
│ 3.11     │   7.12% │  25,246,114 │
│ 2.7      │   4.32% │  15,308,015 │
│ null     │   3.54% │  12,557,543 │
│ 3.5      │   0.17% │     602,821 │
│ 3.4      │   0.14% │     487,594 │
│ 3.12     │   0.02% │      76,309 │
│ 3.13     │   0.00% │       3,362 │
│ 3.3      │   0.00% │         678 │
│ 3.2      │   0.00% │          27 │
│ 2.6      │   0.00% │          12 │
│ 3.1      │   0.00% │           3 │
│ Total    │         │ 354,758,629 │
└──────────┴─────────┴─────────────┘

shy echo Aug 22, 2023, 10:36 AM

#

These would be based on what client is used to send download requests to PyPI as a whole, so there's that.

#

Beyond that, the rationale for this has been to avoid ending up in a situation where we have a substantial number of users who can't be accommodated to by a new release -- packaging tooling needs to be generous in the support cycles rather than aggressive.

(which, as folks here might know, is something that I don't enjoy and actively push back on at every opportunity)

foggy forum Aug 22, 2023, 11:41 AM

#

3.7 is ludicrously popular

#

pydantic are ignoring Amazon Linux installs as that seems to be the source of large numbers of 3.7 installs

#

https://github.com/pydantic/pydantic/pull/7188#issuecomment-1687916319

GitHub

Drop Python3.7 support by hramezani · Pull Request #7188 · pydantic...

Closes #7187

shy echo Aug 22, 2023, 11:43 AM

#

Yea, we might do something like that. I don't have time at the moment to look into what the usage patterns are and what we should be doing here.

stuck girder Aug 22, 2023, 11:56 AM

#

20% is Amazon Linux on 3.7!

(just 1 day's data to save some quota, but still covers 4.5m downloads)

📎 message.txt

shy echo Aug 22, 2023, 11:57 AM

#

That's 20% of PyPI downloads as a whole?

stuck girder Aug 22, 2023, 11:58 AM

#

for downloads of pip

shy echo Aug 22, 2023, 11:59 AM

#

Ah, OK. I think we'd want to check PyPI as a whole.

#

But still, oof.

stuck girder Aug 22, 2023, 12:01 PM

#

for downloads of all packages:
Ubuntu on 3.8 = 18%
Amazon Linux on 3.7 = 15%

📎 message.txt

#

(

❯ pypinfo --days 1 --limit 100 --percent "" system distro pyversion
Served from cache: False
Data processed: 44.29 GiB
Data billed: 44.29 GiB
Estimated cost: $0.22

)

shy echo Aug 22, 2023, 12:03 PM

#

Thanks for running that!

mint patrol Aug 22, 2023, 6:52 PM

#

@stuck girder Could you publish that data somewhere? I think it will be useful for folks who are making version support decisions.

#

Because TIL Amazon Linux is causing a big percentage of 3.7 downloads.

stuck girder Aug 22, 2023, 6:52 PM

#

sure!

stuck girder Aug 22, 2023, 7:25 PM

#

mint patrol <@712244192226574348> Could you publish that data somewhere? I think it will be ...

owzat? https://dev.to/hugovk/why-are-there-still-so-many-downloads-for-eol-python-37-30cp

DEV Community

Why are there still so many downloads for EOL Python 3.7?

Python 3.7 was first released on 2018-06-27 and recently reached end-of-life on 2023-06-27 (PEP...

mint patrol Aug 22, 2023, 7:27 PM

#

Perfect, thank you!

mint patrol Aug 29, 2023, 11:57 PM

#

The PSF is a CNA (covering pip!) 🥳 https://pyfound.blogspot.com/2023/08/psf-authorized-as-cna.html

Python Software Foundation Blog

The Python Software Foundation has been authorized by the CVE Progr...

When a vulnerability is disclosed in software you're depending on, the last thing you want is for the remediation process to be confusing or...

shy echo Aug 29, 2023, 11:58 PM

#

Awesome!

limber ore Aug 30, 2023, 1:17 PM

#

is it possible to get a wheel for a particular package from the cache?

shy echo Aug 30, 2023, 1:32 PM

#

You can query the cache by running pip cache subcommands IIRC.

limber ore Aug 30, 2023, 1:48 PM

#

shy echo You can query the cache by running `pip cache` subcommands IIRC.

I tried but was missing the format argument, thanks!

pip cache list --format abspath PACKAGE

dapper laurel Sep 28, 2023, 6:17 PM

#

do I get it right, that pip's dependency resolver only takes into consideration packages passed in CLI call? It doesn't inspect the environment for the packages that are already installed?

lunar gyro Sep 28, 2023, 6:23 PM

#

dapper laurel do I get it right, that pip's dependency resolver only takes into consideration ...

yes

dapper laurel Sep 28, 2023, 6:36 PM

#

and that works with requirements files too?

lunar gyro Sep 28, 2023, 10:51 PM

#

correct

#

Only caveat is if any of the packages are already installed, pip makes its best effort to prefer the installed version

alpine flower Sep 30, 2023, 5:31 PM

#

While I'm here, I want to highlight this issue: https://github.com/pypa/pip/issues/7863#issuecomment-1741723512
It's about pip download not downloading build depdencies, which currently makes it impossible to automate pre-collecting wheels and source tarballs for offline build environments.

I spent 4h yesterday while building my Flatpak to manually gather all the build dependencies.
Now that more and more packages are switching to PEP517/PEP518 maintaining Flatpaks (which need to be built in an offline environment) is becoming an absolute nightmare, and I was a bit surprised no other Flatpak maintainer seems to really have raised this issue to the pip team yet.
I would love to personally work on implementing this, however a previous comment on the issue (3 years ago) suggested bigger architectural changes would be needed. Is that still the case?

shy echo Sep 30, 2023, 5:42 PM

#

Yea, I don't think it's a giant refactor but it's definitely a decent amount of work to improve this.

alpine flower Sep 30, 2023, 5:43 PM

#

would alternatively a separate command for that be an option?

shy echo Sep 30, 2023, 5:44 PM

#

I don't think that's going to solve things, and making it opt-in via an option is definitely an option if you wanna go down that route.

maiden island Sep 30, 2023, 5:45 PM

#

Yeah, opt in via CLI arg / config would be my vote

shy echo Sep 30, 2023, 5:45 PM

#

And, no, a new command wouldn't be viable unless there's a strong case for it.

maiden island Sep 30, 2023, 5:47 PM

#

I’ve long wanted to steal a resolver that could power this for bandersnatch to auto resolve dependencies for people if it was ever “publically” available as it’s been requested many many times

shy echo Sep 30, 2023, 5:50 PM

#

There's some work being done on https://github.com/brettcannon/mousebender/issues/105 for this (I'd appreciate it if folks don't add noise over there -- my sense is that's where Brett's keeping his notes on the work publicly, and I already feel somewhat guilty about poking it with resolvelib details multiple times). 😅

jovial jasper Sep 30, 2023, 6:49 PM

#

maiden island Yeah, opt in via CLI arg / config would be my vote

--only-build-deps flag is on the table

jovial jasper Sep 30, 2023, 6:53 PM

#

maiden island I’ve long wanted to steal a resolver that could power this for bandersnatch to a...

If you can afford a CLI approach, pip install --quiet --dry-run --ignore-installed --report - essentially exposes the pip resolver.

shy echo Sep 30, 2023, 6:53 PM

#

It does not have build dependencies there though.

#

(yet?)

jovial jasper Sep 30, 2023, 6:54 PM

#

Not yet indeed but I'd consider that in scope.

shy echo Sep 30, 2023, 6:54 PM

#

Yea, same.

fallen scroll Sep 30, 2023, 6:55 PM

#

Huh, wonder if pip-compile could make use of that.

shy echo Oct 1, 2023, 1:22 PM

#

Looks like @jovial jasper and I have collectively DOS'd out GitHub Actions worker queues for the pypa. Sorry folks! 😆

high stone Oct 5, 2023, 1:44 PM

#

hi! i installed py3.12 from the ms store on win11 and trying to pip install opentypespec inside a fresh venv with only current pip fails because No module named 'distutils'. it works in a linux venv... am i overlooking something?

dapper laurel Oct 5, 2023, 1:45 PM

#

Check if opentypespec doesn't use distutils. If so, you should report it in the project repository

high stone Oct 5, 2023, 1:47 PM

#

they have a setup.py and no pyproject.toml, but no direct use of distutils

#

even if i put in a proper pyproj toml (and remove setup.*), it still fails, but only on windows

dapper laurel Oct 5, 2023, 1:48 PM

#

Weird. My guess would be that python version you are using is weird. MS store python is always causing weird issues

#

(At least in my experience)

high stone Oct 5, 2023, 1:49 PM

#

that's what i suspect

shy echo Oct 5, 2023, 1:49 PM

#

https://stackoverflow.com/ is a good place to ask this question, such that the answer to it isn't buried behind a non-indexable page. Consider posting the entire traceback with reproducer instructions there?

high stone Oct 5, 2023, 1:54 PM

#

hm 🙂 ok will do after trying with the python.org version

#

ugh, same error

#

if i install setuptools in the venv, i at least get

ERROR: Can not execute setup.py since setuptools is not available in the build environment.

past pagoda Oct 5, 2023, 2:08 PM

#

high stone hi! i installed py3.12 from the ms store on win11 and trying to `pip install ope...

Important deprecations, removals or restrictions:

gh-95299: Do not pre-install setuptools in virtual environments created with venv. This means that distutils, setuptools, pkg_resources, and easy_install will no longer available by default; to access these run pip install setuptools in the activated virtual environment.

- https://docs.python.org/3/whatsnew/3.12.html

past pagoda Oct 5, 2023, 2:08 PM

#

high stone if i install setuptools in the venv, i at least get > ERROR: Can not execute `...

firHmm

high stone Oct 5, 2023, 2:12 PM

#

yeah i saw that. the venv was created with virtualenv 20.24.5 though: virtualenv --python 3.12

#

the last part of the stack trace is

  File "C:\Users\...\AppData\Local\Temp\pip-build-env-11pb36vg\overlay\Lib\site-packages\setuptools\__init__.py", line 8, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'

so that's why i came here to ask before filing an issue 🙂

dapper laurel Oct 5, 2023, 2:18 PM

#

Setuptools is vendoring distutils in some hacky way, maybe it stopped working for some reason

high stone Oct 5, 2023, 2:24 PM

#

https://github.com/pypa/pip/issues/12200 yeah this is the one

#

oh DUH. i had a SETUPTOOLS_USE_DISTUTILS=stdlib env var lying around from years ago

dapper laurel Oct 5, 2023, 2:38 PM

#

#yetAnotherUserError

limber ore Oct 7, 2023, 8:44 PM

#

FYI a bug on PyPy https://github.com/pypa/pip/issues/12326

limber ore Oct 12, 2023, 12:52 PM

#

does anyone know how the state of a package's metadata could break installation like this?

limber ore Oct 13, 2023, 3:38 AM

#

is this a cache bug? where is it? ThinkingFace

❯ pip install -UI cryptography
Collecting cryptography
  Obtaining dependency information for cryptography from https://files.pythonhosted.org/packages/d7/78/29d8332bebfe3c2d49a63fb23e1c9fc73a13507b5206b98479fda04c993b/cryptography-41.0.4-cp37-abi3-win_amd64.whl.metadata
  Using cached cryptography-41.0.4-cp37-abi3-win_amd64.whl.metadata (5.3 kB)
Collecting cffi>=1.12 (from cryptography)
  Obtaining dependency information for cffi>=1.12 from https://files.pythonhosted.org/packages/5a/c7/694814b3757878b29da39bc2f0cf9d20295f4c1e0a0bde7971708d5f23f8/cffi-1.16.0-cp311-cp311-win_amd64.whl.metadata
  Using cached cffi-1.16.0-cp311-cp311-win_amd64.whl.metadata (1.5 kB)
Collecting pycparser (from cffi>=1.12->cryptography)
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Using cached cryptography-41.0.4-cp37-abi3-win_amd64.whl (2.7 MB)
Using cached cffi-1.16.0-cp311-cp311-win_amd64.whl (181 kB)
Installing collected packages: pycparser, cffi, cryptography
Successfully installed cffi-1.15.1 cryptography-41.0.4 pycparser-2.21

❯ pip cache list --format abspath cryptography

❯

lunar gyro Oct 13, 2023, 4:04 AM

#

pip cache currently only reads the wheel cache (cache storing wheels implicitly built by pip from source), not the network cache.

limber ore Oct 13, 2023, 4:36 AM

#

lunar gyro `pip cache` currently only reads the wheel cache (cache storing wheels implicitl...

oh okay I was very confused. I found a way to get all of the wheels which is to just pip download and it won't hit network since everything is cached

limber ore Oct 15, 2023, 7:06 PM

#

congrats! https://discuss.python.org/t/announcement-pip-23-3-release/36296

#

I briefly got extremely excited about ABI and platform override options thinking everything finally got implemented until I expanded the caveat 🙂

fallen scroll Oct 15, 2023, 7:27 PM

#

I'm quite happy about the new HTTP cache.

dapper laurel Oct 15, 2023, 7:37 PM

#

Is it just internal rewrite or did you switch to another library?

fallen scroll Oct 15, 2023, 7:40 PM

#

I think it's just a different backend from the same library.

dapper laurel Oct 15, 2023, 7:47 PM

#

I will need to have a look 😄

small cove Oct 17, 2023, 6:02 PM

#

pip and pipx together are giving me a catch-22: I have a freshly installed ubuntu 23.10 and with python3-pip additionally installed. If i try to install anything with pip it tells me (correctly) "error: externally-managed-environment" "If you wish to install a non-Debian packaged Python application, it may be easiest to use pipx install xyz, which will manage avirtual environment for you. Make sure you have pipx installed.". Sure! I go to https://pypa.github.io/pipx/ and it tells me to run python3 -m pip install --user pipx which again leads to the aforementioned error message. It also tells me to try installing python3-xyz where xyz is the name of the package, but python3-pipx doesn't exist either

#

(i'm not sure what the right bootstrapping strategy here is, but i expect this loop to very confusing to newcomers)

fallen scroll Oct 17, 2023, 6:05 PM

#

FWIW, the Ubuntu package is just called pipx: https://packages.ubuntu.com/mantic/pipx

ember shuttle Oct 17, 2023, 10:51 PM

#

small cove pip and pipx together are giving me a catch-22: I have a freshly installed ubunt...

There's python-pipx as well

ashen geyser Oct 18, 2023, 11:34 AM

#

Yeah, pipx should definitely explain how to be installed using package managers before pip. Global tools like this should just be on your system once, managed and autoupdated by a package manager.

hard thunder Oct 21, 2023, 2:21 PM

#

I agree that it's best to install pipx through package manager if that's possible on your system. Otherwise, https://pypi.org/project/pipx-in-pipx/ is the second best option

PyPI

pipx-in-pipx

pipipxx (pronounced pipx in pipx): Bootstrap your pipx with pipx.

#

personally I do something like that on Windows (which doesn't have a package), just in a manual way

#

I do an equivalent of this:

venv_dir=$(mktemp -d)
python -m venv "$venv_dir"
"$venv_dir/bin/python" -m pip install pipx
"$venv_dir/bin/pipx" install pipx --python python3 --include-deps
rm -rf "$venv_dir"

#

python3 is the actual executable that you want it to use

#

so that it doesn't point at venv

#

the downside is that it's per-user install

#

but it is managed by pipx at least

#

and you get the latest version unlike with non-rolling distros

fallen scroll Oct 21, 2023, 4:21 PM

#

pipx-in-pipx is terrifying.

dapper laurel Oct 21, 2023, 4:27 PM

#

why?

fallen scroll Oct 21, 2023, 4:41 PM

#

It pretends to be a Python package, but actually does weird stuff to your machine.

#

I think it would make more sense as just a script.

dapper laurel Oct 21, 2023, 4:44 PM

#

I thought about rewriting pipx in rust or go, but the venv needs some sys/sysconfig stuff and I didn't dig deep enough to find out how to get that info without subprocessing to python

limber ore Oct 23, 2023, 1:44 PM

#

is there a way for pip freeze to emit hashes? I can't find an option

trail sun Oct 23, 2023, 1:47 PM

#

limber ore is there a way for `pip freeze` to emit hashes? I can't find an option

don't think so currently, could bump this old issue https://github.com/pypa/pip/issues/4732

limber ore Oct 23, 2023, 1:48 PM

#

ah thanks!

fallen scroll Oct 23, 2023, 1:59 PM

#

What hashes would it emit? It doesn't have access to the original packages.

hazy flume Oct 23, 2023, 2:00 PM

#

the same way pip-tools does, I suppose?

trail sun Oct 23, 2023, 2:07 PM

#

fallen scroll What hashes would it emit? It doesn't have access to the original packages.

I would think pip needs to start tracking that during install, only work on cached wheels, or deny this as a feature of pip freeze. but it's been a long open issue, with some of that already mentioned in there and (at least outside looking in) no progress towards any of those or other options made.

pip-tools (compile) resolves packages to do it, so simply doing what pip-tools does could have surprising results in some cases since freeze should reflect what is installed. (for instance, what if the local package was installed from sdist+patches that didn't modify the version? This is different from just an sdist or wheel, and should not match hashes.)

limber ore Oct 23, 2023, 2:12 PM

#

fallen scroll What hashes would it emit? It doesn't have access to the original packages.

I would assume it uses cached wheels but there may be some edge cases I'm not considering

fallen scroll Oct 23, 2023, 2:16 PM

#

There might not be any cached wheels.

lunar gyro Oct 23, 2023, 2:27 PM

#

Even if there is, most backends don’t build wheels with reproducible hashes so the freeze result would not be particularly useful except for recreating the environment on the exact same machine.

limber ore Oct 23, 2023, 2:51 PM

#

recreating the environment on the exact same machine
that is my current use case at work actually

lunar gyro Oct 23, 2023, 3:16 PM

#

I wonder if it’d be better to have a way to locally clone an environment. That would work whenever the freeze-with-hash method work and would not work if any of the packages can’t be built into a wheel (which would make the hash method not work anyway)

#

The edge case where cloning an environment would not work is if any of the installed file is not relocatable (e.g. badly built dll, but also any other build-time generated file that depends on more than just the relative paths inside the environment). But wheels cannot contain such non-relocatable files in the first place, so those must have been built from source instead. Not supporting sdist (or plain source trees) seems like an acceptable trade-off.

limber ore Oct 23, 2023, 3:24 PM

#

would not work if any of the packages can’t be built into a wheel
how is that possible since installation requires a wheel to be built in all cases?

hard thunder Oct 23, 2023, 3:26 PM

#

fallen scroll pipx-in-pipx is terrifying.

I mostly agree which is why I do that manually on my computers when I need to

#

But figured I shouldn't be omitting it as an option

lunar gyro Oct 23, 2023, 3:27 PM

#

limber ore > would not work if any of the packages can’t be built into a wheel how is that ...

Can’t be built into a good wheel I guess. A wheel would be produced but that wheel will only work for that one environment.

fallen scroll Oct 23, 2023, 3:29 PM

#

limber ore > would not work if any of the packages can’t be built into a wheel how is that ...

FWIW, I don't think installed projects have to come from wheels. They could be installed by some alternative installer.

hoary mist Oct 23, 2023, 3:44 PM

#

the problem effectively becomes that there's no guarentee that "foo-1.0" in an environment matches any particular artifact for foo-1.0 that exists anywhere on the system

#

there is a PEP to add some provenance info that would enable this, but I think the state of that PEP languished after the author changed jobs

#

but I could be wrong on that, I'm not nearly as good as following all the discussions anymore

shy echo Oct 23, 2023, 5:08 PM

#

limber ore is there a way for `pip freeze` to emit hashes? I can't find an option

No, since there's no distribution files to generate the hashes of.

limber ore Oct 23, 2023, 5:09 PM

#

my workaround currently is to freeze, download wheels to a directory from an existing environment, get the hash of each wheel, then inject into the output from the freeze command

shy echo Oct 23, 2023, 5:10 PM

#

You could also do a pip install --dry-run --report to get the artifact hashes.

#

(throwing in the output of the freeze into it)

limber ore Oct 23, 2023, 5:11 PM

#

plus our own package index for wheels that we have to build ourselves, for example we require CentOS 6 still

shy echo Oct 23, 2023, 5:11 PM

#

Ooof.

limber ore Oct 23, 2023, 5:12 PM

#

we are in a unique situation in which the infra is out of our control and determined by whatever customers use

shy echo Oct 23, 2023, 5:13 PM

#

Well, tell them they need to move on and get with the times.

#

Jk jk

limber ore Oct 23, 2023, 5:13 PM

#

I wish lol

shy echo Oct 23, 2023, 5:14 PM

#

"OS too old, you are no longer permitted to be a customer"

limber ore Oct 23, 2023, 5:16 PM

#

my experience working on this has given me an incredible amount of respect for the engineers at Microsoft that maintain their backward compatibility guarantees

#

this is rough

shy echo Nov 5, 2023, 6:05 PM

#

https://github.com/pradyunsg/pip-resolver-benchmarks is a thing that came out of conversations at PackagingCon this year, about how Spack benchmarks their dependency resolution process on top of clingo et al.

iron marlin Nov 6, 2023, 7:05 PM

#

Hi! I'm working on trying to add parallel download support to BatchDownloader. I've already opened one pull request to that effect, and am working on a separate pull request to add a proper progress bar for parallel downloads. I just wanted to make sure that, that sort of work is something that the pip project might actually want.

shy echo Nov 6, 2023, 8:02 PM

#

Hello hello!

#

I think that's useful still, yes.

#

I don't know though how much bandwidth people have to review a PR for this at the moment though, so all I can say is, a PR would be appreciated, please be patient with us and thanks for contributing! :)

iron marlin Nov 6, 2023, 8:08 PM

#

Of course! I just wanted to make sure because I didn't want to waste the maintainers time with unwanted PR's

limber ore Nov 27, 2023, 3:08 PM

#

oh I just came here to post about the parallel download PR. I'm very excited for this!

cyan cargo Dec 5, 2023, 3:39 PM

#

anyone aware of a resolver debugging cheat-sheet - i have a case where a exact pin in a constraints file seems to be ignored and i get warnings about pip searching versions of botocore even tho a exact version is pinned

unfortunately i cant easily extract a demo reproducer yet and in the private case there's a few hundred packages involved and loads of backtracking/doom versioning

shy echo Dec 6, 2023, 1:16 AM

#

Could you try using pip-resolver-benchmarks to create a reproducer?

cyan cargo Dec 6, 2023, 10:24 AM

#

shy echo Could you try using `pip-resolver-benchmarks` to create a reproducer?

thanks for the sugestion, i jsut had pinned enoguh toplevel packages to lates that the resolves finally ran into the pathological case - for some reason i have a package that pinned urllib3==2.0.7 - i now added a new requirement that wil lhopefully find me the error

cyan cargo Dec 6, 2023, 10:50 AM

#

🤦 someone had accidentally removed a python version pin for the locking process - i was locking my stuff with python3.11 on something that was running on python3.9 😱

shy echo Dec 6, 2023, 10:50 AM

#

😂

cyan cargo Dec 6, 2023, 10:53 AM

#

worst of all , time is inappropriate for a disgustingly smoky glass of whiskey now that i know the issue

dapper laurel Dec 6, 2023, 11:13 AM

#

cyan cargo worst of all , time is inappropriate for a disgustingly smoky glass of whiskey n...

It's 4PM somewhere on Earth 😉

cyan cargo Dec 6, 2023, 11:14 AM

#

dapper laurel It's 4PM somewhere on Earth 😉

i have seen what happens to people that apply alcoholic timezone, its not pretty

dapper laurel Dec 6, 2023, 11:15 AM

#

Yeah, I know, I have it running in the family :/

cyan cargo Dec 6, 2023, 11:17 AM

#

my condolences , that one is a "gift" that keeps taking

small cove Dec 8, 2023, 4:07 PM

#

What's the reason PEP 440 doesn't allow relative paths, but does allow absolute paths? I'm trying to understand why it was decided that way when PEP 440 was written

#

Naively, i'd have said that you need relative path for workspaces and vendored dependencies while absolute paths are the special, so i'm surprised the spec has it the other way round

hoary mist Dec 8, 2023, 4:09 PM

#

I think #egg= didn't support relative paths at the time

small cove Dec 8, 2023, 4:10 PM

#

Relatedly, does that mean that PEP 621 doesn't support relative paths at all atm?

shy echo Dec 8, 2023, 4:23 PM

#

I think so?

small cove Dec 8, 2023, 4:32 PM

#

I'm currently working on editable installs (with the eventual goal of building some kind of workspace support) and i'm trying to figure out what the right semantics are

#

I've seen that pip expands environment variables in requirements.txt input https://github.com/pypa/pip/blob/aebc0c5fc321141ede837c80572427ab7b795c3f/src/pip/_internal/req/req_file.py#L503, which some people seem to use as a workaround (https://github.com/search?q=path%3Arequirements.txt+PWD&type=code)

shy echo Dec 8, 2023, 5:11 PM

#

sigh

small cove Dec 8, 2023, 5:14 PM

#

is that a "that's not how you're meant to do that" sigh?

outer ivy Dec 11, 2023, 6:59 PM

#

Hi! I think I'm running into https://github.com/pypa/pip/issues/11896 now that my company has rolled out some network configuration updates. IIUC the maintainers are potentially open to contributions related to this. Is that correct? I'm working with our networking team to better understand what's going on, so if I can upstream something I'd be happy to work on it.

The support typically comes from people like yourself, who have hit issues like this, worked out how to address them, and contribute advice back.
- https://github.com/pypa/pip/issues/11896#issuecomment-1813102579

#

I ask because the thread seems to go back and forth a little.

little kite Dec 12, 2023, 1:39 PM

#

$ python -m pip debug --verbose 2>1 | rg 11_0
$ python -c 'import packaging.tags
             for tag in packaging.tags.platform_tags():
                 print(tag)' | rg 11_0
macosx_11_0_x86_64
macosx_11_0_intel
macosx_11_0_fat64
macosx_11_0_fat32
macosx_11_0_universal2
macosx_11_0_universal

Why do packaging and pip disagree on my compatible tags?

dapper laurel Dec 12, 2023, 1:41 PM

#

different versions of packaging maybe?

little kite Dec 12, 2023, 1:46 PM

#

oh, indeed. pip bundles a 2yo version of packaging

#

why?

dapper laurel Dec 12, 2023, 1:49 PM

#

2 theories:

breaking changes between versions
no one updated it
I think the first one is more plausible

little kite Dec 12, 2023, 1:50 PM

#

it looks like greenlet is overwriting the platform tag as of v3.0.2 with the min supported macOS version being Big Sur (11.0): https://github.com/python-greenlet/greenlet/blob/edbdda27ab1983ac157b588dd0c04816cb31b0ea/.github/workflows/tests.yml#L67C36-L67C36

#

so anyone attempting to install greenlet on an Intel Mac will encounter this

#

IIUC upgrading packaging is blocked on the removal of legacy specifiers: https://github.com/pypa/pip/issues/11715

#

that is really quite unfortunate

jovial jasper Dec 12, 2023, 8:27 PM

#

Yeah it's really hard to upgrade packaging in pip without breaking the world. Last status in https://github.com/pypa/pip/pull/12300

limber ore Dec 19, 2023, 9:09 PM

#

I just realized this date is approaching https://hatch.pypa.io/1.9/config/metadata/#allowing-ambiguous-features

Does anyone know how long pip has supported this? I remember it was blocked on a vendor update to packaging but I don't remember how long ago that was.

shy echo Dec 19, 2023, 9:10 PM

#

The packaging upgrade has not happened yet, so if it was blocked on that, it'll still be blocked.

#

Heads up: I'll be away from the internet over the next bit, vacation time. 😅

limber ore Dec 19, 2023, 9:13 PM

#

that is certainly surprising, I could have sworn the update happened because it was over a year ago and at one point I stopped tracking it which I would have assumed meant it happened rather than I forgot about it...

#

Sep 2022 https://github.com/pypa/hatch/pull/495

#

found the tracking issue https://github.com/pypa/pip/issues/11715

#

(note the most recent comment saying that this would be required for nogil support)

shy echo Dec 19, 2023, 9:22 PM

#

I'm aware. :)

#

Ah, this isn't DMs.

limber ore Dec 19, 2023, 9:22 PM

#

is there anything a contributor such as myself could do to assist?

shy echo Dec 19, 2023, 9:23 PM

#

limber ore is there anything a contributor such as myself could do to assist?

There's an open PR that does the code-base adaptations but we need to add tests and cover cases where the code used to fall back to legacy versions / specifiers and now raise errors -- and handle them gracefully.

#

Taking that PR forward would be welcome, if a bit non-trivial to put it mildly. 😅

limber ore Dec 19, 2023, 9:27 PM

#

https://github.com/pypa/pip/pull/12300

shy echo Dec 19, 2023, 9:27 PM

#

https://github.com/pypa/pip/pull/12300#issuecomment-1751801463 indeed

stuck girder Dec 19, 2023, 9:58 PM

#

And is there something blocking release of packaging that we could help with?

shy echo Dec 19, 2023, 11:07 PM

#

It's not a packaging release, it's a packaging upgrade in pip's vendored copy.

past pagoda Dec 19, 2023, 11:11 PM

#

Hello, I took a look at the repo, I opened a super trivial single line PR
https://github.com/pypa/packaging/pull/752

#

I'd also be happy to help with whatever is necessary, but I've touched this project before, so I'd be slow

shy echo Dec 19, 2023, 11:12 PM

#

limber ore https://github.com/pypa/pip/pull/12300

This is the relevant PR.

stuck girder Dec 20, 2023, 6:08 AM

#

shy echo It's not a packaging release, it's a packaging upgrade in pip's vendored copy.

Right, for pip. But we also need a new packaging release so it can be vendored into wheel. This is currently blocking packages from being built for ~~nogil~~ free-threaded Python builds

hidden flame Dec 30, 2023, 2:33 AM

#

Hmm, I'm running into a handful of test failures on a fresh clone of pip. I'm using a virtual environment based on a 3.11.7 install I compiled from source (manually). Any pointers to what may be causing these issues?

shy echo Dec 30, 2023, 2:45 AM

#

I think we have those in CI too. Haven't had time to investigate why/what changed.

hidden flame Dec 30, 2023, 2:47 AM

#

I took a brief look at the pip repository and saw a green checkmark so I assumed that I'm dealing with a case of PEBKAC meowsip

#

seems like normalization is at least partially at fault

dapper laurel Dec 30, 2023, 2:49 AM

#

that last test fail looks like bad case of normalization

hidden flame Dec 30, 2023, 2:50 AM

#

spoke too late 🐌

dapper laurel Dec 30, 2023, 2:50 AM

#

you sent your message the moment I pressed "send" lol

hidden flame Dec 30, 2023, 2:50 AM

#

great minds think alike

#

although that may be an insult to your mind, haha

#

I'm not very clever

dapper laurel Dec 30, 2023, 2:53 AM

#

I could say the same 😄

shy echo Dec 30, 2023, 3:17 AM

#

It's an intermittent failure, or something that is being affected by a cache maybe?

#

Again, haven't looked closely beyond the notifications that our CI failed at some point and some PRs are red when they shouldn't have been.

hidden flame Dec 30, 2023, 3:18 AM

#

normalization being affected by a cache sounds odd though

shy echo Dec 30, 2023, 3:18 AM

#

hidden flame normalization being affected by a cache sounds odd though

I mean, a setuptools release and a pre-created local dev environment will manifest as a non-reproducer.

#

Until you recreate the environment.

hidden flame Dec 30, 2023, 3:19 AM

#

I presumed CI doesn't cache at all?

shy echo Dec 30, 2023, 3:20 AM

#

hidden flame I presumed CI doesn't cache at all?

Trust, but verify.

hidden flame Dec 30, 2023, 3:24 AM

#

Fair. I'll try to figure this out. Not exactly the way I envisioned how I'd start looking to contribute to pip, but oh well.

#

Thanks for the information, either way!

shy echo Dec 30, 2023, 3:24 AM

#

Sorry, but looking forward to your contributions.

#

And, as you know, feel welcome to ask questions here or on the issue tracker, with the issue tracker being preferred.

hidden flame Dec 30, 2023, 3:31 AM

#

Richard's attempt at debugging test failures on a fresh pip clone

hidden flame Dec 30, 2023, 4:06 AM

#

https://github.com/pypa/pip/pull/12454#issuecomment-1872445714 ... well it was setuptools, I am not surprised :)

stuck girder Dec 30, 2023, 9:43 AM

#

hidden flame Fair. I'll try to figure this out. Not exactly the way I envisioned how I'd star...

Unblocking everyone's PRs is an excellent way to start contributing to pip, thank you!

hidden flame Jan 1, 2024, 9:35 PM

#

Are egg-link filenames standardized anywhere?

#

pip is unable to detect certain egg-link files produced by setuptools 69.0.3 due to underscores which it now preserves

#

ah fun, so technically setuptools is violating the documentation

#

I'll investigate a bit more and report on GitHub.

shy echo Jan 1, 2024, 10:29 PM

#

hidden flame Are egg-link filenames standardized anywhere?

Not that I'm aware of, no.

dapper laurel Jan 1, 2024, 10:31 PM

#

are eggs even relevant anymore?

ashen geyser Jan 2, 2024, 4:00 PM

#

Yeah, pkg_resources? Eggs? Unless you crawl old PyPI releases, you probably don’t want to engage with these.

lunar gyro Jan 2, 2024, 5:00 PM

#

Eggs no, but egg-link and egg-info still lives in legacy editables

hidden flame Jan 4, 2024, 2:24 AM

#

FWIW, I'm waiting on someone to weigh in on https://github.com/pypa/setuptools/issues/4167 to be able to fix pip's CI

GitHub

[BUG] Setuptools 69.0.3 `setup.py develop` generates a .egg-link fi...

setuptools version 69.0.3 Python version CPython 3.11.7 OS Ubuntu 22.04.03 LTS Additional environment information No response Description Happy New Year! 🎉 I was debugging pip's CI which has be...

#

The easier solution would be if setuptools maintained the previous egg-link naming behaviour exibited by <=69.0.2, but I get the feeling the setuptools devs may not like that idea.

dapper laurel Jan 4, 2024, 2:31 AM

#

the fact that setuptools doesn't use major version number as a breaking changes identifier is an issue in my eyes... hard to change anything since there is always someone who depends on the legacy behavior

hidden flame Jan 4, 2024, 2:31 AM

#

well it's setuptools, good luck changing anything

#

semvar will not save you :)

#

but yeah, definitely a risky change to add to a bugfix release

dapper laurel Jan 4, 2024, 2:35 AM

#

hidden flame well it's setuptools, good luck changing anything

as much as I can appreciate the setuptools for years of service, I believe there has to be a do-over, a rewrite of the whole thing if it is to remain any kind of sane solution

#

but that's something for #setuptools I guess

hidden flame Jan 4, 2024, 2:36 AM

#

I personally do not care for setuptools, I'm just trying to fix pip's CI

hidden flame Jan 5, 2024, 5:00 AM

#

while I wait for progress on the setuptools issue, I'm tinkering with pip's improved error infrastructure. It's been lovely to work with, kudos @shy echo ✨

shy echo Jan 5, 2024, 5:08 AM

#

hidden flame while I wait for progress on the setuptools issue, I'm tinkering with pip's impr...

Nice!

hidden flame Jan 5, 2024, 5:08 AM

#

I ran into this error specifically a fair bit while messing around with legacy editable installs (for some reason I do not feel like investigating...)

shy echo Jan 5, 2024, 5:09 AM

#

I've been meaning to backfill the whole set of errors, and I'm glad to hear that others also find it nice. :)

#

I forget if I ended up filing my PR that removed the [present-rich] hack.

lunar gyro Jan 5, 2024, 5:10 AM

#

I’m not sure we want to make the suggestion. IIRC some package managers (RH?) deliberately remove RECORD so pip can’t accidentally uninstall a package it installs. Maybe we should make the message an info and add a documentation pacakge to explain the situation in more detail instead.

shy echo Jan 5, 2024, 5:10 AM

#

We make it already.

hidden flame Jan 5, 2024, 5:10 AM

#

it feels like a nice balance between "is it easy to work on?" and "for hour spent on project, how much does it improve QOL?" task stats

maiden island Jan 5, 2024, 5:11 AM

#

Ichard the pip king these days 🤩

hidden flame Jan 5, 2024, 5:11 AM

#

i have literally filed only one documentation pr lol

maiden island Jan 5, 2024, 5:12 AM

#

Seen lots of chat. For some reason discord notifies me when you do … no idea why

#

I think I only have 1 pip contribution too …

shy echo Jan 5, 2024, 5:12 AM

#

lunar gyro I’m not sure we want to make the suggestion. IIRC some package managers (RH?) de...

I dunno about a dedicated doc page for this -- I'm inclined to throw together the error index page/section and start bundling these into that.

hidden flame Jan 5, 2024, 5:12 AM

#

lunar gyro I’m not sure we want to make the suggestion. IIRC some package managers (RH?) de...

Yeah. I was hesitant to use the "missing" verbage since no RECORD is a valid situation.

#

FWIW, the hint is different if the recorded installer is not pip.

lunar gyro Jan 5, 2024, 5:13 AM

#

hidden flame FWIW, the hint is different if the recorded installer is not pip.

Ah, good touch

shy echo Jan 5, 2024, 5:13 AM

#

(pssst, let's discuss the details when the PR is filed)

hidden flame Jan 5, 2024, 5:14 AM

#

which will be tomorrow because I absolutely should not be up this late :P

#

it's too easy to get caught up in the terminal when you got something good going on

hidden flame Jan 6, 2024, 9:53 PM

#

shy echo I forget if I ended up filing my PR that removed the `[present-rich]` hack.

https://github.com/pypa/pip/pull/12304 you did, don't worry meownod

shy echo Jan 6, 2024, 10:24 PM

#

Good job past me, and thanks for finding that PR#!

gilded prism Jan 7, 2024, 4:57 AM

#

Hey, I just realized that pip download actually runs setup.py and whatever arbitrary code is in there.
https://stackoverflow.com/questions/52486985/pip-download-without-executing-setup-py
I never thought about that and considered pip download safe to run, I thought it only downloaded some wheels. Just like how a browser download (like the "download" feature on PyPI) shouldn't be able to run any code.
Maybe there should be some warning about this in the --help of pip download, or in other places that introduce pip download?

lunar gyro Jan 7, 2024, 5:04 AM

#

pip download needs to run setup.py to figure out a wheel-less package’s dependencies so it can download them too. There are --binary-only and/or --no-deps if that’s what you want

shy echo Jan 7, 2024, 5:05 AM

#

(--no-deps runs the metadata generation too)

gilded prism Jan 7, 2024, 5:05 AM

#

Yes, I understand why it needs to do that. It's just not obvious to a (casual?) user of Python that it executes arbitrary code form the internet, like install

shy echo Jan 7, 2024, 5:06 AM

#

I'd be on board for adding a warning, so please go ahead and file an issue for this (or a PR directly, if you're so inclined).

gilded prism Jan 7, 2024, 5:06 AM

#

I'll make an issue 👍

gilded prism Jan 7, 2024, 5:30 AM

#

shy echo I'd be on board for adding a warning, so please go ahead and file an issue for t...

https://github.com/pypa/pip/issues/12465
Please tell if there's a better way to phrase the issue

limber ore Jan 7, 2024, 5:33 PM

#

how can one define a path with spaces when using the PIP_FIND_LINKS environment variable?

#

oh no 😢

#

mwahaha, I win

#

I'll comment on the issue as a potential workaround

shy echo Jan 7, 2024, 6:08 PM

#

Oh God

dapper laurel Jan 7, 2024, 6:09 PM

#

why would you put spaces in FS names...

limber ore Jan 7, 2024, 6:33 PM

#

in the case of C:\Program Files I have no choice 😉

dapper laurel Jan 7, 2024, 6:40 PM

#

windows is always a special child

limber ore Jan 7, 2024, 6:42 PM

#

true, but in fairness macOS also uses spaces

#

~~Library Framework or something like that~~

/Library/Application Support/

dapper laurel Jan 7, 2024, 6:59 PM

#

that's why (in this developer's eyes) linux is the one and only sane system

trail sun Jan 7, 2024, 7:39 PM

#

You can have spaces in a Linux file path too.

dapper laurel Jan 7, 2024, 7:40 PM

#

you can, but (AFAIK) there are none of those in default paths

trail sun Jan 7, 2024, 7:44 PM

#

Point being that the issue here isn't the os, but tools designed by devs that made assumptions about what was reasonable that didn't match the world, then people get into self re-enforcing loops of "just don't use spaces in a path" because of devs continuing to handle that wrong

shy echo Jan 7, 2024, 8:22 PM

#

dapper laurel that's why (in this developer's eyes) linux is the one and only sane system

This developer disagrees, pointing to his experience dealing with oh-all-the quirks that are distro specific.

limber ore Jan 7, 2024, 8:23 PM

#

trail sun Point being that the issue here isn't the os, but tools designed by devs that ma...

that's pretty much my take on this as well. in this particular case, each OS already has a dedicated character for separating paths so it's unclear to me why spaces were used originally

trail sun Jan 7, 2024, 8:26 PM

#

limber ore that's pretty much my take on this as well. in this particular case, each OS alr...

There's also an argument to allow specifying the separator char as a separate env var to allow portable configuration since that char varies by os, but that's something you reach only after avoid space as a seperator

#

I may add my thoughts/possible pr later, need to think about it more, but I think an env var for custom separator and keeping the current broken behavior as default for compatibility may be a viable way forward on that

limber ore Jan 7, 2024, 8:29 PM

#

the context formatting of Hatch actually allows for the insertion of path separators in a portable way https://hatch.pypa.io/latest/config/context/#system-separators

#

(I would have chosen colon but that would conflict with string formatting syntax so everybody must use the Windows semicolon 🙂 )

lunar gyro Jan 8, 2024, 4:12 AM

#

in the case of C:\Program Files I have no choice
You do, C:\PROGRA~1 (DOS compatibility FTW)

#

All Windows paths (on the local filesystem, not mounted) can be represented in a space-less format because a lot of MS-DOS softwares do not allow spaces in paths

#

But percent-escaping is the standard way to put URL-invalid characters in a URL. Not even a hack, that‘s just the way to do it.

ashen geyser Jan 8, 2024, 8:25 PM

#

My development directory in Linux is a path with a space in it. In the last 10 years or so, there was exactly one problem: a C library that refused to compile when I tried to set up a Ruff dev environment.

I reported it, it got fixed, and when I next tried to contribute to Ruff, there were 0 things in my life that were coded so badly that they broke on spaces.

azure heron Jan 8, 2024, 11:38 PM

#

Is there a way to disable line wrapping in the output?

#

(of pip)

shy echo Jan 8, 2024, 11:39 PM

#

Wait, there's line wrapping in the output?

azure heron Jan 8, 2024, 11:39 PM

#

haha

shy echo Jan 8, 2024, 11:39 PM

#

Do you have a reproducer?

azure heron Jan 8, 2024, 11:39 PM

#

I was just about to say... maybe there's not actually line wrapping

#

maybe it's stylistic formatting

#

I was looking at some output like

The conflict is caused by:
    The user requested b<3.0.0 and >=2.0.0
    a 3.0.0 depends on b==3.0.0
    The user requested b<3.0.0 and >=2.0.0
    a 1.0.0 depends on b==1.0.0

and thought it was wrapped but of course as soon as I ask it occurs it's probably not

shy echo Jan 8, 2024, 11:40 PM

#

Nah, that's each of the items separately.

azure heron Jan 8, 2024, 11:42 PM

#

Cool haha thanks for your help 🙂

shy echo Jan 8, 2024, 11:42 PM

#

😬

azure heron Jan 8, 2024, 11:53 PM

#

Was there a lot of consideration of the formatting of conflicts? (just curious)

limber ore Jan 9, 2024, 2:04 AM

#

if you have to ask then... unlikely 😉

#

it's a difficult problem. as much as I love Rust I've encountered some rather cryptic messages from Cargo about conflicts

shy echo Jan 9, 2024, 2:22 AM

#

azure heron Was there a lot of consideration of the formatting of conflicts? (just curious)

Yes, it was a topic we discussed at some length when working on the dependency resolver in 2020. As Ofek says though, it's quite a difficult problem to solve.

limber ore Jan 9, 2024, 2:42 AM

#

I encountered a massive amount of pain last year (especially regarding error messages) when I had to patch dependencies in order to build on CentOS 6 https://github.com/DataDog/datadog-agent/pull/18303/files#diff-85a53f2362d9a8aeaf81712aeec92218a065f4169f76001f3188ce4ef5b4a83dR545

#

we had to fork a few projects and use Zig to target a specific glibc version, very hacky but it works

azure heron Jan 9, 2024, 2:48 AM

#

I'm working on a package resolver testing tool and going to do a lot of comparing error messages across tools 🙂

stuck girder Jan 9, 2024, 5:55 AM

#

I've not tried this, but here's a new "Python virtualenv manager written in Rust" https://github.com/LilyFoote/lilyenv

Lilyenv is a tool for managing python interpreters and virtualenvs.

lunar gyro Jan 9, 2024, 8:50 AM

#

Why is this not a hard requirement https://github.com/LilyFoote/lilyenv/issues/6

dapper laurel Jan 9, 2024, 8:53 AM

#

stuck girder I've not tried this, but here's a new "Python virtualenv manager written in Rust...

Sadly, it's just subprocessing python -m venv on a specific python version

limber ore Jan 26, 2024, 3:04 AM

#

is there a way to use cloud storage buckets as package indices directly without web serving functionality? specifically, I'm wondering if there's a way to direct pip to automatically append index.html when setting the index URL to .../ since that doesn't happen without a server

limber ore Jan 26, 2024, 4:01 AM

#

ah got it, I can actually rely on blob semantics and treat / as a distinct blob and write the index.html to that instead

limber ore Jan 30, 2024, 1:25 PM

#

question... I know this is hacky but I really need to work around this https://github.com/pypa/pip/issues/11664

what can I monkey patch in the interpreter to make pip think it is on a different platform/architecture?

lunar gyro Jan 30, 2024, 1:36 PM

#

pip just uses this logic https://github.com/pypa/packaging/blob/main/src/packaging/markers.py#L234
So you just need to patch default_environment

limber ore Jan 30, 2024, 1:40 PM

#

that's awesome, thank you!

limber ore Jan 30, 2024, 7:14 PM

#

pip just uses this logic https://github.

dreamy citrus Feb 4, 2024, 4:07 PM

#

good evening. cheers=)

foggy forum Feb 6, 2024, 2:59 PM

#

when does pip do setup.py develop vs pyproject.toml based editables?

#

I'm finding on 3.11 for https://github.com/plotly/dash/ pip -e . does a legacy setup.py develop

#

but for 3.12 it does a pyproject.toml build even though there's no pyproject.toml installed

#

I also have setuptools installed

#

ah I didn't have wheel installed

#

here's the conclusion https://github.com/plotly/dash/pull/2735/commits/de8b94bf92819cc4156302700b2ad24ccac53381

azure heron Feb 15, 2024, 10:51 PM

#

Are there differences between --upgrade and --force-reinstall?

hoary mist Feb 15, 2024, 10:53 PM

#

pip install foo && pip install --upgrade foo will install foo once

#

pip install foo && pip install --force-reinstall foo, will instal foo twice

dapper laurel Feb 15, 2024, 10:53 PM

#

(just a guess) upgrade checks versions and bumps only when newer is available and force-reinstall will always reinstall (could also be useful when for some reason previous installation of the same version is broken)

azure heron Feb 15, 2024, 11:19 PM

#

Makes sense thank you!

shy echo Feb 16, 2024, 12:17 AM

#

(also, force reinstall implies an upgrade)

past pagoda Feb 16, 2024, 12:27 AM

#

shy echo (also, force reinstall implies an upgrade)

It does? It doesn't just reinstall the current version?

#

I would have assumed that using the flag by itself would reuse the same wheel

fallen scroll Feb 16, 2024, 12:36 AM

#

I don't think it's possible to locate the same wheel just from the information in the installed project.

#

Not to mention that the project may not have been installed from a wheel in the first place.

dapper laurel Feb 16, 2024, 12:38 AM

#

Yeah, that would require a lock file

jovial jasper Feb 16, 2024, 8:33 AM

#

azure heron Are there differences between `--upgrade` and `--force-reinstall`?

Note there are many quirks in pip upgrade strategies in presence of direct URLs or editables. See for instance https://github.com/pypa/pip/pull/10564#issuecomment-1484093165

azure heron Feb 16, 2024, 2:17 PM

#

Thank you! I'll study that

topaz pewter Feb 17, 2024, 10:48 AM

#

Hi everyone ...
I am wondering, how pip gets the .whl file, from the html file serverd by pypi server in url https://pypi.org/simple/pandas/

shy echo Feb 17, 2024, 11:01 AM

#

It does a regular http download.

topaz pewter Feb 17, 2024, 11:04 AM

#

yeah .. but there are list of all versions in that html file..
it doesn't even have version attribute in that <a/> tag, how it does parsing this ?

stuck girder Feb 17, 2024, 11:24 AM

#

there's an API: https://pypi.org/pypi/pandas/json

shy echo Feb 17, 2024, 11:24 AM

#

stuck girder there's an API: https://pypi.org/pypi/pandas/json

pip doesn't use that, and we really don't want anyone to rely on that at scale.

topaz pewter Feb 17, 2024, 11:25 AM

#

oh yeah .. I think it is using https://pypi.org/simple

shy echo Feb 17, 2024, 11:25 AM

#

topaz pewter yeah .. but there are list of all versions in that html file.. it doesn't even h...

It's parsing the wheel filename, and the details of what that HTML page looks like is contained in https://packaging.python.org/en/latest/specifications/simple-repository-api/

topaz pewter Feb 17, 2024, 11:26 AM

#

if you observe, the line 561, they are refering to the rel attribute, I am not sure what it is .. !!
https://sourcegraph.com/github.com/pypa/pip@51de88ca6459fdd5213f86a54b021a80884572f9/-/blob/src/pip/_vendor/distlib/locators.py?L561

locators.py?L561

#

yeah and one more, like if we observe the filenames, they are not only name-version .. they are also attached with - win - py3 - py2 stuff like that ..

shy echo Feb 17, 2024, 11:27 AM

#

pip's not using distlib either for this stuff.

topaz pewter Feb 17, 2024, 11:28 AM

#

ohh ..

shy echo Feb 17, 2024, 11:28 AM

#

If you're comfortable looking at the implementation, it's here: https://github.com/pypa/pip/tree/main/src/pip/_internal/index

#

Although, https://github.com/frostming/unearth might be a better choice for thing to look at for understanding things, if you wanna look at an implementation/docs.

topaz pewter Feb 17, 2024, 11:31 AM

#

shy echo If you're comfortable looking at the implementation, it's here: https://github.c...

ohh now I get it ..

#

I was looking for this, in whole repo .. haha xD
thankyou .. I will first go through it, and see if my little brain can handle that much

topaz pewter Feb 17, 2024, 11:32 AM

#

shy echo Although, https://github.com/frostming/unearth might be a better choice for thin...

what is this ?

shy echo Feb 17, 2024, 11:32 AM

#

Neato!

shy echo Feb 17, 2024, 11:32 AM

#

topaz pewter what is this ?

It's a project that pulled out pip's implementation and made it reusable.

topaz pewter Feb 17, 2024, 11:35 AM

#

shy echo It's a project that pulled out pip's implementation and made it reusable.

ohh I see, interesting though ...

wheat iron Feb 17, 2024, 12:44 PM

#

shy echo pip doesn't use that, and we really don't want anyone to rely on that at scale.

why?

#

is it subject to change and the simple API isnt?

shy echo Feb 17, 2024, 12:44 PM

#

Yes. It's an API that only pypi.org implements, not other index servers.

wheat iron Feb 17, 2024, 12:46 PM

#

hmm interesting

wheat iron Feb 17, 2024, 12:46 PM

#

shy echo Yes. It's an API that only pypi.org implements, not other index servers.

so projects currently using it should be fine as long as they dont use another index server?

shy echo Feb 17, 2024, 12:52 PM

#

Which is basically preventing use with devpi if you want to have local caching or with mirrors like bandersnatch-generated indexes or piwheels or... like corporate indexes for stuff that can't be on pypi.org

#

But, yes.

wheat iron Feb 17, 2024, 12:53 PM

#

shy echo But, yes.

cool

#

~~yay i dont have to rewrite my shit that 4 people use~~

small cove Feb 20, 2024, 6:09 PM

#

What's the reason that https://files.pythonhosted.org/packages/8a/6a/19e9fe04fca059ccf770861c7d5721ab4c2aebc539889e97c7977528a53b/pip-24.0-py3-none-any.whl contains a pip3.10 entrypoint?

#

When we're trying to install pip ourselves, this creates an entrypoint mismatching the user's python version. Is there a requirement to special case pip when installing?

foggy forum Feb 20, 2024, 6:36 PM

#

Yes

foggy forum Feb 20, 2024, 6:46 PM

#

small cove When we're trying to install pip ourselves, this creates an entrypoint mismatchi...

pip special cases itself

#

Now there's multiple installers it might be worth getting that wart in the whl spec

#

https://github.com/pypa/pip/blob/3898741e29b7279e7bffe044ecfbe20f6a438b1e/src/pip/_internal/operations/install/wheel.py#L283

shy echo Feb 20, 2024, 6:53 PM

#

I've wanted to clean this up for a while, but it's just not been something I've had time to come around to.

small cove Feb 20, 2024, 6:55 PM

#

foggy forum https://github.com/pypa/pip/blob/3898741e29b7279e7bffe044ecfbe20f6a438b1e/src/pi...

Thanks!

small cove Feb 20, 2024, 6:55 PM

#

foggy forum Now there's multiple installers it might be worth getting that wart in the whl s...

Just for pip or a general "install this tool as tool{python_major}.{python_minor}"?

foggy forum Feb 20, 2024, 6:55 PM

#

pip and easy_install

#

Probably at least worth specifying the current behaviour so uver installers can have a go

small cove Feb 20, 2024, 7:16 PM

#

yeah it'd be great if this was added to packaging.python.org

foggy forum Feb 20, 2024, 8:33 PM

#

small cove yeah it'd be great if this was added to packaging.python.org

Do you have a GitHub issue for UV for this?

small cove Feb 21, 2024, 9:37 AM

#

https://github.com/astral-sh/uv/issues/1593

shy echo Feb 21, 2024, 9:49 AM

#

Could you drop an issue on packaging.python.org as well? Please also mention that I asked for it there (so that I get autosubscribed 😅).

small cove Feb 21, 2024, 10:31 AM

#

I also have another packaging.python.org docs thing, making the direct_url.json url field a valid url. I've put it up as a PR there, not sure if there's a better place to discuss: https://github.com/pypa/packaging.python.org/pull/1506

limber ore Mar 3, 2024, 2:42 PM

#

I don't think this is true https://github.com/astral-sh/uv/pull/2083

Previously, uv would always prioritize the index given by
--index-url. It would then try any indexes after that given by zero
or more --extra-index-url flags. This differed from pip in that any
priority was given at all, where pip doesn't guarantee any priority
ordering of indexes.

I'm almost positive that the --index-url takes priority. Is that not true actually?

#

in testing it appears that way at least

shy echo Mar 3, 2024, 3:16 PM

#

There's no guaranteed ordering there, from pip's PoV.

dapper laurel Mar 3, 2024, 3:28 PM

#

yeah, that was one of the pain points Poetry tried to solve with it's sources (that being mimicing the way pip works, while maintaining some kind of ordering for solver)

shy echo Mar 3, 2024, 3:31 PM

#

I'm pretty sure that there's some arbitrary order but I don't fully remember what and, well, we might have to change it at some point theoretically.

dapper laurel Mar 3, 2024, 3:34 PM

#

a standard (PEP-ified) way to deal with indices other than "point to PEP 503 index" would be great

#

especially mapping dependencies to indices

#

(which is one of the issues that stop Poetry from adapting PEP 621)

trail sun Mar 3, 2024, 3:35 PM

#

thats uhh... kindof happening right now with the lockfile pep

dapper laurel Mar 3, 2024, 3:36 PM

#

trail sun thats uhh... kindof happening right now with the lockfile pep

yeah, but lockfile is an output of resolver. what I mean is standard spec of resolver input

trail sun Mar 3, 2024, 3:36 PM

#

There doesn't need to be a standard spec for that though

dapper laurel Mar 3, 2024, 3:38 PM

#

well, I disagree. currently you can't specify that for example, for system X and python version Y, you want dependency D to be taken from index A and in every other case from index B

#

that could be the outcome of the resolver in lockfile

#

but there is no standard way for users to map that in the input

trail sun Mar 3, 2024, 3:46 PM

#

I don't think there needs to be a standard for that if the lockfile pep process creates a standard format for consuming dependency information, but I'd want to see people explore the use cases they want out of it more before standardizing on that for resolvers.

ripe shoal Mar 3, 2024, 4:23 PM

#

I do think a deterministic way to map requirements to indices as input to a resolver is very important. One of the pain points I see in practice with people using 3rd party package indices is either pulling the package from the wrong index, or general confusion about how to be sure packages will resolve to come from a certain index. Today there isn't really a way to guarantee that a package will come from a 3rd party index (AIUI), without using --index-url which isn't usable if you only want to pull one or two packages from that third party package index.

dapper laurel Mar 3, 2024, 4:25 PM

#

exactly

#

and Poetry solves that

#

but that syntax is not portable to pep 621 requirements

ripe shoal Mar 3, 2024, 4:27 PM

#

I agree being able to declare this in package metdata would be good, but I also think people need to be able to declare this at the pip command line

trail sun Mar 4, 2024, 3:00 PM

#

I agree that it should be specifiable at the command line, but I disagree that we need a standard way to do that, that enforces things about tool design rather than leaving it as "something resolvers should be capable of generating and installers should be capable of consuming", both of which are covered in lockfiles.

iron marlin Mar 6, 2024, 6:28 PM

#

Hi, are ci tests not currently working on PR's? In failed tests I'm getting errors indicating that nox can't find the setup.py, and it was removed in a recent commit on main.
Context: https://github.com/pypa/pip/pull/12388

shy echo Mar 6, 2024, 6:29 PM

#

I believe so.

#

I haven't been keeping up with what changes have been made recently. 😅

jovial jasper Mar 6, 2024, 6:48 PM

#

The fix for CI is ready to merge.

#

And now merged.

iron marlin Mar 6, 2024, 6:53 PM

#

Nicee

vagrant birch Mar 9, 2024, 7:40 PM

#

Has anyone ever needed to call a system tool during the build which itself was written in Python?

#

from what I see pip sets PYTHONPATH in a way that breaks any calls to them and doesn't check which Python is used

#

but even if it would check that the build interpreter is used, PYTHONPATH could contain incompatible libraries and still break things.. meh

#

so, don't write build tools in Python 😛

fallen scroll Mar 9, 2024, 7:44 PM

#

If you need a Python-based tool, can't you declare it as a build requirement?

vagrant birch Mar 9, 2024, 7:45 PM

#

those are not on pypi, they are scripts that are part of system packages

#

on a second though sitecustomize could remove itself from path if it detects it... I guess I'll open an issue with meson-python.

sorry for the ramblings..

#

https://github.com/mesonbuild/meson-python/issues/596

hidden flame Mar 11, 2024, 3:33 AM

#

I've been hacking some more on improving the presentation of pip's error messages. This time I'm working on the requirements related errors. ✨

shy echo Mar 11, 2024, 10:15 AM

#

Oh, they're gonna be even nicer once we can get newer packaging into pip; since it'll make the invalid-requirement case even clearer.

dapper laurel Mar 11, 2024, 10:21 AM

#

Already asked on #pipx but this is probably a better place. Would shipping pipx with pip (or adding that feature to pip natively) be possible? I am willing to do the leg work if maintainers say yes to that

shy echo Mar 11, 2024, 10:22 AM

#

How are you imagining maintenance working for that setup & how are you thinking of installations not going awry because two packages want to provide pipx?

#

I don't know how/if we can make it work, but I also don't really have a strong opinion on this. 😅

dapper laurel Mar 11, 2024, 10:25 AM

#

Well, I am more inclined towards adding native support for that kind of functionality, probably the entrypoint would have to be different or something. It would just make life easier for tools creators

shy echo Mar 11, 2024, 10:25 AM

#

Ah.

#

There's some discussion of a pip run, and I'd be on board for it if we can figure out what shape it's supposed to take.

dapper laurel Mar 11, 2024, 10:26 AM

#

For now I just wanted to know if maintainers would be ok with such feature, the details could be discussed later

shy echo Mar 11, 2024, 10:26 AM

#

I am.

dapper laurel Mar 11, 2024, 10:29 AM

#

shy echo There's some discussion of a pip run, and I'd be on board for it if we can figur...

From what I know, it's supposed to be a solution for single-file scripts to avoid having to make env for their dependencies, rather than isolated env management

#

But I will revisit that

fallen scroll Mar 11, 2024, 12:23 PM

#

IMO, pipx would be a good candidate for inclusion into ensurepip.

shy echo Mar 11, 2024, 12:36 PM

#

It's extremely unlikely to be a part of ensurepip I think.

dapper laurel Mar 11, 2024, 12:59 PM

#

Yeah, pipx has too many dependencies to be just made part of ensurepip

#

Taking a look into the code, I would say that most likely scenario would be to add the functionality of pipx into pip, using patterns and code already available in pip

#

pip x command 😛

little kite Mar 11, 2024, 2:14 PM

#

as far as I can tell, pipx has only got the 2 additional dependencies

dapper laurel Mar 11, 2024, 3:19 PM

#

It has like 5-6 dependencies

past pagoda Mar 11, 2024, 3:22 PM

#

dapper laurel It has like 5-6 dependencies

"additional" [to what pip already has]

little kite Mar 11, 2024, 4:10 PM

#

yeah, userpath depends on click it looks like, so 3 extra in total

dapper laurel Mar 11, 2024, 4:14 PM

#

yeah, that's more than enough overhead

#

having this as part of native pip would be much cleaner solution

little kite Mar 11, 2024, 4:14 PM

#

dunno, not a pip dev, but I don't think a few extra dependencies is gonna be the dealbreaker here

hazy glen Mar 11, 2024, 4:15 PM

#

does pip have a feature for only installing a project's dependencies, without the project itself? like poetry install --no-root?

little kite Mar 11, 2024, 4:15 PM

#

but how do you envision this working as part of ensurepip anyway?

little kite Mar 11, 2024, 4:16 PM

#

hazy glen does pip have a feature for only installing a project's dependencies, without th...

no

hazy glen Mar 11, 2024, 4:16 PM

#

would such a new feature be welcomed as a PR?

little kite Mar 11, 2024, 4:17 PM

#

I think pip's more likely to lean on the upcoming dependency group PEP

dapper laurel Mar 11, 2024, 4:17 PM

#

little kite but how do you envision this working as part of ensurepip anyway?

I don't. That would require patching pipx and/or shipping extra wheels

little kite Mar 11, 2024, 4:17 PM

#

see https://discuss.python.org/t/pep-735-dependency-groups-in-pyproject-toml/39233

Discussions on Python.org

PEP 735: Dependency Groups in pyproject.toml

Hi all! This is my attempt to follow up on Projects that aren't meant to generate a wheel and pyproject.toml . Some notes and questions to seed discussion: I have mostly stayed close to @brettcannon’s proposed simplified solution from that thread, but have made a couple of changes. That’s part of why I listed him as sponsor, rather than...

little kite Mar 11, 2024, 4:27 PM

#

dapper laurel I don't. That would require patching pipx and/or shipping extra wheels

I mean even if it were to be bundled, unless you're installing Python from python.org, it'll have been configured without ensurepip. In fact, I don't think having a different pipx script per Python install makes much sense at all, except for run

hidden flame Mar 11, 2024, 4:40 PM

#

shy echo Oh, they're gonna be even nicer once we can get newer packaging into pip; since ...

Ooo, does packaging 23.* come with better error messages? 👀

#

I briefly tried including the current packaging InvalidRequirement/Marker errors in the pip error and it was less than informative :P

#

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ichard26/dev/oss/pip/venv/lib/python3.11/site-packages/packaging/requirements.py", line 37, in __init__
    raise InvalidRequirement(str(e)) from e
packaging.requirements.InvalidRequirement: Expected end or semicolon (after name and no valid version specifier)
    blac-
        ^

ah very nice

#

I suppose we can surface the error location in the pip error when newer packaging lands... eventually

azure heron Mar 11, 2024, 9:21 PM

#

Not necessarily opposed to the inclusion of pipx in pip but note that we'll be working on pipx for uv in the near future.

dapper laurel Mar 11, 2024, 9:30 PM

#

@azure heron once you get uv installed by default with python installations, it will be fine

azure heron Mar 11, 2024, 9:32 PM

#

But we'll be the ones installing Python 😈

dapper laurel Mar 11, 2024, 9:37 PM

#

pyenv my beloved 😄

hidden flame Mar 12, 2024, 2:48 AM

#

https://github.com/pypa/pip/issues/9657
https://github.com/pypa/pip/issues/10813
https://github.com/pypa/pip/issues/11889
these issues can be closed as they've been resolved

shy echo Apr 7, 2024, 10:59 PM

#

@hidden flame I'm slowly catching up on my notifications for pip -- https://github.com/pypa/pip/pull/12566 is quite nice!

hidden flame Apr 7, 2024, 11:17 PM

#

@shy echo I'm out on a walk, but the short of it is that 1. I don't remember, and 2. I think I was going to check for error messaging/styling consistency between the PRs

#

Evidently I never got around to that (because I was busy on a third error migration PR) but I'll take a look sometime this week if I can :)

shy echo Apr 7, 2024, 11:25 PM

#

No hurries on this, enjoy your walk!

#

Psst, this was the network import optimisation thingie. :P

hidden flame Apr 7, 2024, 11:27 PM

#

I'm a serial perfectionist. Everything's got to be perfect... but right, I think I can address the import PR tonight then, haha

limber ore Apr 12, 2024, 2:57 PM

#

any thoughts on this? https://github.com/astral-sh/uv/issues/3014

shy echo Apr 12, 2024, 2:59 PM

#

Other than pip's behaviour is correct, no.

limber ore Apr 12, 2024, 3:01 PM

#

okay just thought I would bring it up here, I haven't read that spec in a while

#

so I guess the fact that UV always produces a commit hash is not what the spec says?

shy echo Apr 12, 2024, 3:03 PM

#

If pip's behaviour does not align with the spec, the spec needs updating IMO since this is what my understanding of the format is: the whole point of this entry is to reflect the user's request.

#

Of course, opinion subject to change in light of additional information. :)

dapper laurel Apr 12, 2024, 3:04 PM

#

taking a quick look at the spec, it seems that they are within what spec allows

shy echo Apr 12, 2024, 3:05 PM

#

Yup.

limber ore Apr 12, 2024, 3:05 PM

#

I don't know that they are compliant looking at it now https://packaging.python.org/en/latest/specifications/direct-url-data-structure/#vcs-urls

shy echo Apr 12, 2024, 3:05 PM

#

I think the spec language needs improvement.

limber ore Apr 12, 2024, 3:05 PM

#

"requested" really does imply that the user information should be reflected

shy echo Apr 12, 2024, 3:08 PM

#

Made a comment on the issue, to encode the main point of these last few messages there.

limber ore Apr 12, 2024, 3:12 PM

#

they might have used the PEP rather than the living document spec because there is actually something that was not brought over https://peps.python.org/pep-0610/#specification

If the installer could discover additional information about the requested revision, it MAY add a resolved_revision and/or resolved_revision_type field. If no revision was provided in the requested URL, resolved_revision MAY contain the default branch that was installed, and resolved_revision_type will be branch. If the installer determines that requested_revision was a tag, it MAY add resolved_revision_type with value tag.

#

oh even that is not correct that says resolved, my bad

#

https://github.com/pypa/packaging.python.org/pull/1531

jovial jasper Apr 12, 2024, 3:44 PM

#

resolved_revision was left out from packaging.python.org because not implemented by any tool

small cove Apr 12, 2024, 5:11 PM

#

We need the resolved revision for caching

shy echo Apr 12, 2024, 5:13 PM

#

that's exactly what it's for.

#

The issue is about requested revision.

small cove Apr 12, 2024, 5:17 PM

#

Yes, but if we have to make the change from requested_revision to resolved_revision, it would be great to add resolved_revision to the living specification

shy echo Apr 12, 2024, 5:17 PM

#

I agree.

jovial jasper Apr 12, 2024, 6:07 PM

#

small cove We need the resolved revision for caching

The resolved commit that can be used for caching is commit_id

small cove Apr 12, 2024, 6:08 PM

#

good point

jovial jasper Apr 12, 2024, 6:11 PM

#

resolved_revision was meant to be, say, main (the default branch name) when installing from a vcs URL without explicit ref

shy echo Apr 12, 2024, 6:29 PM

#

shy echo that's exactly what it's for.

Ah, for posterity: I clearly misunderstood resolved revision to mean resolved commit hash, which was
wrong.

jovial jasper Apr 12, 2024, 7:08 PM

#

Understandable misunderstanding 🙂 We had quite a lot of back and forth with my co-author on the naming of these things... it's now what it is for good or worse.

limber ore Apr 13, 2024, 4:07 PM

#

I've never seen this error before https://github.com/pypa/pip/issues/12621

small cove Apr 15, 2024, 8:02 AM

#

The ux research results are finally up 🎉 https://pip.pypa.io/en/latest/ux-research-design/research-results/

hidden flame Apr 15, 2024, 9:08 PM

#

I sure hope y'all don't mind me slamming your inbox via the issue tracker :P

shy echo Apr 15, 2024, 9:08 PM

#

Nah, it's appreciated. 🙃

hidden flame Apr 15, 2024, 10:14 PM

#

https://tenor.com/view/mail-always-sunny-always-sunny-in-philadelphia-charlie-day-gif-15762449638585073242

Tenor

#

this this is me for the past few days. You thought I'd stop at just one, two, or even maybe three issues, but nope!

#

Anyway, I'm actually done for today, going to go outside or something.

hidden flame Apr 16, 2024, 12:36 AM

#

https://github.com/pypa/pip/issues/12460 does anyone understand what this person is asking for? I don't think it's very actionable as-is...

trail sun Apr 16, 2024, 12:41 AM

#

might be related to the thing that torch wanted: some sort of virtual selector packages to help users, but doesn't sound like something pip can just guess at even if it is that.

past pagoda Apr 16, 2024, 12:43 AM

#

hidden flame <https://github.com/pypa/pip/issues/12460> does anyone understand what this pers...

Additional context: [...] you will see.
"you will see"
Yeah no. It's the responsibility of the requestor to provide an MRE. Close the ticket and tell the requester that they can reopen it if they edit their comment to contain an MRE.
But even without that part, it's not pip's job to pick which version goes in your requirements.txt, it just dumps out the version that you've installed. This issue is out of scope regardless.

hidden flame Apr 16, 2024, 12:44 AM

#

I cannot close the ticket :)

#

not a pip triager

past pagoda Apr 16, 2024, 12:44 AM

#

oh

fast stump Apr 16, 2024, 2:50 AM

#

past pagoda > **Additional context:** [...] you will see. "you will see" Yeah no. It's the r...

I'll warn the user then 🙂

#

Thanks @hidden flame for mentioning!

hidden flame Apr 16, 2024, 3:06 AM

#

fast stump Thanks <@429445826251587616> for mentioning!

FWIW, it was a feature request, just a poorly written one that wasn't really actionable as-is.

#

I agree the issue should've been closed, but the reasoning is a bit different from the rest of the other issues I've been marking for closure.

limber ore Apr 17, 2024, 7:15 PM

#

does pip follow the recommendation and enable the executable bit for scripts on Linux? https://packaging.python.org/en/latest/specifications/binary-distribution-format/#recommended-installer-features

foggy forum Apr 17, 2024, 7:20 PM

#

limber ore does pip follow the recommendation and enable the executable bit for scripts on ...

https://github.com/pypa/pip/blob/800c1245bd7d5b3ad3fd52fd84c17a230bfa28e4/src/pip/_internal/operations/install/wheel.py#L657 ?

#

Oh maybe it's here https://github.com/pypa/pip/blob/800c1245bd7d5b3ad3fd52fd84c17a230bfa28e4/src/pip/_internal/operations/install/wheel.py#L634

limber ore Apr 17, 2024, 7:23 PM

#

foggy forum https://github.com/pypa/pip/blob/800c1245bd7d5b3ad3fd52fd84c17a230bfa28e4/src/pi...

nice thank you! I encountered a script at work right now with an error but it must be something else we do that's weird

foggy forum Apr 17, 2024, 7:24 PM

#

No, I was mistaken, my first URL is for the installer = pip file

#

pip sets maker.set_mode = True and the magic happens here https://github.com/pypa/pip/blob/800c1245bd7d5b3ad3fd52fd84c17a230bfa28e4/src/pip/_vendor/distlib/scripts.py#L308

#

I can't see what happens if the wheel has scripts but not entry points

hidden flame Apr 18, 2024, 11:48 PM

#

Would help in reducing the PR backlog be welcomed? I was thinking about reviewing the laundry list of open PRs and flagging those that can probably be merged (in a new issue). I also wouldn't mind taking on (some) PRs that have been approved on principal but need a bit more work to be landed (addressing merge conflicts/review comments).

I realize that getting the packaging upgrade landed and release 24.1 cut is the current priority, so I would likely pick this up sometime in May.

arctic pendant Apr 19, 2024, 1:51 AM

#

How do I install extras when doing an editable install via pip? I ask because when running this command...

venv-windows/Scripts/pip install --editable ".[all]"

...I'm told WARNING: libretro-py 0.0.0 does not provide the extra 'all' despite the following section in my pyproject.toml:

# This package's name is libretro.py
[optional-dependencies]
dev = [
    'bandit == 1.7.*',
    'black == 23.*',
    'flake8 == 6.*',
    'isort == 5.*',
    'mypy == 1.5.*',
]
opengl = [
    'moderngl == 5.10.*',
]
pillow = [
    'pillow == 10.2.*',
    'types-Pillow',
]
all = ["libretro.py[opengl,pillow]"]

Any tips?

ashen geyser Apr 19, 2024, 7:31 AM

#

It’s [project.optional-dependencies], not [optional-dependencies]

arctic pendant Apr 19, 2024, 3:10 PM

#

ashen geyser It’s `[project.optional-dependencies]`, not `[optional-dependencies]`

I've since been advised of that in another server, but thank you.

shy echo Apr 19, 2024, 11:30 PM

#

In some fairly different news... This is what my last few days of free time has gotten me:

https://github.com/google/triage-party/issues/306

Will some Googler respond to me? Who knows... 🤷🏻‍♂️

GitHub

Consistent segmentation violation with pypa/pip and a minimal confi...

Description When trying to use triage-party with https://github.com/pypa/pip, I'm consistently getting a SIGSEGV crash. panic: runtime error: invalid memory address or nil pointer dereference [...

hidden flame Apr 20, 2024, 1:23 AM

#

a multi-player triaging party night is not an idea I was expecting, but you know what, that sounds pretty nifty, haha

#

reminds me of the time when Jelle and I spent a few hours cleaning up Black's issue tracker together in 2021

#

pandemic times were weird :)

dapper laurel Apr 20, 2024, 1:25 AM

#

I miss that weirdness :/

shy echo Apr 20, 2024, 4:36 PM

#

The more interesting piece for me is that it'll let me triage with custom filters more powerful than GitHub's built in ones.. 😅

shy echo Apr 24, 2024, 8:45 PM

#

YAYAY!

#

I bodged it, and figured it out.

#

❯ g diff -- **/*.go
diff --git a/pkg/hubbub/item.go b/pkg/hubbub/item.go
index 05833e1..11ac59a 100644
--- a/pkg/hubbub/item.go
+++ b/pkg/hubbub/item.go
@@ -161,7 +161,7 @@ func (h *Engine) createConversation(i provider.IItem, cs []*provider.Comment, ag
                        }
                }
 
-               if !seenCommenters[*c.User.Login] {
+               if c.User.GetLogin() != "" && !seenCommenters[*c.User.Login] {
                        co.Commenters = append(co.Commenters, c.User)
                        seenCommenters[*c.User.Login] = true
                }

Not exactly a patch I understand, but it seems to make it not panic and that's good-enough for me.

hidden flame Apr 24, 2024, 8:46 PM

#

Congratulations!

shy echo Apr 24, 2024, 8:46 PM

#

Thank you!

hidden flame Apr 24, 2024, 8:47 PM

#

I'm glad it didn't turn into one of these :P

#

I may asking the obvious here, but how much of a mess is your inbox? :)

shy echo Apr 24, 2024, 8:50 PM

#

Which one?

#

#

I'm no longer scared of big numbers.

hidden flame Apr 24, 2024, 8:51 PM

#

oh.

#

doing quick napkin math, I'm only responsible for maybe at most 0.7% of those emails which makes me feel better about my activity on the repository

shy echo Apr 24, 2024, 8:52 PM

#

LOL

hidden flame Apr 24, 2024, 8:53 PM

#

honestly though for github, I just click "mark all as read" there is no way I'm reading all of my notifications

hard thunder Apr 24, 2024, 9:08 PM

#

hidden flame I may asking the obvious here, but how much of a mess is your inbox? :)

my corpo inbox is always close to zero, wish I could do the same with my personal one

shy echo Apr 24, 2024, 9:08 PM

#

Hehe, my corpo inbox is also not even close to 0.

hard thunder Apr 24, 2024, 9:09 PM

#

fixing corpo inbox is easy, you can just change jobs if you got it to a bad state and start following the inbox zero practices in the new job /s

#

can't do the same with personal email

shy echo Apr 24, 2024, 9:10 PM

#

The problem isn't me, it's that tools and software that sends me emails and notifications.

#

And, people tell software to send me emails.

hard thunder Apr 24, 2024, 9:11 PM

#

If you have to look at it at least once then just archive it after. If you don't have to look at it at all... filters?

shy echo Apr 24, 2024, 9:11 PM

#

Oh, I have those too.

hard thunder Apr 24, 2024, 9:13 PM

#

one of the Jira boards I have access to is configured poorly so I get email about every created ticket even though it doesn't relate to my team at all so that was annoying until I got to creating a filter 😄

#

archive functionality is great though, just gotta do it eagerly

#

New Outlook app combines threads like Gmail which also helps, I wish the classic app did that too since it is a more complete product.

hidden flame Apr 24, 2024, 9:33 PM

#

It starts with only a few emails that I tell myself "I'll get to them eventually..." which becomes 20 and then 100 :)

hidden flame Apr 26, 2024, 8:12 PM

#

Whelp, pip's CI is failing due to the bump to macos-14 which lacks subversion https://github.com/pypa/pip/actions/runs/8853699641/job/24315160176#step:7:3093

stuck girder Apr 26, 2024, 8:16 PM

#

https://github.com/pypa/pip/pull/12617 will add it

limber ore Apr 30, 2024, 11:27 PM

#

small fix https://github.com/pypa/pip/pull/12665

shy echo May 6, 2024, 9:14 PM

#

Hello folks! I just cut https://pypi.org/project/pip/24.1b1/ a few minutes ago -- please do test it out by installing it and trying it in various environments. It should be generally quicker and a lot stricter about various names and various other details.

PyPI

pip

The PyPA recommended tool for installing Python packages.

hidden flame May 6, 2024, 11:13 PM

#

Already one issue: https://github.com/pypa/pip/issues/12675 pip crashes during its pip self version check when downgrading pip (note that the downgrade is still successful as the check is deferred to the very end of the command). This was my fault 🙃

willow flicker May 10, 2024, 5:25 AM

#

The feature I want most from uv: --exclude-newer DATE. This is so good for building old projects without lock files. 🙂

shy echo May 10, 2024, 5:04 PM

#

Hmm, yea, now that the simple API has this metadata, pip should probably also support it.

lunar gyro May 10, 2024, 5:21 PM

#

I think we actually have an issue in the tracker for this

cyan cargo May 14, 2024, 8:17 AM

#

i think im missing something, i'd like to always enable the truststore usage for a certain virtualenv, but the docs i skimmed don't indicate how to add the use-feature flag to a config file

lunar gyro May 14, 2024, 9:40 AM

#

cyan cargo i think im missing something, i'd like to always enable the truststore usage for...

Good timing https://github.com/pypa/pip/pull/12699

hidden flame May 15, 2024, 11:18 PM

#

https://github.com/pypa/pip/pull/12705 lots of vendoring removals ✨

#

(^ CI is broken due to typing-extensions' incompletesupport for Python 3.13)

foggy forum May 16, 2024, 10:18 AM

#

hidden flame <https://github.com/pypa/pip/pull/12705> lots of vendoring removals ✨

How can this work? https://github.com/pypa/pip/pull/12705/files#diff-7a05c7ad174407ce6cb4af3e50b581952acf0114d989fa874e26b58883a6d0daR41

The retry decorator returns a function without a .calls attribute

#

I think mypy is correct to error here with attr-defined

hidden flame May 16, 2024, 11:43 AM

#

foggy forum How can this work? https://github.com/pypa/pip/pull/12705/files#diff-7a05c7ad174...

It turns out that functools.wraps copies the original function's attributes

foggy forum May 16, 2024, 11:43 AM

#

hidden flame It turns out that functools.wraps copies the original function's attributes

But they won't be mutated?

hidden flame May 16, 2024, 11:44 AM

#

it works as it's a dictionary update

#

I should rewrite the tests either way though

foggy forum May 16, 2024, 11:45 AM

#

Oh it's a list

hidden flame May 17, 2024, 3:08 AM

#

@shy echo Did you intend to share your pip 24.1 beta blog post on DPO? I don't see it currently, but it's probably worth sharing there.

shy echo May 17, 2024, 3:21 PM

#

hidden flame <@740498628854153236> Did you intend to share your pip 24.1 beta blog post on DP...

I did, thanks for the reminder!

shy echo May 20, 2024, 10:02 PM

#

I spent a bunch of time today figuring out what is going on with pip's CI.

#

It's broken because git disabled lazy fetching.

hidden flame May 20, 2024, 10:06 PM

#

wait really

#

isn't that like a core component of blobless/treeless clones? or am I misunderstanding the issue here?

shy echo May 21, 2024, 12:00 AM

#

Yes.

jovial jasper May 21, 2024, 12:51 PM

#

shy echo It's broken because git disabled lazy fetching.

Sounds annoying. Do you have a documentation link about that at hand?

shy echo May 21, 2024, 12:56 PM

#

I made it to https://github.com/git/git/commit/7b70e9efb18c2cc3f219af399bd384c5801ba1d7 for the commit.

shy echo May 21, 2024, 1:38 PM

#

It's changed in the latest bugfix release basically.

#

OK, other than that, I've had a chat with the Faster CPython folks about including the pip requirement parser (from packaging) and the pip resolver (resolvelib) as benchmarks that they're testing against.

#

I've got a bunch of details I need to either write down or implement.

limber ore May 26, 2024, 5:27 PM

#

other than somebody devoting time to it, is there a desire to implement an experimental form of caching which keeps wheels unpacked on disk similar to UV?

#

no promises but like if a giant PR happened one day would that be considered?

hoary mist May 26, 2024, 8:40 PM

#

I think the "problem" ( for lack of a better word ) is it's an observable behavior change, and it's not clear if it's expected or desired for users

limber ore May 26, 2024, 10:06 PM

#

my assumption is that it would be behind flags until such time that it would be deemed desirable to make it the default

#

it would definitely be unexpected, 100%! however I cannot think of any reason where it would not be desirable

#

maybe something to do with hash integrity but I'm stretching

shy echo May 26, 2024, 10:08 PM

#

There was an issue filed by Glyph at some point for the same thing.

limber ore May 26, 2024, 10:13 PM

#

shy echo There was an issue filed by Glyph at some point for the same thing.

ah neat thanks https://github.com/pypa/pip/issues/11092

hidden flame May 26, 2024, 10:32 PM

#

limber ore it would definitely be unexpected, 100%! however I cannot think of any reason wh...

Disk usage would definitely be one of them, especially since pip does no automatic cache housekeeping.

limber ore May 26, 2024, 10:33 PM

#

true good point though I think everyone would still prefer it

#

when distributing there would never be caches anyway like container images where there would be a removal step or shipping pre-built Python installations like I now do for Hatch

#

and on a developer machine my assumption is that it would be a net decrease in disk utilization because as the number of virtual environments increases the net difference would as well

#

I've done absolutely no math on this but I'd say after 4 virtual environments for mostly shared packages there would be a net decrease compared to the wheel cache approach

#

if you're a data scientist/AI researcher/Nvidia user/etc. then the gains probably start at 2 lol

hidden flame May 26, 2024, 11:28 PM

#

So symlinks in addition to unpacked wheels? That's a bit different than your proposal earlier.

limber ore May 26, 2024, 11:37 PM

#

oh yeah definitely you would have to change the installation mechanism to gain the same performance as UV, although as Donald mentioned in that issue it would be an immediate win because copying files is fast

shy echo May 26, 2024, 11:38 PM

#

I'd start with copying files, I've seen more than one project that modifies installed files. 😅

fallen scroll May 26, 2024, 11:40 PM

#

Reflinks would be a good fit.

hoary mist May 26, 2024, 11:49 PM

#

sorry I commented then got distracted by something shiny

shy echo May 26, 2024, 11:50 PM

#

hoary mist sorry I commented then got distracted by something shiny

I relate strongly. 😅

hoary mist May 26, 2024, 11:52 PM

#

FWIW, and take opinion very lightly!, I don't see any real long term problem with having a shared "database" of unpacked wheels that we... copy? symlink? reflink? something?, my statement about things being a problem is more talking about the problem of getting from point a to point b

hidden flame May 26, 2024, 11:53 PM

#

I did actually write a distribution deduplicator at some point. I never used it as the storage savings weren't that great (in fairness, it was definitely poorly optimized) and frankly I like being able to scratch on my venvs sometimes 😅

hoary mist May 26, 2024, 11:54 PM

#

I definitely start yolo editing my venvs occasionally

hidden flame May 26, 2024, 11:55 PM

#

"investigates bugs and starts aggressively inserting print calls into site-packages"

hoary mist May 26, 2024, 11:58 PM

#

https://tenor.com/r7Ks.gif

Tenor

Fuck it. We'll do it live Bill Oreilly

▶ Play video

#

at least 50% of this is because I'm too stupid to use a debugger properly

trail sun May 27, 2024, 12:09 AM

#

pfft. That's a debugger UX problem for most users. Debuggers are a lot less intuitive a tool than they probably could be, and the only time anyone wants to improve them (hyperbole) is while they are in the middle of fixing something else more important.

foggy forum May 27, 2024, 9:20 AM

#

hidden flame "investigates bugs and starts aggressively inserting print calls into site-packa...

I end up debugging a lot by writing all this in site-packages or even the stdlib (eg socket.py)

def __init__(self, ...):
    f = io.StringIO()
    traceback.print_stack(file=f)
    self.tb=f.getvalue()

def __del__(self):
     if not self.closed():
         print(self.tb)

I'm not sure how else to do it

outer falcon May 29, 2024, 8:21 PM

#

I will go check GitHub for full discussions, but @willow flicker suggested that I ask here why

python -m pip --no-cache-dir download --no-deps --no-binary :all: <package name>

is also running the prepare metadata steps. While pip download --help doesn't seem to show it, is there a flag that can be used to skip this and just download the sdist?

outer falcon May 29, 2024, 8:22 PM

#

outer falcon I will go check GitHub for full discussions, but <@740658690893283359> suggested...

Example:

$ docker run --rm -ti python:3.12 /bin/bash
root@27ab9b0779ef:/# python -m venv venv && . venv/bin/activate
(venv) root@27ab9b0779ef:/# python -m pip install --upgrade --pre pip
Requirement already satisfied: pip in /venv/lib/python3.12/site-packages (24.0)
Collecting pip
  Downloading pip-24.1b1-py3-none-any.whl.metadata (3.6 kB)
Downloading pip-24.1b1-py3-none-any.whl (1.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 9.8 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 24.0
    Uninstalling pip-24.0:
      Successfully uninstalled pip-24.0
Successfully installed pip-24.1b1
(venv) root@27ab9b0779ef:/# python -m pip --no-cache-dir download --no-deps --no-binary :all: hist
Collecting hist
  Downloading hist-2.7.3.tar.gz (992 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 992.2/992.2 kB 8.1 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Saved /hist-2.7.3.tar.gz
Successfully downloaded hist
(venv) root@27ab9b0779ef:/#

fallen scroll May 29, 2024, 8:23 PM

#

https://github.com/pypa/pip/issues/1884

outer falcon May 29, 2024, 8:25 PM

#

fallen scroll https://github.com/pypa/pip/issues/1884

Ah. Old old issue. Thanks for this.

shy echo May 29, 2024, 8:31 PM

#

The "1000ft view" is that this is tech debt hitting us. The "download" and "generate metadata" pieces are highly coupled in pip.

outer falcon May 29, 2024, 8:32 PM

#

shy echo The "1000ft view" is that this is tech debt hitting us. The "download" and "gene...

Thanks @shy echo. That's a good enough summary for me.

keen void Jun 6, 2024, 12:47 PM

#

I'm trying to use pip 24.1beta so that I can run the nogil Python builds. It works fine for 3.14, but on 3.12 and 3.13 pip fails:
pip._vendor.packaging.version.InvalidVersion: Invalid version: '3.12.3+'
The failing GitHub Action is here: https://github.com/nedbat/coveragepy/actions/runs/9401046533/job/25891965210#step:8:19
Have I missed an important step?

#

I guess this is because 3.12 reports its version as "3.12.3+" while 3.14 reports as "3.14.0a0"

small cove Jun 6, 2024, 12:50 PM

#

You're probably running into https://github.com/python/cpython/issues/99968

keen void Jun 6, 2024, 12:50 PM

#

small cove You're probably running into <https://github.com/python/cpython/issues/99968>

yes, looks like it. What can I do about it?

#

I'm asked to test pip betas and Python alphas and Python nightlies, but they don't get along.

#

It looks like there might already be a fix: https://github.com/pypa/packaging/pull/802/files and just not released yet?

stuck girder Jun 6, 2024, 12:57 PM

#

following links from that issue, https://github.com/pypa/packaging/pull/802 is a merged PR to fix it, which hopefully will be in pip 24.1, planned for the week of 17th June: https://github.com/pypa/pip/issues/12613#issuecomment-2151171787

keen void Jun 6, 2024, 12:57 PM

#

OK, I'll keep waiting.

#

no second beta?

shy echo Jun 6, 2024, 1:06 PM

#

Not yet, it's blocked on me finding time to cut a packaging release followed by a pip beta release.

#

Or, directly jumping to non-beta depending on what the timelines look like.

keen void Jun 6, 2024, 1:07 PM

#

If i wanted to be really cutting-edge, would I be nuts to install pip from github?

shy echo Jun 6, 2024, 1:07 PM

#

No but, also, not yet. 😅

keen void Jun 6, 2024, 1:07 PM

#

ok!

shy echo Jun 11, 2024, 7:14 PM

#

@keen void You can do so now.

keen void Jun 11, 2024, 8:04 PM

#

shy echo <@424559318617161740> You can do so now.

thanks. Will there be a beta2? Or when is the next release?

shy echo Jun 11, 2024, 8:05 PM

#

keen void thanks. Will there be a beta2? Or when is the next release?

https://github.com/pypa/pip/pull/12763

I need to eat, but I'll cut one after dinner. :)

GitHub

Release 24.1b2 by pradyunsg · Pull Request #12763 · pypa/pip

Toward #12613

shy echo Jun 12, 2024, 6:10 AM

#

@lunar gyro Any idea what might be happening with https://github.com/pypa/pip/actions/runs/9477464511/job/26112070352?pr=12763? It's a windows-specific failure that I don't quite understand (beyond that unc path stuff is implicated and it's only failing on 3.13 😅 ).

 >       assert path_to_url(r"\\unc\as\path") == "file://unc/as/path"
E       AssertionError: assert 'file:////unc/as/path' == 'file://unc/as/path'
E         
E         - file://unc/as/path
E         + file:////unc/as/path
E         ?      ++

#

Nvm me, I found https://github.com/python/cpython/pull/118089 related to this.

lunar gyro Jun 12, 2024, 6:21 AM

#

So looks like C implementation is behaving a bit different from the Python one?

shy echo Jun 12, 2024, 6:21 AM

#

Yep.

lunar gyro Jun 12, 2024, 6:23 AM

#

Bummer

shy echo Jun 12, 2024, 6:24 AM

#

Yea, it looks like we only picked up b2 in the release PR (hehe) -- I'm gonna push through with this release after dropping a comment on the relevant CPython issue.

#

https://pypi.org/project/pip/24.1b2/ is up.

hidden flame Jun 12, 2024, 7:30 PM

#

CI has been red so much in the last two months ducky_australia

shy echo Jun 12, 2024, 7:31 PM

#

That emoji is called ducky australia 😂

maiden island Jun 12, 2024, 7:48 PM

#

Offensive

#

I joke, but yes, most upside down emoji's get branded Australian

#

USA pushed your cricket team @shy echo 😄

#

Wish they had won. Would have been awesome.

shy echo Jun 12, 2024, 7:51 PM

#

I did chuckle that they had two big batters out in the first 15 balls. XD

maiden island Jun 12, 2024, 7:53 PM

#

I bet @hidden flame doesn't even know Canada has a cricket team in the World Cup 😄

hidden flame Jun 12, 2024, 8:07 PM

#

I mean, I'm not surprised, but yes I did not know we have one.

#

I don't follow sports in general :)

limber ore Jun 13, 2024, 4:01 AM

#

https://x.com/Dixie3Flatline/status/1800933933195891098

Kat Cosgrove (@Dixie3Flatline) on X

i don't understand how cricket is played and there doesn't appear to be a sports anime to explain it to me through the plight of a teenager with weird hair so i guess it will remain a mystery

stuck girder Jun 13, 2024, 5:00 AM

#

lunar gyro Jun 13, 2024, 5:13 AM

#

It’s not that difficult to understand what the players are doing but the scoring system is def confusing

shadow shuttle Jun 13, 2024, 5:20 AM

#

97% of scoring is easy. 1 run every time batsmen run across the pitch, 4 if it crosses the boundary rope, 6 if it crosses the boundary rope without touching the ground

shy echo Jun 13, 2024, 8:51 AM

#

shadow shuttle 97% of scoring is easy. 1 run every time batsmen run across the pitch, 4 if it c...

It's all the "extras" that add complexity.

lunar gyro Jun 13, 2024, 10:10 AM

#

Ways to score:

Baseball: 1
Football: 1
Basketball: 3
Cricket: 3?
American football: ????

stuck girder Jun 13, 2024, 10:11 AM

#

American football: 3?

lunar gyro Jun 13, 2024, 10:12 AM

#

It’s definitely more than 3, I can count TD, extra, 2p conv, FG, and I think there are more than one safety rule?

stuck girder Jun 13, 2024, 10:14 AM

#

4 or 5 depending on how you count:

ARTICLE 2. TYPES OF SCORING PLAYS

Points are scored as follows:

Touchdown: 6 points

Field Goal: 3 points

Safety: 2 points

Try after touchdown: 1 point (Field Goal or Safety) or 2 points (Touchdown)

https://operations.nfl.com/the-rules/nfl-video-rulebook/scoring-plays/

hazy flume Jun 13, 2024, 8:38 PM

#

lunar gyro Ways to score: * Baseball: 1 * Football: 1 * Basketball: 3 * Cricket: 3? * Ameri...

there are definitely more than 3
(but this is #pip, not #off-topic :P)

hidden flame Jun 13, 2024, 10:00 PM

#

Who said rules had to be followed :p

shy echo Jun 13, 2024, 10:01 PM

#

Me.

#

Please no chaos.

lunar gyro Jun 19, 2024, 12:38 PM

#

Should make main green (until 3.13.0b2 is released) https://github.com/pypa/pip/pull/12774

shy echo Jun 20, 2024, 9:22 PM

#

Heads up: I'm gonna have very limited internet access for ~4 days starting Saturday GMT since I'm gonna be on a vacation. And, I'm probably gonna cut the release tonight.

ripe shoal Jun 20, 2024, 9:23 PM

#

Enjoy vacation!

shy echo Jun 20, 2024, 10:30 PM

#

I'm probably scarred for life by pip 20.0, coz I'm like... wait, we don't have an issue immediately after cutting a pip release. Did I actually do the release properly?

hidden flame Jun 20, 2024, 10:42 PM

#

I'll be around (although with limited availability). I'll try to handle some issue triage as I'd imagine we'll see some new issues filed as pip 24.1 is picked up.

#

(also, this reminds me that https://www.volunteeramnestyday.net/ is a thing and it's today!)

shy echo Jun 20, 2024, 10:45 PM

#

💯

hidden flame Jun 20, 2024, 10:46 PM

#

shy echo I'm probably scarred for life by pip 20.0, coz I'm like... wait, we don't have a...

I still remember when I introduced a regression on like my 5th contribution (ever) to Black. I freaked out upon reading the newly opened issue :)

shy echo Jun 20, 2024, 10:46 PM

#

FWIW, if there's some showstopper issue and I'm unreachable (because, idk, I'm somewhere in the middle of Scotland), I trust the other maintainers to pick one amongst ourselves to cut an emergency bugfix.

I think this trust is implied already, but figured this is the sort of thing that it doesn't hurt to reiterate. :)

willow flicker Jun 21, 2024, 2:56 AM

#

Usage of betas was high this time around due to needing it for free-threading. Manylinux and cibuildwheel had to ship with 24.1 betas for CPython 3.13 due to that, and I've had to use it in various CI's for CPython 3.13. (IMO, this is the major new feature and it's not mentioned in the highlights!) So, probably going to jinx it, but I think it should be pretty quiet!

hidden flame Jun 21, 2024, 3:41 AM

#

Well, the thing is that there are likely some corporate environments and a wack ton of legacy environments where legacy versions/specifiers are in still in use. Once they upgrade to pip 24.1, we'll start getting pitchforks.

shy echo Jun 21, 2024, 7:24 AM

#

willow flicker Usage of betas was high this time around due to needing it for free-threading. M...

Yea, I hear ya! I can edit the highlights to mention it. :)

hidden flame Jun 21, 2024, 7:21 PM

#

You know, I was expecting worse, but two issues isn't too bad so far.

grand rock Jun 22, 2024, 9:58 AM

#

The CPython change that causes Windows test failures (see https://github.com/pypa/pip/pull/12774) was backported to 3.12.4 which is now included in GHA runners https://github.com/actions/python-versions/commit/1c85886f9c23363cd42fe0d4c2decee499485df0#diff-f69ccbefcbee0c9c68b41a0e58591150a3c87aa13d476d33fc8ebf90588e13c5R377 so it's now causing failures in pip's test suite https://github.com/pypa/pip/actions/runs/9614117106/job/26518252566 (backports for other versions have also been merged upstream, those versions aren't in the runners yet).

hidden flame Jun 22, 2024, 10:57 PM

#

@shy echo you better be enjoying your vacation, submitting and reviewing PRs during your vacation :P

shy echo Jun 22, 2024, 10:58 PM

#

I am! Just figured I'd take a look before wrapping up my day. 😅

#

And, it's like... ~2 minutes of work to file that PR (once I got network working on my laptop).

hidden flame Jun 22, 2024, 11:00 PM

#

Fair :)

azure hazel Jun 23, 2024, 9:59 AM

#

grand rock The CPython change that causes Windows test failures (see https://github.com/pyp...

Attempt at fixing the test failures https://github.com/pypa/pip/pull/12788. It need some more work since there are more failing cases I didn't catch, but happy to hear any feedback on the approach

chrome epoch Jun 24, 2024, 8:12 PM

#

@shy echo random Q for you: does pip install check the file's content against the PEP 503 index-supplied hash, or only against hashes when supplied via requirements.txt in hash-checking mode? i tried to find conclusive evidence in the source but coudn't with a short search 🙂

shy echo Jun 25, 2024, 9:42 AM

#

chrome epoch <@740498628854153236> random Q for you: does `pip install` check the file's cont...

You'll probably only see this when I get network, but the answer is yes.

#

For the first part.

chrome epoch Jun 25, 2024, 1:31 PM

#

shy echo You'll probably only see this when I get network, but the answer is yes.

Much appreciated! Apologies if my message reached you in a place where you’re avoiding computers 🙂

shy echo Jun 25, 2024, 10:23 PM

#

Alrighty, let's see what 24.1.1 needs.

shy echo Jun 26, 2024, 8:33 PM

#

And, it's up!

#

Holler if you see something broken!

hidden flame Jun 27, 2024, 12:49 AM

#

Hmm, do we think we need one for the hardlink one as well? I mean, 24.2 is due in about a month, so ¯_(ツ)_/¯

shy echo Jun 27, 2024, 1:00 PM

#

hidden flame Hmm, do we think we need one for the hardlink one as well? I mean, 24.2 is due i...

I think so, haven't looked at the PR yet.

hidden flame Jun 27, 2024, 1:58 PM

#

Coolio. I'm curious as I got some PRs I've like to see land in the next release cycle, but let's finish 24.1 first.

shy echo Jun 27, 2024, 2:21 PM

#

Nah, go ahead and poke them for landing.

#

main is open for non-bugfix stuff.

hidden flame Jun 30, 2024, 1:43 AM

#

I'm back to hacking on pip errors. This is what I have so far for improving connection failures. I still easily have 80% of the work left, but the prototype is promising. Disclaimer: this already involves some hacks :(

shy echo Jun 30, 2024, 1:47 AM

#

Noice!

hidden flame Jun 30, 2024, 1:49 AM

#

Y'all weren't kidding about it being nontrivial to improve the error reporting in pip. This alone took a few hours (consisting primarily reading urllib3/request source code and documentation, and exploring the exception state that is available to me).

shy echo Jun 30, 2024, 1:49 AM

#

Yea, this in particular is a hairy one and improving it is worth a lot IMO.

hidden flame Jun 30, 2024, 1:50 AM

#

Mhmm. The retrying messages are particularly bad.

#

To potentially pop your bubble, I'm currently relying on installing a custom logging filter on urllib3.connectionpool that "sniffs out" the retrying warning. A slightly less hacky solution would be to patch urllib3 to pass a special flag/enough state in that log.warning(...) call that the filter can look for.

#

I'd like to upstream the extra={...} patch to urllib3 but we can't upgrade to modern urllib3 until 2025 AFAIU due to OpenSSL incompatibilities.

shy echo Jun 30, 2024, 1:53 AM

#

FWIW, I got stuck when I started thinking about what should be logged when we're dealing with flaky indexes and stuff where even "what to write" is different from a regular hand-typed URL... mostly because the guidance would possibly be different and because I couldn't decide who my audience was for the error messages (is it a professional dev, teacher, student, someone-trying-to-do-their-job etc).

Don't end up there, I guess, is the word of caution here. 😅

shy echo Jun 30, 2024, 1:54 AM

#

hidden flame I'd like to upstream the `extra={...}` patch to urllib3 but we can't upgrade to ...

Wait, did I miss something? Why do you say this?

hidden flame Jun 30, 2024, 1:54 AM

#

https://github.com/pypa/pip/pull/12026#issuecomment-1631726998

hidden flame Jun 30, 2024, 1:55 AM

#

shy echo FWIW, I got stuck when I started thinking about what should be logged when we're...

Honestly, I haven't thought that far yet. I'm just trying to raise the baseline to something above "it's truly awful". I'm very much in the mindset of "we can iterate on this later once we get more feedback."

shy echo Jun 30, 2024, 1:56 AM

#

hidden flame Honestly, I haven't thought that far yet. I'm just trying to raise the baseline ...

Yes, please!

shy echo Jun 30, 2024, 1:56 AM

#

hidden flame <https://github.com/pypa/pip/pull/12026#issuecomment-1631726998>

I did miss something!

hidden flame Jun 30, 2024, 1:57 AM

#

AFAUI that's also the primary blocker for parallelizing HTTP requests too, although I'm sure that maintainer time is the biggest :P

#

(I'm not volunteering to parallelize that.)

shy echo Jun 30, 2024, 1:58 AM

#

Yea, I remember that there's a PR somewhere on the repo implementing parallel downloads but couldn't figure out if it was actually correct and all that.

hidden flame Jun 30, 2024, 2:01 AM

#

Yeah.. there are a lot of things to review 😅. We'll get through what we can.

#

I'm simply trying to help move the PRs process along where I can. Improving error reporting is my personal project beyond that.

hidden flame Jul 1, 2024, 12:06 AM

#

There is legitimately no way for me to query the URL hostname or even the full URL from the pre-existing SSLError-related retry warning state. I don't want to patch urllib3 more than I need to, but hmm, this seems more and more necessary.

#

I'll deal with the patching later. I'll try to do as much as I can without it, and then see how much further I can go with some judicious patching.

hidden flame Jul 1, 2024, 2:21 AM

#

OTOH, this looks nice.

hidden flame Jul 8, 2024, 3:17 AM

#

I like that while I'm totally out of the house and without internet, pip 24.1.2 is cut and a flurry of activity takes place on the tracker :)

#

I'm currently working through my notification backlog, but after that I'll take another look at the milestone. I'll try to push the scheduled PRs along as much as I can reasonably can so hopefully this release cycle doesn't drag on forever.

azure hazel Jul 10, 2024, 6:15 PM

#

should (or is it already?) this Discord channel be a documented way of communication, like IRC is?

#

asking since I jumped on IRC before here and noticed it was quite a bit quieter

shy echo Jul 10, 2024, 6:16 PM

#

I don't think it is and... like... yea? I really really don't want to end up with user support questions here though.

azure hazel Jul 10, 2024, 6:35 PM

#

👍 https://github.com/pypa/pip/pull/12840

GitHub

Document `pip` Discord channel by matthewhughes934 · Pull Request #...

Since it appears to be used as much, or more, as the other documented communication channels. The channel URL was taken from the Discord UI: right click channel name -> 'Copy Link'.
Wher...

finite perch Jul 10, 2024, 7:02 PM

#

I certainly would have appreciated the documentation, I only found the discord this week, I'd looked at the IRC several months ago but saw it wasn't particularly active

shy echo Jul 10, 2024, 7:33 PM

#

Whoops

hidden flame Jul 10, 2024, 9:10 PM

#

Yea, alright, let's add it to the developer docs at least then 😅

#

Welcome to the triage team by the way @finite perch! It's good to see you join the team, thanks for all of your help in investigating and improving pip's dependency resolution!

hidden flame Jul 11, 2024, 1:25 AM

#

@shy echo how would you suggest that I go about highlighting PRs that look ready to land, but don't really belong in a milestone? For example, https://github.com/pypa/pip/pull/12572 is purely maintenance only so it doesn't make sense to add to a release milestone, but otherwise, it should be merged at some point. I've left comments to suggest merging on other PRs with limited success.

I realize that this is mostly a problem of "limited maintainer availability" so there's probably no good solution, but I'm curious if you have any thoughts on this.

shy echo Jul 11, 2024, 7:49 AM

#

Awaiting merge is now a label. You should be able to tack that onto things.

hidden flame Jul 12, 2024, 8:11 PM

#

shy echo Awaiting merge is now a label. You should be able to tack that onto things.

Great, thanks!

#

Man, I found another issue I'd like to tackle, but I already have enough tasks on my plate for the time being. Haha.

#

Anyway, we should be nearly ready for pip 24.2 this weekend. All of the remaining items in the 24.2 milestone should be trivial for the most part, barring the truststore PR but it's getting there :)

#

Oh hmm, the changelog entries are definitely going to need some tweaking this release actually.

maiden island Jul 12, 2024, 8:19 PM

#

ichard, always wanting to do more 🥰

hidden flame Jul 13, 2024, 9:43 PM

#

@shy echo I'm about to leave the house for 2 hours (poor timing, heh) but https://github.com/pypa/pip/blob/c2d706f82b1cd9332c70bed170501413240ae008/news/12728.feature.rst should be moved to the process category.

#

That was the main changelog item I wanted to flag earlier.

shy echo Jul 13, 2024, 10:05 PM

#

Good to know. I'll probably copy edit the changelog before the release anyway.

hard thunder Jul 14, 2024, 12:11 AM

#

Can I somehow specify --only-binary such that it applies to all packages except for packages I specify?

finite perch Jul 14, 2024, 12:21 AM

#

You can specify --only-binary :all: and --no-binary {package_name} for all the packages you want the source file from instead.

A less manual approach is to use --prefer-binary which will only use source installation if no binary version is available

hard thunder Jul 14, 2024, 12:39 AM

#

hmm, would --only-binary :all: --prefer-binary {package_name} work?

#

Basically I want to test that I can install all dependencies from wheels except for packages that I know do not have wheels right now.

#

I guess I'll just try it later 😄

finite perch Jul 14, 2024, 12:45 AM

#

hard thunder hmm, would `--only-binary :all:` `--prefer-binary {package_name}` work?

I don't think so, I don't think --prefer-binary takes an argument

hard thunder Jul 14, 2024, 12:45 AM

#

oh, right..

#

I thought it did lol

#

well, thanks for helping, I didn't know that --no-binary will correctly work as an exclude, if --only-binary :all: is used

finite perch Jul 14, 2024, 1:09 AM

#

I'm not sure it's intended behavior, but it works, I even requested that and uv has copied this behavior: https://github.com/astral-sh/uv/issues/4063

finite perch Jul 21, 2024, 1:19 AM

#

🎉 I think I've finally got a fix for backjumping resolution 🎉: https://github.com/sarugaku/resolvelib/pull/155

x-ref: https://github.com/pypa/pip/issues/12317 and https://github.com/pypa/pip/issues/12768

hidden flame Jul 21, 2024, 1:23 AM

#

finite perch 🎉 I think I've finally got a fix for backjumping resolution 🎉: https://github...

Totally uninformed layperson commentary, but how does this compare to changing the contract of resolvelib's APIs so backjumping is safe?

#

This is great news, however. Hopefully it pans out!

finite perch Jul 21, 2024, 1:25 AM

#

I don't quite follow the question, what do you mean by "safe"? And what contract?

I think the only think resolvelib tries to guarantee, is if there is at least 1 solution it will provide a solution

hidden flame Jul 21, 2024, 1:28 AM

#

Sorry, I was referring to https://github.com/sarugaku/resolvelib/issues/134#issuecomment-2180697555.

finite perch Jul 21, 2024, 1:28 AM

#

Ah, yes, this is far superior because we don't need to change the contract

#

And I have not been able to find a solution to changing the contract that I wasn't able to find some resolution that still fails

#

Just spent the last couple of hours starting at diffs I created of the state of various resolutions to see if there was something I was missing, and finally found 1 consistent pattern of when a resolution fails and it shouldn't

hidden flame Jul 21, 2024, 1:31 AM

#

Very neat.

finite perch Jul 21, 2024, 1:33 AM

#

Yeah, once this lands, I can finally get back to my suggestions on improving performance

hidden flame Jul 21, 2024, 1:34 AM

#

Of course :)

#

I don't have any immediate plans for PRing my installation progress patch. I'm currently working on finishing my existing PRs, and then I'll get back to working towards reducing the PR backlog.

#

However, there has been a fair bit of work done by a contributor speeding up the installation step. I'd be interested in seeing you rerun your test and see how much the runtime will improve under v24.2. It should be fairly significant, especially under Python 3.11+ which uses the previously poorly optimized importlib metadata backend.

hidden flame Jul 21, 2024, 1:38 AM

#

hidden flame I don't have any immediate plans for PRing my installation progress patch. I'm c...

I suppose it probably makes sense to PR it once the --progress-bar raw version of installation progress lands, which should occur soon. I've scheduled that PR for 24.3.

finite perch Jul 21, 2024, 1:38 AM

#

Oh yeah, that reminds me, you said:

However, it did not play nicely with the logging stack, so any intervening logs would break the progress bar. To fix this, I had to redo how rich was initialized in the logging stack which took a bit 🙂

And I was a bit concerned that actually maybe this isn't a good idea. I have it set up at work like this, so we can log and have a progress bar in rich at the same time. But it seems like letting rich take full control over the display might not work well in lots of edge cases where people might be using pip 😦

hidden flame Jul 21, 2024, 1:41 AM

#

We already use rich for the download progress bar so those use-cases would be already broken, and AFAIU, rich gracefully handles the "are we running under a modern TTY or not?" question for us.

finite perch Jul 21, 2024, 1:43 AM

#

hidden flame However, there has been a fair bit of work done by a contributor speeding up the...

Yeah, I will rerun tests soon and get new performance numbers and call graphs

hidden flame Jul 21, 2024, 1:44 AM

#

commit fe5f224ab89ca5faf73b80ebaae025f9f23e1358 (HEAD -> installation-progress)
Author: Richard Si <sichard26@gmail.com>
Date:   Mon Jul 15 23:18:36 2024 -0400

    [wip] Add rich progress bar for package installation

 src/pip/_internal/cli/progress_bars.py | 48 +++++++++++++++++++++++++++++++++++++++---------
 src/pip/_internal/req/__init__.py      | 15 ++++++++++-----
 src/pip/_internal/utils/logging.py     | 33 ++++++++++++++++++++-------------
 3 files changed, 69 insertions(+), 27 deletions(-)

It's not that bad of a patch. We just have to enforce that there is only one rich console per stream at use globally. Rich, including rich progress bars actually is able to play nice with non-rich code that prints at will. The problem was that currently pip's rich progress bars are completely unaware of the logging stack, which also uses rich, so the locks that rich implements are sidestepped.

#

This is completely irrelevant, but I have a laugh whenever I talk about rich because it's a nickname of mine that I hear with regularity, ha.

finite perch Jul 21, 2024, 1:47 AM

#

Oh that's funny

hidden flame Jul 21, 2024, 1:47 AM

#

Not a particular fan of it, but I don't really mind it. People like to use it.

finite perch Jul 21, 2024, 1:48 AM

#

I think there are worse nicknames for Richard

hidden flame Jul 21, 2024, 1:48 AM

#

I am well aware :)

#

People have made the jokes before lemon_warpaint .

hidden flame Jul 21, 2024, 1:49 AM

#

hidden flame We already use rich for the download progress bar so those use-cases would be al...

And actually, I'm pretty sure if pip emitted any logs while downloading a distribution, it would break as of right now. This can and should be fixed at some point.

hidden flame Jul 21, 2024, 1:51 AM

#

finite perch Yeah, I will rerun tests soon and get new performance numbers and call graphs

Coolio, TIA!

finite perch Jul 21, 2024, 1:56 AM

#

Oh... I think I might have found a show stopper for pip 24.2

hidden flame Jul 21, 2024, 2:00 AM

#

You can't just bookend your message there, heh.

finite perch Jul 21, 2024, 2:01 AM

#

Oh, I'm confirming I can reproduce and writing up the issue now, it actually looks like it might be fairly cosmetic and purely related to pip development

#

https://github.com/pypa/pip/issues/12864

hidden flame Jul 21, 2024, 2:09 AM

#

Maybe my patch to lazy load the pip self version check module was a mistake 😓.

finite perch Jul 21, 2024, 2:10 AM

#

I feel like that patch would make truststore less likely to be invoked, rather than more likely

hidden flame Jul 21, 2024, 2:11 AM

#

The problem is probably another class of "the pip self version check loads after a new set of pip code is installed, which predictably blows up", although I'm not sure how that manifests here as truststore should be already loaded as the pip install command needs to access the network anyway.

finite perch Jul 21, 2024, 2:12 AM

#

Ah

hidden flame Jul 21, 2024, 2:13 AM

#

I think the easy patch is simply to eagerly load the self version check module at the top of the install command module.

#

I suggested this earlier, but we went with a more targeted patch to only eagerly import it when installing pip.

#

Can you try adding import pip._internal.self_outdated_check to the very top of src/pip/_internal/self_outdated_check.py and attempting to reproduce the crash again?

finite perch Jul 21, 2024, 2:15 AM

#

Sure

hidden flame Jul 21, 2024, 2:16 AM

#

If this fixes it, then this should suffice as a short-term fix to unblock the release, although I'm wondering whether I should just revert the lazy loading entirely as lazy imports are in general pretty dubious in a package installer where packages may be replaced under our feet suddenly.

hidden flame Jul 21, 2024, 2:17 AM

#

hidden flame Can you try adding `import pip._internal.self_outdated_check` to the very top of...

Oh wait, I'm an idiot. Sorry I meant src/pip/_internal/commands/install.py.

finite perch Jul 21, 2024, 2:17 AM

#

Great, because your earlier suggestion didn't work

finite perch Jul 21, 2024, 2:18 AM

#

hidden flame Oh wait, I'm an idiot. Sorry I meant `src/pip/_internal/commands/install.py`.

Still doesn't work

hidden flame Jul 21, 2024, 2:18 AM

#

Hmm, fun.

#

I'll copy n' paste our discussion to the issue, but that's not great.

#

Ah, yeah, that makes sense. The problem is that we're initializing a new (requests) PipSession in the pip self version check. This involves setting up truststore again which breaks as certifi has already loaded and certifi.where() returns the old location for its CA bundle.

#

So, I didn't break this, although this is still not great.

finite perch Jul 21, 2024, 2:28 AM

#

Is that because of the ctx.load_verify_locations(certifi.where())?

I was thinking that line was going to cause problems eventually

hidden flame Jul 21, 2024, 2:28 AM

#

I do wonder why this didn't break earlier when we relied entirely on certifi. It must have been used in a different way.

hidden flame Jul 21, 2024, 2:28 AM

#

finite perch Is that because of the `ctx.load_verify_locations(certifi.where())`? I was thin...

Yup.

#

You should've spoken up, you would've gotten a cookie for being right on the nose 🍪 /lh

finite perch Jul 21, 2024, 2:29 AM

#

Weren't manually calling certifi before right? Leaving all that logic to requests, imagine it's quite battle tested

#

Well, my bigger concern about that line, is people who don't want certifi certificates, and will at some point complain about old/incorrect ceritificates being loaded, perhaps after a real world security incident

hidden flame Jul 21, 2024, 2:32 AM

#

finite perch Weren't manually calling certifi before right? Leaving all that logic to request...

Ah yeah, you're right.

hidden flame Jul 21, 2024, 2:34 AM

#

finite perch Well, my bigger concern about that line, is people who don't want certifi certif...

I don't know what the state of system certificates via Python around the world is, but my gut instinct is that this will probably have to start off as an option.

#

I do remember reading some ancient issues about asking pip to use the system certificates though.

finite perch Jul 21, 2024, 2:35 AM

#

Well, I was more thinking of something that happens in 5 years. But there's also the point that if you use truststore in pip 24.1 you don't get certifi certificates, but in pip 24.2 you do, with no way to turn off

hidden flame Jul 21, 2024, 2:37 AM

#

Hmm, right. I did call out that line in my PR review, but I definitely didn't appreciate the implications at that time.

#

That isn't great...

finite perch Jul 21, 2024, 2:37 AM

#

I just don't have a feel for if anyone is relying on that right now though

#

Probably people who are so paranoid to not use certifi certificates are manually patching pip and the rest of their Python stack, but who knows

hidden flame Jul 21, 2024, 2:39 AM

#

It would be nice to get to a state where hacks like certifi aren't needed, but that'll take some time.

#

If there isn't a currently open issue for weeding us off of certifi, perhaps it would be worth it to create a new one?

hidden flame Jul 21, 2024, 2:42 AM

#

finite perch Probably people who are so paranoid to not use certifi certificates are manually...

Right, because requests uses it internally anyway.

finite perch Jul 21, 2024, 2:43 AM

#

Yeah, by default

#

It's the assumed good store for certificates in the Python HTTP stack

hidden flame Jul 21, 2024, 3:19 AM

#

Thanks for discovering and filing an issue about this @finite perch. It's definitely better to get this fixed before the release. I've left a comment explaining the root issue, a potential solution, and bringing up the bigger question of what pip's relationship with certifi may look like in the future. I will go to bed now as I got work in the morning :P

mint patrol Jul 22, 2024, 3:34 PM

#

Thanks @hidden flame and @finite perch for your effort on tracking down this issue 💜 Is there anything needed from me or Truststore for this fix?

finite perch Jul 22, 2024, 3:52 PM

#

I don't think so, this PR fixes the issue and it's internal pip handling: https://github.com/pypa/pip/pull/12865

GitHub

Better exception handling around self-check by pradyunsg · Pull Req...

See #12864 for context.

finite perch Jul 31, 2024, 4:25 PM

#

@shy echo @idle harness is this a good place to ping for a resolvelib release? Or should I do it on the issue tracker? I'd like to vendor it into pip ahead of 24.3 and make sure there are no unexpected issues

shy echo Jul 31, 2024, 4:50 PM

#

The issue tracker works.

azure hazel Aug 1, 2024, 7:19 PM

#

I was seeing some errors trying to build docs locally with a fresh env: https://github.com/pypa/pip/pull/12886 (issue is incompatibility with new version of towncrier from 2 days ago)

GitHub

Pin `towncrier` in docs requirements by matthewhughes934 · Pull Req...

For compatibility reasons, see comment

#

actually, I was seeing different errors to the one in the linked issue (and that issue is also closed). Digging deeper
EDIT: nevermind, I had too many issue tabs open 🤦‍♂️ the one linked is indeed the problem

finite perch Aug 3, 2024, 2:44 PM

#

Hey, if anyone has examples of very long resolutions or reaching ResolutionTooDeep please send them over to me, I'm back to working on improving algorithmic resolution performance

shy echo Aug 3, 2024, 3:44 PM

#

Home assistant apparently has really bad performance, and IIRC, it was reexploration when we were backjumping.

finite perch Aug 3, 2024, 3:46 PM

#

Home assistant has over 1000 direct dependencies, so not surprising, it's a miracle they managed to move to the new resolver

small cove Aug 3, 2024, 6:06 PM

#

What do you consider very long?

finite perch Aug 3, 2024, 6:44 PM

#

It's not a hard rule, but "seems to be backtracking longer than it should for the amount of requirements I have", could be taking over 5 minutes for a few requirements, or taking over 30 minutes for a lot of requirements

finite perch Aug 4, 2024, 8:10 PM

#

Does it make sense to raise a PR to remove the IRC link from https://github.com/pypa/pip/issues/new/choose ? Is anyone from pip on that IRC?

shy echo Aug 4, 2024, 10:40 PM

#

I used to be, but I think we can safely remove that now.

keen void Aug 4, 2024, 10:43 PM

#

there are 55 users in that channel and the last message was July 29th

finite perch Aug 4, 2024, 10:43 PM

#

I know there are PyPA users in there in general, but I was more focused on if there are any pip devs there, no point sending pip users to a group that can't help them

keen void Aug 4, 2024, 10:44 PM

#

why not change it to point here?

shy echo Aug 4, 2024, 10:44 PM

#

We have links over on the development side to here.

keen void Aug 4, 2024, 10:45 PM

#

ok

shy echo Aug 4, 2024, 10:45 PM

#

I'd mentally modelled this Discord as closer to what #pypa-dev used to be. 😅

#

Maybe that's the wrong model? I dunno.

keen void Aug 4, 2024, 10:47 PM

#

i sure don't know. it's up to you. I come here as a user and get help.

finite perch Aug 4, 2024, 11:00 PM

#

I just got timed out for sharing the typical user support links that gets used to point users to more general help, so I'll just say here is an example at the bottom of my post: https://github.com/pypa/pip/issues/12820#issuecomment-2207206343

hidden flame Aug 9, 2024, 12:55 AM

#

So I started reading the various preparer and resolver PRs filed by Danny. My god is it going to take a while. The preparer <-> distribution <-> resolver interactions are so complex and confusing. 😅

#

This is going to be a multi-day project as I first need to understand how the current preparer impl. works.

finite perch Aug 9, 2024, 12:59 AM

#

Yeah, I want to help out there, I might have some time soon, I'm mostly waiting on a resolvelib release, then then I can vendor and fix some stuff in pip

#

In mean time, I was thinking of looking at them, if it helps detangle this stuff at all I'm all for it

hidden flame Aug 9, 2024, 1:01 AM

#

I don't understand what an AbstractDistribution is and how it differs in usage from a BaseDistribution...

hidden flame Aug 9, 2024, 1:02 AM

#

finite perch In mean time, I was thinking of looking at them, if it helps detangle this stuff...

I mean, realistically any reviews I end up filing are unlikely to be that important, but this is a good opportunity to learn and understand a part of the codebase I've avoided in the past frankly.

dapper laurel Aug 9, 2024, 1:04 AM

#

hidden flame I don't understand what an AbstractDistribution is and how it differs in usage f...

9 times out of 10, when I see AbstractX or BaseX it reminds me of why so many people consider OOP an antipattern

finite perch Aug 9, 2024, 1:07 AM

#

I don't come from a CS background, Python very slowly tricked me in to learning OOP !

hidden flame Aug 9, 2024, 1:07 AM

#

There are like 3 or 4 longish ancient issues on the tracker discussing refactoring the preparer, distributions, and metadata logic, which definitely do not comprise 1/3 of the pip codebase... 🥴

dapper laurel Aug 9, 2024, 1:11 AM

#

hidden flame There are like 3 or 4 longish ancient issues on the tracker discussing refactori...

And then after you do the work it turns out that someone was relying on some quirks of old implementation and you broke their setup...

finite perch Aug 9, 2024, 1:12 AM

#

Who would depend on an undocumented quirk of pip?

#

😉

hidden flame Aug 9, 2024, 1:12 AM

#

can I have a penny for each instance of that?

hidden flame Aug 9, 2024, 1:14 AM

#

dapper laurel And then after you do the work it turns out that someone was relying on some qui...

Well, the goal of the refactoring is that it's actually possible to tweak or add functionality in a maintainable way so the quirks ought to go at some point (or become documented features).

#

I look forward to the day we can remove the legacy resolver although that probably will be at least a few more years out.

finite perch Aug 9, 2024, 1:15 AM

#

hidden flame can I have a penny for each instance of that?

Why would you need that? According to someone posting to the setuptools issue tracker yesterday, everyone who works on critical Python packages are either well paid by tech companies to do so or covered by grants

hidden flame Aug 9, 2024, 1:16 AM

#

It's not like I was talking the cost of security keys earlier...

#

Security keys just magically appear in the hands of critical OSS maintainers.

hidden flame Aug 9, 2024, 1:17 AM

#

finite perch Why would you need that? According to someone posting to the setuptools issue tr...

Haven't read that thread yet though. I'm curious to how badly it devolved.

finite perch Aug 9, 2024, 1:19 AM

#

(oh sorry, actually it was packaging not setuptools), anyway, I was very tempted to give a pretty sharp reply, but decided it was better just to not follow up at all, seems like so far no one else has cared to respond

hidden flame Aug 9, 2024, 1:19 AM

#

lol, yup.

finite perch Aug 9, 2024, 1:20 AM

#

lol

hidden flame Aug 9, 2024, 3:52 AM

#

finite perch (oh sorry, actually it was packaging not setuptools), anyway, I was very tempted...

Honestly, the relationship between volunteer maintainers and corporate users is already dysfunctional. I don't think there is much point in arguing or continuing to engage with them if it's evident they aren't going to realize that this is all a project of love.

#

The optimist within me would love to hope that if we continue listening and accommodating their needs when reasonable, they'll come around, but given how modern corporations work, I don't see that changing ever.

#

I realize that the individual users != soulless mega corporations, but they often come in with the corporate expectations.

stuck girder Aug 9, 2024, 6:12 AM

#

In this case, it comes from a University of Oxford CS lecturer, not a corporate user

rapid blaze Aug 9, 2024, 6:34 AM

#

stuck girder In this case, it comes from a University of Oxford CS lecturer, not a corporate ...

I replied on the issue to answer the original question, but I don't have issue closure privileges on the packaging repo.

rapid blaze Aug 9, 2024, 6:45 AM

#

rapid blaze I replied on the issue to answer the original question, but I don't have issue c...

While I initially resisted, I eventually gave in and added a parenthetical comment about the nature of paid contributions (specifically, almost nobody gets paid to keep slow and unreliable APIs working, they get paid to replace them with something better)

lunar gyro Aug 9, 2024, 7:10 AM

#

hidden flame I don't understand what an AbstractDistribution is and how it differs in usage f...

History reasons and different people want to use Distribution for different purposes but didn’t want to involve in giant refactoring work… Things inheriting from BaseDistribution are for the resolver, while those from AbstractDistribution are for the preparer and evaluator.

shy echo Aug 9, 2024, 7:58 AM

#

And, I do have a couple of half-done refactors that have open issues. It wouldn't be the first thing I started doing in pip that ended up being a case off "bit off a lot more than I can chew". 😅

finite perch Aug 9, 2024, 11:59 PM

#

Okay, so I've found using compileall.compile_dir to provide quite a big speed up over compileall.compile_file, it knocks 25 seconds off installing Airflow on my computer.

However, when I use workers=0 I can not get the output to be quiet, I get a bunch of SyntaxWarnings even if I set quiet=2: https://docs.python.org/3/library/compileall.html#compileall.compile_dir

Anyone know if I can filter a subprocess from printing to stdout or stderr even if I don't have access to the subprocess object?

finite perch Aug 10, 2024, 12:40 AM

#

ChatGPT confidentally gave me a series of wrong answers 🙂

hidden flame Aug 10, 2024, 1:14 AM

#

finite perch Okay, so I've found using `compileall.compile_dir` to provide quite a big speed ...

A question, will this negatively impact installation of one to a few small packages? Multiprocessing startup is not free.

finite perch Aug 10, 2024, 1:27 AM

#

Possibly, it's worth benchmarking

#

Between that and it looking like I can't use compile_dir, a lot more work than expected 😦

shy echo Aug 10, 2024, 8:07 AM

#

@finite perch Are you testing on Linux? IIRC, Linux has the lowest process startup overheads of the big three platforms and those overheads will be about as small as they get there.

finite perch Aug 10, 2024, 2:42 PM

#

Yes, but I will test on both Linux and Windows before making a new PR

finite perch Aug 13, 2024, 1:07 PM

#

FYI a user posted the same issue twice after I closed the first one as a duplicate: https://github.com/pypa/pip/issues/12910. Closed it asking them politely not to do that.

finite perch Aug 14, 2024, 11:07 PM

#

Was very confused when git push wasn't working, took me a while of tinkering to realize github is down

hidden flame Aug 14, 2024, 11:20 PM

#

finite perch Was very confused when `git push` wasn't working, took me a while of tinkering t...

You force pushed too hard ducky_australia

shy echo Aug 14, 2024, 11:21 PM

#

It's a rite of passage I feel like. You know you do a lot of open source stuff when you have GitHub is down as the reason you can't do a thing you were hoping to do.

hidden flame Aug 14, 2024, 11:26 PM

#

(I forgot the top text)

finite perch Aug 14, 2024, 11:27 PM

#

Yeah, not too familiar with this, as all the comapnies I work for very explicitly never rely on anything on the Internet

ripe shoal Aug 14, 2024, 11:29 PM

#

Fortunately full Github outages aren't tooo common

hidden flame Aug 14, 2024, 11:29 PM

#

We'll blame the next one on you :D

finite perch Aug 14, 2024, 11:31 PM

#

Also, gave me a moment to run tests before opening PR and find out it broke about 30% of pip's tests

finite perch Aug 15, 2024, 12:03 AM

#

Hmmm, pip accepts _ to indicate a post version, e.g. 0.1_1 gets converted to 0.1-1 by pip, which is equivalent to 0.1.post1, this is not PEP 440 compliant, going to write up an issue, as this prevents moving some package parsing out of pip into packaging

#

https://github.com/pypa/pip/issues/12914

hidden flame Aug 16, 2024, 12:35 AM

#

lunar gyro History reasons and different people want to use _Distribution_ for different pu...

That is definitely not confusing at all...

finite perch Aug 16, 2024, 12:42 AM

#

Just make very long names for things BaseDistribution -> BaseResolverDistribution, AbstractDistribution -> AbstractPreparerDistribution, some generic class that inherits from both BaseResolverAbstractPreparerDistribution. OO development solved.

hidden flame Aug 16, 2024, 12:43 AM

#

@finite perch Another idea for reducing the subprocess penalty for precompiling would be to spin up a pool of subprocesses at the start of the installation process that could batch compile the installed Python files. You'd probably want to gate this both on the maximum processes allowed (e.g., on my system, os.cpu_count() returns 16!!) and also require a minimum amount of files to avoid slowing down "tiny" installations ... which yeah, is a lot of complexity.

#

Alternatively, you could do all of the batch compiling at the end, but that would change how packages are installed.

#

I'm not going to propose enabling --no-compile, but I am curious, why do we compile on installation? I wasn't able to find a clear reason in the original PR (from a long time ago).

finite perch Aug 16, 2024, 12:45 AM

#

I had the same idea, however it would require a significant refactoring to know ahead of time how many Python files need to be installed as to whether is worth it.

Where as currently the compile_file step happens at a point where it's knowable how many files are available, so it's easier to put in limiting heuristics.

hidden flame Aug 16, 2024, 12:46 AM

#

Yup.

finite perch Aug 16, 2024, 12:49 AM

#

I had planned to make a simple benchmark and test this all this week, but then I got distracted when I was digging through the code for this reported issue (https://github.com/pypa/pip/issues/12911) and found a big regex that can be eliminated (https://github.com/pypa/pip/pull/12917/files#diff-f9e3db06303cb2538cdc54229e6e76ee3b8821d399b655204923ca598c1f3cfc)

hidden flame Aug 16, 2024, 12:51 AM

#

Haha, I relate to that.

#

I try to not get too distracted as I find that working on a bunch of scattered things at once results in nothing being finished :P

#

Of course, that has the equally bad side-effect that I'm "procrastinating" reviewing the large PRs I've been meaning to review.

finite perch Aug 16, 2024, 12:53 AM

#

hidden flame I try to not get too distracted as I find that working on a bunch of scattered t...

I refuse to accept this consequence, might take me years to finish a small task, but I'll get there

rapid blaze Aug 16, 2024, 12:57 AM

#

Stay involved in open source long enough and you may get to see ideas you once had actually amount to something a decade or more later 🙂

hidden flame Aug 16, 2024, 12:59 AM

#

As both a maintainer and contributor, I feel conflicted when PRs languish. On the one hand, I acutely understand the reasons why PRs can stall and take forever to land, but OTOH, it is discouraging to see a PR (or worse, have your own PR) sit untouched for weeks or months on end.

finite perch Aug 16, 2024, 7:25 PM

#

GitHub issues needs a confused emoji reaction

hidden flame Aug 16, 2024, 7:55 PM

#

finite perch GitHub issues needs a confused emoji reaction

is that not a confused emoji?

#

If you hover over it, it says "x reacted with confused emoji"

fallen scroll Aug 16, 2024, 8:01 PM

#

It looks more sad than confused.

hidden flame Aug 16, 2024, 8:01 PM

#

Fair

finite perch Aug 16, 2024, 8:51 PM

#

hidden flame is that not a confused emoji?

On my screen it doesn't have eyebrows, which makes it look less confused and more unamused or disappointed

shy echo Aug 16, 2024, 9:01 PM

#

Each OS has different emoji designs. Plus, those evolve too.

https://emojipedia.org/confused-face#designs

Emojipedia

😕 Confused Face Emoji

A yellow face with open eyes and a skewed frown, as if scrunching its cheeks or chewing its lips. A look of feeling unsure.

While it can convey confusio...

ember shuttle Aug 17, 2024, 12:17 AM

#

I cycle through all of those in the course of a day.

finite perch Aug 17, 2024, 9:42 PM

#

It's not obvious windows is slow until you run a benchmark which involves creating a lot of files and processes, and then wow... :/

finite perch Aug 17, 2024, 10:08 PM

#

shy echo <@511013383101743110> Are you testing on Linux? IIRC, Linux has the lowest proce...

So yup, my benchmarking shows Windows would generally not benefit from using multiprocessing compile_file

#

So instead I've raised the question, why not turn off compiling by default: https://github.com/pypa/pip/issues/12920

inland creek Aug 18, 2024, 12:19 AM

#

finite perch It's not obvious windows is slow until you run a benchmark which involves creati...

Have you tried running them on dev drives?

#

Though that will sort of artificially inflate the benchmarks I guess

finite perch Aug 18, 2024, 12:22 AM

#

Yeah, I don't think I have that feature on Windows 10 anyway, and the idea was to look at the worst performance pip users could commonly face

ashen geyser Aug 18, 2024, 2:37 PM

#

shy echo Each OS has different emoji designs. Plus, those evolve too. https://emojipedi...

That used to really be the case until everyone started copying Apple's design, which in some cases is exceptionally ugly and doesn't fit the Unicode description at all. But Apple decided that this is the emotion that it really should be

finite perch Aug 18, 2024, 2:40 PM

#

Androids old Jelly Bean emojis certainly had a lot of character that don't exist any more: https://emojipedia.org/google/android-4.4

Emojipedia

Google Emoji List — Emojis for Android [Updated: 2024]

Every emoji supported by Android and other Google platforms such as ChromeOS, Meet, and Gmail. Compare Google emojis with Apple, Samsung and Windows. Update to ...

hidden flame Aug 20, 2024, 2:36 AM

#

If someone with the admin bit for pip could delete this issue that would be appreciated. Someone opened a bug report with their company account which turns out was not OK. They already opened a new one with their personal account. https://github.com/pypa/pip/issues/12905#issuecomment-2278927272

shy echo Aug 20, 2024, 7:06 AM

#

Deleted

pale epoch Aug 20, 2024, 6:18 PM

#

i made an issue providing context + an outline of all my metadata resolve work. i just converted all the open PRs to draft except for 3: https://github.com/pypa/pip/issues/12921#issuecomment-2299454418. i implemented parallel wheel downloads a couple days ago but the parallelism one is drafted as it depends on changes to progress bar output (one of the undrafted ones) which i expect we'll want to discuss since it drastically changes the way pip looks

GitHub

metadata resolve workstream · Issue #12921 · pypa/pip

What's the problem this feature will solve? The 2020 resolver separated the resolution logic from the rest of pip, and made the resolver much easier to read and extend. With --use-feature=fast-...

#

the parallel downloads stuff is totally separate from metadata caching and doesn't touch the requirement preparer logic. one of the open PRs does touch the preparer logic, but it's short and part 1 of 3 which cleans up a lot of the preparer logic to make the caching logic much less intrusive and complex

#

interested in anyone's input on any of them, including ones in draft mode. i believe all of these PRs are extremely high-quality work and all of them include thorough test cases and docstrings, i'm using the draft status to make it more clear for maintainers which would be helpful to focus on

pale epoch Aug 20, 2024, 6:53 PM

#

i was asking for input on mastodon on whether to download bigger wheels first for batched downloads and a friend just suggested making downloads resumable if interrupted which is a GREAT idea and should be orthogonal to the rest / not have any dependencies on other PRs

#

was linked to this issue for context https://github.com/pypa/pip/issues/4796

GitHub

[Improvement] Pip could resume download package at halfway the conn...

Pip version: 9.0.1 Python version: 3.6.2 Operating system: macOS 10.13 Description When I have poor internet connection (the network is cut unexpectedly), updating pip package is painful. When I re...

hidden flame Aug 20, 2024, 7:02 PM

#

pale epoch interested in anyone's input on any of them, including ones in draft mode. i bel...

I can't speak for the other maintainers, but it is on my to-do list to review your PRs. The blocker is that it all involves parts of pip that I'm wholly unfamiliar with. 😅 I've been reading and tracing through pip's codebase so I'll have the foundational knowledge needed to evaluate your changes.

pale epoch Aug 20, 2024, 7:07 PM

#

please don't feel rushed to do so at all, nothing has been bitrotting. sorry for that framing, i only meant to provide context on their readiness for review by non-maintainers in case they were curious or had any specific performance-sensitive use cases they wanted to ask about

#

i have been using a specific install command to benchmark resolve/download performance but until someone mentioned spotty downloads and resuming interrupted requests a few minutes ago i hadn't considered that angle at all

hidden flame Aug 20, 2024, 7:09 PM

#

There is an ancient PR for autoresumption of interrupted downloads, but I have no idea what state it is in.

pale epoch Aug 20, 2024, 7:10 PM

#

hidden flame There is an ancient PR for autoresumption of interrupted downloads, but I have n...

will check it out

hidden flame Aug 20, 2024, 7:10 PM

#

The feature is wanted, but the implementation may be not trivial as A) our cache backend may or may not play nicely with interrupted downloads, and B) partial downloads should probably not persist in the cache forever.

pale epoch Aug 20, 2024, 7:11 PM

#

yes! 🙂 i was almost upset bc it's obv useful but expect it to be a lot of work and i thought i was done for now

hidden flame Aug 20, 2024, 7:12 PM

#

The caching story should improve by the start of 2025 as someone has committed to contributing to pip on related changes which involves the cache, and it was suggested they improve the caching management story.

pale epoch Aug 20, 2024, 7:13 PM

#

would def recommend they read through the comments on my resolve workstream thread then as i go into the rationale for why metadata caching is likely to be more appropriate than e.g. parallelism and how it relates to pip use cases (CI vs laptop etc)

#

hmm unless you mean caching of big downloads

hidden flame Aug 20, 2024, 7:14 PM

#

The thing about parallelism is that it's cool and all, but pip currently has zero parallel logic. So, it really should be approached separately as it has its own host of edge cases to consider.

pale epoch Aug 20, 2024, 7:14 PM

#

the metadata caching PRs add new cache subdirs

pale epoch Aug 20, 2024, 7:14 PM

#

hidden flame The thing about parallelism is that it's cool and all, but pip currently has *ze...

yes this was my argument too

hidden flame Aug 20, 2024, 7:15 PM

#

pale epoch would def recommend they read through the comments on my resolve workstream thre...

@limber ore

pale epoch Aug 20, 2024, 7:15 PM

#

this issue https://github.com/pypa/pip/issues/12921

limber ore Aug 20, 2024, 7:23 PM

#

I don't have time to look right now but if the caching has not improved then in Q4 I do plan on making PRs for an experimental new type that stores wheels unpacked (like UV)

#

parallelism is still useful but I haven't accounted to spend time on that other than reviews because there is an open PR

hidden flame Aug 20, 2024, 7:23 PM

#

if only there were 72 hours in a day

finite perch Aug 20, 2024, 7:26 PM

#

My intuitive feeling is the caching of metadata files (also something uv does) will have a bigger effect on user feeling of snappiness for most users using PyPI, than caching unpacked wheels, it's near the top of my list to try cosmicexplorers PR that implements this

pale epoch Aug 20, 2024, 7:27 PM

#

pale epoch yes this was my argument too

was doing dishes but actually my argument is a little different; i actually do not believe parallelism (except for parallel downloads of large files, which i have already implemented) is likely to achieve any performance that caching cannot

hidden flame Aug 20, 2024, 7:28 PM

#

Also, parallelizing the initial index page fetch probably help a fair bit with cold but easy resolves during CI builds.

#

Of course, CI systems generally have low-latency connections anyway so this is mostly moot :)

pale epoch Aug 20, 2024, 7:28 PM

#

hidden flame Also, parallelizing the initial index page fetch probably help a fair bit with c...

actually you can cache this too https://github.com/pypa/pip/pull/12257

hidden flame Aug 20, 2024, 7:29 PM

#

I believe the vast majority of people do not use pip caching in CI.

#

But this is mostly theoretical discussion, caching comes first and then parallelization can come second.

pale epoch Aug 20, 2024, 7:30 PM

#

limber ore I don't have time to look right now but if the caching has not improved then in ...

i interviewed for astral and they liked me until i made the mistake of asking about their monetization strategy lol. you actually can store wheels fully zipped and use them for caching, i did it for pex https://github.com/pex-tool/pex/pull/2175

pale epoch Aug 20, 2024, 7:31 PM

#

hidden flame I believe the vast majority of people do not use pip caching in CI.

yes, the idea is that the metadata cache can be stored separately from the wheel cache so CI systems can avoid large cache dirs but still achieve resolve perf

#

totally open to parallelization too but this caching approach would also reduce PyPI bandwidth costs

hidden flame Aug 20, 2024, 7:32 PM

#

Pip currently stores downloaded wheels as cachecontrol (i.e. msgpack) HTTP responses. A small optimization that could be done to store the wheels directly, instead of HTTP responses to avoid decoding. I have no idea how much this would help, but it may have a nontrivial impact on large (and I really do mean large) wheels.

limber ore Aug 20, 2024, 7:33 PM

#

pale epoch i interviewed for astral and they liked me until i made the mistake of asking ab...

not with binary extensions

pale epoch Aug 20, 2024, 7:33 PM

#

hidden flame Pip currently stores downloaded wheels as cachecontrol (i.e. msgpack) HTTP respo...

will definitely investigate this

pale epoch Aug 20, 2024, 7:33 PM

#

limber ore not with binary extensions

hm how so?

finite perch Aug 20, 2024, 7:34 PM

#

One positive side effect of storing HTTP Responses is the names are all hashes, wheel names can sometimes get so long they exceed Windows file path length when in a directory that's also a little long, something to be aware of when touching

limber ore Aug 20, 2024, 7:34 PM

#

extension modules must be loaded from disk

hidden flame Aug 20, 2024, 7:34 PM

#

finite perch One positive side effect of storing HTTP Responses is the names are all hashes, ...

ah yeah, right, windows

#

🙃

#

nothing can be considered a small change here

pale epoch Aug 20, 2024, 7:37 PM

#

limber ore extension modules must be loaded from disk

i was mistaken earlier, i believe pex also stores all wheels unpacked for this reason. the reason my technique applied there was because pex was interested in reconstituting zipapps from cached zips, so there was an optimization possible to avoid decompressing entries. sorry about that and very interested in your work on this

pale epoch Aug 20, 2024, 7:38 PM

#

finite perch One positive side effect of storing HTTP Responses is the names are all hashes, ...

pants and spack regularly overrun maximum file path lengths on linux too, i'm the first one to bash windows but this happens a lot in CI

#

not sure if it would ever be useful for pip itself but have been doing a lot of work on the rust zip crate after getting really into zip file stuff for pip https://github.com/zip-rs/zip2/pull/236. this PR makes zip file extraction much much faster and i haven't been able to find any prior art for similar techniques

hidden flame Aug 20, 2024, 11:16 PM

#

pale epoch not sure if it would ever be useful for pip itself but have been doing a lot of ...

There is a PR for pip that implements custom zip extraction code to speed up installs, but once again, it's a lot of complexity for pip to take on. None of us are experts on the ZIP format (AFAIK) and we don't want to (half) implement another zip library in pip.

azure heron Aug 20, 2024, 11:44 PM

#

This is interesting... https://github.com/astral-sh/uv/issues/6279

finite perch Aug 20, 2024, 11:57 PM

#

I'm a bit confused about what the user wants to happen, and if it happens with the latest pip / pip-tools or not

pale epoch Aug 21, 2024, 12:02 AM

#

hidden flame There is a PR for pip that implements custom zip extraction code to speed up ins...

thanks! makes sense. definitely excited to see @ofek's work on caching and wheel extraction, i'm not as familiar with performance of python loading or how to rearrange stuff outside of wheels

#

oh yeah for example the metadata caching stuff currently writes each dist's metadata into its own file, which is then gzipped but that's still likely not very space-efficient. could probably save a lot of space by merging metadata for dists from the same project together. that may also be faster

hidden flame Aug 21, 2024, 2:06 AM

#

I just added some additional ram to my laptop, and I was curious to how running the pip test suite performs. I honestly don't remember the previous performance so I can't say if it's better, however, my god are the keyring tests awfully slow. These results are on a 2021 8-core AMD CPU:

pytest -n8 ~> 518.62s (0:08:38)
pytest -n8 --dist=worksteal ~> 383.90s (0:06:23)
pytest -n8 --dist=worksteal -k "not keyring" ~> 204.82s (0:03:24)

There are a lot of expensive tests (the keyring tests, ahem) and they seem to be often crammed on a few cores to a single core in the last third of the run which is inefficient, the worksteal distribution method seems to mitigate this fairly well. And yup, at least on my system, the keyring tests seem to nearly double the test suite runtime...

azure heron Aug 21, 2024, 3:27 AM

#

finite perch I'm a bit confused about what the user wants to happen, and if it happens with ...

Basically they want the packages following an invalid extra to be ignored, in addition to the invalid extra declaration.

finite perch Aug 21, 2024, 3:47 AM

#

azure heron Basically they want the packages following an invalid extra to be ignored, in ad...

I don't think the front end installer should do that without a spec, btw, apache-airflow declares a development extra that they don't export to their metadata, I think they do that through some build magic, but I've not looked at it

#

I can sit down and take a look some time later tomorrow, I got a backlog of user tickets I know I can help people with this week I haven't had the chance to get to

rapid blaze Aug 21, 2024, 5:44 AM

#

That "private extra" weirdness sounds to me like a project that should be using a dependency management tool with support for declaring unpublished dependency groups...

rapid blaze Aug 21, 2024, 5:46 AM

#

hidden flame if only there were 72 hours in a day

Europa has ~84, but the comms latency is brutal 🙂

hidden flame Aug 25, 2024, 3:22 AM

#

FYI, I do plan on providing an additional write-up for https://github.com/pypa/pip/issues/11457 so any people that encounter the deprecation warning and follow the link will hopefully have an easier time understanding what it means and what to do. I was going to do that yesterday, but I got side-tracked into writing a larger blog post about pip 24.2 (because it's been a while since I wrote something and this release has a bunch of cool changes... I'm well aware it would be nearly a month late). :P

GitHub

Deprecate `pip install --editable` calling `setup.py develop` · Iss...

There is now a standardized mechanism for an installer like pip to request an editable install of a project. pip is transitioning to using this standard only instead of invoking the deprecated setu...

hidden flame Aug 26, 2024, 4:16 AM

#

OK, while it's still a draft, the blog post is in a state where I'm happy to share it for feedback. The optimization section is incomplete, but otherwise, it's mostly fully written. https://hugo-draft.floralily.dev/blog/2024/08/whats-new-in-pip-24.2/

What's new in pip 24.2 | Richard Si

In version 24.2, pip learns to use system certificates, receives a handful of optimizations, and deprecates legacy editable installations.