general

[lyric hedge]

lol, 1800 PRs would be brutal

I know that there may be a pyproject.toml nested somewhere, etc. etc.

[brittle token]

Huh, I just realised that https://github.com/jazzband/contextlib2 is probably one of the legacy projects in the first set of numbers (no pyproject.toml, no release in the last 12 months, but I'm not sure if it's still in the top 10k or not)

[steady lion]

Looking for a pyproject.toml anywhere in the tree only gets you down to 1,831

[lyric hedge]

fun

well this is why I have six months to do this :P

[steady lion]

cat .\extracted\co\contextlib2\__sdist_metadata__.json
{
"url": "https://files.pythonhosted.org/packages/c7/13/37ea7805ae3057992e96ecb1cffa2fa35c2ef4498543b846f90dd2348d8f/contextlib2-21.6.0.tar.gz",
"filename": "contextlib2-21.6.0.tar.gz",
"upload_time": "2021-06-27T06:54:40.841178Z",
"download_count": 7207106,
"rank": 864
}

[lyric hedge]

I'm not actually going to file that many PRs. I imagine there are lot of projects where I won't/can't figure out what their build-system setup needs are.

Is this the top 10,000 packages?

[steady lion]

To be honest, I would also only focus on the top 1,000

[lyric hedge]

mhmm

[steady lion]

There is such a long tail

Top 1k: 46,399,709,821 downloads (89.28%)
Top 2k: 49,257,082,936 downloads (94.78%)
Top 3k: 50,297,551,272 downloads (96.78%)
Top 4k: 50,798,677,480 downloads (97.75%)
Top 5k: 51,111,267,932 downloads (98.35%)
Top 6k: 51,338,091,521 downloads (98.79%)
Top 7k: 51,496,118,123 downloads (99.09%)
Top 8k: 51,609,999,060 downloads (99.31%)
Top 9k: 51,696,951,609 downloads (99.48%)
Top 10k: 51,766,085,227 downloads (99.61%)
Top 11k: 51,821,571,091 downloads (99.72%)
Top 12k: 51,867,593,290 downloads (99.80%)
Top 13k: 51,906,710,205 downloads (99.88%)
Top 14k: 51,939,961,957 downloads (99.94%)
Top 15k: 51,969,188,189 downloads (100.00%)

[brittle token]

Eh, I'm gonna blame OpenStack for that download rank still being so high 🙂

[lyric hedge]

man, this is all very cool

[steady lion]

This is using the numbers from Hugo's JSON file, but looking at PyPI's dashboard, the total at 15k and the total for all PyPI downloads are near identical

So looking only at the top ~2k gives you 95% coverage

[lyric hedge]

all I've done is this sad script to check the wheel filenames on PyPI as we also deprecated non-standard filenames: https://paste.pythondiscord.com/E3QQ

[steady lion]

Filtering again to the top 2,500 projects and released in the last year:

Total projects: 1,682
Pure-python projects: 1,417
Projects with no PKG-INFO: 0
Projects with no pyproject.toml: 360
Projects with no [build-system] table: 164

[lyric hedge]

ah, that's not nearly as bad

if you could send me a copy of the modified script you're using to get these statistics, I'd greatly appreciate that

[steady lion]

The script is

import datetime as dt
import json
from pathlib import Path

NOW = dt.datetime.now(tz=dt.UTC)

projects = list(Path('extracted').glob('*/*'))

recent_projects = 0
count_pure = 0
no_pkg_info = set()
no_pyproject = set()
no_build_system = set()

for p in projects:
    metadata = json.loads(p.joinpath('__sdist_metadata__.json').read_bytes())
    date = dt.datetime.fromisoformat(metadata['upload_time'])
    if metadata['rank'] > 2500:
        continue
    if (NOW - date).days > 365:
        continue
    recent_projects += 1

    pure_python = True
    pyproject = None
    for f in p.rglob('*'):
        if f.suffix in {'.c', '.cc', '.cs', '.c++', '.cpp', '.f', '.go', '.java', '.rs', '.s'}:
            pure_python = False
        if f.name == 'pyproject.toml':
            pyproject = f
    if not pure_python:
        continue
    count_pure += 1
    if not p.joinpath('PKG-INFO').is_file():
        no_pkg_info.add(p.name)
    if p.joinpath('pyproject.toml').is_file():
        pyproject = p.joinpath('pyproject.toml')
    if pyproject is None:
        no_pyproject.add(p.name)
        continue
    if not b'[build-system]' in pyproject.read_bytes():
        no_build_system.add(p.name)
        continue

print(f'Total projects: {recent_projects:,}')
print(f'Pure-python projects: {count_pure:,}')
print(f'Projects with no PKG-INFO: {len(no_pkg_info)}')
print(f'Projects with no pyproject.toml: {len(no_pyproject)}')
print(f'Projects with no [build-system] table: {len(no_build_system)}')

Just run at a local checkout of https://github.com/AA-Turner/top-pypi-sdists-full

[steady lion]

The download numbers here are from:

import json; from operator import itemgetter; from pathlib import Path
data = json.loads(Path('top-pypi-packages.min.json').read_bytes())
total = sum(map(itemgetter('download_count'), data['rows']))
for x in range(1, 16):
    count = sum(map(itemgetter('download_count'), data['rows'][:x*1000]))
    print(f'Top {x}k: {count:,} downloads ({count/total:.2%})')

[lyric hedge]

It's only going to double the size of my user drive :P

[steady lion]

I currently have one checkout in WSL and one normally, which doesn't help

[lyric hedge]

OK, I lied. I was looking at system drive, but still:

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p6  159G   84G   68G  56% /home

[steady lion]

I also have https://github.com/AA-Turner/top-pypi-sdists-2000, which is like the -full version, but searchable on GitHub -- they won't index the 10k version!

Again, this still covers ~95% of total downloads

[lyric hedge]

Yeah, I'm not going to need all of the top 10,000

You must be filtering a lot of data files out. The top 2000 repository is tiny!

[steady lion]

The filtering is quite agressive

We only keep files under 5MB that are 'potentially interesting' or metadata files (PKG-INFO/pyproject.toml)

(basically, only keeping source code files, and removing all directories that don't look interesting/relevant)

[lyric hedge]

Traceback (most recent call last):
  File "/home/ichard26/dev/oss/top-pypi-sdists-2000/stats.py", line 17, in <module>
    metadata = json.loads(p.joinpath('__sdist_metadata__.json').read_bytes())
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python3.12.4/lib/python3.12/pathlib.py", line 1019, in read_bytes
    with self.open(mode='rb') as f:
         ^^^^^^^^^^^^^^^^^^^^
  File "/opt/python3.12.4/lib/python3.12/pathlib.py", line 1013, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'extracted/jellyfish/benchmarks/__sdist_metadata__.json'

Hmm, there doesn't seem to be any metadata in the 2000 dump.

find extracted/ | grep "__sdist" also returns nothing

[steady lion]

I only added it to the full one an hour ago!

It's regenerating now: https://github.com/AA-Turner/top-pypi-sdists-2000/actions/runs/14698014663/job/41242560952

[lyric hedge]

ah

[steady lion]

Try git reset --hard && git fetch --all && git pull --rebase now

[lyric hedge]

AFK, will be back later

[lyric hedge]

Still running into the same crash. Not every project has a metadata file.

[steady lion]

Are you able to try with -full?

[lyric hedge]

sure, gimme a few minutes as my download is not that fast

Actually, at 2MB/s, 13+ minutes.

find extracted/ | grep "__sdist" | wc -l
1446

Only 3/4 of projects have metadata.

Oh, there are only Processed 1446 sdists (146.878s) projects anyway. https://github.com/AA-Turner/top-pypi-sdists-2000/actions/runs/14698014663/job/41242560952

[steady lion]

I misnamed the repo, it's actually the top 1500

It took some tinkering to find the limit where GH would stop indexing for code search

Sorry for the confusion

Does the script itself work? Or do you still get OSErrors

[lyric hedge]

Total projects: 1,043
Pure-python projects: 872
Projects with no PKG-INFO: 0
Projects with no pyproject.toml: 228
Projects with no [build-system] table: 99

if I skip the packages w/o sdist metadata, yeah

It seems like all of the projects that are NOT nested under a prefix directory do not have the sdist metadata. The projects that are nested aren't being skipped.

[steady lion]

Oh I see what happened

[lyric hedge]

I modified the script to handle the mixed nesting already, FWIW.

[steady lion]

I added the prefix scheme for the top 10k to reduce individual directory size, but seemingly the old directories weren't properly deleted when I synched it to -2000.

Let me delete them manually now, sorry again for the teething trouble.

[lyric hedge]

find extracted/ -type d | grep "/rich/"
extracted/ri/rich/rich
extracted/rich/rich

Ah yup, you're right.

[steady lion]

The prefixed directories should all have the metadata

[lyric hedge]

I do wonder where 500 projects went though 🤔

[steady lion]

^

[lyric hedge]

My total is 1043. Is the repository supposed to be named top-1000?

[steady lion]

Oh!

[lyric hedge]

Well if you were planning to share these to DPO later, it's good to sort out these issues beforehand. Saves you some embarrassment :)

$ cd top-pypi-sdists-full
[WARN] - (starship::utils): Executing command "/usr/bin/git" timed out.
[WARN] - (starship::utils): You can set command_timeout in your config to a higher value to allow longer-running commands to keep executing.

whelp

[steady lion]

I had to disable starship too

[lyric hedge]

With the full repository, I'm getting the same results so that part looks good.

Total projects: 1,682
Pure-python projects: 1,417
Projects with no PKG-INFO: 0
Projects with no pyproject.toml: 360
Projects with no [build-system] table: 164

[steady lion]

Though I'm on Windows, which is known for slower small-file / git operations

[lyric hedge]

oh wait, that's why I'm missing 500 packages, the script filters out any that haven't cut a release in the past 365 days

[steady lion]

So sans-filters the numbers are as expected?

[lyric hedge]

Mhmm. Yet another case of PEBCAK 😅

[steady lion]

Well, it proves the metadata is working...

[lyric hedge]

   6. certifi
  12. grpcio-status
  15. s3fs
  16. six
  34. jmespath
  39. pytz
  49. google-auth
  68. requests-oauthlib
  77. oauthlib
  80. requests-toolbelt
  86. google-cloud-storage
  96. et-xmlfile
  97. openpyxl
 105. python-dotenv
 112. google-cloud-core
 113. sortedcontainers
 118. websocket-client
 125. google-resumable-media
 126. asn1crypto
 129. smmap
 132. gitdb
 133. msal
 140. wcwidth
 144. paramiko
 155. fastjsonschema
 171. google-api-python-client
 174. asgiref
 176. pkginfo
 179. google-auth-oauthlib
 180. py4j
 184. xmltodict
 187. types-requests
 192. google-auth-httplib2
 200. msal-extensions

Within in the top 200 for no pyproject.toml at all.

I filed a PR for certifi earlier today. That's what gave me the idea to start this investigation in the first place.

 187. types-requests
 247. types-python-dateutil
 502. types-pytz
 595. types-urllib3
 739. types-redis
 792. types-paramiko
 830. types-tabulate
 849. types-deprecated
 905. types-aiofiles
 926. types-docutils
 956. types-dataclasses
1010. types-pymysql
1090. types-pyopenssl
1099. types-cachetools
1167. types-toml
1187. types-cffi
1408. types-jsonschema
1486. types-pillow

hmm, where are the templates for typeshed packages ?

[steady lion]

There's lots of monorepo-type projects, e.g. azure, aws, alibaba, typeshed that lack pyproject

[steady lion]

https://github.com/typeshed-internal/stub_uploader/pull/165

[lyric hedge]

Ah nice, they simply need to cut new releases.

OK, OK. I should really go $actual-work now, but this has been cool. I'll definitely burn some hours tomorrow hacking on this :)

[lyric hedge]

https://github.com/PyCQA/pyflakes/issues/793#issuecomment-1837484687 *sigh*

sometimes I wish the default pyproject.toml didn't exist

[mystic stag]

Make a PEP to depricate / remove it 😄

[lyric hedge]

if you want me to leave the ecosystem forever, sure why not!

I 100% expect an insane amount of pushback and I have zero patience or energy for that these days

[fading veldt]

Yeahh I have a hell of a lot of respect you're finally tearing off the bandaid

Also I forgot that pyproject.toml files don't need a [build-system] table, so a check on PyPI would actually just be checking for the existence of the file.

[lyric hedge]

???

[lyric hedge]

I mean, technically this is valid, but I absolutely do not want to encourage this. Python packages in 2025 should be declaring their build system.

[fading veldt]

Oh neither do I!

[brittle token]

Huh, maybe the starting point here should be to update setuptools to inject pyproject.toml into the built sdist? So even if it isn't present in the source repo, it will be present in the sdist. That way only building from VCS URLs and from old PyPI sdist releases would break.

[lyric hedge]

I thought they used to do that?

[fading veldt]

People relying on fallback pyproject.toml behavior is a bandaid over not having a pyproject.toml

[lyric hedge]

OK, maybe not. I thought I'd read something to this effect in the changelog earlier today.

[brittle token]

Not sure, but if they took it out (or never did it), then pip dropping the client side fallback would be a decent motivation for adding a publishing side one. I do wonder if this should be a PEP, though - there are certainly significant ecosystem level ramifications for the change.

[fading veldt]

I think this just moves where the breakage will be seen (which might be good!). A number of projects rely on not having a pyproject.toml to have implicit --no-build-isolation

[lyric hedge]

We aren't removing this bandaid.

We are moving to forcing PEP 517 processing (and yes, build isolation) for all builds starting in pip 25.3.

I actually expect the PEP 517 part to not cause that much breakage. The build isolation 100% will.

[fading veldt]

Sorry, poor wording on my part but that is what I meant

[brittle token]

So a missing pyproject.toml and a blank pyproject.toml will both still be interpreted as "build with setuptools"? That's a much less drastic change than I thought you were considering.

[lyric hedge]

Yes.

IIRC this is also spelled out in PEP 517. It's untenable to drop this fallback.

It's just there's another fallback to setup.py bdist_wheel that we want to kill.

Sigh, is the deprecation issue not clear enough?

I spent so many cycles with the legacy editable install trying to explain what really is deprecated, and I don't think anyone actually understands what's being deprecated.

[brittle token]

All good, this one's on me - I thought I knew what you meant, so I didn't click through to the issue description. I did not, in fact, know what you meant.

[fading veldt]

Ah yeah it's this section in PEP 518 https://peps.python.org/pep-0518/#build-system-table

Because the use of setuptools and wheel are so expansive in the community at the moment, build tools are expected to use the example configuration file above as their default semantics when a pyproject.toml file is not present.

Tools should not require the existence of the [build-system] table. A pyproject.toml file may be used to store configuration details other than build-related data and thus lack a [build-system] table legitimately. If the file exists but is lacking the [build-system] table then the default values as specified above should be used. If the table is specified but is missing required fields then the tool should consider it an error.

[lyric hedge]

Sorry, PEP 518 is the one

[fading veldt]

I think I may have made assumptions about the deprecation when I read it

So that is on me

[lyric hedge]

Should I explicitly call out that pip will assume a default pyproject.toml once PEP 517 processing is enforced? I implicitly said this in the editable install deprecation issue, but I'm unconvinced it will alleviate any fears.

I did mean to include a "here's the explanation of the larger context" section in the deprecation issue but it was past midnight when I finished the more simplified part of it, so I never got around to writing that part.

[fading veldt]

Let me read through the issue again and I'll be able to give more detailed accurate feedback

[lyric hedge]

Fundamentally, this involves some low-level technical crap so I'm rather pessimistic about being able to explain anything properly here. I instead focused on giving clear, actionable advice so people felt like at least they had a path forward.

[fading veldt]

I think this sentence is maybe confusing:

This deprecation only affects projects that are packaged with setuptools and not have been updated to use the modern PEP 517 mechanism, which is enabled by the presence of pyproject.toml.

[lyric hedge]

Yeah, that's fair.

Go and read it. I'll rewrite the whole issue tomorrow.

[fading veldt]

Sorry to make work for you 🤦‍♀️

[lyric hedge]

It's fine, I'm just really frustrated with it all

I know the editable install issue is still terrible. I should probably rewrite it again.

Thanks for taking a look. Should've gotten a second opinion way earlier.

[fading veldt]

I totally understand the frustration. Don't beat yourself up about the second opinion though. It's the curse of open source maintainership to be siloed until you start talking about breaking things 🫠

(regardless of how much you try to warn users)

[queen hornet]

You found about 60% with pyproject.toml in the top 10k, https://py-code.org/stats shows a bit lower proportion of ~50% across all PyPI

[brittle token]

@lyric hedge It's definitely a hassle to write user facing release notes for "down in the weeds" technical changes that ideally wouldn't be noticed, but definitely have the risk of causing problems that are visible to users. I'm wondering if something like the following might work:

The pyproject.toml specification covers how to build projects that include the file without specifying the optional [build-system] table. pip now also uses this standard fallback mechanism when the pyproject.toml file is omitted entirely (replacing its previously deprecated non-standard ad hoc fallback logic for that case). Most projects which omit pyproject.toml will continue to build without problems after this change, but it's possible some projects will fail to build when using the standardised fallback mechanism (for example, due to undeclared build dependencies).

[oak quiver]

not sure what you mean? there's default behaviour for projects that lack one, I suppose, but I don't think it's specified in terms of default values for contents (except e.g. the [build-system] table for projects that do have one but not that table)... ?

(looks like there's quite a bit here for me to catch up on)

[fading veldt]

@oak quiver see this

[oak quiver]

The problem "how do you get Pythonistas in general to migrate to a new, better way of doing things en masse" has been an unsolved problem for about 14 years now (accounting for 3.0 and 3.1 having significant issues that made them not fully usable from my recollection)

right, so you were indeed talking about the [build-system] table.

[oak quiver]

yes, because by default it will end up trying to build projects with fresh Setuptools versions that are dozens of versions newer than the project itself.

(that's basically what the Setuptools 78 debacle was about)

[blazing lantern]

I'm giving a keynote on pylock.toml at EuroPython and then there will be an accompanying blog post afterwards

[lyric hedge]

sounds good! I think I accidentally made some more people aware of pylock.toml via the pip 25.1 release, so it'd be good to make sure there are good resources out there

[blazing lantern]

I don't know if I would go that far. One interesting side-effect of the index API having an upload-time is you could say that for any sdist created after a certain date, the lack of [build-system] means one isn't defined instead of falling back. Add in a long period for tools to warn about this and then another long time before the date requirement simply goes away and it should trickle out to only be projects that are no longer receiving updates anyway.

[lyric hedge]

I said that I wouldn't consider dropping the default build-system table fallback.

Did I misspeak?

[blazing lantern]

Nope, I'm just saying I have thought about dropping the default and I think it's feasible

I just don't have the packaging time to drive such an endeavour

[lyric hedge]

Ah, you meant the other way.

Yea, I don't either.

I want these setup.py deprecations to be over and then stop with the deprecations for the time being.

[inland prawn]

there will always be deprecations and removal schedules as long as software will be written

[lyric hedge]

oh yeah, but it's tiring to be constantly doing them

[mystic stag]

How it feels right now

[lyric hedge]

three seconds later: kaboom 🧨

[lyric hedge]

progress so far, the script seems to work okay!

[lyric hedge]

Projects where --use-pep517 succeeds: 207
Projects where --use-pep517 fails: 7
Projects that don't build at all: 15

that's not bad at all! I'll need to do this again with more projects as I'm still trialing this code, but this is encouraging.

[fading veldt]

That's pretty good!

[lyric hedge]

it only takes a few minutes to check the top 1500 projects, filtered by those that are Pure Python and had a release in the last year. I guess I'll have to expand this to the full 10000 dataset!

[steady lion]

I've opened https://github.com/youtype/mypy_boto3_builder/issues/358 for the mypy-boto3 and types-boto3 projects, c. 400 in total

Other prefix-offenders are django (121), google (106), azure (90), pytest (67), flask (36), flake8 (34), dagster (33), colcon (22), oslo (20), sqlalchemy (19), mkdocs (18), edx (17), aws (16), robotframework (15), requests (14), zope (14), streamlit (13), dash (12), djangorestframework (12), sphinx (12), databricks (11), json (11), alibabacloud (10), drf (10), opencensus (10), aliyun (9), py (9), autogluon (9), jupyter (9), click (8), dbt (8), grpcio (8), nbdev (8), pandas (8)

With Google, Azure, AWS, Alibaba, etc, it's probably possible to create a single request that they update templates, but I haven't investigated the level of co-ordination between e.g. every django- or pytest- prefixed project, I imagine it's much lower, though

[oak quiver]

the real extremist approach is to just remove the extant non-conforming packages from the index

[oak quiver]

more seriously: I endorse this sort of idea. It would have the side benefit of getting new projects to actually think about the choice of build backend, too.

[lyric hedge]

Thanks! I was planning to be more generous with the explanation though. I don't want to assume that the projects know what PEP 517 is or even what pyproject.toml is. If I'm going to filing issues/PRs on mass, I want to explain what I'm up to avoid confusion and backlash.

In addition, I haven't decided what to do for the projects that build correct and will continue to build once --use-pep517 is always on. Technically, they don't need to add a pyproject.toml, although it'd be nice to get them to stop depending on the compatibility fallback spelled out in PEP 518.

I'm just wary of any potential backlash. I suppose for these projects, I could offer/propose that they adopt pyproject.toml, explain the benefits, but clearly indicate that it is not mandatory so they won't feel forced to do something

[lyric hedge]

OK, so I had some deficiencies in my script. I'd passed --no-index to avoid extra downloading, but I forgot about setup_requires. I also had some corrupted sdist downloads for some reason. Fixing both of those issues results in even better stats:

Projects where --use-pep517 succeeds: 225
Projects where --use-pep517 fails: 0
Projects that don't build at all: 4

Time to try with the top 10000 packages I suppose!

[lyric hedge]

Projects where --use-pep517 succeeds: 1739
Projects where --use-pep517 fails: 16
Projects that don't build at all: 78

Looking at the error logs for all 16 projects where --use-pep517 fails, it's due to declared (via setup_requires) build-time dependencies that I didn't predownload.

These results aren't too surprising, though. Either you declare your build-time dependencies which means PEP 517 processing (w/ build isolation) should not fail your build, or you don't declare them, and the build is going to fail now and later.

Seems like the setuptools PEP 517 implementation is largely backwards compatible with setup.py bdist_wheel so most of my attention will be spent on getting projects to properly declare their build dependencies.

[oak quiver]

someone could write a really good blog article promoting the idea and then that could get shared around?

[lyric hedge]

that blog post may already exist! although I haven't tried looking

[oak quiver]

I'm certainly interested in writing that sort of thing, but I might need someone else to define the scope for me :/

[oak quiver]

afaik the problematic cases are the ones where there are dependencies required in order to execute setup.py, before setup actually gets called

which I guess is rare enough to explain why it took so long to get fed up with the legacy approach...

[hollow trout]

Why even bother removing the fallback? Isn't it basically just supplying a default in a dictionary lookup?

Doesn't seem like a good use of anyone's time.

[lyric hedge]

yup, that'll be included in the "projects that don't build at all" category

[lyric hedge]

what fallback? We're removing the setup.py bdist_wheel fallback, not the default pyproject.toml fallback.

[oak quiver]

Because the default doesn't provide a way to specify a version of Setuptools, which recently has mattered a few times and potentially theoretically matters with every new Setuptools release.

(There may be other reasonable ways to address that - for example, choosing a version based on the distribution's release date.)

[hollow trout]

I was referring to the (hypothetical) removal of the ability to build setup.py-only projects via the setuptools.build_meta:legacy backend (which is what I believe @oak quiver is talking about).

[lyric hedge]

That isn't what we're deprecating though?

[hollow trout]

I know. 🙂

[lyric hedge]

well now I'm confused

I'll leave the conversation, sorry!

[hollow trout]

How many projects set a specific version of setuptools, though?

[oak quiver]

That's the point. They don't, but their build can get broken by later versions. Which Pip and other installers will automatically pull in for isolated PEP 517 buids.

Oh, you mean among projects that do have pyproject.toml? Well, they're much less likely to break, because breaking changes in Setuptools tend to be oriented around trying to clean up legacy stuff (and failing and reverting the changes because it still breaks too much...).

New projects would probably do well to upper-cap Setuptools IMO; other build backends such as Flit explicitly recommend doing so in their documentation, and the usual reasons against upper caps shouldn't apply (as far as I can tell) to isolated build environments.

[high stone]

In case you missed it, today's the last day to sign up for and propose topics for https://discuss.python.org/t/packaging-summit-at-pycon-us-2025/87031/

[mild dew]

PSA: wanted to share some cool work my colleague Alexis did to slash PyPI's test suite runtimes by 80% 🙂

https://blog.trailofbits.com/2025/05/01/making-pypis-test-suite-81-faster/

[lilac tree]

Very nice writeup, thanks! our already parallelized pytest job went from ~3min to ~2min (1000 tests) because I enabled the experimental coverage 👍 🚀

[jovial agate]

Hello. Sorry if this is a common question, but I’m just wondering how big the warehouse/pypi repository would be after syncing? I need an offline copy for a school project.

[uneven karma]

https://docs.github.com/en/rest/repos/repos?apiVersion=2022-11-28#get-a-repository has a size attribute, which is described as --

The size of the repository, in kilobytes. Size is calculated hourly. When a repository is initially created, the size is 0.
If you look at https://api.github.com/repos/pypi/warehouse, the size is 407760.
That's ~407mb, just under half a gb.

[jovial agate]

That’s all of the packages themselves?

[uneven karma]

Ah, you don't want the GitHub repository, you want to mirror all packages on PyPI?
Check out https://pypi.org/stats/, you would need 27.3 terabytes

[jovial agate]

OK cool. That makes way more sense lol

I guess I’ll have to have a think about how to cache them. Essentially I have students each with their own micro VM to work with. I’d like them to be able to “pip install” stuff. The whole VM and system is completely offline and cannot connect to the internet.

[uneven karma]

https://github.com/pypa/bandersnatch

You can configure which packages you want it to sync

See-also: #bandersnatch

[jovial agate]

Amazing. Thank you for your help, @uneven karma

[opal wolf]

@mellow cargo how long are you at PyCon? I meant to find you to chat about the thing you shared at packaging summit but the organizers were too good at packing the schedule with interesting topics 🙂 I'm around for all four days of sprints

[mellow cargo]

I will stay for 2 days in the sprints!

[frigid jewel]

any sprints around the PyPA domain that i should be aware of?

[opal wolf]

it sounds like there's going to be a good bit of wheelnext sprinting, I'll probably be in that room in some fashion

[fading veldt]

Great!

[high stone]

Whomever gets to it first -- can we block a room for packaging overall, like we did last year?

I'm not sure how/where we're tracking the sprint room allocations, and it'd be good to get all packaging folks in one room.

Preferably the room next to CPython please. 🙃

[fading veldt]

Looks like the form to submit sprints is https://us.pycon.org/2025/sprints/project/submit/

[opal wolf]

FYI there is an ad-hoc intro to sprints, 6-7 in 310-311, I don't know what happened to the official one but it would be good to have people there to announce sprints (though I will be there and announce it for packaging if nobody else does).

[fading veldt]

The official one seems to have been canceled?

[high stone]

(thread!)

[opal wolf]

Is someone submitting the packaging sprint to the website or should I do so?

submitted something, feel free to DM me edits to the content

[high stone]

I did in on the in person board.

Room 317

[opal wolf]

Are we sharing with WheelNext / should we?

[high stone]

Making a thread for PyCon US 2025 sprints to stop spamming people with notifications...

[mystic stag]

It was great meeting you all at PyCon ❤️

[echo ruin]

Is there a reason importlib.metadata.distributions() doesn't appear in the importlib docs? It's a bit hard to answer "what's the current API for iterating over all installed distributions" on the page, is that function (still?) the current way to do so?

[fading veldt]

Perhaps you are looking for https://docs.python.org/3/library/importlib.metadata.html#importlib.metadata.packages_distributions ?

[echo ruin]

I saw that -- I can use it, but it seems unnecessary

(by which I mean I don't care about the values)

So of course from a programming perspective it probably gives me the data I want

I'm just wondering if it deprecates distributions -- I don't see indications of that explicitly

(And the source doesn't seem to say the function is deprecated either from what I saw: https://github.com/python/cpython/blob/main/Lib/importlib/metadata/__init__.py#L963-L968)

[fading veldt]

Oh interesting, it's also in __all__, and referred to in the docstrings of several other types in the module, so it is definitely intended to be public. Perhaps it was missed when documenting. I'll open a cpython bug and add docs for it.

[shut quest]

does anyone know what the README in ruamel.yaml is talking about? https://sourceforge.net/p/ruamel-yaml/code/ci/default/tree/

my guess is there is just some misunderstanding, but would be good to clear up the confusion if so

[west drift]

i think the author got this email and mistook the PEP 625 requirement to normalize the filename as a deprecating the namespace package (while distribution name and distribution contents are actually decoupled through the normalization)

[shut quest]

yeah, i think this has inspired them to go do something weird with their RECORD
the 0.18.7 release doesn't have the mismatch
(context for others: https://github.com/astral-sh/uv/issues/13666)

i can see how that email would be confusing if you don't know that "source distribution" refers to a very specific thing

[oak quiver]

letting "sdist" be the weird random jargon term almost seems better in that context. at least it avoids ambiguity

[west drift]

wrote a ticket for the ruamel.yaml story https://sourceforge.net/p/ruamel-yaml/tickets/544/

[oak quiver]

I notice that the current version PySide6 only has wheels for up to cp39, but it can apparently install just fine in environments with a more recent Python. How exactly does that work?

(well, 3.9 specifically for the latest; other Python versions for older releases)

[opal wolf]

it's tagged as either cp39 or abi3, the latter is the stable ABI: https://docs.python.org/3/c-api/stable.html

meaning that the wheel only requires a subset of CPython symbols that are guaranteed to still be present and have a binary-compatible interface in all future Python 3.x versions

[bright parcel]

The latest revision of PEP 772 (Packaging Council governance) has been published. Many thanks to @tall fog and @opal wolf for the great discussions at PyCon and in the PR: https://discuss.python.org/t/pep-772-packaging-council-governance-process-round-2/93904

[oak quiver]

but they're different kinds of tag. I had thought that the wheel should be rejected on the basis that the Python code couldn't be guaranteed compatible. But I guess Python code is always assumed forwards-compatible (hence people being able to install Tensorflow and have it break on identifiers named async...)

[tall fog]

And thanks to everyone who improved this PEP. It took everyone's thoughtful input. 💐

[brittle token]

Python version compatibility is in the first part of the tag. py3 is the forward compatible tag, equivalent to abi3. It's theoretically possible to use more specific tags there, but Requires-Python is generally a better tool for that purpose (albeit one with its own limitations given the lack of a Python version specific yanking mechanism on index servers)

[oak quiver]

what I'm confused about is, what exactly is the restriction imposed by the cp39 tag?

[opal wolf]

oh, right, the tag is a three-part thing - Python version(s), ABI(s), and platform(s). so a more common case is something like cp39-none-manylinux1_x86_64. so ignore what I said about "either cp39 or abi3"

however, packaging has special code for abi3 saying that if you're on Python version 3.x, then you support py3x-none-... and also all of py3x-abi3-... down to py32-abi3-...

this isn't in the spec but probably should be at least mentioned there, even if only non-normatively? I agree with you that this now imposes an assumption on the Python code that might not be warranted!

in other words, for cp39-abi3-..., beacause of the abi3 tag, the cp39 part means that it's compatible with Python 3.9 and above.

[oak quiver]

Aha. Does resolvelib understand this too?

[inland prawn]

Resolvelib is just resolving versions, the abi stuff is for choosers that get the list of wheels for a given version

[lusty quarry]

wasn't this fixed at some point in pip, or build, or pyproject-hooks, or somewhere? I can't find where but I could have sworn some project took exceptions into account and made a fix https://github.com/pypa/hatch/issues/1987

[blazing lantern]

You have to include it as part of the data you provide resolvelib: https://github.com/brettcannon/mousebender/blob/pep/mousebender/resolve.py

[oak quiver]

mm. so I'll realistically need the packaging logic as well...

... it'd be nice to have a way to express more abstract requirements that could be provided by multiple different packages. Or even just express "x or y" in pyproject.toml dependencies. One of the things I want to do for the ecosystem is provide stripped-down versions of some packages, such as Numpy, which are still sufficient for a lot of purposes; and then clients should be able to express "if the real Numpy is already in the environment, the stripped-down version isn't necessary"

the closest thing I'm aware of is runtime detection (i.e. try to import from one then the other, and give an error saying to install one if neither is found)

[mystic stag]

This was discussed a little in https://discuss.python.org/t/pep-508-any-plans-for-conditional-dependency-versioning/90792

[oak quiver]

oh, I remember that thread now

[opal wolf]

tt'd be nice to have a way to express more abstract requirements that could be provided by multiple different packages
this would be great. it's pretty common in Linux distro ecosystems, it might be hard to do it in a non-curated ecosystem like pypi, but virtual packages or claiming to "provide" some other package has a lot of good use cases.

concretely it feels like something that would help the general dynamic-library problek, somehow (with a lot more details to be worked out) - it would be nice to say "I need libcudart.so.12, I can use the one from the nvidia-cuda-runtime-cu12 wheel, I can also use one that is installed on your system by your CUDA drivers"

I think you can kinda fake it with default extras, now that defaulte extras are a thing? someone can create a "libcudart12" package with a default extra of [wheel] that depends on nvidia-cuda-runtime-cu12, or a non-default extra of [i-promise-its-installed] that doesn't. or a "minimal-numpy" package whose default extra depends on normal numpy and whose non-default extra depends on your thing

[peak ocean]

Hrm, virtual packages in a non-curated ecosystem do sound like something that would have to be carefully designed. Risk-wise, it makes the typosquatting seem insignificant in comparison.
I could imagine an approach where virtual packages are registered and individually curated, with an allow-list of packages that can provide them.

[brittle token]

Yeah, folks already get nervous enough about the disconnect between distribution names and import package names, so nobody has been prepared to try and tackle the nightmare that would be virtual dependencies with a permissive package pool.

It might be possible to find inspiration in the variants work (although I think the draft design of that uses concrete dependencies to find the variant plugin processors, so it may not be that helpful)

[west drift]

it would help a lot to have the option to rename something on installation so packages wouldn't need to contain modules of the wrong name

similar to https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#renaming-dependencies-in-cargotoml

we do get regular reports of users who have a project that non-deterministically has broken installs cause the have two packages installing the same module, maybe with PEP 794 we can give better warnings for that ahead of time

[oak quiver]

To be clear, I wouldn't want virtual packages to empower an installer to go "hey, I can satisfy requirement with small-package instead of big-package" - especially given the need for PyPI to provide the information about what packages provide the requirement.
I'm not even looking for "hey, the environment already satisfies requirement because it contains big-package, so I don't need to fetch small-package (for that to be useful broadly, would require the big-package authors to go back and come up with a list of abstract requirements they fulfill, and then for others to make use of that somehow)

I'm really only looking for the ability to put small-package OR big-package, explicitly (in preference order, with whatever syntax), in a dependency group.

This would also address the "choose a backend" thing that the optional-extras thing is partially motivated by

as for the name-collision thing, it has deliberate uses. (like if I want to offer several distributions that work together and let you import from a common namespace)

I think all an installer really needs to do is check, before installing a foo.bar package, whether an existing site-packages/foo/ already contains a non-empty bar/

(... ugh. I think that doing proper acceptance testing for PAPER is going to require setting up bandersnatch locally... ?)

but yeah, it may be nice to have Import-Name metadata, but not actually necessary to give warnings. Unless you want to check before the download :)

[brittle token]

Oh, yes, "at least one of" dependencies are something I've occasionally wished for as well.

Essentially the package defining its own virtual feature of "I need a library that does X; I have code to handle X that uses libraries A, B, or C, so as long as I have at least one of them available, I'm all good".

[oak quiver]

Yes, that's precisely what I have in mind.

[abstract nebula]

Won't default extras get us most of the way there?

I guess it would still require the user to opt-out

[mystic stag]

It definitely has use cases, but I would want any PEP to have buy in from all the main Python package resolvers. It quickly becomes non-trivial to resolve when you have a few dozen requirements with various OR statements .

Basically I'm not willing to commit the time to updating pip/resolvelib to do be able to handle that 🙃

[brittle token]

I have a more constrained version of the problem on my venvstacks todo list (because CUDA), but yeah, a fully general solution seems like it would be a nightmare for resolvers

[oak quiver]

By the way, do you see mousebender as relevant to https://github.com/pypa/pip/issues/13334 ?

[blazing lantern]

No. My PoC in mousebender was all about getting the functionality to create a lock file. What you need to make an installer work is in 'packaging' and 'installer'.

[oak quiver]

Ah, I meant because you listed a longer-term goal to "Consume a lock file for the platform to install the specified wheel files"

[blazing lantern]

Those goals also pre-date uv, so some things have changed. 😉

[oak quiver]

Understandable. (But now they've kinda made their way onto my list x.x)

[woven yarrow]

I thought this page would say "This is version 2.4" near the top: https://packaging.python.org/en/latest/specifications/core-metadata. is there some other way i can tell what version it describes?

[hollow trout]

I think it technically describes all of them (since it notes the lowest supported version for every field).

[woven yarrow]

hmm, well the history mentions version numbers. It seems odd to not make it clear what the latest version is.

[oak quiver]

I agree. (As is, we do have Version of the file format; legal values are “1.0”, “1.1”, “1.2”, “2.1”, “2.2”, “2.3”, and “2.4”. near the start; but explicit is better than implicit)

[woven yarrow]

I made a PR: https://github.com/pypa/packaging.python.org/pull/1871

[brittle token]

Queued for merge 🙂

[oak quiver]

I just noticed that Anaconda documentation still claims that Pip doesn't do dependency resolution: https://www.anaconda.com/docs/tools/working-with-conda/packages/pip-install#understanding-conda-and-pip

[brittle token]

I saw a similar misconception in a recent newstack article (their description felt like it predated the addition of the backtracking resolver)

[oak quiver]

"newstack"?

... speaking of which, is the catastrophic backtracking issue still present, where Pip looks at a bunch of old versions explicitly because of legacy metadata?

(I guess I start at https://stackoverflow.com/questions/65122957 to investigate....)

I guess it still potentially downloads a lot of .metadata files, but they're smaller than the full wheels...

[mystic stag]

It's not to do with legacy metadata, it's just that resolving is hard, I'm collecting examples of current issues here: https://github.com/pypa/pip/issues/13281, I expect this to be better for most users in 25.2 if I can get a resolvelib release and vendoring before then,

[oak quiver]

ah, right, I remember that issue

actually let me take something to #pip

[brittle token]

Just a misc software driven tech infrastructure news/opinion site, started back when scale-out-over-scale-up was still a novel idea (folks back at Red Hat used to read it, and apparently I read enough of the links they sent through that Android's algorithm still occasionally lobs some of their programming language and DevOps articles in my direction).

[oak quiver]

ah
It turns out I have rather little understanding of "devops" in general and would usually rather design systems to avoid having to think about it

(unless you count toolchains that run locally to accomplish tasks that run locally. Then that's actually what I like making)

[oak quiver]

Maybe there's a specific channel for this question but
Is there any kind of standard for where Python applications installed from wheels should put their config/logging/etc. files? Supposing that it's something that can be expected to change relatively frequently.

Should the application assume it will be installed at a user level and use a user-level config dir (e.g. per platformdirs), independent of the install environment? Or use the .data folder of a wheel to make something up? (Maybe .data/share/appname?)

(Maybe it also makes a difference whether the application wants to provide an initial state directly, or compute it on first run... ?)

[lusty quarry]

Or use the .data folder of a wheel to make something up?
this should only be written to by package managers i.e. files should be static, platformdirs as you mentioned is what you want

[oak quiver]

I've discovered that there is a https://python-packaging.readthedocs.io/en/latest/index.html , and the advice there appears to be very out of date. Can/should anything be done?

[mystic stag]

Make issue or PR at https://github.com/storborg/python-packaging ?

[spiral urchin]

Is this an actual thing going to happen? (Not listed under events like other summits) https://ep2025.europython.eu/session/packaging-summit
@high stone @hidden flame

[frigid jewel]

out of curiosity, what qualifies to go under the "others projects" channels section?

[hidden flame]

Yeah, not sure why it’s not listed elsewhere

I’ll ping them again

[vital pier]

Looks sound to me! 😂

At this time, this documentation focuses on Python 2.x only, and may not be as applicable to packages targeted to Python 3.x.

[inland prawn]

I'd say that those are projects that are related to packaging in one way or another, but are not governed by pypa org directly

[merry rune]

https://inspector.pypi.io is down?

[lime ocean]

I can use

[inland prawn]

yeah, it's back, but it was down

[uneven karma]

502 for me right now

[merry rune]

Still 502 for me (and https://github.com/scikit-build/scikit-build-core/actions/runs/15888107541/job/44804927820?pr=1109)

[frigid jewel]

never had taken a good look at scikit-build-core before but whilst i was there

looks scary

[frigid jewel]

if something like Nuitka applies then i'd be happy to host a channel and help people out with it if they ever need help

[merry rune]

Yeah, that’s the GitHub toml highlighter mad that that’s not valid toml. Not sure how else to show that compactly, though

[brittle token]

Yep, exactly. The Discord instance admins make the final call on whether a requested addition makes sense or not.

[lyric hedge]

And usually how it goes is that one admin decides they want to add a non-PyPA project channel ... and they simply make one. We operate very informally.

[cerulean drift]

Hello, Im new to linux as a whole and want to install build, hatchling, installer and pip. I didnt want to go into all the channels to ask the same question so i hope that its fine that i ask here. I dont really understand how to install them at all. Ive tried and also googled for it but cannot find anything. So as a last resort im here to ask for help.

Thanks to anyone who can and is willing to help.

Nevermind i uh, Didnt think it existed on the manager thingy. Disregard the question. Dumb me.

[oak quiver]

If you're installing these tools as a Python developer then I don't recommend using the system package manager for them (except Pip if you're on a Debian derivative, because you need something to bootstrap the install process). In practical terms you will need to understand virtual environments to do everything safely and sanely, and then you can easily set those tools up with native Python packaging within such an environment.

most home-desktop Linux distributions come pre-loaded with some version of Python and a bunch of packages that the distro maintainers think you will find useful and/or are essential for system functionality. However, that Python can be missing pieces of the standard library (especially on Debian) and your system package manager will offer only a small fraction of what's available on PyPI.

(meanwhile, trying to use Python ecosystem tools to install packages in the system environment can cause a serious mess and interfere with your system package manager.)

(Also, installer is unlikely to be useful to you if you're in the situation you describe. It's the sort of thing you would include in your program if you were trying to make an alternative to pip)

[high stone]

[[generate]]  # array of tables
# ... 
path = ""

[merry rune]

But that messes up the following parts, they would also need a header. I’d have to sort it last. Could try, though

[queen hornet]

It's now listed with the others 🎉

[hidden flame]

Yeah, I also just set up the registration form (not yet on the EP site): https://discuss.python.org/t/packaging-summit-at-europython-2025/97706

[oak quiver]

https://discuss.python.org/t/2-packages-the-each-require-different-packages-with-a-module-with-the-same-name/97703

There are ways to hack around this, of course, but it'd be nice if packaging could handle it somehow
Unfortunately, an installer can't just make up a namespace and put the installed package in a subfolder, because the package code could be using absolute imports for its own content (which I personally strongly feel should always be relative, at least barring some weird custom plugin system or something like that; but there doesn't seem to be strong buy-in for that in the ecosystem)

... Ugh. It looks like the two packages do basically the same thing, with the same top-level name, but different interfaces.

Even if we had a solution for the idea of specifying "alternatives" in dependencies, the clients would need to opt in and create their own adapters or something.

[vital pier]

I started thinking about this at PyCon US this year, and Dan and I even cooked up a basic starting point, and my opinion has evolved to that the import syntax needs to change before "packaging can handle it"

[lyric hedge]

Congratulations @mild dew on your PEP acceptance!

[mild dew]

thanks @lyric hedge!

[hidden flame]

Hey all, in case anyone is attending EuroPython 2025 in Prague and wants to chat about Python Packaging, there is going to be a packaging summit on Tuesday afternoon: https://discuss.python.org/t/packaging-summit-at-europython-2025/97706

[woven plover]

Is there a remote option?

[hidden flame]

Sadly no, the summit doesn’t have a streaming setup, and it wouldn’t be practical for the round table part

[indigo token]

when I go to the website there is a fat orange banner linking to https://ep2025.europython.eu/remote

Can’t join us in Prague? Attend remotely — online options available!

[brittle token]

The setups are generally different for the summits and open spaces (highly interactive, very rarely online or even recorded) vs the actual conference talks (relatively passive audience, often recorded, and potentially available live while being recorded)

[oak quiver]

What are the intended implications of PEP 639 for the actual LICENSE files?

• If I have a project.license in pyproject.toml with a valid license expression string, is the resulting wheel expected to include a corresponding file anyway?

• If a build system encounters a project with such a pyproject.toml and no LICENSE files, is it expected to generate one for the wheel?

• Contrarily: is a build system permitted to read a provided LICENSE file, detect what license it is, and replace it with a license expression?

[hollow trout]

I don't think the build system is supposed to be that smart. Whatever is in the license field in pyproject.toml goes into the License-Expression field in the distribution metadata. Whatever is in the license-files field goes into the License-File field (and matching files are copied into the distribution).

They are basically independent.

[oak quiver]

but this does allow me to omit a license file from the wheel, yes? (It seems that including a license in a repository is still a good idea on GitHub, but.)

[mystic stag]

Yes, it provides no guarantees for whether the project or distribution is actually correctly licensed

[oak quiver]

what I'm trying to figure out is, is it "correctly licensed" if I have no LICENSE file but I do have a valid SPDX license expression in METADATA?

but maybe that requires a different kind of expertise

[mystic stag]

The problem is in the term "correctly licensed", define that and the you will have your answer 🙃. The PEP does not give a definitive answer for that.

[oak quiver]

yeah. :(

I intend for bbbb not to include the file unless it's explicitly requested - either by using license-file or by the tool-specific mechanism for file inclusion.

(which, er, last time I paused work on it, it was because I was struggling with the design for that...)

[spiral urchin]

Some conversation that happened during ep25 from the top of my head…

• Specifying external dependencies
• Locking build-time requirements
• Include lock file in wheel? (for e.g. uvx install)
• License in lock file?

[oak quiver]

"ep25"?

[inland prawn]

europython 2025

[blazing lantern]

Yeah, it all depends on the license itself. Some explicitly require you include the license file, some don't.

[oak quiver]

ahh, I hadn't even considered that angle. my planned approach should allow devs to do what they need easily enough, though, I think.

[lyric hedge]

is anyone aware of a publicly available wheel that is larger than 4 GB? I'd like to see if this bug still exists: https://github.com/pypa/pip/issues/9549

(yes, I am just as concerned as you are that these wheels do apparently exist out in the wild)

[oak quiver]

iirc the biggest wheels are for things like astronomy data

[lapis hound]

Pytorch Linux 🤣 🤣 🤣

[fading veldt]

$ curl -I https://pypi.nvidia.com/isaacsim-extscache-kit/isaacsim_extscache_kit-4.2.0.2-cp310-none-win_amd64.whl#sha256=a84dd40f2d2e43bb6ad738190bb492643d2eb7153e4db7cf4029e082cbf9ddff
HTTP/2 200
content-type: binary/octet-stream
content-length: 4395485553
...

[tardy isle]

chonky

[bright parcel]

DPO thread for the just published round 3 of PEP 772: https://discuss.python.org/t/pep-772-packaging-council-governance-process-round-3/100181

[lyric hedge]

🚨 PSA: there is a email phishing attack targeting PyPI users: https://discuss.python.org/t/pypi-org-phishing-attack/100267/

[uneven karma]

QoL nitpick: I didn't set attestations: true, it set itself (ie, its on by default)
IDK much about GitHub Actions inputs, I wonder if it could explicitly say whether I'd enabled attestations or if they're just defaulting to on

Status: Downloaded newer image for ghcr.io/pypa/gh-action-pypi-publish:76f52bc884231f62b9a034ebfe128415bbaabdfc
Warning: The workflow was run with the 'attestations: true' input, but an explicit password was also set, disabling Trusted Publishing. As a result, the attestations input is ignored.
Warning: The workflow was run with 'attestations: true' input, but the specified repository URL does not support PEP 740 attestations. As a result, the attestations input is ignored.

[blazing lantern]

Sorry, this wasn't a reply to anything, so what's this in reference to?

[uneven karma]

Oh just a ramble after I got confused this morning

I couldn’t figure out why it was telling me I had set a flag I didn’t set
Took me a couple minutes to realize its just defaulting to on

[inland prawn]

@uneven karma you might want to go with that to gh-action-pypi-publish repo

[uneven karma]

Fair

[oak quiver]

Is pypistats.org giving 502 to anyone else?

[fading veldt]

Yes, me too

https://github.com/crflynn/pypistats.org/issues/82

[frigid jewel]

Companies can create organization accounts as a paid service while community projects are granted complimentary access.

how does this apply to freemium model type of projects?

[queen hornet]

https://astral.sh/blog/introducing-pyx

[blazing lantern]

I'm curious as to how they are handling the GPU-aware part of their blog post as that's not in any spec right now (e.g. variants aren't a thing yet and I believe the WheelNext folks have plans).

[lime ocean]

My somewhat educated guess is they're building off their --torch-backend=auto approach and perhaps adding some automation server side to make that easier (besides having specific accelerator suffixes)

@wide stump is also here so maybe he or some other uv folks can shed some light?

[astral shale]

stay tuned tomorrow morning 😄
https://pytorch.org/event/pytorch-live-2-8-release-qa/

I'll post a large update tomorrow morning on DPO

[honest geyser]

The GPU indexes on pyx aren't using variants right now

We're taking packages that are current built and hosted across several different third-party indexes with varying strategies (hacks) to encode metadata about their dependencies (e.g., cuda or torch versions), and building them ourselves into a cohesive index with consistent metadata.

[viscid rune]

hello

[oak quiver]

Am I alone in perceiving that usage of the Python Help section of PDO has been trending downward lately?

[blazing lantern]

This seems a bit off-topic for this server

[brittle token]

Does anyone have strong opinions on CLI options for configuring index URLs? I'm leaning towards a uv-inspired --index and --default-index spelling (since this will be mapping to uv pip compile config settings internally), but --index-url and --extra-index-url have a pretty well established history at this point, so I could potentially be persuaded that maintaining consistency with that was a better way to go.

It just feels like one of those potential inertial shifts where if enough tools were to start adopting a different preferred spelling, it could eventually turn back around and create pressure on pip itself to change its preferred option spelling.

("Eh, just pick one, it'll be fine" also feels like a potentially valid response here...)

[mystic stag]

IMO the main issue is users assume the options --extra-index-url means that it will have a lower priority than --index which is not how pip works and has caused a lot of confusion (and a dependency confusion attack), so first would need to understand your behavior and then choose options that are semantically appropriate

[brittle token]

True, the fact the options will have uv's semantics is a concrete argument in favour of using uv's preferred spelling.

[high stone]

Does anyone know who the owner is of [e-mail] mailing list?

I'm trying to audit the list of subscribers -- I thought I was an owner and could do it on my own but evidently not. 😅

[fading veldt]

You should be able to email [e-mail] to contact the owner(s)

[high stone]

Done. 😅

Was more in the, hey, I'm not a 100% sure if the owner is gonna read the emails.

[fading veldt]

Yeah, worth a shot. The owner might be Alyssa, since she was the first to post and referenced another thread about the creation, but I guess we'll see!

[high stone]

I'm an owner now.

[mortal shore]

apparently I'm (one) of them, but looks like you got it taken care of 😛

[lime ocean]

+1 this feels like such a gotcha and was actually the cause of the PyTorch dependency confusion attack. Ambiguous index priority is the source of a lot of confusion.

[oak quiver]

my general feeling is that you can't really just configure indices; you need to configure strategies for which index is used (or whether to look in a cache, in a local folder, ...) and the general case of this is unreasonably complex for the command line. my design is to offer some pre-configured common strategies (like "cache first, then PyPI" and "cache only") and the ability to define more.

[fading veldt]

I'm drafting a PEP to add a JSON format core metadata file (in addition to, but to eventually replacing the existing email format core metadata files). I'd like to hear the pain points people have with the email format of core metadata to make sure I cover everything. Please see 🧵

[high stone]

Signal boosting https://discuss.python.org/t/pypa-committer-vote-process-for-pep-772/103309 here.

[high stone]

And... https://github.com/pypa/packaging-problems/issues/989 for the actual committer vote!

[shut quest]

noob q: did pep 772 consider a non-democratic setup, like the one we have for the typing council (pep 729)?

[bright parcel]

Not really, mostly because of the tradition for pypa-committers and because packaging has a much larger and diverse set of stakeholders.

[serene hill]

I'm seeing a really weird behaviour on PyPI right now - either I'm missing something really obvious, or something is going very wrong. Details in the thread to follow...

[shut quest]

makes sense, i find it interesting to think about the social magic that makes these constructs feel legitimate or not 🙂

[bright parcel]

Like so much in Python, things grow organically. They make sense until they don't.

[trim slate]

Whats with the new google antibot measures. Is serp api or paid crawlers the only way left

[frigid jewel]

wrong chat?

[obtuse holly]

Hey, not sure who maintains packaging.python.org but this page is all dead links: https://packaging.python.org/en/latest/specifications/schemas/

[queen hornet]

Thanks, I left a comment at https://github.com/pypa/packaging.python.org/pull/1889#issuecomment-3287539359

[obtuse holly]

Thank you!

Second thing, also not sure where to put this, but https://github.com/SchemaStore/schemastore/ does not reflect the current status of PEP 639. I was considering making a pull request but realised that this may be better left to someone with a bit more standing cause this is the first deprecation of something in a pyproject.toml file, which means that a lot of editors will start complaining with that change.

[mystic stag]

You can see the previous PR: https://github.com/SchemaStore/schemastore/pull/4143 on this, I beleive it's updated via community efforts

[woven yarrow]

sorry to be a total newb about this stuff all the time... Cog (https://github.com/nedbat/cog) won't successfully run tox locally. I get complaints about license format on older Pythons. I can see that the problem is it using old setuptools. I've tried a few ways to force latest setuptools in the tox environments, and nothing is working: download=True, reqs=setuptools>80, VIRTUALENV_DOWNLOAD=1, and so on. No matter what, my Python 3.10 and 3.11 (but not 3.9!?) environments have older setuptools. I'm sure I've fixed this in other repos, but I don't know what to do here.

Any help is greatly appreciated.

[vital pier]

I think license = {text = "MIT"} will work for both older and newer setuptools (or the equivalent license.text = "MIT" ought to work too)

[west drift]

you can fix this by declaring requires = ["setuptools>=80"]

[vital pier]

oh yeath, that too 😄

[west drift]

with requires = ["setuptools"], technically any setuptools version is allowed, so tox uses the pre-installed one. python 3.12 "fixes" that because it doesn't preinstall setuptools, so pip install setuptools gets you the latest version instead of the (old) vendored one

[vital pier]

here ya go: https://github.com/nedbat/cog/pull/57

[woven yarrow]

THANKS! this works 🙂

Tons of people gripe in public about the churn and confusion in Python packaging. Not enough talk about the bunch of helpful people who can help you machete your way through the thicket. You are the best.

[obtuse holly]

I just successfully installed a package that hasn't had any releases in 12 years and it worked out of the box! Python packaging is cool!

[brittle token]

PyPA and "the consent of the governed"

[woven yarrow]

Is there a tool that can look at a package and its dependencies then list the versions of Python it's compatible with? Taking into account where binary wheels are needed, and whether those wheels are available?

[frigid jewel]

not sure where the best part to ask but i figure many of the revelant people are already here.

is there a particular place in DPO where i can just nerd out on what i'm working out?

is it really python-help?

[drowsy moss]

📢 PSA: a new PyPI phishing domain spotted but Gmail classified it as spam just fine — https://discuss.python.org/t/pypi-org-phishing-attack/100267/15

[sinful pecan]

Quick question, is including wheels/artifacts like this normal? I was under the impression that pip handles this.
https://github.com/unitaryfoundation/pyqrack/releases/tag/v1.69.0

[hollow trout]

It's unusual, but there's nothing really wrong with it. It's just an alternate download location.

Ah never mind, I see there's already a discussion in #pip.

[uneven karma]

(They spammed posts across multiple channels and servers, there was a long conversation in #pip )

Yeah

[oak quiver]

I got it sorted over on the main pycord

[lusty quarry]

I encountered something very insecure today and had I not had time to deep dive it would have messed up a lot of our developers. However, I really respect the person's effort. It would have never occurred to me to cache Git data by installing pre-commit hooks on everyone's machine by releasing a source distribution without a wheel 😂

[brittle token]

Well, that goes on the "do not use" list. "It's safe for me to execute arbitrary code on everyone's machine because the current code is visible and arguably benign" is rather missing the point of why requiring arbitrary code execution at installation time is a security concern.

[oak quiver]

(a shame about all those sdists, but nobody has a better way...)

[oak quiver]

apropos of this plus the wheelnext discussion of symlinks

I'm wondering if there's appetite for a general PEP for "platform-independent file that Python treats like a symlink"

which would then cover both the "inherit packages from a 'parent' environment" case as well as "supply multiple effective paths to the giant compiled binary blob"

and maybe even more uses for environment setup that I haven't thought of

[oak quiver]

https://discuss.python.org/t/problem-with-python-in-virtual-environment/104082 If this is reproducible without some hidden issue, that seems like a nasty papercut. Do venvs really need to bundle as many activation scripts as they do?

[hollow trout]

For me it works... I don't understand how.

I suspect this person might not actually have Python installed.

[high stone]

https://discuss.python.org/t/pep-772-packaging-council-governance-process-round-3/100181/61

Signal boosting my call to action for the packaging council saga - tldr: if you have concerns, speak now!

[mystic stag]

I have concerns but I elect to forever hold my peace 😉

[blazing lantern]

No because activation is entirely optional, so we could include no activation scripts. 😉 Otherwise what activation scripts do you propose dropping?

[mild dew]

do we have any PyPA GH org admins around? i have a user who's spamming pip-audit with LLM PRs who i'd like to stop 😅

[mystic stag]

No admin powers outside pip, but I reported their PRs on pip-audit as spam to GitHub at least

[oak quiver]

I mean, can it not filter more precisely according to the user's environment? it's unclear to me why OP of that thread had the Bash version available to try in the Powershell environment. (Is it some "maybe the user has WSL" thing?)

[mortal shore]

uh, I used to be

let me see if I still am

[blazing lantern]

It's because there's no way to know what shell may used at any point; just because you use Bash today doesn't mean you won't use PowerShell tomorrow (which is cross-platform)

[mortal shore]

yea I am, got a link?

also if you know what buttons to push that's helpful too 😅

[mild dew]

thanks! here's one of them: https://github.com/pypa/pip-audit/pull/959

i think the ban options are somewhere in the org settings, but not sure exactly where

one sec, lemme see on another repo

[mortal shore]

found it I think

[mild dew]

thank you!

[echo ruin]

Hi! I don't 100% understand the "right" way to run pip-audit on a library, particularly in light of the pip CVE. Specifically in all my projects I essentially run pip-audit via nox by installing the project to audit + pip-audit into a noxenv and then invoking pip-audit. This has always seemed off to me (for reasons that are becoming important now clearly) namely I know that of course pip-audit may have its own dependencies, notably now pip, and that the results of running it are therefore clouded by potentially flagging CVEs on dependencies of it and not my library. In https://github.com/crate-py/rpds/pull/178 we chose to instead switch to pyproject.toml checking, which is fine there (because in fact rpds.py has no dependencies anyhow) but in https://github.com/python-jsonschema/jsonschema/pull/1416#discussion_r2404589655 that's now not the case, as I assume (but honestly haven't checked) that that pip-audit mode will not do what I want it to do, namely check every possible packaging extra and assure me that each one does not install a vulnerable dependency.

Is there something I'm missing, or should I give up, go on with life and simply add --ignore pip to the way I invoke pip-audit everywhere (which maybe I prefer over ignoring the specific CVE as it may be less likely I ever write a package with a real dependency on pip than it is that pip has another CVE 🙂

[mystic stag]

I've not spent any time with pip-audit so this might be all wrong. But my impression is that I would not have pip-audit do any resolution itself, I would pin all my dependencies with something like pip-compile and then target the outputted requirements file with pip-audit. This might require installing pip-audit into it's own separated virtual environment, using something like pipx.

[echo ruin]

Yeah I considered that (just doing it via a pinned requirements file) but it seems somehow not ideal, but maybe I just need to convince myself

[mystic stag]

Well, if you use that pinned requirements file to installed your requirements in your target environment, then that should give you confidence that your CVE report matches your environment

[echo ruin]

The "not ideal" feeling comes from that the reason to do this at all as a maintainer is to get immediate notification if a new user installing my library is going to be installing something vulnerable alongside it

[echo ruin]

Right but in this case that's precisely not what I want! I want to simulate a user simply installing the library and getting the versions of deps they get at that point in time

[mystic stag]

Ah, right, I'm not sure what you can do as a library, I think of CVEs in terms of applications

[echo ruin]

Yeah so perhaps you're suggesting choice C -- "give up and don't bother doing this at all" 🙂 which I also am considering

[mystic stag]

Or at least, assessing if you're vulnerable to CVEs

[echo ruin]

I mean as a maintainer I want -- if I depend on a vulnerable library -- to pin away from the vulnerable package version.

[mystic stag]

I don't think that's a good idea in general, it's not up to a library to assess the impact of a given CVE for all users, as a user I might need that older dependency, but still want to use a newer version of your library, and I might have assessed that the CVE does not apply to me

There are exceptions, like when the library and dependency are highly coupled to each other, but in general I think it's a bad idea to force users to use newer dependencies on anything other than breaking functionality, and maintance cost for you as a library author

[echo ruin]

Sure, I'm not saying always. Insert some actual decisioning at the point where one finds out there's a vulnerable library

Ok, thanks for rubber ducking at very least, sounds like I'm not necessarily missing anything big and this may just be "implement it if I want it, otherwise give up"

[mystic stag]

Yeah, I think that's reasonable

[brittle token]

For automated CVE notifications, that's where things like Github's Dependabot come into play.

[mystic stag]

Github's Dependbot seems to often just silently break, at least for me, if CVE analysis was important to me it's not something I would depend on.

[mystic stag]

btw, one thing you can do if you're interested in the lower bounds of your library is something like uv pip compile pyproject.toml --resolution lowest-direct -o requirements.txt: https://docs.astral.sh/uv/concepts/resolution/#resolution-strategy, you may or may not find helpful.

[obtuse holly]

I learned yesterday that dependabot has a cli and can be run locally which is fun

[mild dew]

as the pip-audit maintainer: i agree wholly with what @mystic stag said 🙂

(the situation with the current pip CVE is an exceedingly frustrating one, and encapsulates why i think CVEs are such a bad system for communicating vulnerabilities)

[echo ruin]

Yeah I assumed that was not fun enough which is why I didn't ping you directly, so thanks for chiming in!

[echo ruin]

Can I confirm whether you're including the "don't really run pip-audit on libraries" bit in your agreement here 🙂

(Needless to say -- or maybe not needless to say -- you do awesome work so thanks for everything overall!)

[mild dew]

yeah, with the caveat above -- there's no perfect one-size-fits-all prescription, but i agree with the thought that vulnerability scanning in general applies to "leaf" contexts like applications and frameworks and not individual libraries

and thank you! i don't mind being pinged FWIW, i only realized how many people were being affected by the advisory by catching up on the backlog here 😅

[brittle token]

Huh, I only just noticed that the "Named pylock.toml" regex disallows extra .'s in the name portion (it came up in migrating venvstacks over from requirements.txt due to pylock.cpython3.11.toml failing to validate against it).

[blazing lantern]

Yep as I figured it made parsing a bit easier and it matches project name restrictions (I've probably also had to stare at wheel file names for too long and have their restrictions imprinted on my brain).

[fossil gulch]

https://tenor.com/view/hello-gif-18027547882538584652

[obtuse holly]

https://www.theregister.com/2025/10/27/python_foundation_abandons_15m_nsf/

[obtuse holly]

So I've noticed that python 3.8-3.13 had signed git tags for their releases, but starting with python 3.14 and continuing into python 3.15, there aren't signatures on the git tags of their releases.

Is anything codified about signing tags or did I just get accustomed to something extra that happened?

[fading veldt]

I checked on this and the answer is PEP 761 removed gpg signatures for release artifacts and that includes tag signing

[obtuse holly]

Fair. I think. It's just weird to me to not see the comitter/author have a verified tag on GitHub

The last unsigned release I can find is 3.12.0rc1

[blazing lantern]

Releases are signed with sigstore now

[brittle token]

Current status (content warning, specifically for Brett): wishing we had mandated the use of POSIX path separators in lock file path fields.

Thinking I might file a uv issue anyway, as the way uv export publishes local file paths on Windows is somewhat painful to consume no matter what the standard does and doesn't specify (and I'm not sure their file URLs on Windows actually have a valid path segment)

[mystic stag]

Yeah, could just be uv bugs, I don't think a lot of users are consuming the output of uv export from other tools yet

[brittle token]

Given uv pip install was the only generally available pylock.toml installer when I went looking, it honestly wouldn't shock me if I was first. I'm only tinkering with it because I want Windows and Linux to generate consistent lock files, which means relative paths with POSIX separators (and Windows ends up restricted to storing the lock file and the wheels it references on the same drive)

[brittle token]

I filed it as a feature request here: https://github.com/astral-sh/uv/issues/16514

[fading veldt]

I think it's good we didn't. I'm sure there are people who have package source trees on UNC windows shares that could be hard to express otherwise

[blazing lantern]

The file format is also versioned for a reason, so updating it with some path requirement is doable. But I agree that it's more of a uv feature request.

[oak quiver]

...I'm a little surprised that one still writes uv pip install for a feature pip doesn't have yet and might design differently... ?

at some point, the pip in those uv commands becomes noise

[brittle token]

Forward slashes work in UNC paths as well

[brittle token]

From a uv point of view, it was the logical place to plug it in, and the pip implementation was far enough along to know the expected UX

[mystic stag]

But it is true that uv pip and pip will likely to continue to diverge from each other, for example the semantics of installing groups from pyproject.toml files are not the same

My PR that restricts packages by upload-time has chosen different semantics than uv pip and I'm working on a new feature to restrict pre-releases which I'm not sure uv pip will adopt

[obtuse holly]

That's frustrating

[oak quiver]

I feel quite the opposite, honestly. The point of designing a system with open standards, and not officially blessing a single tool(chain) that everyone is expected to use, is so that competing tools can express competing opinions like that.

(I mean "competing" in a loose, friendly sense.)

[mortal shore]

I think the thing is that uv pip was supposed to be a drop in replacement for pip, and then uv <whatever> would have the competing opinions, but of course real world happened and diverges came about 😉

[tardy isle]

I hope that uv install is coming one day, maybe they’re waiting to come up with a nice UX proposal before committing to something?

[abstract nebula]

What do you expect uv install to do?

Poetry had a poetry install and they ended up renaming it to poetry sync

[inland prawn]

To be precise we have both install and sync

[abstract nebula]

Do they have different semantics?

I was under the impression that poetry install is an alias to poetry sync for compatibility reasons

[oak quiver]

Heh. I guess I'm doing everything wrong if I want broad adoption of my project ;) I'm doing my own UI design more or less from scratch (rather than trying to maintain compatibility with something I'm explicitly trying to improve upon); I'm focused on things like simplicity and modularity that nobody seems to care about any more

and worst of all I'm wayyyy behind uv's development timeline ;)

[blazing lantern]

I mean if people really cared we could try to define an "installer CLI API" that all installers agree to support so they can all be swapped in/out in other tooling

[mortal shore]

I don't think it's really that big of a deal tbh and I think trying to define a cli like that is more harmful than positive, just kinda a weird state when the "pip compat" layer ends up diverging so the uv pip install ... ends up really being uv install ... just with extra letters to type

but it's not really a problem to solve or anything, just a possibly funny wart

if an extra 4 characters is the biggest problem uv users have, they're doing pretty good 😉

[oak quiver]

(manually moving from #sboms , with a bit of editing)

Fields defined in the following specification should be considered valid, complete and not subject to change.

if pyproject.toml states that dependencies are dynamic, generally that would be for a reason that can't be resolved until build time (i.e. inspecting the end user's environment, not something done during creation of the sdist), right?
... Which at first seems to mean that you can't make a new-style sdist, because you can't write an accurate set of Requires-Dist fields, and not writing any appears to mean that the project has no dependencies. Actually you can have Dynamic entries in core metadata and not just pyproject.toml, but:

If a field is marked as Dynamic, it may contain any valid value in a wheel built from the sdist (including not being present at all).
the notion of METADATA differing from PKG-INFO seems like a contradiction to "complete and not subject to change".

[hollow trout]

I suspect that the first quote was written before the introduction of Dynamic and is not accurate anymore.

[inland prawn]

Install installs dependencies into the env, while sync also removes dependencies that are not in lockfile

[hollow trout]

Yep, it was added in 2017 in https://github.com/pypa/packaging.python.org/commit/6c1cc8634aa16c9aabe70395829343d2d4991c36#diff-9134793a71d204fc2296a94316e2a3ee1ccd4e9a4fba2cd4b5d30ba441ba60a2. Dynamic was added by PEP 643 in 2020.

[oak quiver]

mm

I feel like quite a bit of the prose on packaging.python.org needs review, but

because I generally understand things well enough to use them myself, I end up not spending time thinking about PRs for that.

[manic belfry]

And we have not even started talking about the install from pylock CLI 😁

[obtuse holly]

Oh I agree. But uv pip and pip diverging is the frustrating part. I know myself because uv pip had something (pylock support) I expected pip to have the same support. That was confusing. Trying to find uv's pylock support since they were the first to have it and end up having to go into the theortically deprecated namespace of pip commands which are supposed to be the low level interface for a pip equivalent, but pip not having support..... it was confusing.

[spiral urchin]

When uv pip was first introduced, many people suggested it should be called something else to avoid exactly these potential problems, but the uv folks outright shut down the possibility. They simply don’t think these are serious issues.

[obtuse holly]

😔

[mystic stag]

I will say, at least so far, between the two projects it's uv that has been getting all the user support questions about why uv pip doesn't behave identically to pip. I think in hindsight using the phrase "drop-in replacement" was probably not good wording.

[spiral urchin]

I doubt the people at Astral feel the same

[west drift]

imo that's just a uv bug, we should totally write relative unix paths like we do elsewhere

[brittle token]

I don't think that would really work, as some of the divergences are in "which packages will it pick?" because the resolvers work differently. I think it's fine though - the uv pip layer is more the "installation toolkit" layer at this point (vs the top level project management layer).

[obtuse holly]

revision bump and mandate posix?

[honest geyser]

This isn't an accurate take. We did not shut this possibility down outright, we spent time discussing it both internally and externally.

In the end, we didn't think it made sense to change it, but that's different than thinking it's not a serious issue.

[brittle token]

macOS folks: what happened in macOS 14 to make wheel sizes so much smaller? (numpy's 14+ wheel is 10MB smaller than its 11+ one, and there's a similar size drop from 12+ to 14+ for scipy).

While these are nominally arm64 wheels in both cases, the lack of dedicated x86_64 wheels targeting the older versions makes me wonder if those were actually dual-arch binaries that weren't explicitly labelled as arm64.x86_64 because the build tools didn't do that automatically.

[honest geyser]

(You can even see that I created an issue for discussion and left it open from Feb -> June before closing it https://github.com/astral-sh/uv/issues/1657 — I've also opened https://github.com/astral-sh/uv/issues/12604 to discuss further problems that people are encountering)

[blazing lantern]

I'm not thinking of byte-for-byte sameness, just "can I choose my installer for this tool to use" without having to special-case every installer (e.g. how is Tox or Hatch supposed to support using either pip or uv based on my preference if there's no unified API to drive those tools as installers?)

[brittle token]

Is it really feasible to do better than uv pip has done in that regard? I guess we could write down the "subset of the subset" as an informational PEP for the basic parts

[blazing lantern]

That's exactly what I'm suggesting: what can we agree on so that you could say "use pip" or "use uv pip" and running a subprocess with that or some other installer won't fail for some subset of CLI commands because those flags don't exist?

[lyric hedge]

@mild dew what was the status on producing RFC9457 error responses for PyPI? Or was it in a draft PEP somewhere? I vaguely remember RFC9457 being mentioned somewhere, but I can't remember where. For context, someone is proposing that pip parses RFC9457 error responses which is pointless until an index starts producing such responses.

[queen hornet]

https://github.com/python/peps/pull/4635 was just merged to update draft PEP 807

[mild dew]

hey! yeah, the PEP in question (PEP 807) is only for Trusted Publishing APIs, not for the 503/691 index or other responses. i would like it if those index responses also standardized on RFC 9457 for errors, but i haven't put the hours into writing that PEP yet 🙂

with that said, pyx uses RFC 9457 for error responses, so pip could check for the application/problem+json content type on error responses and progressively enhance at least there. but IMO it would be more than justifiable to wait until a standard actually defines index error responses instead

(i believe some other internal/private index implementations also use RFC 9457, but i have no particular insight there beyond pyx...)

[brittle token]

An issue I reported to UV about environments failing to lock highlighted that we don't really have a formal standard for checking CPU architectures in environment markers: https://github.com/astral-sh/uv/issues/16589

The problem is with the platform_machine marker: that defers to the stdlib docs for platform.machine() which say "The output is platform-dependent and may differ in casing and naming conventions."

[tardy isle]

this is pretty related to the discussion around 32/64 bit architectures and ABI flag environment markers

in the PEP 780 discussion

[oak quiver]

(FWIW, I'm making my installer deliberately with a programmatic API, with the intent that other programs that have to create environments can declare that wheel as a dependency, and also not install my own UI. It's not really clear to me why tools like that need to offer a "choice of installer")
(... but I'm also planning to make the setup of wrapper/activation scripts, etc. a bit different, so....)

... I guess that something along the lines of PEP 780 is going to be necessary for PEP 725 to work... ?

[oak quiver]

Is there a history behind sample package names like timmins in Setuptools documentation, beaglevote on packaging.python.org etc.?

[lusty quarry]

Workspace support has finally landed in Hatch thanks to the valiant efforts of @spark fulcrum! The hugest of shoutouts to him. Now development can recommence and the main branch is no longer unreleasable 🙂

[brittle token]

Beaglevote is from AMK choosing a Peanuts theme for https://peps.python.org/pep-0241/

No idea on timmins (if it's been there from the start, PJE presumably picked it, if it's more recent, it likely originated with Jason).

[oak quiver]

@brittle token I'm watching your venv pre-PEP with great interest, BTW. for PAPER I intend to create custom environments that would definitely benefit from that kind of signaling, and will probably need config files that I would have otherwise put directly in the venv root.

I'm -.9 on standardizing pylock.toml there, since it seems to me that any plausible "expectations" of the venv-contents will be tool-specific. I can imagine saying "pylock.toml is optional; if present, contents are up to the named MANAGER's discretion" but I don't know if there's a point to that (beyond respecting the PEP 751 naming convention?)

but if I wanted a lockfile there I also could just call it pylock.paper.toml.

[brittle token]

The reason I'd like to standardise the significance of an internal pylock.toml is so any tool can check if a nominally locked venv has been modified since its last sync

[lavish rapids]

macOS folks: what happened in macOS 14

[west drift]

<@&815744082823348274> ^ spam thanks for deleting

[lyric hedge]

Banned

[oak quiver]

Did this get addressed in pyoxidizer btw?

[oak quiver]

... the pre-pep discussion seems to be going rather far afield of what I expected. I thought this was basically just about setting some conventions for "if there's an optional file/folder here (where existing venv wouldn't put anything), here's what it means", so that tools can put the files there and interoperate. I don't see how that would require core support since I didn't interpret that venv would be required to start adding any files; it's trivial for tools that want to put a file in the venv-info folder to create the folder if it doesn't exist yet.

[brittle token]

@oak quiver The attraction of stdlib involvement is the potential reach: it would be nice if someday it was genuinely hard to create a venv, stick it on a USB key or network drive, and NOT get helpful feedback when trying and failing to run that venv on an incompatible system. As a packaging-only PEP, it's still a useful idea (and that's most likely where I'm going to start), it just won't necessarily help beginners all that much (expect indirectly via tool improvements).

[oak quiver]

ah, so the hope is that this allows core Python to give better diagnostics?

(although I honestly would prefer if such venv cloning worked more often than it does, which is basically never because of all the absolute paths)

[vital frigate]

I’m looking at trying to get new versions of libclang released for https://pypi.org/project/libclang and have reached out to the current maintainer about continued maintenance, and opened a PEP 541 request where it was suggested that having it live under an org such as pypa so there are more stakeholders would be good. So my question here is: would the libclang pypi package be something that could live under the pypa umbrella or is it too far removed from packaging to be accepted?

If LLVM is willing to handle it that would be preferred, but at the current rate I don’t expect that would happen for a couple years.

[inland prawn]

why would you want it under pypa umbrella? like you said, LLVM org would be better suited for such project, but does it need to live in any org in particular? would it be an issue if it was to live under your user on GH?

[vital frigate]

I’d be okay with it living under my user on GH, someone had suggested that there might be people in pypa with interest in its maintenance, which I interpreted as being under the pypa org — which would help minimize the current sole maintainer issue if any one person disappears.

[inland prawn]

I am by no means a pypa admin, so don't take my words for meaning anything but my own opinion, but in my eyes it doesn't look like a project that should live under pypa

[oak quiver]

Maybe https://github.com/jazzband would be interested?

[uneven karma]

Or the PSF org if it’s a big enough deal

[fading veldt]

I'd also reach out to the maintainer of https://pypi.org/project/clang/, who probably also is motivated to make the libclang bindings a thing. I don't think pypa is the right vehicle for this package, I'd suggest either jazzband or ideally llvm itself.

[queen hornet]

I don't recommend Jazzband, there's an admin bottleneck https://github.com/jazzband/help/issues/196

[oak quiver]

oh, that's unfortunate.

[vital frigate]

Yea, I’ve talked to the maintainer of the PyPI clang package some (also falling behind on updates) — we’re working towards getting the packaging for the clang PyPI package moved into LLVM, but they don’t have many people that feel knowledgeable enough about packaging to approve changes.

[obtuse holly]

Curious what everyone's personal preference is for a frontend for building pure python wheels.

Context is I'm working on a deployment workflow that follows all of the best practices

[lyric hedge]

pip wheel is more for package consumers and python-build for package authors

So in short, use https://pypi.org/p/build.

[obtuse holly]

yee, I started with https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/ as a base but I have feelings about it

I like it for demo purposes but putting the build and publish in the same workflow file seems odd to me?

plus that implementation means anyone who can push can publish

[lyric hedge]

You can restrict who can publish using GitHub environments which are gated on required reviews.

You can totally go with a manual trigger workflow, too.

[inland prawn]

You could even skip the CI and always push manually from your machine 😉

[oak quiver]

it seems like most backends now are part of a suite that includes a corresponding frontend anyway...

like flit-core makes sense for pure Python but you if you get the full flit package then that is also a front-end with its own cli and everything. And something like hatch(ling) is much larger than that

and yeah, to me "CI" is a big scary thing that hobbyists rarely even have a good reason to think about, much less consider. I've been doing all my publishing from local and it's kinda hard to imagine not doing so

[mystic stag]

https://discuss.python.org/t/clarify-that-arbitrary-equality-is-supposed-to-be-case-insensitive/104885

tl;dr I plan to add clarifying language to the the spec for arbitrary equality (===). ASCII versions shall be compared in a case insensitive way, non-ASCII behaviour is unspecified, This is to match existing legacy systems and not get into a protracted discussion of what case insensitive means for non-ASCII. If anyone has an objection to that please post there.

[oak quiver]

... the standard accepts non-ASCII package names?

https://packaging.python.org/en/latest/specifications/dependency-specifiers/#names

(to be clear, I support the idea either way)

[mystic stag]

Standard versions don't, but arbitrary equality is special: "This operator is special and acts as an escape hatch"

[oak quiver]

I hadn't understood that to override what's allowed in the string in the first place, but.

[oak quiver]

(FWIW, after reading through the thread: I agree that the spec does currently describe a case-sensitive comparison. But I also agree that changing the spec to match behaviour, on a deprecated legacy thing, is better than changing the implementations, absent objections from legacy maintainers.)

[mystic stag]

I mean, I also came to the same conclusion when I started fully implementing it in packaging, it was only because it broke backwards compatability, and then I realized the spec described it's own intent, and then I investigated the history and found that's how it's always intentionally been (with tests and everything!)

So this led me to the conclusion the spec needed updating

It's also really weirdly worded if there was no normalization intended, like "simple equality operations" instead of "exact equality operations", but I don't want to tackle that section of the spec as a whole, just this small bit

[oak quiver]

That was specifically what I was getting at, actually. There is phrasing like "these are simple equality operations" and I parsed it like "any instance of this is a simple equality operation"; and I considered that there is only one meaningful "equality operation" in context and that "simple" is meant as descriptive rather than restrictive.

which I suspect is how your interlocutor in the thread parsed it, too.

[mystic stag]

Yeah, language is hard 😔,and I'm particularly not great at it

[strange knot]

does anyone remember the recent-ish example of a pypi project with wildly inflated download counts and the root cause ended up being some mis-configured container image that kept downloading it over and over? I want to reference that but I can't find it 🙁

[honest geyser]

uhh cloudflare workers were at fault

[uneven karma]

(it was #pypi message)

[honest geyser]

thanks!

also https://x.com/samuel_colvin/status/1952003062198890583

Samuel Colvin (@samuel_colvin)

Looks like @Cloudflare is absolutely hammering @pypi.

Two cloudflare packages, and one package which I guess is a dep of the others are being installed almost 3 billion (!!!) times a month.

And there was I thinking AWS was bad for 1.3b downloads.

Is this workers python runtime

X · Aug 3, 2025, 1:46 PM

[oak quiver]

I feel like the top pypi packages look quite a bit different from how they would if everyone had well considered setups

(I mean e.g. where CI setups can actually locally cache properly and such)

[oak quiver]

I've started writing some user-facing stuff about packaging (i.e. installation and environment management); there's basically just a "plan" here so far, but feedback is appreciated
https://github.com/zahlman/installing-python-code/wiki/Overview

[oak quiver]

A query from the main Python Discord:

is there a way to, in [pyproject.toml], specify that a dep[endency] should not be installed when the python interp[reter] is free threaded?

[tardy isle]

nope, that’s PEP 780

[mystic stag]

Oh look, the dependency specifier grammar was never actually able to parse arbitrary equality requirements: https://github.com/pypa/packaging.python.org/pull/1969 🙃

[mystic stag]

For the morbidly curious, I found this investigating why a comma could be used in a version in arbitrary equality for a version specifier but not for a dependency specifier: https://github.com/pypa/packaging/issues/1000#issuecomment-3599840944

[tardy isle]

it's over; it's done: https://github.com/numpy/numpy/pull/30340

^ numpy.distutils is gone in numpy main, woohoo!

[oak quiver]

I look forward to the reduction in wheel size

[oak quiver]

I also see some new financial support for Python: https://vercel.com/blog/investing-in-the-python-ecosystem

[lusty quarry]

What would be most appropriate wheel tags to use for projects whose sole purpose is to distribute prebuilt binaries? For simplicity, let's assume that the minimum supported version of glibc for the Linux binaries is unknown and that the platform support matrix is limited to Linux, Windows and macOS each running on either x86_64 or ARM64. Here are 2 examples:

• Ruff https://pypi.org/project/ruff/#files
• DotSlash https://pypi.org/project/dotslash/#files

I see two differences, one being musllinux versioning (1.1 vs 1.2) and the other being multiple tags for the Linux wheels. What would these affect in practice?

[blazing lantern]

py3-none-* without introducing some way to say, "Python doesn't even matter" for the interpreter part of the tag.

[lusty quarry]

Would you recommend adding that second wheel tag on Linux to support old versions of pip that don't recognize modern PEP 600 tags?

I'm not sure when support landed and by extension which distributions would be unsupported.

[blazing lantern]

https://mayeut.github.io/manylinux-timeline/ should tell you how far back you may want to care

[west drift]

depends on whether you have strong performance requirements. if not you can use statically linked binaries and label them with the lowest manylinux and musllinux tages you like (cause there no proper staticlinux tag). musl plus a custom allocator can be a few percent slower than glibc in our benchmarks. maturin uses this.

otherwise, musllinux 1.2 is fine for musl (it works at the scale of ruff), for glibc i can't say what version to target

[blazing lantern]

https://github.com/pypa/packaging/blob/5de9865be26d93b726f4d85da7f8dda6480d2fb5/src/packaging/_manylinux.py#L208-L215 has the mapping manylinux tag names to glibc version if you statically compiled glibc into a binary and you just need some number

[oak quiver]

(original context:)

I have actually been thinking about a tool that pre-packs existing .pyc files into an atlas that can be read into memory ahead of time, and then a loader that retrieves the needed bytes from there for the marshall.loads call

but I guess .pyd is Windows-specific... ?

[narrow snow]

Hi, I’m Sanchit. I’m exploring pip/PyPI for GSoC 2026 and long-term contribution. I’ll begin with the “good first issues” and follow the contributing guide.

[blazing lantern]

Yes

[oak quiver]

https://sethmlarson.dev/deprecations-via-warnings-dont-work-for-python-libraries

[mystic stag]

It's true, no one reads DepreciationWarnings better to have your own class if you want someone to actually read them

[oak quiver]

Is there any documentation anywhere of the intended semantics of trove classifiers? Often I read through and can't decide whether a particular classifier should apply to my project.

[indigo token]

It's simple: just use FutureWarnings (or subclasses thereof), they're visible and mean the same thing.

Definitely don't create a Warning class for deprecations that doesn't inherit from either FutureWarning or DeprecationWarning, that's not semantic.

[inland prawn]

I was actually thinking about some kind of IO stream that build backends could use to emit info about warnings/deprecations that would be printed out on package installation (kinda like what npm ecosystem has)

[mystic stag]

Are you aware of https://github.com/pypa/pyproject-hooks/issues/157 ? There's an open issue but once we solve it and vendor it back ends will be able to publish warnings by default to front ends during installation

I've been meaning to work on it for the last few months, but I keep getting distracted

[inland prawn]

wasn't aware there is an issue about it, but I am not surprised someone already thought about that. The only concern I have is about cases when installation is done from a pre-built wheel, since IIRC the pyproject-hooks won't be part of that pipeline (maybe some DEPRECATIONS.txt file that installers can read? don't know, shooting blind here)

[lyric hedge]

Right, but we also don't want to spew out a ton of warnings that are inactionable. If pip is installing from a wheel, it's unlikely a build backend deprecation warning is useful since the user is probably an end user.

I suspect that a first iteration of build backend warning forwarding would be gated on local project build/installs.

[rare shard]

Hello,
I'm reaching out because the talon-v3 package, which is a library from Mailgun for parsing emails, seems to have been removed from PyPI very recently.
Was the package deleted by the maintainer (Mailgun) or was it removed by PyPI admins for a security or policy reason?

Any official confirmation or link to a public statement would be extremely helpful as I could not find any.
PS: If there is specific channel for this kind of query, then please lmk.

[oak quiver]

#pypi ?

[woven yarrow]

I need to fix how coverage.py finds the stdlib and the installed third-party packages. https://docs.python.org/3/library/sysconfig.html of course seems like the way, but here's what I don't get: How do I know what "scheme" to look up with sysconfig.get_paths(scheme)?

I had been using all the schemes that contained os.name, but "venv" is important. Should I use os.name plus "venv", or be more selective, and if so, how?

[west drift]

is it possible to read all packages in sys.path? this also handles a bunch of odd cases that are outside std and the venv

[woven yarrow]

I can't skip everything in sys.path, because the code you are trying to measure is in sys.path.

Maybe the right thing is to just iterate all of the schemes, and take all the *lib paths.

I think part of my problem is that sysconfig paths seem focused on helping with "where should I write files?" not, "where did files get put?"

[woven yarrow]

also, there seem to be times that some third party packages are in sys.path in directories that aren't mentioned by sysconfig at all, which makes sense, since sys.path can be quite arbitrary.

[oak quiver]

the fact that sys.path is just a list does seem to cause some problems.

there's no semantic grouping of path types, and there's also no shortcutting of "x module should only be in y path"

(I guess the latter is by design; certainly it empowers the namespace packages that were added later. but people hardly use that now it seems)

[tardy isle]

there’s far too much unmaintained code messing with sys.path to do much about it without a lot of pain I think

[woven yarrow]

In this case the problem isn't code messing with sys.path. Generally people who do that do it to get "their" code importable, and they want their code measured by coverage.py

The problem is that uv sometimes installs third-party modules in unusual places. Details are in this bug report: https://github.com/astral-sh/uv/issues/17041#issuecomment-3643218514

[west drift]

without knowing much about coverage.py, is it possible to detect what is first-party code, and remove that from the list you get from sys.path?

e.g. first party code is usually installed as an editable, or at least from a local path (vs. from a registry)

[woven yarrow]

I don't think it's generally true that first-party is installed as editable.

[tidal kiln]

hey everyone, I need to raise an question with the core team regarding what kind/shape/scenarios of installations we support, which I think is a good opportunity to explore maybe "soft-deprecating" in some shape cross-directory native dependency trees

meaning certain packages would be guaranteed to share the same site-packages

making it possible to use rpath

does anyone have thoughts on this?

there are multiple different things a proposal like this touches, so I'd like to hear from the community

[west drift]

i meant it less for editables specifically and more in the sense whether there's some heuristic to tell what in sys.path is first party wrt to coverage.py

for the uv/coverage.py bug, it seems that the venv layering that uv does for --with is confusing coverage.py's logic

[woven yarrow]

this is absolutely true. I think the answer might be that when things get complicated (like with uv --with), then people need to configure coverage to specify exactly which modules to measure.

[peak ocean]

Can you clarify what a cross-directory native dependency tree is?
Does this limit all third party libraries to being in a single site-packages tree? If so that breaks virtualenv --system-site-packages
Maybe it would be helpful to understand where you think rpath would be useful, but is currently impossible to use.

[tidal kiln]

dependencies of projects shipping libraries need to be in the same directory as their dependency

so that they can set rpath properly without it breaking

yes, --system-site-packages wouldn't be available in some use-cases

but isn't that what you as a distro want? no mixing system and PyPI dependencies?

distros keep patching stuff to better suit their use-cases, and that's fine for their packages because they can patch/control them to avoid things from breaking

but it creates a fundamental incompatibility with PyPI

and I am honestly super super tired of trying to bridge things and try to achieve a reasonably good UX across everything

regarding Debian in specific, I have put a tremendous amount of energy trying to make things better, but I don't think anything I have ever done has materialized in anything

so yeah, if I can't do that, I can try to put a reasonably scoped barrier between the two ecosystems, to avoid the current bad UX

I have done a lot of outreach, and am a member of the DEI WG, which cares very much about UX, and Python package managing has always been one of the top issues for new users — the first thing everyone says is do not use the Python from your Linux distribution, install it from somewhere else

I am tired

sorry for the burst, I am just super socially, mentally, and physically tired right now

the proposal above isn't even about distros, that just happens to be the major side-effect

[peak ocean]

@tidal kiln: Thanks for your efforts. As to whether they made anything better, it's hard to tell. I have been putting effort into trying to bridge between the upstream Python world and Debian, too, in the last few years. I think we've managed to get past some of the vitriol, and persuade people that engaging across communities helps. After all, we're all playing in the same space and if we don't work together, it's our users who get hurt.

Many of us are stuck with decisions from a long time ago. Not stuck permanently, but there's a huge inertia that makes change hard. PyPA is right in the middle of that.

Here I was still just to clarify exactly what you were talking about, so I could even engage in your question.
If I understand it correctly, you're proposing that when a Python project is installed (via pip/whatever) into a Python virtualenv, every dynamic library that gets dlopened will be in the same directory tree (sys.path entry) as anything that imports it. Is that correct?

[oak quiver]

that seems compatible with system-site-packages actually? since you'd be accessing any system site packages via the Python wrappers, which then import the C implementation part from their own path entry
but it puts constraints on how the distro's Python is arranged

[peak ocean]

But if something depends on one of those system libraries, is that a problem?

the first thing everyone says is do not use the Python from your Linux distribution, install it from somewhere else
OT for this channel, but FWIW, if that's a team's first response to users, we should talk about why that is and try to improve on that. Because (from the distro point of view) I don't think encouraging users to install things from outside their distribution's package manager is a great idea. New users need to learn how to administer their systems before they can safely do that without breaking things.

[tidal kiln]

yes, that's the problem, but we can have packaging managers kindly tell the users that isn't supported

it would push us so much further on standardizing, and officially supporting "properly" projects with external dependencies

right now, our real world solution is to trick the linker to add a path to the library search path

[oak quiver]

if "everyone" == "people trying to teach new Python programmers how to participate in the PyPI-powered ecosystem" then it's definitely understandable
in particular, by "use" people may have in mind "install packages into the environment of" (i.e. what PEP 668 is about)

also e.g. on Debian family even if you make a venv, the system Python might fail to provide a chunk of stdlib stuff without additional Apt packages (notably all the tkinter stuff)

[tidal kiln]

which while seems to be surprisingly easy on GLIBC, but making work on Musl for eg. involves dynamically generating a stub ELF with a DT_NEEDED entry pointing to the absolute path of the we want to be loaded next

[oak quiver]

(honestly I think we'd be better off if the distros pre-made a venv and had their system tools use it, by default)

[tidal kiln]

no no, that's not a team first response, that's the general first response from almost everybody that is helping someone get started with Python

I think that's the magnitude of the problem you are failing to understandand

I am not saying distros are the only ones responsible, the native dependencies on PyPI problem is also a key factor, but distros patching Python, and the way they are distributing it, is a very big part of it

[peak ocean]

That's what the system-python proposal was all about. But we've made 0 progress on that...

[tidal kiln]

because that was never the real issue

[peak ocean]

We don't want to be patching it in a way that causes problems for anybody. Our patches are there to solve real problems.

[tidal kiln]

I participated in the discussions that lead to that, and that was the only actuable thing we could agree on

[peak ocean]

Anyway, this whole thread really belongs in a #distros room or something. It's distracting us from discussing your proposal.

[tidal kiln]

it's all connected really

[peak ocean]

OK, I still don't understand exactly what problem you're trying to solve with rpath

my expectation was that Python libraries that need C dynamic libraries ship them and use rpath, already

[tidal kiln]

when I tried upstreaming a way to standardize a vendor-specific install scheme, from what I remember Debian wasn't convinced and wouldn't commit to saying they would use it

that pretty much killed the proposal

https://gist.github.com/FFY00/625f65681fbcd7fc039dd4d727bb2c2f

all of the issues Python people were worried about, like startup time, I can fix right now

and now have enough weight to push this forward

so if we can get Debian to commit to using that, or something like it, we can get it implemented

[peak ocean]

Sorry, this brings back vague memories, but I've forgotten most of the context of that issue

[tidal kiln]

IIRC Debian's problem was that the vendor scheme needed to share the same prefix

or rather, the original scheme

[peak ocean]

Our custom install schemes do cause havoc in all build tools.
They seemed like a good idea at the time (a hack around distutils not really understanding prefix), but not convinced they've paid off in the long term

[tidal kiln]

they wanted it going to /usr/local

[peak ocean]

Yeah, we really want to keep non-package-managed stuff out of the package-manager's part of the FHS

I think all distros probably feel similarly

[tidal kiln]

that's the issue, I am not putting in question that you are fixing an issue, just that when you do this kind of stuff, you might not be aware how much it could impact the rest of the ecosystem

that's why I wanted to bridge things

so that I can do my best to design things in a way that helps you fix your issues, but doesn't impact the PyPI ecosystem badly

[tidal kiln]

but that's against the FHS

[peak ocean]

Right, but do understand that those decisions were all made long before I was involved here, so please don't ascribe any of the problems they cause to me. I'm interested in helping to fix them or make things less sharp.

[tidal kiln]

right, I am not saying you are responsible

[peak ocean]

What do you mean about it being against the FHS?
/usr/local is for the sysadmin to put things in.

[tidal kiln]

just that things have always been complicated

[tidal kiln]

nothing in /usr/local should be used by a program in /usr

bc then you get comflicts

which you did

so you renamed site-packages dist-packages

[peak ocean]

Our layouts were designed by core python people, who were involved in Debian. We (the rest of Debian python community) thought they knew what they were doing. And I still assume that they did. I'm sure many bigger plans were not followed through with (like getting similar mechanisms adopted upstream / across distros)

Re dist-packages - that was to permit /usr/local/bin/python to be something you install yourself. And it doesn't even attempt to share any of the system's modules.

[tidal kiln]

from what I have talked about with people that were involved at that time, nothing was really very thought very carefully

it was a design from another time

[peak ocean]

nothing in /usr/local should be used by a program in /usr
I think that's a fairly extreme interpretation. I think Debian's approach here is more in line with the principles of the /usr/local hierarchy.

[tidal kiln]

it made sense, but as things started getting bigger, it stopped

[peak ocean]

Possibly so, but there's huge amount of inertia behind it now

[tidal kiln]

it's not IMO, when you do that you directly conflict with a Python installation in /usr

there's a namespace for a reason

[peak ocean]

I'm not following. What conflicts?

[oak quiver]

I think this would be easier to understand with concrete examples of what can go wrong
But yeah especially now that distutils is gone it does seem like the distro's fault if they don't do anything about sysconfig
My guess is they wouldn't naturally feel much responsibility here because their own packaging systems will have .deb, .rpm etc. that already directly say where things are going to go; and they aren't expecting pip etc. to put anything in the system environment now that there's been the PEP 668 agreement. But there are still problems apparently?

[tidal kiln]

if you make users install packages to /usr/local but then use them in /usr Python

[peak ocean]

/usr/local/pythonx.y/dist-packages (or whatever it is) is not /usr/local/python3.x/site-packages/

[tidal kiln]

because you had to rename it

you are solving your issue with mangling the Python install layout even more

[peak ocean]

Yes, but it solved it.

[tidal kiln]

but created other issues

[peak ocean]

(and it's been that way for decades now)

[tidal kiln]

like we talked about

https://ffy00.github.io/blog/02-python-debian-and-the-install-locations/

this is one of them from the migration

but I talk about the FHS thing there

[tardy isle]

now there’s a desire to also depend on system build and runtime dependencies in a hopefully organized way

[tidal kiln]

but I agree the FHS is really not meant for software like this

[tardy isle]

more than the ad-hoc way it works now

[tidal kiln]

it's old and outdated

[peak ocean]

That it may be, but it's what we have

[tidal kiln]

okay, but I just now proposed to put up the work in the upstream to get us to a middle ground that works better for the end user

[peak ocean]

Which proposal are we talking about?

[tidal kiln]

Debian would just have to concede the /usr/local thing basically

[tidal kiln]

this basically

[oak quiver]

you mean, like, the PEP 725 stuff?

[tidal kiln]

updated but essentially that

[oak quiver]

yeah if people want to not have to vendor stuff all the time then that does add a certain urgency...

[tardy isle]

ideally pip would be able to say “please install libxyz otherwise the command you just did will fail” instead of failing 20 minutes later with an obscure error

[oak quiver]

there's kind of a problem in that if you're expecting system dependencies then they can't be "virtualized" even if your package is installed in a venv

oh is this specifically about build-time requirements then?

[tardy isle]

both because libxyz is needed at runtime too, with a wheel it’s bundled statically via the delocate tool

[peak ocean]

Conceding to lettin gpeople break the package manager is a no-go

I think the compromise there would be to make PEP-668 harder

[tidal kiln]

no?

how would it break the package manager?

[peak ocean]

Read the intro to PEP-668, it explains the issues

[tidal kiln]

this would have absolutely no affect on PEP 668

[peak ocean]

At least if users install things into /usr/local, they don't mess with the package manager

If they were to install things to /usr, it's not that you might break things (the way things are now) but that you almost certainly will

[tidal kiln]

what's different here if python is picking up the path from /usr/local anyway?

[tidal kiln]

I agree

[peak ocean]

I thought you were asking us to remove /usr/local from our path?

That's how I interpreted:

Debian would just have to concede the /usr/local thing basically

[tidal kiln]

my request is:

• you don't change the default scheme, pip install goes to site-packages (with the same PEP 668 protection)
• debian packages go to a new debian-packages directory

and then I would also like to propose a cmdline flag to disable non-vendor customizations

[peak ocean]

you don't change the default scheme, pip install goes to site-packages (with the same PEP 668 protection)
So pip would install to somewhere not on sys.path. That sounds terrible for users.

[tidal kiln]

it would be on sys.path...

[oak quiver]

(but debian packages already go to dist-packages?)

[tidal kiln]

all under /usr

[peak ocean]

Our sys.path doesn't contain any site-packages. We have the single /usr/lib/python3/dist-packages, and the /usr/local/pythonX.Y/dist-packages

[tidal kiln]

and users can install to the latter?

[peak ocean]

Yes

[tidal kiln]

then what is different other than the paths??

[oak quiver]

fwiw I'm on Mint

$ /usr/bin/python
Python 3.12.3 (main, Nov  6 2025, 13:44:16) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/usr/local/lib/python3.12/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.12/dist-packages']
>>> 

no site-packages here.

[tidal kiln]

it's the same thing, just with all the paths under /usr

[peak ocean]

So, you want users to install with pip into /usr/python3/dist-packages

That's a directory that is full of package-managed things

[tidal kiln]

no, site-packages the default directory

on every system

[peak ocean]

That's not on our sys.path

[tidal kiln]

because you are patching Python to remove it!!!

[peak ocean]

You're asking us to completely change how we ship Python

[tidal kiln]

don't

yes

[peak ocean]

That's not something we can just change on a dime

And I really don't see any need to do the huge amount of work to make it so.

[tidal kiln]

I am a distro packager, I understand the amount of work I am asking of you

[peak ocean]

I'm not seeing any benifit. AndI'm seeing a lot of downsides.

[tidal kiln]

I will repeat myself then

[peak ocean]

Our layout solves real problems.

[tidal kiln]

.

[peak ocean]

I really don't understand how the other distros don't have similar issues with Python's default layouts

[tidal kiln]

we already went over this

[peak ocean]

No, not really. I noted that I've spent a lot of time and effort adding support for our layouts to build tools. And we appreciate the build tools for taking these patches.

But I'm not seeing how changing any of this helps.

[tidal kiln]

because next time something changes, it's the same thing!

[peak ocean]

Yeah, but it's simple

[oak quiver]

(again, I think concrete examples of user stories would be helpful here)

[peak ocean]

And the alternatives are way worse

[tidal kiln]

new build tool! oh you forgot to patch for Debian's quirkiness

this is not good

[peak ocean]

Rigth, but how often do new build tools arise

And... ideally we'd have this more standardised upstream in Python, so it isn't a Debian quirk

But a standard thing that the whole world does

[tidal kiln]

yes, I have given you basically one condition

all paths need to be /usr

pip and debian can install to different paths

[peak ocean]

That goes against FHS, so it's a hard sell within Debian

[tidal kiln]

it doesn't

what you are doing does

[peak ocean]

User-installed (from source) things into /usr are against our FHS

[tidal kiln]

but I guess we won't get anywhere here again

I tried... again

same result

[peak ocean]

I think your interpretation of FHS is not the same as other people's

[tidal kiln]

your interpretation is literal and conflicts with the conceptual models established by the FHS

prefixes are namespaces

[peak ocean]

I think your interpretation of there being hard walls between the namespaces is not the goal of the FHS

[oak quiver]

"user-installed (from source)" is meant to include third-party sdists?

[peak ocean]

Debian's interpretation (which I'm sure you understand) is that /usr/local is "for the sysadmin" and /usr is "for the package manager" (in very rough terms)

https://www.debian.org/doc/debian-policy/ch-opersys.html is our documentation on the topic

[tidal kiln]

yes but that has never worked for software with their own package managing

[peak ocean]

Right, we're at the intersection of package managers here. It's "fun" 😛

But the Python package managers don't intend to control the "system" rather they install things into a Python

We make a space available for them to install to, inside /usr/local where a sysadmin would expect their own installed bits to be

[tardy isle]

fwiw, this is why I always use a pyenv of conda userspace python install, and I thought that’s what the debian maintainers wanted

[tidal kiln]

we are going in circles

like I said, I tried again

I don't want to spend more time re-iterating my position

[peak ocean]

@tardy isle : We don't want anything really. We just provide the things that we think are useful. (And not all the things that would potentially be useful, because manpower...)

[tidal kiln]

you know my position, if anything changes, feel free to let me know

[oak quiver]

.

[peak ocean]

@tidal kiln If you can help my by ponting to concrete issues, I can try to discuss things within our community. But my gut feeling here is that what you're asking for is against FHS, so it's going to be a hard sell.

It would really help if I had motivation for the change

Let's start from first principles - user stories

[peak ocean]

I'd say anything installed from source (the old unix way) or using a third-party package manager.

@tidal kiln : Similarly with PEP739 - knowing how to handle it would be helped with some concrete examples of expected use-cases.

[tidal kiln]

your patching of Python delayed the distutils deprecation for probably years

every tool needs custom handling for debian

etc.

[peak ocean]

And it all needed to change when distutils was replaced with the sysconfig way of declaring these things. Yes.

But, the sysconfig way is much cleaner, thanks!

[tidal kiln]

no, there was a direct path

[peak ocean]

(and urgh at _distutils_hack etc.)

[tidal kiln]

the issue was debian

debian was patching the old way and not the new one

everything I said in my blogpost

I am gonna stop engaging on this particular issue here

[peak ocean]

I think that's a case of us not understanding the urgency and necessity of the change, until tools started to use it.
That's our bad, and we had to fix it, yes.
Maybe more communication about the changes would have helped us align sooner?

[tidal kiln]

there are tons and tons and tons of user stories on the bug tracker of every python package manager

[peak ocean]

You could help me by pointing to some examples

RIght now I'm in the dark, really

[tidal kiln]

sigh, just go into pip's bug tracker and search debian

[peak ocean]

And yes, your blog post mentions many things about what we patch, and how it caused you pain. But it's such a big grab-bag, it's not actionable in any way that I can see

[tidal kiln]

this is draining energy I don't have

I have made my offer

[peak ocean]

What have you offered?

[tidal kiln]

again, the gist above

[tardy isle]

https://github.com/pypa/pip/issues/13208 seems about right

[tidal kiln]

you say it's against the FHS and it's a non-starter, let's stop wasting time then

let's talking about something maybe more productive

multiarch

[peak ocean]

I can reply to that with advice (as could I'm sure @tidal kiln who was also a PEP-668 author, which is what that ticket is about)

[tidal kiln]

do the multiarch versions of python in debian share site-packages/dist-packages?

[peak ocean]

Firstly, there aren't "multiarch versions"

[tidal kiln]

I am gonna re-iterate that I never thought that PEP would solve anything

just make things slightly better

[peak ocean]

Multiarch is the mechanism that allows packages from multiple architectures to be installed concurrently

[tidal kiln]

okay, what about PyPI packages?

you can control the distro packages to work there

[peak ocean]

So yes, all Python architectures (if you could have multiple installed concurrently, which you can't have for the interpreter itself, but you can have embedded in other things) will use the same dist-packages

[tidal kiln]

but PyPI packages will break, right?

[peak ocean]

If you install a PyPI package it will go into /usr/local/python3.x/dist-packages which will be used by all the architectures of Python

If you're installing something with extensions, they'll only be built for one arch, obviously

[tidal kiln]

so, yes

[peak ocean]

So won't be importable on others

[tidal kiln]

it will break

[peak ocean]

Same goes for packaged python modules

[tidal kiln]

that's my fundamental issue with your multiarch implementation

[peak ocean]

Multiarch is way wider than Python. It's far from perfect or fully-implemented everywhere. It's been a slow incremental development.

[tidal kiln]

same thing as this

https://github.com/python/cpython/issues/127294#issuecomment-2501787837

awful UX

[tardy isle]

which is really annoying to practically deal with

[peak ocean]

Yes, exactly

And we'll have the same issues with free-threaded builds

And multiple python interpreters (pypy + cpython) etc.

[tardy isle]

https://github.com/pyca/cryptography/pull/13954 just a couple weeks ago

[tidal kiln]

no, because their runtimes should be separate

[peak ocean]

The problem spaces are just much wider than our package manager can reasonably represent and follow all of the dependency trees for

[peak ocean]

You can say that. But we say that it makes our lives a lot harder. By sharing runtimes, we're able to much better stay on top of new upstream python releases

[tidal kiln]

and by not doing that you create an horrible UX

[peak ocean]

I don't disagree that it has downsides

[tidal kiln]

I truly do understand the amount of work I am asking of you

[tardy isle]

please don’t handle free-threaded python like that, if you ever get around to setting packaging for it

[tidal kiln]

but IMO that UX is unacceptable

[peak ocean]

@tardy isle : The alternative is WAY harder for us

I mean, hundreds of times more work

[tardy isle]

well, just telling you it’ll cause real issues for your users

and python is changing its ABIs…

[peak ocean]

That (changing ABIs) doesn't concern me much

[tidal kiln]

PyPI is fundamentally incompatible with that

[peak ocean]

How so?

[tidal kiln]

you can't install multiple variants of the same package

we can work on it

but it will never be implemented in time

you will cause havoc on users

[tardy isle]

user a installs free-threaded numpy with pip

user b imports numpy with GIL-enabled python

seg fault

i think in 3.15 it’ll be a runtime error?

[peak ocean]

This is why you have t in the filename

[tidal kiln]

the extension suffixes should cover that

[peak ocean]

It's just not importable, not a segfault

[tidal kiln]

but it will be the same thing as the issue I linked

[tardy isle]

on windows with python.h not setting Py_GIL_DISABLED things can sometimes go wrong in confusing ways

c.f. the cryptography PR above

which was only problematic building conda packages on windows because of that packaging quirk

maybe all this is windows-specific and other linux differences mean it’s less bad

i’ve only ever seen this awfulness on windows

[tidal kiln]

and we have this on windows because of technical limitations

hopefully we will be able to fix it soon

[peak ocean]

I don't see anywhere that that cryptography bug was actually figured out. It was papered over.

[tardy isle]

how so?

I spent a day or so debugging…

[peak ocean]

I am just not finding the comment that explains what's going on

[tidal kiln]

if you are going this route, my recommendation, make it impossible for external installers to work

this only works for you

[peak ocean]

That is what PEP-668 is about

[tidal kiln]

no

that is a guard rail

make it a hard error

[peak ocean]

There are users who are depending on being able to override it

[tidal kiln]

and it will be horrible

[tardy isle]

oh I see, I never did figure out how it hapenned to work before, but setting Py_GIL_DISABLED there is definitely correct on Windows and it wasn’t happening before. I’m not sure why it was possible to call into a GIL-enabled C extension. Somehow the pyobject layout doesn’t matter?

[peak ocean]

And honestly, if they are in an environment where their system package manager doesn't need to work reliably any more, or they understand exactly what they're doing, it's OK.

[tidal kiln]

it's not

users will override it without understanding

[tardy isle]

so I’m taking exception to you calling that PR “papering it over” a little, hope you understand where I’m coming from

I’m not a windows expert and debugging on windows is unpleasant

[peak ocean]

Yeah, I do understand that. Sorry 🙂

My point was it's hard to take that as an example of the problems you run into, when I don't see a clear explanation of a problem.

[tardy isle]

the problem is that python.h doesn’t set Py_GIL_DISABLED on windows, because all builds share a site-packages directory

[peak ocean]

They could use a real package manager on Windows 😛

I guess the lesson there is that python.h needs to differ for the free threaded build

[tidal kiln]

that's the plan

[tidal kiln]

yes, and we have that already on linux

[tardy isle]

also because all builds share a site-packages directory, pip-installing stuff can lead to broken installs

[tidal kiln]

the problem is package installation

[tardy isle]

https://py-free-threading.github.io/installing-cpython/#__tabbed_1_1

these instructions go out of their way to tell people to install the nuget builds

which don’t do that

[peak ocean]

There's not a whole lot we can do about that. We initially did PEP-668 without the escape hatches, and there were a lot of complaints.

[tidal kiln]

@peak ocean what I hear from you here is "it's too much work for us, so we rather let users have an horrible UX"

which to me is sad

perhaps you don't have enough resources, I don't know

[peak ocean]

That's a fair point, and there's a lot of nuance to be found in this space.

[tidal kiln]

but I'd think a Canonical-supported distro would

I agree

[peak ocean]

You're on the front line of dealing with the users in a way that we distro-maintainers are not.

[tidal kiln]

I definitely think this is something you might want to take to the Debian higher ups, however they are, I am not sure of the structure, before you implement it

[peak ocean]

Typically only very techincal people make it to Debian's bugtrackers (maybe because it's email driven 😛 )

There are no higher-ups

We're a community project

[tidal kiln]

I am both

[peak ocean]

Right, but I'm not involved in pip upstream

I happen to work for a company that funds work on Debian. And there are a few of those, Canonical included (although they prioritize working on Ubuntu)

But almost all of the work in Debian is going to be done by individuals on their own time

[tidal kiln]

then ask something of Canonical or something like that??

this will be a really bad look for Debian IMO

what you are doing affects them, via Ubuntu

[peak ocean]

And I think the people who care about the usability of Python for new developers in the Python upstream have a very different idea of how to approach things than we do

Until very recently the python packages in Debian were maintained by a Canonical employee. He's retired now.

[tidal kiln]

right

[peak ocean]

I think they have someone on the foundations team who has some python responsibility

But I don't think they have the time to massively re-invent anything

And for doko and me, looking after the interpreters. Just staynig on top of bugs and new upsteam python releases is a fairly large time sink

[tidal kiln]

I am just saying, this will be horrible for users

[oak quiver]

did Windows have analogous problems before the introduction of free-threaded builds? I can't recall any

[tidal kiln]

IMO it would be better if you boarded up and straight up broke Python package managers

which is a very high bar for me to say

[peak ocean]

I really think that's what we did with PEP-668

I guess you disagree because you get to deal with the users who override that 😛

[tidal kiln]

no, that's just a silly little command line flag 😜

[oak quiver]

I remember Windows having a separate site-packages per Python version; but no-GIL introduces additional builds of the same version and that causes the problem, is that it?

[tidal kiln]

yes