general

[lusty quarry]

omg I dislike them so much

they call me once every 4-6 weeks to ask questions about stuff related to Python

I always ask to be taken off their list but it never happens

and they try to acquire my work email for some reason

(and yes I block the numbers)

[lusty quarry]

FYI related to that article, I'm going to release something Monday or Tuesday for what I view as the final solution to distributing Python applications

and by next Friday this is what Hatch will use and will become the new recommended installation mechanism

it is something I've done at work but I wasn't quite confident in, then I spoke to Armin over the weekend and he advocated for something similar which updated my perception

and then, finally, I can go back to feature development 😄 I was fixated for far too long on this distribution problem

[uneven karma]

[indigo token]

I'm both excited and slightly bemused by the word choice “final solution” (maybe hits different when one is a German lol)

[lusty quarry]

you will see when I release it and you get to try it out but I'm pretty sure it actually is the best possible solution lol

[woven plover]

possibly overbearing ideas - what if there where some logic gate environment marker that enabled fun use cases like

• when_installed ensure that if a optional dependency is installed it meets at least that version
• when_requested adds dependencies only when something is directly requested (like extra bells and whistles for a cli)

in the same vein what if the +/- signs could be used to define extras that install by default or get a opt-out

for example

[project]
name = "fun"
[project.optional_dependencies]
"+cli" = [ "rich" ]

then asking for fun should imply fun[cli] but asking for fun[-cli] would opt out of the auto-add

[woven plover]

possibly overbearing ideas what if there

[hidden flame]

Is that https://github.com/ofek/pyapp?

[lusty quarry]

yes

announcing tomorrow

[hidden flame]

nice, keeping my feet still

ah, also python-build-standalone based!

fascinating, we should get indygreg involved in this

if you haven't yet

[lusty quarry]

I haven't, he is very busy

[distant briar]

The inability to remove self commands is surprising

[lusty quarry]

what do you mean?

[distant briar]

I mean that you're forcefully putting self subcommand

Like, the executable acts the same as if you run python -m yourmodule unless the second argument is "self" in which case there are some custom subcommands added by pyapp

[lusty quarry]

I don't know of a better way to self-fix stuff

without that command everything is hidden

[distant briar]

The lack of any sort of customizability is what bothers me, I guess?

I don't know if it's technologically possible to allow the wrapped app to somehow call (I'm guessing no?) this self command such that it's up to the wrapped app how it decides to implement it but it would certainly be the most helpful. If not then I'm not entirely sure what kind of customizability I would expect and I'm sure that would be your next question (as would be mine if I was the author 😄)

[lusty quarry]

my plan for Hatch is to only show a top-level self command when installed with this method

[lusty quarry]

okay, I guess I could have a build time option that prefixes the command with an underscore and the environment variable that I'm going to add for detection could be, rather than just 1, the path to the executable

how does that sound?

[distant briar]

I don't really understand what you mean by the second part with the env var

[lusty quarry]

when invoking the actual application it will set an environment variable. previously it was just going to be a flag but I can make it be the path to the executable instead to serve both purposes

also please note that while the update command could be invoked by the application to the wrapper, the restore command could not on Windows because Python would be open and you can't remove open files

[distant briar]

You can rename if you're that specific process but other than that, yrah

[lusty quarry]

interesting, what do you mean by that?

[distant briar]

Ah, so it's to help find the executable you need to runs self subcommand on

[distant briar]

Like, if you're x.exe and want to update yourself, you can do rename("x.exe", "x.exe.old") and then create the new x.exe

I don't think you can really do it fully atomically but at least you could do it with two atomic operations

I thought you can only do the rename from that running x.exe but now I'm not so sure such a limitation exists

[lusty quarry]

actually it looks like I am able to use compiler time variables as part of the command generation so rather than prefixing by an underscore I will just add a new option to set the name of that command, defaulting still to self

thanks for the feedback, that's a good suggestion!

[distant briar]

Looking at it again, I realised why I initially just talked about the way to disable it - the Starship command seemed kind of odd and had me a bit worried about what other commands I could suddenly start exposing in the future in my executable when using pyapp if I don't check that it hasn't added something

[lusty quarry]

Also that starship command is hidden

[distant briar]

Hidden as in it doesn't show in help, I guess?

[lusty quarry]

yes

I believe in the readme I differentiate as well

(or at least I intended to, don't remember if I did)

[distant briar]

I would consider that to be similarly hidden to self command in itself then since by default, it is not visible when running help unless you explicitly add it to help in your own app that gets wrapped

It's just that I'm not sure that this really changes my feelings about it since it's still there. But I might just be a tad pessimistic :P

[lusty quarry]

FYI this is now a built-in build target in Hatchling and makes it a bit easier to use

[distant briar]

I'm still setuptools and sometimes flit guy 🙃 modern setuptools though :)

[lusty quarry]

fair enough! have you tried Hatchling?

[distant briar]

Maybe? I don't want to lie and I'm not entirely sure since it's really a the same when using pyproject.toml-based config

And I make new projects from time to time and try new things

So it's possible but probably not to a big extent

I'm not big on using project managers such as hatch (not hatchling) so I haven't really felt the need much

But yeah, idk

[lusty quarry]

really a the same when using pyproject.toml-based config
actually no! metadata is the easy part, the hard part to expose in a simple way is file inclusion, which I've tried to make as easy as possible

[distant briar]

I guess that's fair, I blindly rely on default inclusions unless I have data files

Honestly I'm just looking at pyapp from a perspective of how I would want to carefully prepare the exact CLI interface for the user

[lusty quarry]

And that's what most people do, I think Hatchling has the best defaults, in part due to the use of version control exclusion files

[distant briar]

I also have (at least?) one app where I ship multiple entrypoints which doesn't seem to be the target here since it obviously couldn't be a single executable then

But that's whole other thing to think about lol

[lusty quarry]

that's why I made the Hatchling built-in plugin!

it builds an executable for each entry point

every executable will use the same distribution behind the scenes

[distant briar]

Same distribution version?

Like, updating one updates all?

[lusty quarry]

yes the location on the file system is based on the environment variables for the project name and version (and Python distribution obviously)

[distant briar]

Interesting

I missed that

I guess it would be nice if I could decide that only one of those would have the self command

[lusty quarry]

I can add that to the plugin but it won't be until next week (feel free to open a PR after I add that environment variable renaming command option tonight)

[distant briar]

With the multiple command scenario, I actually wanted to have dedicated myapp-update command for proper updates prompting the user for anything they should know about before proceeding, etc

So that would work nicely with that, if I were to use pyapp

[lusty quarry]

@distant briar part one of your feature requests https://github.com/ofek/pyapp/pull/4

[lusty quarry]

part two https://github.com/ofek/pyapp/pull/5

[golden falcon]

if you skip_install I suppose you have to provide a custom Python distribution, or how would this work with distribution_embed?

(this is fantastic)

[lusty quarry]

Skip install assumes the project is already set up in the distribution

embed just determines whether the distribution is fetched at runtime

[golden falcon]

yeah, but how do you set it up in the dist?

[lusty quarry]

that would be up to the user I guess, basically setting up a distribution for each platform with everything they need installed then turning that into an archive that they host

[lusty quarry]

hey just FYI I postponed the announcement for a few days for 2 reasons

one is that I'm getting a fully working GitHub Actions config ready so users can start building apps immediately (I also had to do some learning about how to make cross compilation work)

the other is I am waiting for Ee to give me access to the official Python Apple developer account so I can make keys to distribute Hatch without that annoying security pop-up. we meet on Monday

I fully anticipate that by next Friday Hatch will have binaries for every platform and users will experience a frictionless installation experience finally

[obtuse holly]

... so hatch will be signed with the same keys as the python.exe for apple?

[lusty quarry]

it will be signed officially by the same org but the actual key and associated certificate will be different for security purposes

[obtuse holly]

whew good

though it feels weird to me, as it makes hatch feel officially endorsed or even official

[lusty quarry]

I guess it's a trade-off?

the feeling of endorsement versus requiring a maintainer to pay for an Apple developer account which costs $100 annually

the other option of course is to not sign which I view as a non-starter

under that option, the trade-off is the current hassle of installing Python versus an executable that requires fiddling with a system setting to run

and finally I suppose there is a high level trade-off that we are continually encountering

erring on the side of not recommending versus solving pain points for users

[lusty quarry]

cross compilation is now successful! https://github.com/ofek/hatch-showcase/releases/tag/v0.5.0

Hatch won't look exactly like that, I plan to put each binary inside an archive and also there will be an MSI for Windows and a DMG for macOS

also I forgot to chmod +x the non-Windows executables lol

but yeah, we're so close now

[lusty quarry]

Hatch is now signed and notarized for Apple https://github.com/pypa/hatch/actions/runs/4982663709/jobs/8918883759#step:18:28

Shout out to Ee for setting up the credentials this morning

[inland prawn]

did I read it right, that packaging.metadata will eventually replace the usage of pkginfo?

[tidal kiln]

I guess so

they both implement the core metadata standard

[blazing lantern]

If all that package does is parse PKG-INFO/METADATA and use built-in types from Python, then it can already replace it. I opened https://github.com/pypa/packaging/pull/686 to get feedback on an enriched API to returns more structured objects, but I'm waiting on @high stone to provide feedback or to just close it do to lack of interest

[inland prawn]

nice, would be nice to drop another dependency 😄

[mortal shore]

I ahve it on my TODO list to look at this btw, sorry haven't gotten around to it yet

[strange knot]

Hi folks, please let [e-mail] know if you find other instances of this user violating the code of conduct: https://github.com/pypa/setuptools/issues/3340#issuecomment-1561632849

[rapid glacier]

Which user in that relatively long issue thread issue thread is meant by that? There are a few participating.

Be aware if you click the embed instead if the link directly, you are sent to the start of the thread and not the intended comment. Discord is special.

Edit: with something like that it could be beneficial to manually disable link embed by putting the link inside of <> for example -> <https://google.com>

[hollow trout]

You should be able to just remove the embed.

[elder kettle]

How can you write all that text, spending all that time thinking about what you want to communicate, and still pollute your message with so much unnecessary hostility? That was so hard to read.

[strange knot]

Ack, thanks, removed the embed

[blazing lantern]

As long as I know someone will look at it that is already a big help (I have had this sitting on my work iteration plan for two months now, e.g. https://github.com/microsoft/vscode-python/issues/21196 😅); I'm starting to get pressure at work to solve the WASI installation "problem" (via pure Python wheels-only installer) and I was "stealing" work time to push upstream standards stuff as much as I can and this is part of it, but I'm starting to lose that goodwill around this specific project in order to get a solution sooner rather than later

[inland prawn]

I could take a look, but I am a total outsider 😄

[blazing lantern]

Totally happy to hear your opinion on the API! I don't need a code review but just feedback on whether my proposed API meets people's (general) needs (the raw parser support that's already in packaging.metadata provides a nice escape hatch to not feel the need to please everyone all the time 😉)

[mortal shore]

I can give you a light commitment end of week

[trim haven]

Hello fellow packagers! I built a website that you might find useful! It's called Manylinux Inspector an it's an index of the manylinux images, showing what software (and what version) is in each of them. https://manylinuxinspector.joerick.me/

[obtuse spire]

Hello, there may be a more specific place to ask this and if there is, please point me in that direction. … with that being said, does anyone know if there is a “pyinstaller” equivalent that can be used on iOS? I’m trying to make a simple python script executable. Trying to accomplish this 100% on my iPhone

[spiral urchin]

The “simple Python script executable” part may be difficult since Apple requires all apps to have a graphic user interface, not command line. But assuming you’ve figured that out, Briefcase can package code for iOS fairly well. There are still several hurdles for you to jump over (mainly due to iOS development, not Python) but it can be done. There are a few community links in the linked docs that can point you to the right people for this topic.

https://briefcase.readthedocs.io/en/latest/

[obtuse spire]

Thank you sir!

[tranquil meteor]

hey

https://github.com/pypi/support/issues/2910

Will it take one month to recover?

[lusty quarry]

are there usage numbers for PyPy? I don't see a need to test it in CI anymore (generally speaking)

[inland prawn]

I have never heard of it being used in production

[queen hornet]

last time I saw numbers it was pretty low, but I think it's good to support another implementation, in a similar way it's good there are options for packaging, and adding a 6th Python version to a CI matrix of 5 usually isn't too much overhead

[lusty quarry]

it wouldn't be so bad if it wasn't so slow compared to the other jobs

[queen hornet]

yep, that's why I put it as the first one in the matrix

[tidal kiln]

well, it's the best alternative for python 2

[woven plover]

For test and build tools it's really nice to have

I wish they had more manpower, I'm pretty Sure that there's a lot of potential and PhD opportunity in speeding it up and getting rpython more versatile Plus python 3 ready

[inland prawn]

I didn't even know pypy is still on python 2

[tidal kiln]

pypy supports 3.9, 3.8, and 2.7 atm

they said they will keep 2.7 around for the near future at least IIRC

[inland prawn]

inspiration came

[queen hornet]

How long will PyPy support Python2?

Since RPython is built on top of Python2 and that is extremely unlikely to change, the Python2 version of PyPy will be around “forever”, i.e. as long as PyPy itself is around.

https://doc.pypy.org/en/latest/faq.html#how-long-will-pypy-support-python2

[mortal shore]

I'm not sure I understand the reason for that, but I may just be too smol brained to understand. AIUI RPython is the language PyPy is written in, so I'm not sure I understand why that influences what version of the language RPython emits. But I might just be dumb 🙂

[spiral urchin]

My assumption without knowing any of the implementation details is they might need to maintain a working RPython interpreter, that interpreter is already working and fully Python 2 compatible, so Python 2 maintenance is practically free

[queen hornet]

https://twitter.com/pypyproject/status/1160209907079176192

There is no plan to deprecate the 2.7 version of PyPy, basically for as long as PyPy exists. RPython is based on Python 2, which is not really realistic to change, so we need PyPy2 to bootstrap.

The PyPy Project (@pypyproject)

There is no plan to deprecate the 2.7 version of PyPy, basically for as long as PyPy exists. RPython is based on Python 2, which is not really realistic to change, so we need PyPy2 to bootstrap. Documented here: https://t.co/SGmZpOqPbW

Twitter · Aug 10, 2019, 3:22 PM

[mortal shore]

oh that's right, I remember now

RPython isn't like, it's own thing I think, I think it's a thing you invoke with a python interpreter, that then produces a PyPy

and if you invoke it with a previously produced PyPy, it's much faster

[woven plover]

rpython is pretty much in-interpreter metaprogramming - plus a toolchain to turn that "metaprogram" into a graph to transform and turn into the actual program

so one starts by importing the implementation and then handing the "entrypoint" to rpython
rpython then meta-interprets the bytecode until that metainterpreter has a full graph of everything

then that graph is transformed until it has jit, gc, and all the other fun and is serialized to an exceedingly funky c file which compiles to the pypy executable

so unfortunately the 2 major road-kills for porting rpython to python3 are

a) all the importable stuff for the meta level
b) the actual meta-interpreter that deals with the byte-code to graph

[torpid veldt]

It's not really self hosting if py3 needs py2 though right

[drowsy basalt]

Is there a good guide for creating a simple pip(x) install-able package in modern python aka add dependencies, do stuff, pip(x) install ., run from PATH?

[nimble bough]

be in the same folder as the pyproject.toml/setup.py and pip(x) install ., it'll build and install the package

[drowsy basalt]

Turns out creating a trivial pyproject.toml and not doing anything the tutorial suggests works out of the box, because there's a fallback on setuptools which automatically recognises the src layout and everything works out.

[noble matrix]

This is the canonical tutorial; it was updated relatively recently and is IMO pretty good now (except for some potential confusion on choosing your backend): https://packaging.python.org/en/latest/tutorials/packaging-projects/ If you have issues, please let us know!

With the src layout any modern backend (Hatch, Flit, Setuptools, PDM, etc), you only a few lines in the pyproject.toml (and nothing else, no setup.py, setup.cfg, etc) to get a minimal working setup for any basic project. I.e. something like:

[build-system]
requires = ["YOUR-BACKEND>=VERSION"]
build-backend = "YOUR_BACKEND.ENTRYPOINT"

[project]
name = "YOUR_PACKAGE_NAME"
version = "0.0.1"  # Don't even need this is you're using VCS-based versioning

Poetry is the only commonly-used backend that doesn't yet support the standard [project] table and requires configuration via its own bespoke mechanisms, but they're working on it, so I hear.

[inland prawn]

yes, we are working on it. we had a lot of decisions made before [project] table standard and it's not easy to get rid of some stuff, since it's rooted deep into the code

[uneven karma]

Every single person I’ve sent this tutorial too has immediately come back with “so uh… which backend do I use?”
A quick note there would be very helpful.

Also, yesterday, I was looking at helping someone make their first package. It turns out that Hatchling is now the first backend suggested. They had followed the tutorial exactly, and… published their venv inside their package.
It turns out, Hatchling included everything in their CWD, including their venv and editor project files and more. Adding those to a .gitignore got Hatchling to ignore them properly.
I don’t think I’ve never not had a .gitignore, but I don’t believe Setuptools has the same behavior.
This should be called out too.

[queen hornet]

This guide gives advice on picking a frontend and backend https://www.pyopensci.org/python-package-guide/package-structure-code/python-package-build-tools.html

[blazing lantern]

Hatchling includes everything git would by default in your sdist, while Setuptools only includes what you specify.

[noble matrix]

Yeah, there is a distinction here in Hatchling's approach vs. many of the other VCS-supporting backends/plugins—many of the others include what is actually checked in to your repo, while Hatching (and perhaps others) uses what would be checked into your repo following your Gitignore (i.e. if you did the usual git add . or git commit -a). Though I am curious if Hatchling respects the user/system level .gitignore like Git does, or just the local one.

@lusty quarry ?

[lusty quarry]

it does not

[lusty quarry]

My reasoning is that a VCS checkout is not guaranteed and is in fact the rarest of all situations. If a project is only published on PyPI in CI/CD then that assumption is met but then fails as soon as a user installs the sdist or attempts are made to build for other ecosystems like Conda or Fedora that use either the sdist on PyPI or the tarball release artifact on GitHub.

[drowsy basalt]

I initially tried that, selected hatchling as the backend as it was the first one, and then it failed: I assume hatchling does not detect src layout so while it pipx install-ed the package just fine what it put on the sys path was the src folder, and thus the invocation of the entry point blew up.

Repro is be pretty easy: create a directory (e.g. plop), add a pyproject.toml:

[project]
name = "plop"
version = "0.1"

[project.scripts]
plop = "plop:main"

and src/plop.py:

def main():
  print("hello, world")

then pipx install .

After testing all 5 options

• default, setuptools, and pdm: work out of the box
• flit: refuses to build because there's no description (despite that being an optional field). works after adding one
• hatchling: install succeeds, "ModuleNotFoundError: No module named 'plop'" when trying to run the script

[spiral urchin]

Flit’s behaviour is intentional, I asked about it a while ago and it’s about best practice. The field is technically optional but you should have it so it’s made required.

[noble matrix]

Does it work if you import plop? Perhaps the issue could be due to handling of single-file modules...does it work for an import package? @lusty quarry , ideas?

[drowsy basalt]

import plop where? The location where the code is is plop itself, hatch just doesn't package it such that it can be found.

[noble matrix]

Sorry, I'm just asking if you try to import plop in a Python interpreter in an environment with plop installed via Hatchling, does that also not work, or is the problem only with the entrypoint?

[drowsy basalt]

Sounds fair, but would be nice if it provided an error message rather than vomit a traceback on your lap like an ornery cat though:

error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [16 lines of output]
      Traceback (most recent call last):
        File "pipx/shared/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "pipx/shared/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "pipx/shared/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-4agea3k3/overlay/lib/python3.10/site-packages/flit_core/buildapi.py", line 23, in get_requires_for_build_wheel
          info = read_flit_config(pyproj_toml)
        File "/tmp/pip-build-env-4agea3k3/overlay/lib/python3.10/site-packages/flit_core/config.py", line 79, in read_flit_config
          return prep_toml_config(d, path)
        File "/tmp/pip-build-env-4agea3k3/overlay/lib/python3.10/site-packages/flit_core/config.py", line 106, in prep_toml_config
          loaded_cfg = read_pep621_metadata(d['project'], path)
        File "/tmp/pip-build-env-4agea3k3/overlay/lib/python3.10/site-packages/flit_core/config.py", line 630, in read_pep621_metadata
          raise ConfigError(
      flit_core.config.ConfigError: description must be specified under [project] or listed as a dynamic field
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

The error is visible about a quarter up, but is not easy to notice at first glance.

[noble matrix]

That's due to there being no clear current standard for backend-frontend comms for this, but see https://discuss.python.org/t/adding-a-mechanism-to-forward-user-facing-warnings-from-build-backends-to-build-frontends/28106/ for an active proposal to fix that

[drowsy basalt]

Same diff', and looking at the site-package for the venv it's not surprising:

~/.local/pipx/venvs/plop/lib/python3.10/site-packages
├── build
│   └── lib
│       ├── plop.py
│       └── __pycache__
│           └── plop.cpython-310.pyc
├── pipx_shared.pth
├── plop-0.1.dist-info
│   ├── direct_url.json
│   ├── entry_points.txt
│   ├── INSTALLER
│   ├── METADATA
│   ├── RECORD
│   ├── REQUESTED
│   └── WHEEL
└── src
    ├── plop.py
    └── __pycache__
        └── plop.cpython-310.pyc

hatchling does not see the src layout, so it just puts src itself on the syspath (I assume). And also some leftover build garbage. By comparison PDM recognises the layout and only packages the module:

~/.local/pipx/venvs/plop/lib/python3.10/site-packages
├── pipx_shared.pth
├── plop-0.1.dist-info
│   ├── direct_url.json
│   ├── entry_points.txt
│   ├── INSTALLER
│   ├── METADATA
│   ├── RECORD
│   ├── REQUESTED
│   └── WHEEL
├── plop.py
└── __pycache__
    └── plop.cpython-310.pyc

[noble matrix]

As mentioned, I'm pretty sure this is due Hatch not supporting a single-flie module combined with the src layout, as I would presume it should (unless there is some reason for it—@lusty quarry ?). If you use it with an import package instead (or include the module at the root of the flat-layout), then I assume it should work.

[lusty quarry]

https://hatch.pypa.io/latest/plugins/builder/wheel/#default-file-selection

https://github.com/psf/black/blob/e7783e9ab27e95cd9b50c2ef84174f4e1c9d60cd/pyproject.toml#L115-L117

[tool.hatch.build.targets.wheel]
only-include = ["src"]
sources = ["src"]

[noble matrix]

Is there a reason to not insert src/<NAME>.py in the default search order after <NAME>.py, like the other backends do and to parallel with the import package case? Is there any realistic additional ambiguity that would create in new packages?

[lusty quarry]

no just an oversight and I'd be open to a PR

[noble matrix]

Gotcha; any pointers to where the relevant code lives in Hatchling? A bit of browsing the source tree didn't find an obvious place.

[lusty quarry]

https://github.com/pypa/hatch/blob/d25637599dc51eec918a0a1b7e478be28a8a402c/backend/src/hatchling/builders/wheel.py#L172

I appreciate the help but I would very much prefer that any free time you might have be devoted to PEP 639 🙂

[noble matrix]

Ah, I looked in sdist.py but not wheel.py. Seems like it would just be adding one more branch to the loop plus adding the appropriate test case under TestDefaultFileSelector in https://github.com/pypa/hatch/blob/d25637599dc51eec918a0a1b7e478be28a8a402c/tests/backend/builders/test_wheel.py#LL36C3-L36C3 , which should be mostly a copy/paste job.

[noble matrix]

Free time? Wuts dat? (In all seriousness, will do now that I'm finally catching up on my post PyCon backlog)

[lusty quarry]

yeah each target has its own defaults https://hatch.pypa.io/latest/plugins/builder/sdist/#default-file-selection

[noble matrix]

@drowsy basalt Seems like this might be an easy one for you to help with?

[short copper]

I'm not sure what would be the best place to ask, so excuse me if this is the wrong channel.
Is it "legal" to build wheels starting from binary libraries that have already been built by CMake + ninja upfront?
So that I would only use a python tool to create the wheel itself as the last step that CMake runs, rather than having to call pip install . (probably using some scikit-build-core) to build everything from scratch?

[mortal shore]

I think most or every build system builds the .so files prior to building the wheel, so that sounds fine to me? also just in general the wheel spec doesn't really care how you build things, it just describes a file format

[short copper]

The only thing I would like to do is avoid "reinventing the wheel" 😉. I now wrote my own stupid script to assemble the pieces, but it would be nice if wheel and record file etc. could be generated by some existing library. I feel somewhat stupid to do this "by hand" (well, with a script, but still).

[lusty quarry]

Hatchling can do what you want https://pypi.org/project/hatchling/

[short copper]

Thank you very much, diving into the docs now ...

[short copper]

Just one more thing that puzzles me. I tried using pybind11-stubgen to generate pyi files. But I'm puzzled by chicken-and-egg problem. I need to run pip install mypackage first before being able to run stubgen. But I need the resulting pyi to build the final package. I'm probably mixing something up. (I could build preliminary wheel, install it, generate pyi, build final wheel. But there sure must be a shortcut to this?

[shut quest]

wheel pack will generate RECORD file for you https://github.com/pypa/wheel/blob/461f7a3a2d4824009ca4855b3300d61e6d405dc4/src/wheel/wheelfile.py

[merry rune]

@short copper I think the right way would be to have a step in scikit-build-core were you can run a "post processor" like stubgen. py-build-cmake supports this, so we probably can add that to scikit-build-core soon. When I work on a plugin for hatchling / extensionlib, that might be available there out-of-the-box, it sounds like. (First priority for plugins though is setuptools, since I want to retire scikit-build (classic)'s code and have it use scikit-build-core.

[manic belfry]

Hi, we are trying to unlock the upgrade of the vendored packaging in pip. To test the deprecation messages, I'm looking for examples of PyPI projects that are installable today and have non standard version numbers or dependency specifiers. If you know some, can you please share at https://github.com/pypa/pip/pull/11945 ?

[queen hornet]

Maybe you can query https://github.com/sethmlarson/pypi-data to find some?

[noble matrix]

I remember Paul Moore had a tool and personal archive that allowed him to query things like that among all PyPI packages, which he did for e.g. PEP 685 (extras names), and IIRC PEP 625 (sdist names). Maybe you could ask him about that on the issue.

[mortal shore]

Note that PyPI has validated pep440 for about 10 years now iirc

Unless my memory is bad at least lol

[manic belfry]

Thanks folks. @mortal shore I was not aware that the validation was in place since so long. Do you know if PyPI validates dependency specifiers too?

[mortal shore]

Uhh, whatever packaging.requirements.Requirement(requirement) does

Note: PyPI doesn't check the METADATA file, just what is sent alongside the file on upload, which in theory is the same as what is inside the METADATA file, but in practice sometimes differs

[blazing lantern]

doesn't check the METADATA file yet 😉 (having thumb issues since May, else I would be farther along with my relevant packaging PR)

[mortal shore]

[zealous swan]

@mortal shore Reading the above conversation makes me think of https://blog.vlt.sh/blog/the-massive-hole-in-the-npm-ecosystem, are there any parallels there?

[mortal shore]

Yes and No:

Warehouse does not get it's internal metadata from the artifact, it gets it similarly to npm, alongside the artifact itself, with no mechanism to prevent those from drifting. However, this is only it's "internal" metadata, which is the metadata stored in the database. This metadata appears on the /simple/ page in the form of "requires-python", nothing else from the internal metadata is currently exposed in /simple/.

Thus installers that only use the simple API, are protected from this type of attack/confusion in all cases except for requires-python, which is a non trivial hole, but in the attacks mentioned, it's only the downgrade attacks that are possible (or generally DoS attacks).

However, some installers don't use only the simple API (poetry is one, not sure of others) and instead opt to use things like the "legacy JSON API", which the legacy JSON API just publishes the internal metadata that Warehouse stores.

[inland prawn]

FYI I am currently working on switching us to Simple JSON API with Legacy JSON API used as fallback only (by us, I mean Poetry)

[zealous swan]

Thanks for the clarification on Poetry @inland prawn, I was just about to ask to make sure I had that right 😅

[mortal shore]

PyPI has an extra complication in that conceptually our metadata is per artifact, but Warehouse does this "wrong" (inherited from legacy PyPI) where it stores metadata per release rather than per artifact. In practice this means that the first artifact for a version to be uploaded "wins" for which metadaa gets exposed (this is why twine and other uploaders tend to try to upload wheels first, they tend to have more complete metadata).

However historically, pre-warehouse, PyPI would use the last release, and even had a command (register) that would allow people to mutate the internal metadata without uploading anything, which people often did use and there have historically (not anymore) been cases where administrators have "fixed" the metadata for a release.

So our internal metadata is not very trustworthy

[zealous swan]

Got it, I remember being confused by that in the past. Thanks for clarifying that. Might be something worth documenting in PyPI' security model

The whole "first dist wins" part is the sticky one, coming from the API it feels authoritative at the level of a release. The Simple JSON API has this behavior too?

[mortal shore]

uhh

Yes I think that's right

[inland prawn]

yeah, I thought that might not be that clear right after I send that 😛

[mortal shore]

note that's only for things that are stored on the Release model in Warehouse, which is basically all the core metadata. I think the File model has File.requires_python, but I think we just denormalize it over from the Release model and it's stored on File to avoid having to do a join

The simple API doesnt' expose much metadata so the issue doesn't really affect it much

well "much metadata", besides the PEP 658 METADATA file which has all of the metadata :V but that's extracted from the artifact that is being uploaded, and is stored per file not per release

[zealous swan]

Yeah and if the biggest risk is a package DoS-ing... itself, that's fine. I can see it being more interesting if say dependencies are affected by the issue too.

[mortal shore]

There's an open issue to fix the way we store metadata

so that it's per file not per release

it's just been blocked on:

• Figuring out the answer to questions like "our UI and legacy JSON treats things as per version, how do we handle that?"
• Avoid remaking the same mistake where we can't actually trust the metadata because it's sent "alongside" the artifact, so we want to start sourcing the metadata from the artifact instead.

[mortal shore]

sorry lots going on ATM so forgot to say, this is great news to hear 🙂 Hopefully that didn't come across as calling poetry out, just trying to accurately describe the state of the world.

[inland prawn]

nah, it's fine

[placid wyvern]

As a gee whiz question, is it a known issue that ensurepip flags against Microsoft's security definitions?

[tidal kiln]

@fickle hound would know

[fickle hound]

That one's new to me, but I can check it out. Normally the false positives get dealt with fairly quick, just not as quickly as the true positive detections get added

[fickle hound]

@placid wyvern I can't reproduce this easily - exactly which version of Python are you seeing the detections against, and which files?

[placid wyvern]

Hmm, I haven't personally experienced the issue, but it's highly common in the Python Discord. The next time it pops up, I can have the individuals dump out their Python install folders if that would be helpful.
Typically it presents thusly in my experience:

• User installs Python 3.11
• Through whatever reason, pip is not distributed in the installation process.
• Troubleshooting steps dictate that we have them invoke py -m ensurepip.
• This process fails to complete due Windows Defender/Microsoft Security blocking the action.

I enumerated my installs as well and couldn't replicate the behavior, but it's frequent enough at this point that I'm fairly confident that it's beyond user error at least.

Here's the most recent one that prompted my question here.

[inland prawn]

sudo maybe? or whatever windows equivalent is?

[placid wyvern]

Mmm, I don't think privilege escalation should be required for this
The issue is common enough, however, that searching ensurepip within the help threads of Python Discord actually reveals this with incredible frequency.

[fickle hound]

Wow, this is not something I was expecting at all

I guess it's scanning the command line arguments? Because this code is just re-running sys.executable

What would be most helpful next time this arises is to get a screenshot from Protection History

(that's a different trigger, not the ensurepip one)

But I suspect it'll be the same cause

[placid wyvern]

Will do! Appreciate the compass!

[whole violet]

Is there a good channel to ask questions about build/packaging issues and questions using build and meson-python?

[inland prawn]

#build #meson-python or just general if you can't pinpoint which is better for your problem

[zealous swan]

Wheel build numbers: is there a use-case for those beyond being able to put out new wheels for the same release?

[lusty quarry]

for CLIs like Hatch is there any benefit to supporting every non-EOL Python version?

since I have standalone binaries set up which will be released next time I'm thinking about just supporting 3.11

maybe some enterprise organizations have requirements that they support a single environment and dump everything in that?

[queen hornet]

75% of June downloads are 3.8-3.10

┌──────────┬─────────┬───────────┐
│ category │ percent │ downloads │
├──────────┼─────────┼───────────┤
│ 3.10     │  27.77% │    44,104 │
│ 3.9      │  26.80% │    42,576 │
│ 3.8      │  20.25% │    32,173 │
│ 3.11     │  18.57% │    29,504 │
│ 3.7      │   5.37% │     8,537 │
│ null     │   1.17% │     1,860 │
│ 3.6      │   0.04% │        57 │
│ 3.12     │   0.01% │        19 │
│ 2.7      │   0.01% │        10 │
│ 3.5      │   0.00% │         1 │
│ Total    │         │   158,841 │
└──────────┴─────────┴───────────┘

[lusty quarry]

oh I see, I guess Linux users create virtual environments from whatever happens to be available on their distribution's package manager

thanks for the stats 🙇‍♂️

hopefully if I feature the binaries prominently as the preferred way to install for Linux then those numbers will decrease

| python_version | system_name | download_count |
| -------------- | ----------- | -------------- |
| 3.10           | Linux       |         38,504 |
| 3.8            | Linux       |         35,887 |
| 3.9            | Linux       |         33,293 |
| 3.11           | Linux       |         19,764 |
| 3.9            | Windows     |          6,957 |
| 3.11           | Darwin      |          6,299 |
| 3.7            | Linux       |          5,922 |
| 3.9            | Darwin      |          2,778 |
| 3.11           | Windows     |          1,909 |
| 3.8            | Darwin      |          1,791 |
| 3.8            | Windows     |          1,664 |
| 3.10           | Darwin      |          1,662 |
| 3.10           | Windows     |          1,561 |
| 3.7            | Windows     |            730 |
| 3.7            | Darwin      |            565 |
| Total          |             |        159,286 |

[spiral urchin]

if I feature the binaries prominently as the preferred way to install
Unlikely, knowing how Linux users tend to think. Manually downloading an opaque binary is close to the least (if not the least) preferred way. But we can hope.

[inland prawn]

Might be a pain for containers

[lusty quarry]

how so?

(I will keep supporting every version I'm just curious why it would be an extra burden for containers since it doesn't matter what is shipped in the image)

[lusty quarry]

maybe I'm just bad at system administration? I prefer binaries or installers from the source, whether that is the organization's website or GitHub releases

ripgrep, aws cli, really I can't think of a counterexample where I would prefer the system's packages

[inland prawn]

Putting app based on for example 3.9 in the container for py3.9 and having Hatch restricted to 3.11 could be a pain

I mean, extra steps of installation etc

[mortal shore]

I generally prefer things come from.. some kind of package manager, if only because then I have a consistent way to keep everything updated and recreate an environment

[lusty quarry]

perhaps it's my error in communication: by standalone I mean at runtime it does not depend on the environment it does everything for you to get 3.11 (likely I will embed the distribution)

[inland prawn]

I second that

[lusty quarry]

that makes sense and like I said perhaps I have insufficient experience maintaining deployment environments. in my mind having build scripts are preferable so I can choose the precise versions of software for better security scanning and also allows me to get faster upgrades

[mortal shore]

I always think of it like this:

• Things where I care a lot about specific versions, I want to control things tightly and this will likely be maintained outside of a package manager
• Things where I mostly don't care about specific versions, I just need a version of it, I want these inside of a package manager because I have too much stuff installed to sit there and carefully manage the versions of everything

[distant briar]

Yeah, I avoid using anything that can't be installed through package manager and be updated with it

Using https://github.com/topgrade-rs/topgrade to upgrade everything on my personal machine

[rapid glacier]

That becomes interesting when the package manager is able to get stuff from github directly, like the „new“ windows package manager. I guess github being owned by Microsoft has this advantage.

[distant briar]

Not really, it still is a package manager that can manage installed software and update it

Nonetheless, I vastly prefer Chocolatey

Though it is less streamlined with how you have PowerShell scripts

[lusty quarry]

I do like winget

[distant briar]

But that also has benefits

[rapid glacier]

Yeah it just blurs the line between something „you get from github“ and something „you install from your package manager“

[placid wyvern]

I always seem to end up with scoop, chocolatey, and winget managing various packages.
At this point I've just relinquished control of my packages to whatever's most convenient at the time. 😅

[spiral urchin]

If you replace “GitHub” with “the internet” that’s basically Go, and no I don’t really feel it’s a particularly good idea. GitHub still lacks protection regarding immutability and now they’ve opened the gate it’ll be difficult to close it.

[distant briar]

Winget is just direct downloading installers/zips and verifies the checksum it has for it in its repository

[rapid glacier]

I mean node packages got hijacked (by the author but still)

So thats a vunerability that is sadly everywhere, lets hope that hey are working on closing it, whatever the package manager is

[distant briar]

I'm fine with just chocolatey but that's just system package managers, I still use pipx, cargo, or npm on Windows.

[rapid glacier]

It would be really interesting how many pypi downloads there are, with frozen version and without.

[distant briar]

As for Linux, I add snap, flatpak, nix (actually not sure if I use any package from it right now as I might have replaced the packages I used from it with sth else but it has a very big collection of software and is a package manager which I'll always prefer over unmanaged stuff) and obviously a plethora of, in my case, apt repos

Probably forgot about sth, the important thing is that I can use topgrade to upgrade everything at once.

[spiral urchin]

Yeah I’m not saying Winget is worse, just that how it does things (in this particular area) doesn’t really move the needle much. I do like Winget though, it seems to get the actual packaging part right, and having MS at its back makes it a trillion times easier to push that best practice to projects (or just spend a ton of resource repackaging things), which is a big advantage over other solutions.

[mortal shore]

time to make a package manager manager

[lusty quarry]

I really want to try Nix but I haven't had the time, it looks so interesting and useful albeit complex

[storm canyon]

asdf kinda is that 😛

[spiral urchin]

asdf is a package manager to end all package managers (which works out as well as you’d expect) not a manager manager

[storm canyon]

fair

[distant briar]

I consider topgrade to be kinda that, at least to the extent that I most care about

you can't search across multiple package managers but I rarely search through the package manager rather than google so

upgrading everything at once is really enough for me

okay, it can't manage the installations of managers which I guess was what you actually meant by manager of managers lol

[placid wyvern]

https://inspector.pypi.io/project/sigmaepsilon-math/0.0.1/packages/fb/c0/2511f82e9c9b889cfe6a71e78560c02cd19f74419bd7632fabc76f589e48/sigmaepsilon.math-0.0.1.tar.gz/sigmaepsilon.math-0.0.1/.math-pub/Scripts/Activate.ps1
Swinging in here, anyone have any... idea of what the heck is going on here? I've seen plenty of venv's uploaded, but never one with a certificate file/signature embedded.

[distant briar]

That's standard powershell activate script (may not be a thing in older Python versions, not sure on that)

It allows you to activate venv when you have RemoteSigned execution policy

https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_execution_policies?view=powershell-7.3

[placid wyvern]

Yeah, tracking on it being a standard activation script, but I don't know that I've, across well over 50,000 package reviews at this point, ever come across signatures being included in the Activate.ps1 itself.

Looks like you're right on the money though. That's neat.

[blazing lantern]

The Windows installer has had a signed Activate.ps1 since Python 3.9 (it's why a friend of mine rewrote that script to be static instead of generated like for the other shell scripts)

[placid wyvern]

That's... actually quite unfortunate for us lol.

Interesting that I haven't come across it yet though.

We have a rule that flags on Activate.ps1 quite frequently, and we've been slowly expanding the whitelist hashing to try and accomodate for the mountain of different Activate.ps1's that seem to exist out there.

[inland prawn]

I got to ask. any reason for uploading a venv?

[placid wyvern]

Oh I have no idea, but it happens about 300 times a day and I get pinged on every. single. one.

[queen hornet]

Probably a mistake. I reported it to a couple of projects once, and they hadn't realised

[placid wyvern]

It would be handy but without some sort of automated notification process, I'm not sure that we could accomodate notifying every time a venv is uploaded. 😅

[lusty quarry]

that's one of the reasons I was strongly against PEP 582

[placid wyvern]

I haven't followed PEP 582 and the general discourse around it that much, I'll do some reading and form an opinion afterwards however.
I'm generally for prohibiting the upload of virtual environments period from a security standpoint-- for people doing it accidentally, it represents a potentially large swathe of unneeded data. And for individuals doing it intentionally for malicious purposes, enumerating the venv to try and discern whether or not the countless files in that code may be malicious is exceedingly difficult. (Bordering on needle in a haystack level.)

However, that might be an opinion formed out of ignorance, or based on my novice perspective. I haven't seen a lot of compelling reasons to allow it so far, but I'm also not sure how you would go about preventing that type of behavior programmatically, and I'm unaware of any practical applicability of it.

[indigo token]

How do you manage that? I go to great lengths to make sure that absolutely nothing that’s globally installed on my system is installed without a package manager.

I can’t think of any reason why anyone wouldn’t want to have something like ripgrep managed by a package manager.

[quartz crystal]

Hello tool authors!
I've put up PEP 721 standardizing how to unpack sdists: https://peps.python.org/pep-0721/
Discussions to: https://discuss.python.org/t/pep-721-using-tarfile-data-filter-for-source-distribution-extraction/28928

tldr:
Per the accepted & implemented PEP 706, unpacking tarballs in Python 3.12-3.13 will give you a DeprecationWarning for pending change in behaviour: Python 3.14 will change the default security/features trade-off toward security.
PEP 721 tells you to use the future default (the new data filter), and to not worry too much about security on Pythons that don't have the feature backported.

For backwards-compatible implementation tips, see: https://docs.python.org/3/library/tarfile.html#supporting-older-python-versions

[blazing lantern]

If you want help preventing this: https://github.com/python/cpython/issues/83417

[placid wyvern]

What did this get hung up on?
It seems... idiomatic to me. The comments seem to indicate that Conda introduced similar behavior with no real hiccups.

Forgive me, I'm not wildly spun up on core Python processes. Just some dude running a third party antimalware service.
I've got a mile long list of gripes that, frankly, all seem to have discussed to some degree in the Python discuss forums or on GitHub, but rarely do they ever seem to pan out to an actionable idea/PEP.

[blazing lantern]

It got hung up on Vinay voting -0.5, which negates my +1 on the idea. So with no one else speaking up it's a core dev stalemate (at best; Vinay also created venv, so going over his objections feels slightly icky). But if more people came forward saying they thought it was a good idea it would be helpful to help make a case that it's an idea worth doing

[placid wyvern]

Let me do some thinking on it.

I'm probably not a good advocate for it from a Python perspective, but from the security scanning/third party perspective, we stand to benefit extraordinarily well from such an implementation. Some of the internals of the venv are written in a manner that makes it extraordinarily difficult to perform heuristics analysis of a package without also whitelisting venv files. The most correct way to do that from our perspective is hashing, but the caveat there is that it starts to get pretty computationally expensive compared to the actual ingest of packages.

[lusty quarry]

I commented, perhaps you can state your use case/experience there

[placid wyvern]

Tossed my thoughts up there. 😅 Don't eviscerate me, I am but one guy on the internet.

[distant briar]

for direct downloads, this could be helpful (I haven't used it but encountered it at some point and it's just sitting there in my gh stars): https://github.com/xermicus/blindspot

no clue how people manage "build from source" apps though, nix is an option but if it's not in their repo (I guess not that common) then you have to deal with a new unfamiliar syntax to figure out how to use it which is quite a lot of effort. + nix without nixos doesn't work all that well for some apps (especially GUI ones)

[mortal shore]

In general most things get hung up on "someone needs to sit down and figure out exactly what is being proposed to change, and then actually propose it"

(which of course makes sense, volunteers and all that)

[tidal kiln]

before I post to https://discuss.python.org, does anyone have feedback?

https://github.com/FFY00/peps/blob/cross-builds-info-pep/pep-0720.rst

this is an informational PEP documenting the state of cross-compilation and other relevant information

[queen hornet]

preview at https://pep-previews--3192.org.readthedocs.build/pep-0720/

[lusty quarry]

very nice!

[queen hornet]

https://fosstodon.org/@ucodery/110741753669416050

https://ep2023.europython.eu/session/an-unbiased-evaluation-of-environment-management-and-packaging-tools

[golden falcon]

Different size circles make this difficult to parse

[fallen shuttle]

That title for a talk is very provocative (almost clicky-baity?)...
It is almost impossible to be 100% unbiased and most of the times we all have our implicit biases (it is something perfectly normal and we might have to try very hard to rise above them when they are impacting negatively society or other people)

There is a lot of academic interes in the area of biases btw, e.g. https://harvardmacy.org/blog/can-we-be-consciously-unbiased

The diagram is very nice

[lusty quarry]

they definitely have a bias for Rust (so do I though...)

[queen hornet]

Yep, the speaker accidentally said she really likes one tool, then cursed herself for being biased! 🙂

[lusty quarry]

which one? lol

[queen hornet]

I don't remember, we'll have to wait for the recording!

[lavish patrol]

Hello! Would you be interested in knowing if a pip user's host is on WSL, from a statistical perspective?

[distant briar]

I would for my own packages

PyPI statistics aren't the most reliable for actually knowing how many actual users you have but it's something

[mortal shore]

They reliably tell you something, but it may not be something you want to know 😉

[distant briar]

Yep

Does PyPI get enough info for me to be able to discern between specific distros?

Free frontends for the PyPI big query (pepy.tech/pypistats.org) don't expose it so I assumed that you only collect data about the generic linux/darwin/windows and arches but I don't think I've actually looked into it more than that

[mortal shore]

Uhh

I think so yes

https://github.com/pypa/pip/blob/9ca96dd8a8f40a824cb8315f5b9c080fccf24ca9/src/pip/_internal/network/session.py#L109

[distant briar]

Now that you showed me these structures, I remembered that I did look up those stats in the past, just forgot

Nope, I did not, I thought I did but based the stats on views of particular install guide, not PyPI data

But yeah, this is very cool then, I'll have to look into those

[elder tinsel]

I'm not gonna go out of my way to test on WSL. If WSL isn't behaving equivalently for some reason, I consider that WSL's problem not mine, but it could be neat information to have if people are writing things which are basically cli tools that might be setup with pipx or something, especially if it's something that doesn't directly support windows for some reason.

[gritty roost]

@tidal kiln missed your message about cross compiling. I'll take a look at the PEP draft when I've got some time. I've spent a while working with cross compilation over at https://github.com/jvolkman/rules_pycross, which is a Bazel ruleset. E.g., these numpy and psycopg2 examples can cross compile between MacOS and Linux (build linux wheels on macos, build macos wheels on linux): https://github.com/jvolkman/rules_pycross/tree/dev/external/examples/external_linking/deps

[echo ruin]

I want to specify a dependency on a package only if there is an available wheel matching the current environment

How impossible is this without a bunch of hard coding copy paste nonsense

[inland prawn]

Impossible I would say

Metadata just doesn't allow for such spec, since it's unaware of different package artifact types

[echo ruin]

I didn't have high hopes

bleh ok, I guess I get to look into maintaining a copy pasted list of which architectures I'm building wheels for and duplicate that

[blazing lantern]

Is it just the single package, so other packages can have sdists but not the single, specific one? If it's general then pip has a --only-binary :all:

[echo ruin]

@blazing lantern it's just a single package yeah -- but that's a user flag, I'm the library author

the use case is "I have a package, it has a dependency requiring a rust toolchain, but when one is likely to not be present I want 'fallback to a pure python dependency' not 'blow up on install'"

[mortal shore]

we don't gave a good story around "optional, but eager"

it's either mandatory or "optional, but opt in"

[echo ruin]

understood yeah

selfishly I think "blow up on install" is better than "every user needs to know to add an extra"

which is why I went this way in the first place

but maybe I'm ok with the pain of "explicitly write environment markers for the platforms I've built rust wheels for" in order to help out users on other ones, I haven't decided yet

[indigo token]

This has caused pain in other situations as well, such as psycopg / psycopg-binary.

It might be solvable by having a provides mechanism, that allows multiple packages to provide alternative implementation of the same thing. The base package <name> could be the fast one, and <name>-purepython could have provides = ['<name>'] so it would be installed if the other one can’t be.

It would be a big discussion though, since it has security implications: If someone would create a malicious package with provides = ['numpy=99.99.99'] people would get that if they accidentally depend on numpy >= 1.26 or so (numpy’s version at the time of writing is 1.25.1).

[echo ruin]

@indigo token I like that idea for other reasons (and I think have brought it up independently in the past) but it seems like a bigger "decision" than this sort of thing

I guess for the reasons you mention

[lusty quarry]

interesting feedback Is anyone using PyPy for real work? https://news.ycombinator.com/item?id=36940871

mattip

Is anyone using PyPy for real work?

I have been the release manager for PyPy, an alternative Python interpreter with a JIT [0] since 2015, and have done a lot of work to make it available via conda-forge [1] or by direct download [2]. This includes not only packaging PyPy, but improving on an entire C-API emulation layer so that today we can run (albeit more slowly) almost the ent...

Hacker News · Jul 31, 2023, 10:53 AM

[queen hornet]

one of my earlier PRs to https://www.pypa.io broke incoming links by inadvertently switching Sphinx builds from dirhtml (e.g. https://www.pypa.io/en/latest/specifications/) to html (e.g. https://www.pypa.io/en/latest/specifications.html)

please could someone check my fix at https://github.com/pypa/pypa.io/pull/90 so we can fix other websites that link to it?

[high stone]

pypa.io RTD dirhtml

[peak ocean]

So far we've been shipping .dist-info/RECORD files in Debian, but they aren't compatible with multi-arch (different .so files on different archs). Would you have any concerns with us dropping the RECORD files in Debian, entirely? I can't think of any tooling that would be depending on them.

I'm guessing this would cause trouble for anyone wanting to pip upgrade anything that the distro ships (which they'd already have to be overriding EXTERNALLY_MANAGED, to do)

[hollow trout]

Shouldn't different architectures have different lib directories?

[spiral urchin]

It is fine for a dist-info to not contain RECORD and Python-specific installers will automatically refuse to uninstall the dist
Last paragraph in the RECORD section: https://packaging.python.org/en/latest/specifications/recording-installed-packages/#the-record-file

I assume Red Hat and related distros already do this since the feature was proposed by @quartz crystal

[peak ocean]

Not for Python in Debian. All (public) python modules are in /usr/lib/python3/dist-packages/. Architectures are differentiated by the tag on the .so. Debian's multiarch mechanism allows the same package to be installed for 2 architectures and share files as long as they are identical. The issue here is that RECORD isn't identical.

[peak ocean]

Thanks for confirming that.

[woven plover]

Wondering, would it be a option to have dist info in a platform specific directory?

[peak ocean]

That could be an option, if we needed RECORD to exist. But the spec doesn't leave any space for platform specific directories.

[hollow trout]

Where do you put the .so files, then?

[woven plover]

then the spec may need a update to allow platform specific dist info when its contents are in fact platform specific

[hollow trout]

It does allow that. Pretty sure .dist-info can be located in the platlib directory.

[woven plover]

if thats allowed then the issue seems solve-able pretty easily

[peak ocean]

Right next to each other

You guys are assuming that the platlib differs across platforms, we don't do that

[spiral urchin]

It’s probably just easiest to not have RECORD. From what I can tell even if you successfully come up with a way to have a RECORD somewhere any tool trying to “upgrade” the package would seriously corrupt the system anyway

[peak ocean]

Yep

[high stone]

I think we mentioned this to someone else involved in Debian when this was implemented/changed -- a concern raised was that pip's error message wasn't great when there's a missing RECORD on like, 5 year old version, and when we asked what would be better in the latest version, we didn't get a response.

I might be misremembering though, and I definitely think this is a good thing to do (separately from EXTERNALLY-MANAGED).

[peak ocean]

I note the spec also talks about pip giving a decent error message when INSTALLER = apt/dpkg, but I don't see any code implementing either of those things. We are using INSTALLER = debian.

[high stone]

Do you have a link?

[hollow trout]

I'm guessing it's this:

For example, if a tool is asked to uninstall a project but finds no RECORD file, it may suggest that the tool named in INSTALLER may be able to do the uninstallation.
(https://packaging.python.org/en/latest/specifications/recording-installed-packages/#the-installer-file)

[high stone]

Ah, right. It does say "may" but that's nitpicking. I think a PR doing that would be very welcome in pip.

[hollow trout]

(Incidentally, the phrasing in that section suggests that dpkg would be a better value than debian, since debian is not a tool.)

[peak ocean]

Hrm, that isn't the wording I remember, but I can't find it now

We used debian, because we're building Debian packages. dpkg is the installation tool, but it doesn't have to be the installation tool. (Not that there are any real competitors)

I can't change it to dpkg. We're not doing anything with the value at the moment...

[mortal shore]

IIRC we more or less decided that the INSTALLER file was a bad abstraction for anything but an informational thing, because the real differentiator isn't "who installed this thing" but "what ecosystem is managing this install"

[quartz crystal]

Yup, Fedora & derivatives remove RECORD. It deters Python-specific installers, but it's also done because we can't easily generate it reliably. We can't keep distro packagers from tweaking files after the wheel is unpacked, so RECORD could get out of sync, and generating/updating it somehow from RPM metadata would be a lot of trouble for very little gain.
IMO no record is better than an unreliable one :)

[echo ruin]

wheels for armv6 don't exist (i.e. aren't uploadable to PyPI) right?

[high stone]

See https://github.com/pypi/warehouse/blob/5134d25e390021eb5ffd8aba78d65bded744ef57/warehouse/forklift/legacy.py#L109-L126

[strange knot]

Also see https://github.com/pypi/warehouse/issues/3668

[echo ruin]

thanks! google had found me this one before asking but I didn't understand your comments at the end

specifically I understood you to be saying that uploading those is or would in the future be deprecated

but I didn't understand the bit about moving to manylinux2014 -- it doesn't have support for armv6 AFAI was able to understand

so they're simply deprecated without replacement if I follow, yeah?

(the specific question for me is "should I try to build these", for which if they're deprecated or discouraged, then that makes it an easy "no")

I've also already responded (on my issue tracker) and said "probably go ask piwheels", though it seems piwheels actually themselves don't support rust yet, so oh well...

[inland prawn]

(since there is no channel for packaging) any idea when to expect next packaging release?

[high stone]

packaging release

I've started a brainstorming thread about Python packaging decision making processes: https://discuss.python.org/t/32997

Please share your opinions (and/or ❤️ the ones that you agree with) on there.

[fast skiff]

Hi

[zealous swan]

Would resolvelib bindings for this help other projects here? I know pip can't use binary deps: https://twitter.com/prefix_dev/status/1700109553516327135

[spiral urchin]

I’ve wondered implementing the resolver in Rust or Zig (not difficult at all) but currently the bottleneck is so much in downloading metadata it wouldn’t really move the needle much. Maybe after PyPI backfills PEP 685 for most packages it’d start to be interesting

[inland prawn]

I think you mean pep 658 (serving metadata)

[spiral urchin]

eh right

[lusty quarry]

wow that's some uncanny timing... I was actually going to open up a rather long issue on their repo talking about the potential of that and my willingness to help

the short answer is yes Hatch would definitely benefit from that and this coming quarter at work I'm going to be devoting some time to lock file stuff (in coordination with Brett since he has the most context)

[high stone]

I had a discussion with these folks around resolvelib as well -- they attempted a port of resolvelib to Rust (https://github.com/prefix-dev/resolvelib-rs) -- it seems to me that their focus is on a compiled-binary ecosystem (i.e. conda-like, which makes sense since these are the folks behind libmamba as well). The motivation was the resolvelib's error message provide a lot more "visibility" into the failures. Their current resolver is in https://github.com/mamba-org/rattler, which has a libsolv based resolver, which assumes static metadata with complete visibility of the entire graph IIUC. That can't work for the PyPI model of sdist + wheels, since sdist dependencies are "dynamic".

[high stone]

Signal boosting this again on a weekend -- I'd appreciate folks making more suggestions here. I think this is going to be an important piece in figuring out what the next steps for Python packaging would be and I'd like to for everyone who wants to have a say in that to use this to state their opinions/preferences.

[high stone]

Resolvers, and prefix.dev

[stiff girder]

Heya!

Is there any functionality on PyPI to tighten version bounds after a package is published?

I'm dealing with a dependency that violates SemVer and published breaking changes on a patch/micro version. With Hackage (the Haskell package server) that's how I'd solve it.

The only other solution I can see is to yank all versions and publish a post-release version to replace each one, but I'd rather not do that.

[inland prawn]

that's the way to go

once you publish the version, it can be yanked, but can't be changed

[woven plover]

is there a howto for setting up package docs under pypa.io ? i just merged the setuptools_scm mkdocs branch and i want to put up release/dev docs before cutting a 8.0 release

[lusty quarry]

I believe if you use GitHub Pages then it's automatic

[alpine tendon]

just wondering, are there any plans for the next manylinux? I can't seem to find anything on that topic

[spiral urchin]

There’s no “next” manylinux. The manylinux platform is now defined by glibc version so it grows with the mainstream distributions

[alpine tendon]

I see. I meant the official docker images, since their newest target now is 5 years old. Not that I care much about it, the question just came up elsewhere and I couldn't find any roadmap.

[spiral urchin]

The point of manylinux is to support many linux distributions so it’s arguably better if the target is old? Unless there’s a reason to bump the minimal requirement

[alpine tendon]

ok, makes sense, thanks!

[lusty quarry]

given the discussions about revamping governance, do we have a (soft) freeze for PyPA acceptance of new projects?

[queen hornet]

I think that would require a PyPA committer vote to modify the PEP https://peps.python.org/pep-0609/#updates-to-the-governance-specification-processes

[blazing lantern]

I've just started working on wrapping resolvelib in an API to make it easier to customize (https://github.com/brettcannon/mousebender/issues/105). If Prefix's package maintains API compatibility it could be a drop-in replacement for resolvelib.

[inland prawn]

I wonder if we could tweak Poetry's resolver to have compatible API

[spiral urchin]

That’s what I tried before resolvelib was a thing. It’s been a few years since then, many the code is better now.

[inland prawn]

The code is now better I guess, but from what I know about it, it's quite coupled with models we share between Poetry and Poetry-core

[spiral urchin]

That was the biggest problem so I guess things didn’t change much

[lusty quarry]

does someone on macOS mind testing out a new Hatch feature real quick?

pip install -I "hatch @ git+https://github.com/pypa/hatch.git@python"
hatch config set dirs.python shared
hatch python install all --private
hatch python show
hatch python remove all

[queen hornet]

I'm just about to head AFK, but I got ValueError: Could not find a default source for name='3.7' system='macos' arch='arm64' abi=''

full log

[lusty quarry]

thank you, hmm

oh, there literally is no 3.7 for Apple ARM

I will change all to ignore that error silently

[lusty quarry]

fixed, if somebody wouldn't mind trying again

[queen hornet]

better this time:

New setting:
[dirs]
python = "shared"

[lusty quarry]

it looks like the last 3 commands didn't show in the output

[queen hornet]

the last line of output was python = "shared" then an empty line and back to the prompt

[lusty quarry]

I'm not sure how copying and pasting works in your shell but try to please manually run the last 3 commands as I think they didn't actually get picked up

[queen hornet]

oh I see

one moment

[lusty quarry]

the configuration command was the final one that got executed as far as I can tell, unless there is an error and I'm hiding output somehow

[queen hornet]

they ran but no output:

/private/tmp
❯ hatch python install all --private

/private/tmp
❯ hatch python show


/private/tmp
❯ hatch python remove all


/private/tmp
❯

[lusty quarry]

fixed now, thank you for testing! lmk what you think

[queen hornet]

/private/tmp
❯ hatch config set dirs.python shared
New setting:
[dirs]
python = "shared"

/private/tmp
❯ hatch python install all --private
Incompatible distributions: 3.7, 3.8, 3.9, 3.10, 3.11, pypy2.7, pypy3.9, pypy3.10

/private/tmp
❯ hatch python show

/private/tmp
❯ hatch python remove all

[lusty quarry]

ughh okay I assumed on macOS with ARM it returned aarch64 but it actually returns arm64

I wish there was an enumeration somewhere... anyway I will fix after this meeting

[queen hornet]

platform.machine() returns 'arm64' if that's any help

[lusty quarry]

does that also hold true for Linux machines? because on there I also assume ARM is aarch64

should I also change that?

[queen hornet]

I get 'aarch64' in a Docker container with Debian running on this Mac, best to check with native Linux

[lusty quarry]

fixed

[lusty quarry]

after merging, the next PR will automatically do this behavior for environments

to be specific, when an environment requests a version of Python that is not found it will maintain its own location internally of distributions. so say you have a matrix with 3.7-3.11, and none of them are on PATH, it will download during environment creation and work automatically 🙂

[queen hornet]

looking better now:

New setting:
[dirs]
python = "shared"
Installed 3.8 @ /Users/hugo/.pythons/3.8
Installed 3.9 @ /Users/hugo/.pythons/3.9
Installed 3.10 @ /Users/hugo/.pythons/3.10
Installed 3.11 @ /Users/hugo/.pythons/3.11
Installed pypy2.7 @ /Users/hugo/.pythons/pypy2.7
Installed pypy3.9 @ /Users/hugo/.pythons/pypy3.9
Installed pypy3.10 @ /Users/hugo/.pythons/pypy3.10
       Installed
┏━━━━━━━━━━┳━━━━━━━━━━┓
┃ Name     ┃ Version  ┃
┡━━━━━━━━━━╇━━━━━━━━━━┩
│ 3.8      │ 20230826 │
├──────────┼──────────┤
│ 3.9      │ 20230826 │
├──────────┼──────────┤
│ 3.10     │ 20230826 │
├──────────┼──────────┤
│ 3.11     │ 20230826 │
├──────────┼──────────┤
│ pypy2.7  │ 7.3.12   │
├──────────┼──────────┤
│ pypy3.9  │ 7.3.12   │
├──────────┼──────────┤
│ pypy3.10 │ 7.3.12   │
└──────────┴──────────┘

[inland prawn]

Are those cached globally or per project?

[queen hornet]

they were downloaded into ~/.pythons

took 29s for the pip install, install all the Pythons, and then remove them all

15s to install them all

[blazing lantern]

I think the trick is that would require maintaining some mapping since the OSs giving the inconsistent naming, and since the stdlib isn't the most fast-moving thing, it would probably need to be 3rd-party

[indigo token]

That's awesome! Rye is too opinionated for me, so this should serve nicely.

I'd prefer platformdirs instead of yet another ~/.something

[lusty quarry]

you can put the installations wherever you want and also there are two named keys that map to special paths. isolated corresponds to the platformdirs internal path like you desire https://hatch.pypa.io/dev/config/hatch/#python-installations

[lusty quarry]

dumping thoughts because I don't know where else to put it:

I just spoke with the folks from https://prefix.dev to talk about dependency resolution and lock files for PyPI. It seems like their in-progress implementation is working quite nicely (saw a live demo in a terminal) and there is a plan to open that within the next month or two. After that they will do bindings which I hope to be able to help with as part of my work next quarter.

The downside is that there is no support currently for source distributions because, like we all know, that's a hard problem. They are interested in doing that but it's just not something that is prioritized currently.

The lock file will have multiple embedded lock files keyed by for example (Python version, system, architecture, ... and potentially other things like GPU (although no PEP for that yet). I noted (which is definitely fine I think) that the file would have much boilerplate versus Poetry that keeps the markers alongside the dependency. This ultimately doesn't matter because nobody reads lock files except for machines and disk space is cheap.

Some other stuff I forgot but that's basically what happened

They also noted that (which is something I suspected based on a cursory reading) the resolver pip currently uses does not use state of the art techniques in SAT solving. They were originally just going to port that to Rust actually but ended up requiring a new implementation

[blazing lantern]

While disk space is cheap, some people do read lock files in PR diffs to verify no security issue in terms of what's going into the lock file

[inland prawn]

also lockfiles allow for diagnosing "why there is a package X in version A installed, when there is already version A++ available?" We had multiple of those in Poetry

[woven plover]

lock files can have a git diff filter and a github action to make diffs comprehensibly printed

[inland prawn]

the part people mostly complain about (and that doesn't really have a good solution from what I know) is locking a hash of lock base (so a hash of pyproject the lock was made from), since it forces to relock everything on conflicts

[lusty quarry]

are there any projects shipping ARM wheels for Windows?

[alpine tendon]

only in CI for now: https://github.com/pygobject/pycairo/actions/runs/6022088276

will be part of the next release (and completely untested, because I lack the hardware)

[lusty quarry]

interesting thank you! can consumers buy Windows machines like laptops that run on ARM?

I haven't seen any (but I haven't looked in a year or two)

[inland prawn]

There is Microsoft Volterra, but I have never looked into its availability for general public

[alpine tendon]

There is a thinkpad x13s, as well as the surface pro x or something like that

in my case I just added arm because all the meson/cibuildwheel things are in place now, so why not

[merry rune]

cmake and ninja are both provided for Windows ARM. I have a local dev machine for testing, you can get one that looks like a mac mini.

Cross compiling is painful, it can be done but you have to manually handle the extensions in CMake (cmake/ninja are executables, so those work out of the box). I've got some work with the nanobind-example for Windows ARM and am trying to improve the experiance (are you sure meson + cibuildwheel supports it?)

[blazing lantern]

Yes, e.g. https://www.microsoft.com/en-ca/d/surface-pro-9/93vkd8np4fvk?activetab=pivot:fulltechspecstab

[blazing lantern]

Why do you ask? I believe @fickle hound can put you in touch with people if you need something

[lusty quarry]

I was building binaries for Hatch and was thinking about getting rid of the 32-bit ones when I realized that I did not build ARM which made me think that I actually had never seen such a machine for Windows

that's very awesome it actually exists!

are there benchmarks compared to the latest Intel or Ryzen?

oh it is Intel

actually I'll have to do more research because I have no idea what I'm looking at, there is no indication that they are ARM (maybe that's a marketing decision?)

[queen hornet]

(funny bug for that link: I'm in Finland, and it reasonably asks if I want to go to the Finland store. the right link is to an /en-fi/ page. I click it, it takes me to the Sweden store at /sv-se/ and I get an infinite loop of being stuck in Sweden)

[blazing lantern]

Search for "ARM" on that page; there are multiple versions of that device and one of them is ARM, the others are Intel (i.e., the SQ 3 version is the ARM one)

[lusty quarry]

oh interesting, thanks!

[rapid glacier]

you uncovered windows' plot to restore the Kalmar Union

[queen hornet]

EuroPython videos are out, here's this talk https://youtu.be/3-drZY3u5vo?si=c2wkksHvEe7Bpiz_

[alpine tendon]

released now: https://pypi.org/project/pycairo/#files

[lusty quarry]

I found a really good example of what I've been trying to advocate for with #extensionlib https://github.com/nomic-ai/gpt4all/blob/main/gpt4all-bindings/python/setup.py

extensions are built by not-the-build-backend and simply instructed where to ship them

it's so clean

[woven plover]

whats the recommended library to invoke a build backend in a environment where its ok to run it

[inland prawn]

build?

[woven plover]

what if i want a library that jsut returns me the resulting metadata? (i'd like to avoid replicating the mechanics myself)

[torpid veldt]

build.util

[torpid veldt]

build.util.project_wheel_metadata

https://pypa-build.readthedocs.io/en/stable/api.html#build.util.project_wheel_metadata

[indigo token]

OK, @lusty quarry @wide saddle I tried out both and I think they have nice approaches.

• rye has by far the most thought-through management of python toolchains, integrated with its pipx replacement (rye install) for installing global tools. However, its reliance of a bunch of conventions makes it interoperate poorly with other tools. If you want to use its lockfiles and single env per project, it’s bliss, if you have a hatch project, it’s hard to tell hatch where to find the installed toolchains.
• hatch has more simplistic python toolchain management that integrates well with other tools (just add the directory containing the python3.xs to $env.PATH). It is extremely configurable and therefore easy to make behave with other tools. Its tox-like environment matrix is much more flexible than what rye does with shims.

In summary: Once we have a lockfile standard and multi-toolchain standard, I’d love to use rye. Until then, hatch excels in project management, while pipx and hatch are decent for user-wide tool and toolchain management.

[lusty quarry]

@indigo token how can Hatch improve? (other than the planned dependency management commands and locking)

[indigo token]

flying sheep how can Hatch improve

[inland prawn]

where did name "wheel" come from? I was trying to find some info on that but there is not much to go on

[torpid veldt]

Reinventing the

[queen hornet]

it's quite simple...

PyPI used to be called the Cheese Shop from the Monty Python sketch

http://en.wikipedia.org/wiki/Cheese_Shop_sketch

what do you find in a cheese shop? wheels of cheese!

[inland prawn]

interesting

thanks

[queen hornet]

nice history here https://www.youtube.com/watch?v=AQsZsgJ30AE / https://dustingram.com/talks/2018/10/23/inside-the-cheeseshop/

[inland prawn]

oh, great

I plan on making a talk about packaging in python for my local python meetup and was looking for some resources

[torpid veldt]

https://discuss.python.org/t/where-the-name-wheel-comes-from/6708

Wheel of cheese from the cheese shop?

The name "wheel" came from a "wheel of cheese": you put wheels in a cheeseshop.
I like to think the authors of the spec really got this name right, because in addition, nobody can say they're going to "reinvent the wheel".

[merry rune]

If you need to teach packaging (vs talk about it), then https://intersect-training.org/packaging/ might be useful.

[bright parcel]

PyPI used to be called the Cheese Shop from the Monty Python sketch

Indeed... I named it that, way back in the day. The joke was that the MP cheese shop had no cheese, but the Cheese shop had packages! There was a period of time where Python was de-humor-ized and it was felt that "cheese shop" wasn't professional sounding enough. We had a lot of MP references all over the place. Many of the early machines that hosted various Python infrastructure were named albatross, ximenez, etc.

[lusty quarry]

period of time where Python was de-humor-ized
much of society sorely lacks humor currently

[mortal shore]

I was digging through the very early catalog-sig records the other day

the PyPI name did come first as far as I can tell, but Richard claimed it was pronounced "Pie Pee"

which no

I do not accept this

[lyric hedge]

we need more humour

[rapid glacier]

I demand a PEP to bring the humor back.

|| /s ||

[mortal shore]

There is https://pypi.org/404.html

[queen hornet]

Look up PEPs 404 and 801

[distant briar]

I read PyPI as "pie pie" but I know it should be "pie pee eye" (pronouncing P and I separately) since "pie pie" could also mean PyPy (though with context it should probably be obvious which you're talking about)

I think that PyPI and PyPy being so close is in its own way humorous 😛

am I forgetting some 3rd project with similar (or erronously similar) pronouncation?

PyPA has similar spelling but not pronouncation

[queen hornet]

PyPI, PyPy, PePy, Mypy, pyPI...

[distant briar]

pyPI
I hope you just repeated yourself and this is not a thing

lol

[queen hornet]

it is a thing! some scientific package. unfortunately I didn't note down the URL so it's impossible to find!

I remembered grep.app does case sensitive search!

pyPI (v1.3): Tropical Cyclone Potential Intensity Calculations in Python

https://discourse.pangeo.io/t/september-29-2021-introducing-pypi-tropical-cyclone-potential-intensity-calculations-in-python/1822

they've since wisely renamed it to tcpyPI 🙂
https://github.com/dgilford/tcpyPI/issues/5

[vital pier]

I always thought it stood for "Pyrfectly Packed Items" 🤣

[lusty quarry]

the links do not point to real files until the next minor release but this is the new installation page, particularly the first two sections are new https://hatch.pypa.io/dev/install/

I would appreciate any feedback 😄

[lofty aspen]

But then where do eggs come from!? 👻 🕵️

[woven yarrow]

Pythons hatch from eggs. BTW, https://cheeseshop.python.org still works.

[lofty aspen]

Terry Gilliam hatched from an egg. Got it. 😁

[woven yarrow]

If a project wants to end completely, is there a good way to get the attention of people using it? py2neo completely deleted itself from GitHub and PyPI: https://hachyderm.io/deck/@nedbat/111210327621710300

[inland prawn]

They shouldn't be able to remove themselves from PyPI I think...

[elder tinsel]

if it being practically abandoned while looking for someone to take over for 2 years yielded nobody, I think deletion's all that's left for people to notice it isn't being maintained anymore. ¯_(ツ)_/¯

[woven yarrow]

at least leave GitHub.

[elder tinsel]

It's their project with their choices to make for it. They left it up for someone else to take for 2 years with nobody taking interest, and people only care now that it's gone. people aren't entitled to free labor of others, we should do more to be thankful to the projects we use and be mindful of what that entails.

[woven yarrow]

I'm not saying he has to maintain it. I'm wondering what the best way is for him to end the project. Now we have broken builds, and have to scramble to find a janky fork to use as our own. Is there a way to force people's attention to the coming end of a project other than "all of your builds are now broken"?

[elder tinsel]

No, clearly there isn't because he was looking for someone else to take it for years, and it was EOLd months ago

[woven yarrow]

"EOLd" meaning he commented on the project. Open edX has ~150 dependencies. We don't read all the issues in all of those repos. Deleting the PyPI project got our attention. If the GitHub repo hadn't been deleted, we'd have an easier way forward today. That could be a recommendation, no? Delete one thing at a time?

[elder tinsel]

Any reccomendation you have here is just that, a recommendation. Do you really think an official statement saying "don't delete things on github" would have changed something here? People are free to do with their personal projects what they want.

[woven yarrow]

I'm just trying to think through how best to do this. I think if there is a more careful path, and people can find recommendations about it, they might follow the recommendations. Of course we can't force people to do it, but that doesn't mean we can't think through a better process to recommend.

[elder tinsel]

I think deletion was the right call. If nobody cared enough in 2 years, including people with complex apps using it, either it will be replaced or things will move on without it.

and maybe wake people up to keeping an eye on what they depend on and how.

[inland prawn]

I guess there could be an addition to package indices standard, that adds some kind of "EOL" field with a message/date that allows maintainers to declare project being EOL and installers to raise warning about that

[inland prawn]

I wouldn't say that. Deleting a project from PyPI is like blowing up a car in a full parking lot, because you no longer care about it. There is a lot of collateral damage

Declaring EOL or project being archived? Sure, that's maintainers right. But deleting a publicly available project is not fair

[elder tinsel]

how so?

are you entitled to their work?

If people showed thanks and were involved as is the ideal, we probably wouldn't be having this discussion. Clearly the ideal of open source failed in this case a long time before reaching deletion.

[uneven karma]

Sinbad do you feel the same way about, for example, leftpad?

[elder tinsel]

What point do you think you're making by bringing up leftpad here?, because I think the author of leftpad was absolutely justified given the preceeding events of a package name being siezed from them to benefit a company paired with that nobody should have needed that dep to begin with.

and I dont think anyone should publish on a platform that doesnt allow them control over their own works, that includes the choice to delete it.

[uneven karma]

No point, just listening to the perspectives.
I’m still trying to decide where I sit.
On the one hand, I agree with you that the authors should be able to do whatever they want with their products.
But on the other hand, I also agree with Secrus et all that it’s not particularly “fair” for something that upwards of possibly millions of people depend on to suddenly disappear with no recourse.

[elder tinsel]

The people benefiting off of something freely provided can't trump the individual rights of authorship. To view it otherwise, you'd have to have had a guarantee from them that it will continue to be provided. If you want that from someone, pay them, or manage a fork yourself. You can even put it on a cron job to just pull from upstream, and be in a position to maintain it if you need.

There are tools on the other end for people relying on open source to do so without being disrupted by a deletion, while still respecting the choice of the original author to not be involved publically anymore with it, which may be more involved (and even political in some cases) where leaving it up would leave an idea of association.

And in the case of having "over 150 dependencies but not watching their issue trackers", that's a recipe for disaster waiting to happen again and again, especially if you're just installing from a public index and not a locally managed one

relying on open source is a two way street, that should entail contributing back when you can, and so on, but just as it's something given freely, people truly dependent on it should use the license offered to their advantage in ensuring their future success in the case that it stops being available.

[lusty quarry]

there is already a thread on this topic, I firmly maintain my stance https://discuss.python.org/t/stop-allowing-deleting-things-from-pypi/17227/54

folks should definitely not be allowed to delete projects on PyPI, imo

[woven yarrow]

@elder tinsel let's look at it a different way: I'm a maintainer. I want to EOL my package. What steps can I take to cause the least disruption possible? What can PyPA do to help with that? I like the idea of an EOL declaration that pip could warn about. Does pip warn if I install a yanked version?

[lusty quarry]

Does pip warn if I install a yanked version?
yes

[woven yarrow]

OK, cool, can I make it fail on a yanked version?

If i could say --fail-on-yank, i could do that every night in CI, to get advance warning about something I need to take care of.

[uneven karma]

I deleted one of my first packages
I felt bad after and archived the repo

Since then, I've just archived the repos and not touched the PyPI packages at all

[uneven karma]

I feel like there should be a different name for that
Yanking a version declares that that version is "invalid"
But that version is still perfectly fine
We just want to declare that there won't be any more versions
... maybe if we could yank the package itself?

[woven yarrow]

in this case, they could have yanked every version. No one new could install it. Everyone pinned would still get what they had been getting. pip would warn.

yanking the package as a whole could be a new thing PyPI could support.

which would have the same effect.

[elder tinsel]

I don't think this is the place of a package installation tool to solve. It's largely a social issue, and trying to add a layer of technology at this part of the process isn't really going to help. There's a baked in assumption with that that the maintainer cares about notifying people anymore. Some may not and could just delete without warning because they are done for whatever reason. There's the unfortunate reality of death of a sole maintainer as well where a package will just stop working with future things at some point due to version incompatibilities eventually. It's better solved by getting people to be more conscious of their deps.

maintainers can already do something noisy here with an import hook that warns if that's their goal.

[uneven karma]

Don't most people not pin their dependencies?

[woven yarrow]

thanks, that's an example of an idea I am looking for. I understand that the universe is a chaotic mess and I have no control over maintainers. I'm looking for ways we can improve things. I think we both believe that better is possible.

[woven yarrow]

yes, unpinned is a larger problem, but at least the people who are pinned would have an information channel.

[elder tinsel]

I think we both agree there, but may disagree a bit on how to do it. I really don't want to see pypi being unable to delete as a result of things like this. not all the details are public, but the author of the current package in question wanted to break things off after a dispute between them and neo4j, deletion as a form of expression should not be compromised in open source.

that they delayed and tried to let someone else take it up first was their choice, but someone else in such a dispute should have just as much recourse to make a clean break when it happens.

[lusty quarry]

deletion as a form of expression should not be compromised in open source
the trade-off is not worth I would argue for the people negatively impacted

[elder tinsel]

the maintainer's rights matter more as it's their work.

the users getting it for free have a means to maintain it themselves under a permissive license (it has to be at least that permissive to even publish on pypi)

Back to constructive bits:

In the case of intending to inform, I think it's possible to have the build system warn already (you'd need to do it in setup.py or an extension) and import definitely is possible. with an import hook.

[lusty quarry]

I simply disagree, I'll quote myself:

People who participate in open source are volunteers and all volunteers implicitly enter a social contract. Namely, that one is providing value and the beneficiaries owe the volunteer nothing for that value.

By a volunteer taking away what they’ve already provided they are breaking that contract. We wouldn’t tolerate a medical professional with Doctors Without Borders to remove sutures or a builder with Habitat for Humanity to remove bricks if they decided they wanted to improve their conditions or if a beneficiary suddenly came into wealth. So, some limits exist and are justified.

[elder tinsel]

that's a disingenous comparison

[lusty quarry]

please explain? in my mind it maps on precisely

or better, comment on that existing thread if you want 🙂

[elder tinsel]

with open source, I've given the code, and you have the means to take a copy. My choice to stop publicly associating with the code did not stop you in the past from keeping a copy to use.

you continuously relying on it being available is you not availing yourself of what the licence allows

the distinction matters here, as code is often also speech or politically adjacent, espcially with the concerns of machine learning in the current socioeconomic situation

[lusty quarry]

source code removal I think is generally fine, I am speaking specifically about distributions on PyPI, NPM, crates.io, etc.

[elder tinsel]

I think crates is fine, they were day 0 "deletion isn't an option" policy.

I dont think you can retroactively change such a policy and have it be "okay"

people have to know they are getting into "Forever" in advance of doing so

[lusty quarry]

if Rust is able to do so legally then such a policy definitely does not break the contract of any licenses and therefore is only a matter of a handful of folks being annoyed, who themselves will benefit from the policy since if one is a maintainer then it is highly likely one is also a user of dependencies

so I think the trade-off is worth it

[woven yarrow]

@elder tinsel if you are ok with crates doing it, we could do it on PyPI for packages uploaded after today.

[elder tinsel]

I will strongly advocate that people never publish on pypi should that ever come to pass. With crates.io, it was clear from the start where the line was and what you were agreeing to with that. Changing such a policy about deletion may have benefits you see as worth it, but I think it's a direct attack on the people who power open source. A better way forward would be a new index with that as an explicit goal and made clear to those uploading in advance rather than changing the deal on them.

Could be as simple as forever.pypi.org , and serving the same files if they also exist on pypi.org (not duplicating them)

This is a fundamental change to what an author has the rights to do if changed.

[woven yarrow]

yes, but it could be announced, with a grace period, etc. if it's ok for Rust, why is it not ok for Python?

[elder tinsel]

because it's being changed out from under people.

[woven yarrow]

it wouldn't affect anything that had already been uploaded.

[elder tinsel]

are you going to ensure everyone has to have read the new policy before they can upload?

(this matters, some people have automated nightly builds as pre-releases)

[woven yarrow]

the shoe is on the other foot now: you want me to monitor 150 issue trackers, but package authors can't keep up with PyPI policy changes?

PyPI could email every address in every package metadata.

[elder tinsel]

considering this is a change involving authorship and legal rights, and could potentially interact with things like the right to be forgotten since it will be associated with a user on pypi not just the code itself under the license...

yes, it's different

[uneven karma]

Also on that note
According to Snyk, one of my orgs has almost 2_500 dependencies
Is that number ridiculous? Probably. Should it be reduced? Probably.
But it's physically not possible for me to maintain an active and ongoing mental understanding and awareness of that many packages

[elder tinsel]

I mean from a legal standing of pypi here on that by the way, not just a "right thing to do". I don't have the answers on many things, but I do know that legal agreements can't be enforced in some cases and there's potentially conflicting rights at play. Such a policy change probably requires more if you want it to actually serve the goal you want, where a clean new index (that pypi could even have an opt in "duplicate anything I upload to pypi.org here" checkbox for to make it easy) would not.

If your goal here is ensuring continuous availability, don't open up a legal maze when you don't need to.

[woven yarrow]

i'm looking for ideas for processes and tools that can let maintainers bow out with the least disruption (if they want to avoid disruption).

[elder tinsel]

would something like forever.pypi.org not be that? It's a clear seperate index, it could be configured such that names on pypi.org being taken imply the same name to the same user on forever.pypi.org (bidirectionally), and that uploads to forever.pypi.org require one additional agreement. Once agreed to, there could be a confugration setting that automatically makes uploads to one act as to both, and being served from the same org, this could be done without duplicating storage.

pip could then be configured to also know this index by default, and prefer it.

[woven yarrow]

yes, that could be a possibility, thanks for it.

[lusty quarry]

that really doesn't change anything because signing up has nothing to do with indices

[elder tinsel]

it does though, people that need the "Forever" can choose to only use packages that are uploaded with that intent

[lusty quarry]

wasn't your main issue the terms that you accept on the sign-up page?

[elder tinsel]

no

I'm not fundamentally opposed to a forever index like crates.io, just to changing one that isn't forever to forever in a way that isn't 100% airtight. forever is complicated legally, and there are a lot of issues that can come from terms that aren't clearly agreed to and forever each on their own. I don't think changing this is going to be worth it even socially (I do see the benefits, but I think changing it is difficult to do well), and what will the costs be to ensure no legal issues arise from it?

[queen hornet]

this is how npm does it: https://docs.npmjs.com/deprecating-and-undeprecating-packages-or-package-versions
you set some metadata, the installer prints a warning, the website shows one. they strongly recommend you don't unpublish=delete files from the registry

[elder tinsel]

is a policy update that goes out by email enough to legally get around things like the EU's right to be forgotten?

probably not, but im not the lawyer that would have to answer that.

there's clearly a demand (And good benefits for users) for a forever index, that doesn't need to mean pypi becomes that, could just be pypi also offers that.

[lusty quarry]

EU's right to be forgotten
I am in favor of such policies but I think this wouldn't apply. The only thing I can think of would be that PyPI could prevent showing the author on the project page. Internally, metadata can't be changed because then the contents of the artifacts would be different.

[elder tinsel]

I dont know if it's the only thing at play here, but if it is, the author bit on pypi is a little bit of an issue.

[woven yarrow]

IANAL, but the EU right to be forgotten is about "personal data", and has exemptions for things like "The data represents important information that serves the public interest, ... and where erasure of the data would likely to impair or halt progress towards the achievement that was the goal of the processing." I don't think it applies to published Python packages.

[elder tinsel]

Even if it doesnt apply, what's the costs of ensuring that's the case with a lawyer? What happens if someone decides to sue pypi anyway? (lawsuits can be expensive even with a win).

Then there's the social costs of such a change on maintainers that may feel they didn't sign up for their work being indefinitely provided.

Then, on the other side, how easy is it to have another index that once a user opts in for a package serves the same files, but ceases allowing deletion for packages that have opted in?

a lot of devs (I'll include myself here) don't have an issue with it if it's clear, done with consent and agreement of all involved. But I don't think a policy change is itself clearly requiring the consent and agreement of all involved by default. There are ways to make it so, such as preventing new uploads until agreeing to new terms that are presented in plain language, what's the cost of those?

This may not be legally required in some cases, the code clearly has a license to be copied, distributed, etc, but socially it's the right thing to do.

[woven yarrow]

I don't know the answers to those things. I hope other PyPA people will see this discussion and consider the options.

[mortal shore]

My plan is still what I wrote in that thread

To eventually disallow deletions on PyPI without administrator help, but there's a lot of things I think we need to do before that is a reasonable option

[placid wyvern]

Could both parties be placated by describing a fair deprecation cycle that respects both the author's wishes and the community's desire to maintain a non-volatile supply chain?
I would think that implementing a method whereby packages intended to be no longer supported or deprecated could emit warnings on installation through a package manager.

Obviously we have 'yank' that does... kind of this, but not really. I'm more thinking of the 'loud import hook' idea whereby the communication between PyPI -> packaging manager (pip) emits some warning about an impending deprecation/deletion, and after <x> period of time, that package is removed...

Brainstorming in the abstract here, I think both sides of the discussion hold a lot of validity.

Everyone would thereby hopefully be aware or made aware of the intention to delete with clear timelines that are communicated as effectively as possible across and upwards within the dependancy chain.

[mortal shore]

EU's right to be forgotten exempts data that is needed for legitimate business operations AFAIK, so I don't believe that we'd have any issues. Before we worked out a way to anonymize IP addresses before we stored them, we stored IP addresses indefinitely, even if the user deleted their account and our lawyer at the time advised us that was fine WRT the right to be forgotten because we had a legitimate business need to do so.

It's also legally fine to disallow deletions, even of existing artifacts. Whenever you upload something to PyPI you've already agreed to allow us, and any of our users, the right to distribute it indefinitely.

[woven yarrow]

We have today's deletion as an example of causing disruption. Anyone have examples of maintainers EOL'ing a package in a way that avoided disruption?

[elder tinsel]

no such thing. someone will depend on it, someone won't end up notified, or if notified, will be annoyed by it and continue using it anyhow, then when it stops being updated and eventually breaks, that was still disruptive to the user.

[woven yarrow]

ok, how about "minimizing disruption", "trying to avoid disruption"? Work with me here.

[elder tinsel]

we can come up with ways to make EOL more visible, but it's..

yeah

just want to be clear on the goal

[woven yarrow]

I've tried to be clear.

[elder tinsel]

I don't think "minimizing disruption" is a quantifiable or necessarily a directly actionable goal.

Does a goal of "have an easy to use, easy to find way to inform users of end of life/deprecation" with the secondary goal of this being used by users to make informed decisions going forward work for what you want out of something here? (I think it does, but just to be sure...)

[woven yarrow]

yes, it would be great to have more ways to inform users of a package of an impending EOL.

[elder tinsel]

I think something two-fold on that would be relatively easy then. standard library thing or a recipe for an import hook + one additional metadata field + tool support for that field should be able to cover both on install and on import

[mortal shore]

I also contest the idea that this is some sort of "attack on the people who power open source".

The people who are most harmed by deletions are people who power open source. When a project is deleted from PyPI, it's the rest of the open source community of authors who have to make a mad scramble to run around and fix their now broken packages that used to depend on X.

The second group of harmed people are simple every day users of open source, which of course also includes other authors (as noted above). PyPI is in an interesting place where it has to balance between giving power to authors to allow them to run their project as it sees fit, and restricting power from authors to give users the ability to rely on PyPI as a useful thing for them.

A number of things we've done over the last 10 years or so have been reducing the power that authors have to create an overall better experience for everyone involved.

The only people I don't really care about are random businesses who are affected, but those pople are also the people who the current situation is least painful for, because they can simply setup a mirror of PyPI (which many of them do anyways for other reasons) and be completely insulated either way.

[elder tinsel]

The situation today involves a developer being burned by a coporation ("random business") and ceasing support for a library that interacts with that random business's tool.

There's a lot to balance, but that someone may not want to be associated with something anymore and that the ability to not be is being put into question where it wasn't before, even if legally sound, isn't neccessarily a good thing.

[woven yarrow]

"where it wasn't before": you might want to check https://pypi.org/policy/terms-of-use/, which says, "by uploading I grant or warrant that the PSF is free to disseminate the Content, in the form provided to the PSF."

[elder tinsel]

I'm aware

again, "even if legally sound" is seperate here. Those terms to most people will be read as "obviously they need that to be legally allowed to do what I'm asking them to", and there is no written policy preventing deletions, and people can delete things currently.

[woven yarrow]

I will grant you that there might be some people who are bothered by it. Enforcing 2FA caused some people to be upset. PyPI can't please everyone.

[elder tinsel]

people agree to things they view as reasonable contextually, I don't think changing this overnight or retroactively, even if legally sound, is a good idea.

[mortal shore]

The thread that was posted is long, so I don't fault anyone for nto reading the entire thing, but one of the features outlined as something we should put into place before restricting deletions is some explicit features:

• The ability to "yank" at the project level, rather than having to do one release at a time.
• The ability to apply some sort of "notice" (without the yank semantics) to surface warnings in the UI and in installers, which may or may not include making the project read-only (similiar to GitHub's archive feature).
• The ability for authors to distance themselves from either a project, or from PyPI all together (likely implemented as some sort of "abandon project" feature).

https://discuss.python.org/t/stop-allowing-deleting-things-from-pypi/17227/71?u=dstufft

https://discuss.python.org/t/stop-allowing-deleting-things-from-pypi/17227/78?u=dstufft

those two posts still reflect my thinking on the subject

[woven yarrow]

Those all sound like great improvements.

[mortal shore]

Important to also note this post: https://discuss.python.org/t/stop-allowing-deleting-things-from-pypi/17227/73?u=dstufft

[elder tinsel]

to be crystal clear on my position:

• I'm in favor of a forever index, I see a lot of value here.
• I'm not in favor of existing uploads automatically having this behavior applied to them.
• I'm fine with new uploads having this happen if done in a way that ensures users are aware of the new behavior first.

More just a "we legally can, but lets try and make sure everyone is still intending to participate this way before changing the behavior" communication and clear consent with context for what is being consented to is important for it not feeling like this is just placing one group over another.

the goals are good, I just don't want anyone feeling burned or that pypi used broad legal terms to do something that wasn't clear they intended to do, etc.

[mortal shore]

I don't want to say we wont do it, but I'm -1 on some sort of time based filter on what files are allowed to be deleted or not. The distinction is relatively meaningless once a project has uploaded even one additional file, because then they can't delete some files on their project, but can other files, and I think overall it just creates a more confusing atmosphere for everyone.

As the last post I linked says, any change like this would come through a PEP, which would serve as both the avenue for authors who disagree with the change to weigh in, and help mold the ultimate policy (or if enough folks were against it, while not enough were for it, to block the change completely). But importantly, it would also serve as a notice to the community that the policy is changing, and if they don't agree with that, then they should take action to distance themselves now.

[elder tinsel]

are most users aware of PEPs to the point that they see discussions on them prior to acceptance?

(I'd say no just on volume)

[woven yarrow]

again, this seems ironic: you expected me to monitor all the issue trackers of my dependencies, I thought?

[mortal shore]

this is of course, all my opinion, an doesn't represent any sort of consensus from the PyPI admins, and until a PEP is written and decided upon, doesn't represent anything more than just what I think 🙂

[elder tinsel]

I'm saying that defensive dependency management is different from trying to do right by a community, it's different goals

You depend on something? try and keep an eye on it
You manage something for a community and you want to actually serve those users? Make sure you communicate with them

[woven yarrow]

once the PEP is written, I'm sure there will be much discussion about it. Interested people will participate in various forums to the extent they can. What else should we do?

[woven yarrow]

package authors depend on PyPI, right? They should keep an eye on it.

[elder tinsel]

they should, but it's an inversion here that you somehow arent getting. you should (because they may do something that harms you if you dont)

conversely, to avoid harming users, pypi should be proactive in notifying (The same way you want a more visible EOL)

[woven yarrow]

I will leave this in @mortal shore's capable hands. I am a PyPI user, not admin.

[elder tinsel]

the PEP process isn't something most users are aware enough of to ensure that alone gets wide visibility that isn't just people already heavily invested.

I'll keep thinking about this, and I actually get notifications for new PEPs, so I'll chime in at that time, but I think there needs to be more here. Not because it's legally required or not, but because it's changing expectations and we don't want anyone to feel slighted by process when it shouldn't be neccessary.

[mortal shore]

I'm sure that not every single person follows every single PEP.

PEPs like this often include additional communication mechanisms in them (for instance, when we removed the ability to upload eggs, we emailed anyone who uploaded an egg recently, which was part of the process decided upon in the PEP). Presumably this PEP would also contain communication as part of the process of deprecating and ultimately removing deletions.

We have things like the PyPI blog, the announcement mailing list, etc, that we'd be able to utilize as well.

At the end of the day, PyPI is itself an OSS project, and while it has a unique and privileged position, it's still bound by the overall limitations of most OSS projects. If you care about how PyPI is developed, operated, and what features it does/doesn't have, then you need to follow the channels of development. If you choose not to do that, then we'll still make an effort to inform you, but we're also not likely going to move heaven and earth to ensure that every single person who ever uploaded to PyPI is aware of the changes.

[elder tinsel]

then we'll still make an effort to inform you

This part here's the main bit, as long as this isn't just flipping a switch without real prior notice, I'm mostly fine with it. I'd want to see what that effort looks like in this case, casue this is a much bigger change than no longer uploading eggs, that was visible in a "this stopped working, but you have other options that should work, here's how to use them" this would only be visible to a user who missed notification when they went to delete, which could in theory be anywhere from seconds after, to years later, to never.

while it has a unique and privileged position
This part is also relevent. This isn't just "do I trust depending on OSS", this is "are the people running the primary OSS ecosystem for this language making an ecosystem I want to participate in and place trust in"

I see very high value in this particular change, but improper notice based on the weight of a change even if I agree with the change raises "okay, so what happens if it's a change I dont agree with?"

Anyhow, I'll get off the soapbox, but pypi being in such a central position and the goal (I hope) being to try and serve all users fairly while balancing practical and security concerns isn't easy. I'll try and contribute to a real plan of notice when the pep comes around.

[lusty quarry]

no longer uploading eggs
this was handled very carefully by everyone involved

[mortal shore]

Changes like this are the hardest ones to communicate, because we also have to be cognizant of the fact that people who signed up for PyPI didn't necessarily do so in order to get emails from us everytime we made some change, and in fact whenever we add a new notification email we tend to get a burst of spam reports from people who don't want to get those emails.

Like you said, a change like this could possibly impact anyone, but in reality the impacted people are going to be a small portion of PyPI, because we can't see the future and know which users were going to use this feature and which were never going to use it, but the flip side of that is we have to take care not to burden the users who don't care with over communication. Users untrust us with their email, and part of what they're trusting us for is that we're not going to send them unneeded or unimportant things (at least to the extent we can).

The same is true with eggs of course (or any change), we don't know which people were planning to upload eggs in the future, but we had a pretty obvious qualifier we could use to separate out people who were likely to be impacted from the set of all users who could possibly be impacted.

Doing that for deletion will be harder, because there's not any clear/obvious qualifier, but most people (I believe anyways, I haven't dug into the data to verify this yet) will never use the delete button, so simply blasting everyone would likely be too broad of a campaign, and would be misusing the communication channels we have (but maybe I'm wrong! maybe everyone is deleting things!).

So I don't have a "This is how we'll communicate it" sorted out yet, because we don't have the blocking features implemented yet so I haven't really dug into the actual nuts and bolts yet, beyond getting some level of broad agreement in that thread that this is an avenue we should pursue.

(and I say "pursue" on purpose, because it's not a fait accompli, it's a thing I think should happen, and I, of course, have a position that means that things I think should happen are more likely to happen... but I want to be very clear that this is still no more decided than an idea that happened to be discussed on discuss.p.o and people at that time were generally on board with it)

[elder tinsel]

Sometimes it seems like a lot of things are on a track that they will inevitably happen. I understand that process is still going to happen, but this seems to be one of those things. I don't say that to imply any mishandling of process, I say it because I generally believe it's both the right thing to do, well motivated, and easy to argue for. However, my experience with such obviously good things is that they are often done in the obvious way, when there may be either a better way or a better way to communicate on the changes, and the way a lot of people talk about preventing deletion doesn't really sit well with me.

[woven yarrow]

It's clear to me that @mortal shore is approaching this thoughtfully. In terms of inevitability, it's been in discussion for 15 months, so clearly there's no headlong rush to get it done.

[elder tinsel]

FWIW, This is exactly the kind of change I'd expect to be emailed about in advance of it going into effect, otherwise why do you need me email if you aren't going to use it for important changes? This is a change that has permanent consequences on everything I use pypi for once that switch is flipped. (if, etc).

[mortal shore]

Not to like, toot my own horn or anything, but things that I think should happen generally do happen if/when someone does the work... so like it's definitely more likely it'll happen than if some random twitter user was posting it?

Of course part of that is because of the position I'm in, but I think (or at least I like to think :D) that part of that is because I tend to aggressively shift the goal posts for "what I think should happen" as discussions happen and new information arrives. If, when the PEP is written, the tide of sentiment is against it, then it won't happen, whether I think it should or not.

Anyways, I don't think it's super useful to litigate the exact communication mechanism atm, since we don't have a PEP written yet and we're probably still a good length of time away from it since we have several blocking features that are as of yet unimplemented. I only mention it because it's one of the things that make communicating tricky

[elder tinsel]

I mean no offense by this, but thoughtfulness has limits based on what we are consciously considering. The very fact of where we are having this discussion biases the conversation to people heavily invested in packaging, and it can be hard to consider cases that are outside our knowledge. For something as permanent as "no longer allowing deletion", it's exceptionally hard, to the point where I don't know if we can realistically include the viewpoints of everyone who would be impacted.

This isn't easy, and I'm not trying to throw blame around on it, I think we all want long term ecosystem improvement or we wouldn't be here right? I just.. I don't know how we do this in a way that feels okay retroactively, which is why the whole opt-in second index for "forever" came to my mind.

(may not be a good solution for reasons I can't see either)

[mortal shore]

The opt in second index is a non trivial amount of additional, ongoing work and maintenance not just for PyPI, but for the entire Python ecosystem as they have to differentiate between what index people are using

and in general it's not a solution that scales, we've "changed the rules" a number of times over the years, each time some people were happy, some people were upset, overall I think those changes made things better, but if we had to split out a seperate index for each one of them, we'd be up to like 6 or 7 indexes by now

[elder tinsel]

I was thinking something entirely automatable at this point where once a package opts in, anything on pypi is just also visible on the second index, and just configuring pip (or other tool) to point at it, but I see the point still.

the second index then just provides "these packages have commited to being available as provided for as long as pypa is around"

wonder if metadata could do the same job though

[mortal shore]

right, but the mechanism for doing that is something that has to keep working.

Like as an example, PyPI actually does run a second index currently. It's not directly exposed to end users, but we run a bandersnatch mirror which is intended to be a fallback that Fastly will automatcally fall back to whenever the main PyPI is down. That second index basically stops working for months or a year+ without anyone noticing and that is one of the best cases for us because we don't actually have to maintain the software, we just have to keep the server running and the cron job going.

the reality is any service that is operating requires ongoing maintenance

[elder tinsel]

I'm trying to think of a less retroactive answer to the general problem to avoid the social issues I can forsee without seeing a way to satisfy everyone if the answer is just "dont allow deletions"

one way could just be "nodelete=true" in the metadata. + tools having a way to only install things that have that set.

maybe it just needs to change and we do the best we can to communicate it though

[mortal shore]

Our strategy thus far for any changes like this is to do the best we can to communicate it, and ultimately make the change. I think it's worked reasonably well so far. Of course not everyone has been happy with every decision, and that includes people who have left PyPI to publish their projects elsewhere (though I think the ones I know of who did that kept the name on PyPI to prevent name squatters).

It's definitely a balancing act, and there's hardly ever a perfect answer that doesn't harm someone.

Also on just general grounds, I think part of defining a coherent platform is deciding which things you say yes to and which things you say no to. I think a fundamental problem that a lot of OSS projects have is they struggle to say "No", so every time something changes, they tend to collect these configuration knobs to support the old way and the new way.

Each one of those knobs are, in isolation, typically relatively low impact, but the more of them you collect, the more difficult the project becomes to understand, to work on, to use, etc. There's an old blog post that I can never find again that goes into this and I wish I could find it again because it was great (it was about the gnome software project, etc).

So I'm generally opposed to configuration knobs, or rather I think there is, and should be, a high bar for configuration knobs to exist, and if we can get away without one, then we should strive to do so, because I think the alternative ends up looking like this:

[elder tinsel]

I think it depends on the knob. That's a great example in counter, but I think something like this is something that (should) be set and forget in a global pip config based on if you just want what's there now or if you care about only using things that will always be there.

but... i see where it leads and agree to generally fewer knobs where possible.

[mortal shore]

yea I'm not a religious zealot about it, some knobs are important 🙂

This particular knob I don't feel is a very good one, because I suspect it ends up being a cargo culted "make pip do the thing I expect it to do" knob, not one that people spend any time thinking about it

[elder tinsel]

hmm

[mortal shore]

Because I think it will basically be a knob nobody uses, until something gets deleted, then they get mad, find the knob, and set the knob forever and never think about it again

until the next person has the same thing happen to them

or they get a new laptop, or whatever

[elder tinsel]

and on by default would basically just be ecosystem saying "We arent Forcing it but..."

yeah, no this knob isn't good

[mortal shore]

right, on by default is basically just making the change, but being dishonest about it

[elder tinsel]

Each one of those knobs are, in isolation, typically relatively low impact, but the more of them you collect, the more difficult the project becomes to understand, to work on, to use, etc. There's an old blog post that I can never find again that goes into this and I wish I could find it again because it was great (it was about the gnome software project, etc).

Just use monads when designing tools that consume options and it just works.

(of course at the point where this option makes sense, the project is already more difficult to understand to those whom it doesn't make sense)

[mortal shore]

(because I imagine exactly 0 people would ever purposefuly opt into the "please let things get deleted and break my build option")

[lusty quarry]

I understand why PEP 665 chose to postpone support of source distributions until later. At work right now I have to resolve dependencies for a bunch of platforms with conditional logic and build and upload whatever doesn't have an existing wheel to a new internal index which I have to create still.

This is supremely sucky

if everything had a wheel >60% of my time would be saved

[lusty quarry]

update: I stand by my claim that this is quite tedious but I'm approaching the problem from a different angle now and will have multiple indices instead which will make things a little easier

there is no easy way to have a single index that serves every platform that the Agent supports. our existing build pipeline is quite intricate and has built up conditional logic over time such that the wheels that we build differ based on the platform due to compiler flags sometimes and other times features that we wish to exclude

example of differing feature https://github.com/DataDog/datadog-agent/blob/0cc9d2f90e49a7524f2e6d4aa995449e024eaa94/omnibus/config/software/datadog-agent-integrations-py3.rb#L230-L233

example of differing compiler flags https://github.com/DataDog/datadog-agent/blob/0cc9d2f90e49a7524f2e6d4aa995449e024eaa94/omnibus/config/software/datadog-agent-integrations-py3.rb#L174-L187

[bright parcel]

@lusty quarry I can definitely understand why having to install build dependencies and ensuring that sdists/sources can be built into a consumable format would be painful. Is it true even in the face of pure-Python libraries? I would think those would be easy, but likely not sufficient in most practical cases.

[lusty quarry]

it would be much easier if we controlled the platforms of course but we have to support everything our customers run on

[feral lake]

are the manylinux Docker images supposed to require authentification for pulling? eg. quay.io/pypa/manylinux2014_x86 ?

Trying to pull it says Error response from daemon: unauthorized: access to the requested resource is not authorized and opening it via the web interface gives me a 403

where can I find publically accessible versions of the manylinux images?

[feral lake]

Ah I see, the name in the repo seems to be wrong, it's manylinux2014_x86_64

[blazing lantern]

One possibility is using the Trove classifiers representing the project's development status; there is an Development Status :: 7 - Inactive one to flag that development has ceased, which might signal it's time to vendor

[woven yarrow]

That would be cool if that status were flagged by pip (optionally). I doubt many people use that status.

[mortal shore]

we'd have to add it to the simple API

[elder tinsel]

I wonder if more people would intentionally do a metadata only post-release to get the project marked with that if it meant tools would use that to inform people doing pip install (etc) or even in other tools like pip-compile of the project now being inactive.

[mortal shore]

though I don't know that classifiers are a great way of doing it since you have to make a new release

although I have some random sketches around here of an idea to allow metadata patching

[elder tinsel]

After the last discussion here, I started looking at metadata, and I think we could safely allow patching a lot of metadata if it had to be done with an additional file (no modifying original) and were careful to keep the things allowed to be changed relatively constrained.

• Adding a specific "package != version" without otherwise changing the range (This coincidentally hits a nicer balance on the permissive version range discussion we had earlier)
• Changing non-install related metadata like trove classifiers
• changing project urls/email
• editing a description

[high stone]

Metadata patching

[blazing lantern]

In case people didn't hear: https://prefix.dev/blog/introducing_rip . I'm not sure of the true state of things as the blog post mentions "a high-performance wheel cache and the ability to actually unpack and install the wheel files", while the README at https://github.com/prefix-dev/rip says that installation is not done yet

[lusty quarry]

In case people didn t hear https prefix

[echo ruin]

Some of you -- at very least @mild dew -- must know about this already, but I did not! https://cfp.packaging-con.org/2023/schedule/ -- next week.

[mild dew]

I’ll be there IRL, so say hi if you’re also there!

[echo ruin]

Nice! Enjoy it, would love to be there myself if I was in Berlin next week but alas, next one...

[high stone]

I'm going to be at PackagingCon as well. :)

[coarse iris]

Any chance push a release of shellingham to PyPI for fixing a bug on MacOS? the action secrets seem invalid @spiral urchin @lusty quarry

[lusty quarry]

I don't have access to update secrets, unless I publish it locally

[spiral urchin]

Let me regenerate one

OK done, as PYPI_API_TOKEN

I also configured Trusted Publisher on PyPI but don’t know how to use it tbh

[coarse iris]

Thank you both for your help. The new release is out.

[blazing lantern]

Once you have it set up on PyPI, this is how I have been using it: https://github.com/brettcannon/microvenv/blob/main/.github/workflows/release.yml

[high stone]

I've gone ahead and created #welcome to direct all the "person has joined" messages to, since that seems to be the bulk of activity in this channel which has buried the discussion here. :)

[torpid veldt]

Where is the up to date spec for PEP 685? It says it's a historical document and it seems pip and pkg_resources don't implement it

But of course poetry does implement it so seems incompatible

[spiral urchin]

The PEP has two parts, one on producing and one on comparing the produced output. Most backends should implement the producing part now. Not sure about the comparison part, the interest seems low in general.

[inland prawn]

Is it bad that we implement it?

It's a standard after all

[torpid veldt]

Currently I have an issue with visions[type_image_path]

When I export to requirements.txt I get visions[type-image-path] in there and that breaks liccheck and pip install warns about a missing extra

The problem happens if I do a poetry add ydata-profiling and do a requirements export

[torpid veldt]

No I don't think it's bad that you implemented it, it's bad that not everyone implemented it though

[indigo token]

There’s an infinite recursion error here:
The PEP refers to the PyPA specs page:

This PEP is a historical document. The up-to-date, canonical specifications are maintained on the PyPA specs page.
The PyPA specs page refers to the PEP (on https://packaging.python.org/en/latest/specifications/dependency-specifiers/)
Restrictions on names for extras is defined in PEP 685.

It also seems that the PyPA specs page doesn’t define how to compare extras, unless I missed that.
I think to get rid of ambiguities, we should

• replace the reference to PEP 685 with a definition and a footnote (“as originally specified by PEP 685” or so)
• clarify on the PyPA specs page how comparison of extras works.

[queen hornet]

https://pip.wtf

[inland prawn]

I don't get why people don't like TOML so much. I'd say it's the best currently available config language

[rapid glacier]

Fairly certain it is because of the perceived story of:

„You have to create a setup.py“

„No, best practice is setup.cfg“

„Now the thing to do is pyproject.toml“

Which probably makes a lot of people hesitant to change to pyproject.toml, because they think it will be as short lived as the ones before. Maybe even some anger about changing practices projected onto the thing that is the latest change (pyproject.toml).

[distant briar]

Does it need to hit a decade before people realize that it's not going away

[woven plover]

how optimistic

[lusty quarry]

there wasn't an actual standard until pyproject.toml, so I think the situation is different

[inland prawn]

Yeah, previous "standard" was setup.py, which was later inherited by setuptools

[rapid glacier]

I mean, a standard can often be something descriptive and both setup.py and setup.cfg was used as a defacto standard. When I started with python, each and every book/blog/course/text on random restroom walls/… was saying the standard is setup.py. Most still do.

(To say it clearly before someone assumes something because of the above: I like pyproject.toml)

[blazing lantern]

It's defined in https://packaging.python.org/en/latest/specifications/core-metadata/#provides-extra-multiple-use, so https://packaging.python.org/en/latest/specifications/dependency-specifiers/ can probably get updated (or just made to be past tense). The core metadata spec also defines how to interpret things with older metadata versions that don't record the normalized name which implies comparison rules.

[blazing lantern]

Obviously didn't participate in the PEP 722/723 discussions

[queen hornet]

They link to them in the rant, so at least knows about them if not participated

[blazing lantern]

Then I have no sympathy. They just seem grumpy we didn't solve this a decade ago for old versions of Debian

[indigo token]

Speaking of which, there’s a little implementation of it: https://github.com/ThatXliner/idae

Also @merry rune implemented it for #pipx, right? So one can do pipx run path/to/script.py?

[blazing lantern]

I think that support hasn't been released yet while we figure out PEP 723 and the whole dependency group discussion (the original implementation pipx mirrored what pip-run does which is different slightly than either PEP 722 or 723)

[gritty roost]

is there a PEP that describes the selection priority when multiple wheels match a platform? E.g., when there is both a cp312 wheel with a platform-specific native extension and a py3 pure-python wheel

[lusty quarry]

I don't think so, but in practice I believe this is what determines priority nowadays https://packaging.pypa.io/en/stable/tags.html#packaging.tags.sys_tags

[blazing lantern]

Correct and correct; no PEP specifies priority order, but packaging tries to stick with what pip generates as the order (essentially most restrictive to less restrictive with some quirks thrown in just to make my life difficult 😉)

[high stone]

And, this situation is better than the old one where all the tools had copies (AKA forks) of a file that had diverged and they all did things slightly differently. :)

(thanks again @blazing lantern for championing the consolidation)

[gritty roost]

I'm currently using pip's TargetPython.get_sorted_tags for my use case. Guess I should switch to packaging. I don't ever want tags for the host system, though (or if I do, I want to indicate that explicitly), and it seems like there's a lot of "default to host" in packaging.tags. To be fair, there's some of that in TargetPython as well.

[lusty quarry]

I don't know if it's true now but I thought pip could not provide any backward compatibility guarantees outside of the CLI i.e. anything internal that you import is not meant to be used

[high stone]

It's still the case. I reserve the right to move and rename things as per the needs of the project, the phase of the moon, the position of jupyter in the night sky and whatever my home planet is.

[gritty roost]

Right. Recently get_tags split into get_sorted_tags and get_unsorted_tags. But sometimes you do what you've gotta do...

[mortal shore]

yea, afaik pip does not go out of it's way to break things for people, nor does it go out if its way to not break things 🙂

[queen hornet]

https://github.com/python/cpython/issues/87211 points to a Stack Overflow post -- they started at https://docs.python.org/3/installing/index.html then moved on to https://packaging.python.org/en/latest/tutorials/installing-packages/ but still got stuck

there's been lots of updates on python.packaging.org, does this issue still look valid?

[hollow trout]

It's hard to tell what that user's problem actually is, since there are few details in the SO question.

[queen hornet]

yeah: "This did not work either" isn't very helpful. if there's nothing actionable, let's close it

[blazing lantern]

The API has been specifically small for that module for ease of maintenance. But if you have a use case for something then please open a feature request at https://github.com/pypa/packaging and we can consider adding it.

[queen hornet]

some discussion about generating SBOMs for manylinux images: https://discuss.python.org/t/create-and-distribute-software-bill-of-materials-sbom-for-python-artifacts/39293/8
leave a comment there if you have insights

[high stone]

@mossy pulsar I approved the pre-commit.ci request for pipx. Noy sure why you made the same request on the UI multiple times. 😅

[woven plover]

I'm wondering, would anyone be interested in working out a Way for packages to ship locks/constraints

I have some flaky tooling at Work to generate a Baseline of pins for the external dependencies of around 100 internal packages and it's still feeling like a shipwreck (maybe mileage can get better if hatch workspaces get done)

[inland prawn]

what about what Poetry offers with poetry.lock? unless you want to build artifacts with pinned deps

[lusty quarry]

if hatch workspaces get done
that's the next feature after the current release going out soon

[woven plover]

I need locks for only 3rd party packages as baseline,

[inland prawn]

so, nix?

[woven plover]

Nope ,not a fit for the organization,the teams and the other surrounding tooling

[inland prawn]

conda? never used it but I heard it supports both python and 3rd party packages (and I think I have seen some kind of lock file there too)

[woven plover]

Currently I use a makeshift pip compile clone+ wrappers to ensure pip gets invoked with the constraints

(which is alright, pip compile ought Not to have to support the use case I have)

[gritty roost]

Bazel?

[woven plover]

That one is completely orthogonal and solves different problems

[gritty roost]

I probably don't understand the use case.

[lusty quarry]

I agree with Secrus, your requirement sounds like a fit for Conda

[woven plover]

i need to pin the outer dependency set of a few hundred packages without pinning those packages, as far as i understand thats not part of how conda works, in particular if those deps arent mirrored on conda forge yet

[lusty quarry]

what do you mean by pinning without pinning?

[woven plover]

@lusty quarry i want to pin only the 3rd party dependencies to define a baseline, but nothing of the internal items that update often (this is primarily to ensure everyone is on the same externals while moving the internals faster)

[lusty quarry]

oh I see

excuse my ignorance as I might be misunderstanding, but don't you just want to provide a builder image e.g. as a basic example, a Dockerfile with a bunch of pins on top?

[woven plover]

the pins are shared between > 20 teams with different responsibilities and differend local/docker/k8s workflows and different operating systems for development

before the pins and enforcing them we had numerous "fun" effects with staleness

the eventually betters solution wil lbe to eventually use toolbox and devcontainers as base, but that takes time to migrate all the pices to

[lusty quarry]

at work we use the concept of "golden base images" where at the deepest level a team or two is responsible for maintaining the security and core OS, then teams add layers continually on top

[woven plover]

(the pining tooling started at a point where we didnt even have ci containerization, the ci side is much better already but the dev experience for local dev is not yet lifted up

the "golden base images" is basically what would be the base images we'd use for toolbox/devcontainer images to enforce devtool parity for the most part

[lusty quarry]

with more understanding of your situation I think the other suggestion from above of using nix would satisfy your constraints, but I understand as you mentioned that it wouldn't work for your org

[woven plover]

tbh, with nix i hit a level where i strongly want to avoid confronting most people with it - its fine when all is right, but its a grave mess for anthing else due to how modules and package overrides work with it,

its incredibly hard to debug and safely maintain at times - and anything that needs to sanely manage state versions is practically unable to use nix without massive hackish workarounds

[ivory phoenix]

recently i've sorted some packages by popularity, but only taking into account how often a package had been downloaded up to 30 days before.
as i started thinking about assessing how "popular" a package is, or maybe how "important", in fact, i wrote this short stream of consciousness https://github.com/jaraco/skeleton/issues/98#issuecomment-1839865665.

is there actually any existing solution for evaluating what third-party packages are the most critical for the entire community, taking convergence and download statistics into account? i don't have much knowledge in this topic and it got me thinking.

[blazing lantern]

PyPI uses download count to identify critical projects

[lusty quarry]

FYI at some point this page got broken https://peps.python.org/pep-0621/

until very recently there was a banner that directed you to the living document

[queen hornet]

yeah, packaging.python.org re-organised their pages, and broke the intersphinx ref we were using at peps.python.org and the banner was deleted to get the builds passing

I've got a PR to put the banner back, let me get that sorted and merged

and it's back! thanks for the reminder!

[lusty quarry]

awesome, thanks!

[queen hornet]

Graphs of backends used by PyPI projects https://framapiaf.org/@fcodvpt/111533565837454708

[ivory phoenix]

that's a lot of backends.

[inland prawn]

Many projects use their own variations

[queen hornet]

yep, here's graphs with backends removed if they're used less than 5 times https://framapiaf.org/@fcodvpt/111540079686191842

goes something like: setuptools, poetry, hatchling, flit, pdm, maturin

[lusty quarry]

poetry backend adoption I reckon is mostly from enterprises that have projects that require (or think they require) lock files and they just go full poetry

actually, forget me. enterprises wouldn't be publishing to PyPI and therefore wouldn't be tracked here

[peak ocean]

Poetry was the first of the new PEP-517 backends, that I heard of, I think that drove a lot of users to it

[lusty quarry]

oh interesting, I always thought Flit was the first

[peak ocean]

Could well have been, but poetry was the first I heard people talking about adopting

[distant briar]

Poetry has its own philosophy behind it, "packaging reimagined"

[mortal shore]

flit was first if I recall, but never had much marketing

[inland prawn]

flit didn't have much marketing + it was just packaging, while poetry also installed packages, locked them, managed project venv and provided dev-dependencies (why is this not a PEP-ified thing is still amusing me). Basically, at the time it was the all-in-one tool for Python

[inland prawn]

first work on Poetry pre-dates acceptance of pyproject.toml standard (used to be poetry.toml) 😄

[inland prawn]

had. this is now outdated with how packaging evolved beyond just setuptools