PyPA and "the consent of the governed"

[bright sequoia]

We learned the hard way that interoperability standards are only meaningful if tool authors care enough about them to actually implement them.

We got the first standing delegations for the PyPA era in place by way of gathering a pile of packaging tool authors in Santa Clara for PyCon US (so they could assent to the initial standing delegations), alongside the Python Language Summit that year (so we could talk to GvR about creating a standing delegation mechanism for packaging PEPs, since all prior delegations were made on a per-PEP basis).

Succession planning for the standing delegations (which we only used once, when I handed my side of things over to Paul) was via nomination by the incumbent and ratification by GvR, and later the SC.

Setting up the Packaging WG wasn't democratic either - that was getting a few of us that the PSF Board were happy to trust together in order to give the PSF technical advice on ways to spend money that would be most helpful to the packaging community.

PEP 609 was an initial step towards a more general PyPA governance model, but it didn't change the standing delegations or the funding administration arrangements.

PEP 772 is the whole ballgame - taking the ad hoc arrangements we made a decade ago, and replacing them with something that allows tool authors to more directly influence how the standards and services they rely on are managed.

(Addendum: the way I worded that makes Santa Clara sound more planned than it was. Having a whole pile of tool authors there was planned, since the previous path forward had failed when the planned distutils replacement was dropped from Python 3.3 as being unworkable in practice. However, we worked out while we were there what we thought would make sense as the next steps, settling on the refrain "./setup.py install must die", and everything else we proposed was just what we thought would help us get to that outcome)

[fringe blaze]

I heard the calling setuptools setup.py directly must die sentiment a lot as I've got into the packaging world over the last 5 years, so it surprised me that rereading PEP 517 recently, due to pip 25.3 being scheduled to drop calling setuptools directly, that it's not against the spec:

tools should revert to the legacy behaviour of running setup.py (either directly, or by implicitly invoking the setuptools.build_meta:legacy backend).

[bright sequoia]

The "install" part in that shorthand phrase is key. Build tools are inherently in the business of running arbitrary code at build time, so the only setuptools command that was intrinsically problematic was ./setup.py install (since it would run arbitrary code at installation time, potentially with elevated privileges, and with no reliable way of statically analysing what the install would do).

The setuptools devs had their own reasons for discouraging other direct setup.py invocations, but there was no ecosystem level imperative for that aspect of the changes.

[bright sequoia]

Huh, I was coincidentally reminded that I actually wrote a more contemporary history of those events, unhindered by the extra decade of separation in time: https://python-notes.curiousefficiency.org/en/latest/pep_ideas/core_packaging_api.html#a-bit-of-python-packaging-history

[pale dagger]

For a long time there wasn’t really any decision making process, it was just tools doing whatever they wanted (and then other tools trying to emulate that).

I vaguely remember the burst of PEPs around wheel, version numbers, PyPI hosting, etc that really started the idea that the PEP process could be used, though I wasn’t at SC PyCon so I don’t recall if the idea got pushed on the mailing list first or what.

I do remember personally being really against using the PEP process (which is funny to me now given my position 😅 ), and it being @bright sequoia who really convinced me that it was a good idea.

[bright sequoia]

It was Tarek that restarted the use of PEPs prior to 3.3 (standardising the package dependency metadata and the installation metadata to support uninstalling packages). My role was then making the argument that we could tweak the PEP process to make it less annoying for that use case (primarily via the standing delegations, so we could accept PEPs directly on the packaging mailing lists without having to repeat discussions on python-dev with folks that didn't necessarily care much about packaging topics)

I'm not sure of the exact sequence of events, though. The three big things I remember converging before Daniel & Marcus Smith set up the first packaging mini conf in Santa Clara are:

• Éric Araujo deciding that distutils2 not only wasn't ready, but wouldn't solve the problems we had, so it got dropped from 3.3
• Daniel Holth proposing the wheel format and platform tagging mechanisms
• your creation of crates.io as a demonstration of what PyPI could be

I had also talked to Richard Jones (for those that never met Richard: core dev, original creator of PyPI, founder of PyCon AU) about what we could do to make your ideas the way the actual PyPI worked.

I personally called getting involved my "keep Daniel & Donald from burning out" project, as you both had excellent proposals, but were also clearly getting understandably frustrated at the lack of an obvious way to get consensus on what we should do about it.