pypi

[oak pebble]

Knowing that it should be 5 mins is helpful.

It's consistently taking 40 mins for me. It seems to bog down a lot in the Legacy tests.

I thought I had noticed the tests take quite a while on my PR so I thought it was normal for it to happen on my pc. I'll double check that.

My setup is Windows, i7 11th gen, 8 processors, 32 GB mem.

I run in WSL2. 8GB mem, 4 processors.

I watch the processors and memory in htop while the tests run and processors are totally fine.

Memory was running around 7 so I doubled to 16 for a run but it still ended up running for 39 minutes. Now that I think about it though, I didn't check htop for that run so maybe it was maxing out the 16 GB.

I'll double check the following
:

• how long tests took in my pr
• how much memory tests used when I bumped wsl to 16 GB
• if there is info I can use in the coverage report when limiting tests to 1 file.

Thanks for the response @tribal sedge!

[tribal sedge]

Thanks, helpful details. I don't have a ton of experience on Windows/WSL, so my assistance might not be too helpful.
At the end of test runs, there's also a duration table of slowest tests, but it sounds like something else is wrong.

Our getting started docs for Windows show using Docker Desktop, do you do that as well?
https://warehouse.pypa.io/development/getting-started.html#installing-docker

[valid flame]

I think the issue might be in the way WSL syncs the filesystem. a colleague of mine had similar issues (normally fast ops taking a long time) working on WSL

[oak pebble]

Its been 5 months since I got through the Getting Started docs so I can't recall if I went through the docket section but I am using Docker For Desktop. I THINK that's where wsl2 runs

[oak pebble]

Very true! I'll check this. I hit this very quickly when I started development I wsl so I never share files between host and "VM". But definitely worth checking.

[fair wasp]

WSL stuff still hits any filter drivers in windows (notably, windows defender). There was a big thing with rustup and this being one of the things slowing it down.

[oak pebble]

I'll dig a bit more into the tests that are slow and see if something jumps out at me. Maybe something to do with the company VPN or something random like that. Maybe some how is looking at files in Windows instead of WSL.

Again, helpful knowing how long they are expected to run! Thanks for that!

[oak pebble]

I can't imagine that's enough to slow things down by 8 though? It perhaps. I'll keep that in mind as well!

[fair wasp]

It depends, in rustup's case, at one point in time it was the overwhelming majority of time taken.

https://www.youtube.com/watch?v=qbKGw8MQ0i8&t=2446s Has more context, and NTFS was not the issue at any stage (hence the title of that one)

an open issue on this: https://github.com/Microsoft/WSL/issues/873

[oak pebble]

Oh awesome! Thanks for the details. I'll take a look at this for sure then.

[unborn nacelle]

pypi sends

cache-control: max-age=600, public
for simple json responses such as https://pypi.org/simple/django/?format=application/vnd.pypi.simple.v1+json. Doesn't that mean that it takes 10min from when i publish a package till pip or any other client picks it up?

I see how this massively reduces the number of requests, i'm trying to understand the tradeoff/user experience here

[fair wasp]

9 minutes 59 secondsif they ask at Exactly the second before it's updated.

the page isn't 10 minutes behind, but pypi asks that clients keep a 10 minute cache

[unreal jewel]

the cache-control header isn't a mandate, it's telling clients how long they can cache the response for before they should attempt to revalidate it. Browsers and many general purpose http clients which implement caching will typically, as mentioned, re-use the same response for up to 10 minutes, which will only be 10 minutes if you happen to be requesting the same URL multiple times in a 10 minute span, but they're not required to.

For pip in particular, I believe that they instruct the caching library to set a max-age of 0 to everything, which will still cache things, but only for use with ETag / conditional gets.

[unborn nacelle]

thanks! i'll copy what pip does, ignore max-age but using the other http cache semantics

[oak pebble]

Yes, I've been doing this as I worked

[oak pebble]

Following up on this: I don't think I saw that the docs recommend installing both Docker For Desktop in Windows and also Docker in Linux when running WSL. I just tried setting everything up from scratch but am now unable to get passed make initdb due a missing image.

[oak pebble]

I just realized that I'm picking on Dustin a bit with my PRs. Is anyone else available for a review on this PR where I'm refactoring the OIDC code a bit? https://github.com/pypi/warehouse/pull/14063

[rare wadi]

#267624335836053506 message (links to https://discord.com/invite/python)

User wanting to upload their first package is confused about registration being restricted
Is there anything we can can tell them about when registrations will be allowed again?

[oak pebble]

Anyone know off the top of their head if there is a short hand you can use with
dcker compose run --rm web python -m warehouse db downgrade [enter short hand] like head~1(i tried this one and it doesn't work)

[acoustic panther]

https://github.com/pypi/inspector/pull/157

[tribal sedge]

We pass through to alembic, so downgrade -1 ought to work. Ref: https://alembic.sqlalchemy.org/en/latest/tutorial.html#relative-migration-identifiers

[tribal sedge]

Sorry about that, no specific date yet, but hopefully early next week.

[oak pebble]

Perfecto, thank you! And thanks for the Alembic docs link.

[acoustic panther]

@merry valve Hi Dustin! Mind taking a look at this when you get the chance?

[oak pebble]

Any chance I could get a review on this PR that refactors the OIDC token endpoint code to make it easier to add additional handler code? https://github.com/pypi/warehouse/pull/14063

[acoustic panther]

Testing out Monaco for the inspector. Thoughts?

CC: @merry valve

[tribal sedge]

I found out today that this needs to be specified like this: python -m warehouse db downgrade -- -1

[oak pebble]

OH i never actually got around to trying as my migration was working as I expected. I'll give it a try meow.

Partly worked. The argument makes into the alembic call but it fails do figure out what to do with it I guess.

alembic.script.revision.RevisionError: Relative revision -1 didn't produce 1 migrations

Not a biggy. My migration, as I said, is working and I need to jet of from work now any way.

[tribal sedge]

hm - you might have to rebase that migration, it might have two heads, especially if a new migration has been added since your down_revision was set

but all in good time

[tribal sedge]

Signal booster: https://fosstodon.org/@brettcannon/111564223485456257

[rare wadi]

Just curious, has PyPI seen the same outcries of frustration that the likes of GitHub have?
It seems like practically every day now I see someone on Discord or Reddit or wherever saying that "MFA is evil and they "NEED" to be exempted".
But I don't think I've seen a single person complain about PyPI doing the exact same thing.
Have I just missed it?

[meager umbra]

Pretty sure I've seen complaints.

https://lucumr.pocoo.org/2022/7/9/congratulations/

[unreal jewel]

I've only really seen complaints over the inital MFA requirement which was targeted at the top N projects

[valid flame]

any TL;DR why people don't like it?

[rare wadi]

TL;DR: it's too slow to pull my phone out every time I need to login
Also lots of people saying that that "my code doesn't mean that much to me, so I don't care if my account gets hacked [, so I shouldn't be forced to take precations about something that I don't care about]"

Also way more people than I expected saying that it's "too hard"
Which I can't wrap my head around
You can figure out software development, but you can't scan a QR code??
Is it just because you're refusing to try?

[lone summit]

a) For some people it's security theatre: All my passwords are stored in my password manager. I don't know them. If someone gets my PyPI password, it means they have all my other passwords as well. It's a very unlikely scenario, and at that point it's game over for me, whoever managed that probably also has access to my phone.
b) It's not helping me, only (potentially) people using my packages. At this point all a MFA requirement does is slightly annoy me, without improving security for me. Depending on how likely you think this attack scenario is, it might slightly improve it for anyone using my packages. It smells of responsibility for stuff I'm not getting paid for, which leaves a bit of a sour taste.

In the end, it doesn't even slightly improve security for anyone, even (potential) users, since the MFA secret is just another bit of information that I feed to my password manager, so it really is only annoying, nothing else.

(I get why it's done on an organisational level, what with password reuse and less careful individuals, it's just annoying on a personal level.)

[meager umbra]

Why would getting your PyPI password imply getting your other passwords?

[unreal jewel]

presumably you use other Python packages, and it does benefit you to have those people use MFA

[lone summit]

Sure. It just doesn't benefit me to use MFA myself.

[lone summit]

I only ever use the account from computers I control which my password manager runs on. If some malware is active there, it will have access to my password manager, which contains all my other passwords. The only other possibility I see is a breach of the PyPI database (and subsequent cracking of hashes), which MFA won't protect against.

[unreal jewel]

If you constrain the situation tightly enough, MFA does not offer a lot of advantage yea

I dunno if I'd call it zero, but it's getting into the weeds

[fair wasp]

I agree that it feels like more work that doesn't provide benefits over my existing security, but it's a small enough amount extra given there's not a hardware token requirement. Should that ever happen, I'd just stop publishing to pypi.

should my password to pypi ever be compromised, a lot more would be compromised first, including dev environments where someone could much more subtly compromise a compiler used in wheel building.

[unreal jewel]

You can imagine like... a bad browser addon that can read and exfil content from a web page but not get into the secrets in a password manager, or MITM boxes with malicious certs or what have you. Those are probably pretty unlikely

[fair wasp]

I dont think it goes quite to the level of security theater, but I do understand why some people may view it that way. If people are already handling secrets properly, the level of compromise needed probably includes compromising the devices uploading wheels and packages, which could then just interfere with normal uploads by moifying twine, or a build process, or a compiler, etc.

it provides a little extra assurance that even people arent handling secrets properly, they are a slightly harder target though.

[pliant obsidian]

this is an excellent point. I may think I have "perfect" security, but does every other PyPI publisher whose package I depend on? and their dependencies?

[oak pebble]

I may think I have "perfect" security
It's these people that worry me.

I'm assuming you're using the "royal I" there @pliant obsidian . I don't mean to call you out specifically. I do worry when someone tells me they should be exempt because they have perfect practices when there is no way for me to confirm that.

[latent pier]

As someone whose made extensive use of both password managers (LP Premium and then Bitwarden Premium) and hardware token MFA (Yubikeys and now U2F), personally I'd find it much more likely that a single password gets leaked, compromised or intercepted at rest or in transit any of the numerous other places that could potentially vs. my password manager itself getting hacked (which would be an existential risk). Furthermore, I would never store MFA secrets in my password manager to begin with, as that defeats much of the value of MFA, I use hardware U2F tokens whenever possible (which was helped by PyPI giving them out for free) and otherwise a persistently encrypted Aegis vault on my phone for TOTP, and store backup codes to anything that actually matters in my safe and also in that of a trusted family member for off-site backup.

[obtuse torrent]

are there any plans to allow 2FA authorization when trying to uploading with i.e. twine instead of just using tokens that completely supersede 2FA?

something like what npm has

[tribal sedge]

Seems somewhat timely and relevant.
https://blog.pypi.org/posts/2023-12-04-account-takeover/

[oak pebble]

I THINK that would be up to Twine to handle the respective API calls and prompt a user for a verfication code. I don't know the PyPI login API code at all though. I haven't looked at those endpoints yet.

[obtuse torrent]

It needs support from PyPI's side as well

For the authorization flow

So I figured I should be asking here

[unreal jewel]

I don't think we've spent much time thinkign about doing that

unless I'm misremembering

[tribal sedge]

This topic might be helpful: https://discuss.python.org/t/2fa-usability-on-pypi-and-with-packaging-tools/38167/9?u=miketheman

[unreal jewel]

but I don't think that anyone is actually against doing that

[oak pebble]

Ok, sounds like it's how 2FA is done in PyPI's API. I've worked with other APIs that would response with a particular error code that indicated that you needed to provide a 2FA code with your next call. That was the context I was thinking of where it's up to the client to understand that error and prompt the user for the code.

[pliant obsidian]

I've been using trusted publishing since it was available (it's really good!), but not yet with environments. is the name I add in this box on PyPI the same as the name: pypi in here?

# .github/workflows/ci-cd.yml
jobs:
  pypi-publish:
    name: Upload release to PyPI
    runs-on: ubuntu-latest
    environment:
      name: pypi
      url: https://pypi.org/p/<your-pypi-project-name>
    permissions:
      id-token: write  # IMPORTANT: this permission is mandatory for trusted publishing
    steps:
    # retrieve your distributions here

    - name: Publish package distributions to PyPI
      uses: pypa/gh-action-pypi-publish@release/v1

https://github.com/marketplace/actions/pypi-publish#trusted-publishing

[proud bison]

yes, you enter pypi.

One thing I haven’t figured out yet how to actually leverage environments to make things more secure. What settings would help for a non-enterprise org?

[pliant obsidian]

I think the key thing is, without envs, anyone who has write permissions on the repo and can tag/release can trigger a deploy

but with envs you can tighten that down to just "release managers"

e.g. https://sethmlarson.dev/security-for-package-maintainers#platforms-and-roles

in many cases, you don't have many maintainers so you might want committers == release managers. I guess envs make it more explicit, but also add an extra step in the release process

[bronze cedar]

Hello! I'd like to ask "what's up" with a package, since it looks interesting to me. However, I cannot find that that package-page has an artifact to download, or linking back to repo etc; nor can I find the users' details in pypi.

What should I do? 😕

[pliant obsidian]

which package is it? is there any contact info in source files in the sdist?

[bronze cedar]

It's https://pypi.org/project/pytest-coverage/

[lone summit]

Version 0.0.1 doesn't appear to have artifacts, but 0.0 does: https://inspector.pypi.io/project/pytest-coverage/

Looks like the author removed all versions; 0.0 also is pretty empty and contains a hint to use pytest-cover instead.

(which was merged back into pytest-cov)

[bronze cedar]

Okay - that is what I guessed too

Should the whole package be archived? (Stay for posterity, but don't allow a "basic" pip install pytest-coverage to get it?)

[lone summit]

That's the decision of the author, afaik PyPA usually doesn't intervene with packages that aren't outright malicious. The package currently depends on pytest-cover, which depends on pytest-cov>=2.0, so in the end it's the same as just installing pytest-cov anyways.

[bronze cedar]

Okay, I see. I was hoping for some cleanup 😕

idk; it feels like the https://pypi.org/project/print/, https://pypi.org/project/pprint/ (was it so?) packages. "Sort of misleading".

[unborn nacelle]

Do you have any recommendations for a list of n thousand popular python packages? I'm currently using https://hugovk.github.io/top-pypi-packages/ but many of them seem to a lot more fringe than expected (e.g. one depends on a .tgz sdist from 2011 or tensforflow nightly builds). I was thinking about e.g. a list sorted by github statistics; I'm not looking for anything scientific i'm just using this to scale our testing and find edge cases

[pliant obsidian]

the author of pytest-coverage is also the author of pytest-cov, you're definitely better off using pytest-cov, or even coverage itself directly

[pliant obsidian]

those are the most downloaded packages from PyPI, no filtering.

Libraries.io has several other sorting options:
https://libraries.io/search?order=desc&platforms=PyPI&sort=rank

[unborn nacelle]

Thanks for compiling that page, the list has been really helpful so far!

Thanks for the libraries.io link, i'll try their API

[rare wadi]

Is that possible?

[pliant obsidian]

could "yank" all the releases?

[formal jolt]

Does anyone use devpi? I'm setting it up for some local experimentation and am curious if there are any gotchas

[spice hull]

Hey Mike, question as I'm kind of catching up here. Is there any internal information on the account compromise rate for some of these? I know that while new user account creation was suspended, we still had a few malicious uploads that we reported and were subsequently actioned. Out of idle curiosity, is there any indicator that these were from compromised accounts?

I know there's one (Urlcut) where we were asked to double check user existing uploads after blatantly malicious behavior was detected in a new package. Their preexisting packages ended up not being malicious. Are accounts that contain legitimate package uploads suspended until an investigation is completed regarding potential compromise?

[tribal sedge]

Thanks for asking! The recent report published is the only known case of an account takeover that I'm aware of in the past 6 months.
If I recall correctly, the other malwares during registration disabled had similar username randomness that we've seen with other malware spam, so it's entirely possible they had performed mass account registration without publishing packages.

I never got a response on the urlcut request btw. 😉

When I look at a report, I tend to look at other packages by that user to determine if it looks like a takeover or something else - there's no hard and fast rule, but more of a "let's see..."

[spice hull]

Complied with, whoops. Our wonderful Microsoft service that relayed all our emails decided that it was now a paid service. Joy.
Appreciate the information, was just a passive curiousity that I had while examining a lot of what we've been dealing with lately.

[pseudo bramble]

i used devpi for mirroring at work and found it had some reliability and performance issues. i ended up just replacing it with https://github.com/hauntsaninja/nginx_pypi_cache and haven’t looked back

[formal jolt]

Ah interesting thanks! I need to be able to upload packages... unfortunately. Separately, I've been looking at running an offline cached mirror of PyPI for a test suite. I was just using mitmproxy though.

i.e. record traffic during the test suite then replay later

[spice hull]

https://thehackernews.com/2023/12/hackers-abusing-github-to-evade.html
https://www.reversinglabs.com/blog/malware-leveraging-public-infrastructure-like-github-on-the-rise
Some fun stuff regarding a package we initially detected and reported on the 9th of December.

CC @tribal sedge since you handled the actual report.

[blazing yacht]

I'm trying my hand at some warehouse issues. The example data in example.sql.xz is 5 years old. I'm finding it difficult to locate representative packages and releases in my local development. https://github.com/pypi/warehouse/blob/main/dev/example.sql.xz
Is there some way to get more recent data? Or is there some other generated example data I could use instead?

[tribal sedge]

Thanks for the article, good read!

[tribal sedge]

Hey Mark! Sorry you're hitting some friction there. We have a semi-related open issue for something related to local test data, but that's kind of hidden in the comments.

Can you share what some of the qualities of the packages you'd be looking for are? It's possible they are there, but harder to find.

[blazing yacht]

I've been working on https://github.com/pypi/warehouse/issues/14549 and I'd like to test that the UI works ok for release pages with many and varied wheel files.
I've found the Cypthon package has a few interesting wheel files, but I'm not sure how to find other release pages with lots of wheel files. I can access and query the local dev database, if you've got some hints for what to look for?

[pliant obsidian]

https://pypi.org/project/Pillow/#files has 53, https://pypi.org/project/ujson/#files has 64. Scientific packages will be good to check too, https://pypi.org/project/numpy/#files has 35

we can query the dump at https://github.com/sethmlarson/pypi-data

here's packages with at least 100 wheels:

$ sqlite3 'pypi.db' "SELECT package_name, COUNT(filename) as c FROM wheels GROUP BY package_name HAVING c >= 100 ORDER BY c DESC;"
pyzstd|213
xxtea|211
pybase64|211
simple-manylinux-demo|198
pyms-nist-search|195
...
$ sqlite3 'pypi.db' "SELECT package_name, COUNT(filename) as c FROM wheels GROUP BY package_name HAVING c >= 100 ORDER BY c DESC;"  | wc -l
     279

[rustic venture]

the dump is based on testpypi data from... a long time ago. so that isn't a representative way of finding things in the example.sql.gz

we haven't updated it since it was initially created and the current test.pypi.org DB is too large to create a new one.

i have considered "fixture" style db creation/filling instead of relying on a dump but never made much progress.

[tribal sedge]

Thanks @pliant obsidian ! That would totally work for production pypi

[tribal sedge]

I guess you've got something going, which is great, but just in case you were still looking for local devl stuff, here's a SQL query you can run agains the dev db to get a sense of releases with many wheels:

SELECT
    COUNT(*),
    p.name,
    r.version
FROM
    release_files rf
JOIN
    releases r
ON
    rf.release_id = r.id
JOIN
    projects p
ON
    r.project_id = p.id
WHERE
    rf.packagetype = 'bdist_wheel'
GROUP BY
    p.name,
    r.version,
    rf.release_id
ORDER BY 1 DESC
LIMIT 10;

Outputs:

 count |      name       |   version
-------+-----------------+--------------
   135 | forexconnect    | 1.6.0
    34 | spherogram      | 1.5.4
    30 | psycopg2        | 2.7
    30 | psycopg2        | 2.7.3.1
    30 | psycopg2        | 2.7.1
    30 | psycopg2        | 2.7.2
    30 | psycopg2        | 2.7b2
    29 | psycopg2        | 2.7.6.1.dev0
    29 | psycopg2        | 2.7.5
    29 | psycopg2-binary | 2.7.5
(10 rows)

For completeness, that would be this page: http://localhost/project/forexconnect/1.6.0/#files

[fleet grove]

👀

(why so many banners?)

its a solid portion of my screen

[tribal sedge]

Dismiss any you don't want to see.

[serene fern]

The bottom two seems to refer to the same thing and one of them should maybe be removed?

[pliant obsidian]

I don't see the bottom red one (in incognito), so no duplication

[rare wadi]

The red one pops up when you click the register button

[pliant obsidian]

that's fair enough. anyway now is a good time to remind everyone to enable 2FA for PyPI and GitHub before 2024! 🔒

[tribal sedge]

https://blog.pypi.org/posts/2024-01-01-2fa-enforced/

[oak pebble]

@finite pulsar if you need to, you can ping me here as well. But we can keep it in the PR as well. Which ever works best for you.

[finite pulsar]

@oak pebble thanks! i'll ping you here if it's something synchronous, otherwise i'll keep it on GH 🙂

[finite pulsar]

@oak pebble hey! are there any public docs for ActiveState's OIDC IdP? i tried googling but couldn't find any 🙂 (this is for prepping some docs once your PRs are merged)

[oak pebble]

Hey @finite pulsar! Not yet but they are being written. Is their delay blocking you or are you just asking to see if you can include them in what you're working on?

[finite pulsar]

@oak pebble nope, not a current blocker! we'll need them before switching the admin flag for the public docs, but that won't be for a bit anyways 🙂

[stiff matrix]

you can also reply to messages

[stiff matrix]

like this

these pings look funny

[rare wadi]

but then Discord complains and tells me to make a thread

[oak pebble]

@finite pulsar , I'm fixing up the tests in my last OIDC PR and I'm considering removing dummy_oidc_payload, but keeping dummy_oidc_jet. I'd rename it to be service specifc though. I'm thinking this because I also want to add a dummy_activestate_oidc_jwt fixture but don't want to add an dummy_oidc\_[publisher]_payload (apologies for the backflash. MD is not cooperating) fixture. Any where the payload fixture is used I'd just drop in the json.dumps call.

Does that make sense to you? I'm obviously open ot other ideas, this is just the best option i'm currently able to thinking of.

Also let me know if you need more info. Happy to fill in details

[finite pulsar]

@oak pebble makes sense to me, and seems reasonable 🙂

(i don't have any better ideas)

[warped bison]

Does anyone know where I can download pypa3?

[blazing yacht]

I can see that the warehouse repo is busy with OIDC and Trusted Publisher work, which is great!

Once there's a chance - no rush at all, I can pick things up again whenever - I'd appreciate some answers to my questions in these two PRs so I can progress with them:
Add yank reason to release history page: https://github.com/pypi/warehouse/pull/15115
Make it easier to see the available wheels per package release: https://github.com/pypi/warehouse/pull/15087

[coral widget]

https://github.com/pypi/support/issues/3527

[stiff matrix]

hi! is is currently possible to render admonitions in project READMEs on PyPI?
in my README, i have the following GitHub-rendered admonition:

> [!Note]
> If you use Windows, it is highly recommended to complete the installation in the way presented below through [WSL2](https://learn.microsoft.com/en-us/windows/wsl/install).

in GitHub, it renders as:

for mkdocs material, i use another syntax:

!!! note
    If you use Windows, it is highly recommended to complete the installation in the way presented below through [WSL2](https://learn.microsoft.com/en-us/windows/wsl/install).

the first syntax in PyPI renders as:

can I make it look like one of the aforementioned admonitions?
ideally without using yet another separate syntax?

i didn't test the mkdocs material syntax, as i am quite sure it's mkdocs material-exclusive

[pliant obsidian]

you can use https://pypi.org/project/readme-renderer/ to check how it would look, I think PyPI uses this tool

[stiff matrix]

thanks. i had checked it, but i didn't see any support for admonitions.

[pliant obsidian]

yeah, I think it doesn't support admonitions. using some raw html might do it

[stiff matrix]

do you think that it might be a good idea to add support for GitHub admonitions @ PyPI readme renderer?

i would already have to use 3 separate syntaxes for admonitions

(i am already working on a project for automatic rewrites between them exposed as pre-commit hooks)

[pliant obsidian]

I imagine it could be useful, many projects are on GitHub and use README.md for PyPI pages

check if there's an issue open on the repo, and open one there?

looks like it supports reST admonitions: https://github.com/pypa/readme_renderer/pull/242

[stiff matrix]

yes, I did see this. unfortunately, I am unable to leverage that in any way as a Markdown user.

[stiff matrix]

also, the styling for these seems quite poor atm.

[pliant obsidian]

yeah, there's still https://github.com/pypi/warehouse/issues/8300 open to improve styling

[stiff matrix]

oh, thanks!

this looks like a good place to start off from.

it could help in Markdown admonitions as well.

i'll try to implement that if there's no PR yet

[oak pebble]

@tribal sedge will you have time this week to take a look at https://github.com/pypi/warehouse/pull/15168 ?

@finite pulsar I'm getting docs tickets going for our docs person and wanted to make sure I'm providing all the right notes. What docs will you link to from PyPI? I'm assuming it's configuration docs that explains how to configure PyPI to work with the respective OIDC Provider? A link to reference material would be a sufficient response!

[finite pulsar]

@oak pebble hey! in terms of docs, i think the ideal state is to mirror something like the ones for GH based publishers: https://docs.pypi.org/trusted-publishers/adding-a-publisher/

(in other words: ideally there'd be a new tab on each of these pages for ActiveState with ActiveState-specific info + contextually relevant links)

we have a draft PR open with a skeleton for that: https://github.com/pypi/warehouse/pull/15192 but it'd be great if you or someone else from ActiveState could file a PR against that branch so add the specifics for your provider 🙂

(similar to what we're doing for GitLab, which is still a WIP: https://github.com/pypi/warehouse/pull/15283)

[oak pebble]

Perfect! Absolutely we can make a PR for that. Thanks for the response @finite pulsar.

[tribal sedge]

I think so

[oak pebble]

Absolutely no pressure. I'm asking so I have a sense of how to prioritize other items at work and so I have more details for my coworkers when asked about the PRs I have remaining. Thanks for the response, Mike!

[tribal sedge]

LOL timing is everything. I'm reviewing now, and turns out the graphql api endpoint is down for maintenance.
@oak pebble does bring up a good review question - should PyPI disallow users from setting up ActiveState Trusted Publishing if the GQL API is down? Is there an SLA for it, and I just hit the wonky window?

[oak pebble]

You have amazing timing, Mike! One of our teams is currently doing a very large data transformation that just happens to be running right now. It's a once in an (app) life time migration.

If that GQL API is down, then yeah, a user shouldn't be able to configure an ActiveState Trusted Publisher.

It should, other than right now (SO sorry that it's right when you happen to put time aside 🤦‍♀️), be up 99 % of the time.

[tribal sedge]

no worries, I can review plenty of more on that PR

[oak pebble]

@tribal sedge can you recommend a tool for that re-formatting of publishing.html and other template files?

I've been wanting something for those files for a while but was too lazy to look for one 🤦‍♂️

[tribal sedge]

hmmm - I don't think we have one in our linter stack right now, otherwise you'd know 😉
I've used djlint on other projects and it was pretty good after I tweaked some settings. https://www.djlint.com/docs/languages/jinja/
I also have a dusty branch somewhere that also adds basic entries to our .editorconfig but need to dig it up and ship it.

Our linter stack does use curlylint, but that's seemingly unmaintained for a while. https://www.curlylint.org/

[oak pebble]

Excellent, that's helpful. Thanks Mike!

[oak pebble]

@finite pulsar , Mike merged https://github.com/pypi/warehouse/pull/15168 this morning (squeals with glee). I now just have this PR remaining: https://github.com/pypi/warehouse/pull/15204

We (ActiveState) has had to push some tickets back 1 (maybe more) sprints for some of our internal work so there isn't a rush to get this reviewed and merged. It's also really small and no refactoring involved so non-blocking for anyone else hoping to introduce a new publisher.

If you all have the cycles, I'll be happy to work to get this reviewed and merged as well but if no one is available I'll be fine to stand by as well.

Let me know what the best option is for the pypi peoples.

[finite pulsar]

@oak pebble awesome! I will absolutely have the cycles for review on that last PR this coming week — ideally, we’d have everything merged on the PyPI side so that we can announce ActiveState support at the same time as Google Cloud and GitLab (and that latter one is still WIP on my team’s side, so there’s similarly no big rush here) 🙂

[oak pebble]

Sounds good! Thanks so much!

[oak pebble]

Is there a timeline associated with this announcement that I should keep our internal work on track for?

[finite pulsar]

Nothing firm — I would like to have the GitLab provider merged and everything documented and locked in by early April, but the actual timeline here will depend heavily on review cycles from Mike and the other maintainers

[oak pebble]

Understood and makes sense. I just wanted to make sure I didn't have to light any fires here at ActiveState to wrap up remaining items.

Thanks so much for the quick response. It's really appreciated.

[finite pulsar]

Nope! I think as long as we’re ready to flip the admin switches on all three new provides by then, we’ll be in great shape

And no problem!

[finite pulsar]

@oak pebble congrats on the merges! i pinged you on https://github.com/pypi/warehouse/pull/15192 for the remaining doc bits 🙂

[oak pebble]

Thanks a lot! I have a ticket for the docs stuff in my queue. So I should push my changes to 15192 when I have the docs stuff ready?

[finite pulsar]

pretty much! my rec would be to branch off of that PR, similar to what we're doing here: https://github.com/pypi/warehouse/pull/15283

[oak pebble]

Can do!

[oak pebble]

@anyone, is it an option to enable ActiveState OIDC Configuration in the https://test.pypi.org/ environment?

[tribal sedge]

Done

[oak pebble]

Thank you very very much, Mike.

[spice hull]

https://github.com/pypi/warehouse/pull/15452
Blew up our prod.

[pliant obsidian]

be happy they didn't merge on Friday 🙂

LGTM, but will wait to merge this until Monday.

[spice hull]

The infinite wisdom of a man who has witnessed that mistake first hand 😅
We're back up and running now.

[unborn nacelle]

How?

[spice hull]

The introduction of a new key in the API response was unfortunately not handled in a dependency. 😔
I've not been in this sphere long but I'm endlessly entertained at how things can kind of cascade into major issues; we were pinned to a version of that dep (which essentially just wrapped PyPI's JSON response) because of an entirely separate reason, so we had to refactor our existing codebase to fit the aiohttp -> httpx transition on the dep, so it wasn't quite as simple as just "fix the unanticipated key".

posted here more for schadenfreude and because... that's an entertaining way to have an outage. 😆

[unborn nacelle]

There's an interesting semver implication about this because you effectively must accept additional fields when deserializing to be forward compatible, but it also means your missing when you're not handling extra fields

[spice hull]

Realistically the solution is to probably move towards in-house handling of the PyPI API response. We only really care about 2-3 fields.

So the introduction of most new keys is probably something we're not going to be interested in.

[tribal sedge]

@spice hull sorry about that, glad it didn't ruin a weekend 😉

I'm hoping that some work I'm doing in the near term may eventually prove helpful as describing API interactions via an OpenAPI document that folks can consume, but we're a bit of a ways off from that reality.

[spice hull]

I appreciate the excitement it brought my Monday. And it uncovered a strange bit of technical debt, so the silver lining was a pretty large value add. 😄

[oak pebble]

Possible silly question: Is it possible to run the warehouse docs locally?

[pliant obsidian]

Yes: https://warehouse.pypa.io/development/getting-started.html#building-documentation

[spice hull]

New Vipyr stats dropped. We've been around for about a year now. Almost to 1 million packages scanned!

[oak pebble]

Thank you very much!

[latent pier]

Looks interesting, you have a link with more info?

[spice hull]

https://vipyrsec.com
https://github.com/vipyrsec
Just static code scanning for antimalware stuff 🙂

[latent pier]

Thanks!

[copper plinth]

vipyrsec mometn

[oak pebble]

Would anyone be willing/able to merge master into this PRs branch https://github.com/pypi/warehouse/pull/15192 ?

We're adding our docs and one of the changes I was going to make was to update the ActiveState form with additional help text but none of the ActiveState OIDC code is present in the branch.

Might actually make more sense if I asked IN the PR 😶‍🌫️ ...I'll do that

Thanks Dustin!

[pliant obsidian]

once upon a time, PyPI releases were hosted somewhere other than PyPI itself. now, all files must be upload to PyPI, and you can't install from those old "self-hosted" releases. there was also a transition period with both kinds of releases

anyone know when installing from "self-hosted" releases was disallowed?

[unreal jewel]

PEP 470

the earlier transition was... 438?

[pliant obsidian]

well remembered! https://peps.python.org/pep-0438/

and 470 was implemented in September 2015: https://github.com/pypi/warehouse/issues/431

thanks!

[rare wadi]

Where’s the specification for what characters are allowed in a package name?

[meager umbra]

https://packaging.python.org/en/latest/specifications/name-normalization/

[rare wadi]

[wicked wind]

Question about package name conflicts. Through $job, we have a package which we'd like to upload at foo.bar however someone unconnected to us already has the name foobar so we're prohibited by PyPI from claiming the foo.bar. I'm attempting to do a friendly transfer of foobar to us, under the assumption that once we control foobar I could delete the package and create the new package named under foo.bar. Is that a correct assumption?

[unreal jewel]

I think so

I'd have to test it to be sure

[wicked wind]

Thanks @unreal jewel - any way I can help verify one way or the other?

[pliant obsidian]

you could try it on TestPyPI

[unreal jewel]

Yea just trying it on test PyPI is probably the easiest thing

[vague spruce]

Hello! Quick question:

Which classifier would I use for a variant of a supported license? As an example, the MIT-testregex license, which is based on the MIT license. Which of the following (or something else entirely) would I use?

(a)

License :: OSI Approved :: MIT License

(b)

License :: OSI Approved :: MIT License
(AND)
License :: Other/Proprietary License

(c)

License :: Other/Proprietary License

Note that this is a general question, so saying "just use a different license" will be of no use to me (I only say this because it has happened before).

[proud bison]

Would be fixed by https://peps.python.org/pep-0639/

Hatch already supports the draft PEP, and there has been some movement to get it accepted recently.

[wicked wind]

Yeah, good idea, I'll give that a try.

[glossy turret]

Is the unit of the size for json api bytes?

[violet fable]

https://github.com/pypi/warehouse/blob/223de96caeded9e5348fe339b7b3cff7a47b6587/warehouse/forklift/legacy.py#L1280 I believe so

around that code sets the size attribute of File objects which are then read here: https://github.com/pypi/warehouse/blob/223de96caeded9e5348fe339b7b3cff7a47b6587/warehouse/legacy/api/json.py#L97

[glossy turret]

Thanks!

[latent pier]

We hope to have PEP 639 rewritten (again) and accepted in the next few months and things are now progressing again toward that goal, but in the meantime, the best I would recommend is the de-facto convention of including the license's SPDX identifier (MIT-testregex in the above case) in the project.license.text key of pyproject.toml, and just not using the classifers at all. Then, once PEP 639 is accepted and implemented (or already, if your backend supports it like Hatch does), you can just use it as the value for project.license, and everything will Just Work™.

[vague spruce]

Ahhh, gotcha. From my understanding, PEP 639 sort of allows for both the SPDX ID and license file to be used, where the SPDX ID will be a replacement for the License classifiers. Is that correct?

[latent pier]

(Of course, if at all possible, in general people should, of course, avoid just non-standard bespoke licenses unless they have absolutely no other choice; some have proposed eventually requiring an OSI/FSF approved free/open license on package repositories, and some already do)

[latent pier]

More specifically, it replaces both license classifiers and the free-text license.text and full-text license.file pyproject keys and their corresponding License field with a single license key supporting an SPDX expression (which can include multiple IDs, combinations, modifiers, etc.). Additionally, it standardizes and extends the existing unstandardized license_files mechanism for adding license files to sdists and wheels to add it to pyproect metadata and list it in core metadata, improve its usability and ability to handle multiple nested licnese files, and avoid naming conflicts.

[vague spruce]

Ahh, gotcha. Very cool! Am looking forward to this O_O

[proud bison]

It is very cool. The defaults make it so that just having a halfway reasonably named license file in the root of your project gets that included without further configuration.

[latent pier]

To be fair, that's already been the case for many years now (thanks in part to a PR I made to wheel with some globs to handle pretty much all common cases, based on what I'd helped with for Conda-Build and the already existing license_files support, which then got added to Setuptools and others as a de-facto standard, and then (re-)adopted in turn by me in PEP 639, and finally implemented as part of that (draft) standard in Hatch, among others). PEP 639 just standardizes and extends that established mechanism to be usable and more ergonomic for the less trivial cases.

[oak pebble]

Has anyone else been noticing the tests taking quite a while to run in PR reploys for the warehouse?

[tribal sedge]

Yes, di has been digging through trying to find what's stalling it. Something to mitigate them taking hours: https://github.com/pypi/warehouse/pull/15582

[oak pebble]

Thank ya kindly @tribal sedge and Dustin!

[harsh raptor]

hey y'all, just recently we started getting upload failures to test pypi like this:

ERROR    HTTPError: 403 Forbidden from https://test.pypi.org/legacy/            
         Invalid API Token: caveat must be an array  

this feels like a potential issue with the recent update to the OIDC auth that was changed?

[wicked wind]

Confirmed it does seem to work on testpypi

[oak pebble]

Hey all. We're doing integration testing with ActiveStates OIDC against test.pypi.org and we're having an issue that I can't repro with a locally running instance of the warehouse (Note: yes updated OIDC_BACKEND=warehouse.oidc.services.OIDCPublisherService and confirmed it's being used)
High level details:

• We can get a token when a publisher is configured on an existing project
• We cannot get a token when the user has a pending publisher configured
• The error we're getting when the config is a pending publisher "errors":[{"code":"invalid-publisher","description":"valid token, but no corresponding publisher (All lookup strategies exhausted)"}]

Before I go into detail of what debugging I've done I wanted to ask; has anyone else hit this before? Does this sound familiar to anyone?

Sorry for the ping but di (ping removed) or woodruff (ping remove) I believe you two are the most familiar and worked the most with the OIDC code.

Also, Is this the appropriate place to be asking this type of question?

[oak pebble]

Ok, appologies for the noise. We appear to be good to go.

We were able to get it to work for another user but same publisher config. So either something was wrong with the origin user's publisher config (multiple people checked it multiple times) or something else.

I'll obviously report an issue if we come across something tangible after further testing but hoping for PEBCAK 🙂

[drifting pollen]

what's the current status of WASI wheels on PyPI?

[glossy turret]

My understanding is that there is no stable ABI in wasi libc so it is not possible to have non-pure python wheels on pypi

[drifting pollen]

that's a bummer – are there any plans to rectify that?

[glossy turret]

Hm actually I suppose wasi 0.2 defines a stable interface

So a PEP might be able to be drafted?

[proud bison]

@quaint pivot has posted a bunch of things here, but few of them resulted in a discussion. If you’re interested in that, maybe talk to him? https://discuss.python.org/c/webassembly/28

[glossy turret]

Ah right there's the larger issue of how to dynamically import components

[wet flax]

https://github.com/astral-sh/uv/pull/2493 -- would linehaul need updates to handle this additional metadata coming from uv?

[tribal sedge]

Maybe a user agent update? https://github.com/pypi/linehaul-cloud-function/blob/main/linehaul%2Fua%2Fparser.py

[formal jolt]

Feel free to ping me if you need anything

[unborn nacelle]

We'll have to do the pip/23.0.0 (uv/0.2.1 like pip) otherwise :P

[unreal jewel]

I always meant to turn that into a standard

someone should write a PEP 😄

[unborn nacelle]

From my pov having a json schema with some comments on packaging.python.org would be the most helpful

The current implementation is a port from the https://github.com/pypi/linehaul-cloud-function/blob/1.0.1/linehaul/ua/datastructures.py dataclasses

We've skipped the setuptools version (we install the latest version in build envs only) and rustc version (rustc --version is slow), otherwise all information should be there

[formal jolt]

so... I've noticed that pip sends macOS patch version numbers 1 but wheels tags never use patch versions just the major / minor version tuple. Is there a reason for this? I don't want to introduce tracking the patch version in uv for this singular purpose. Could / should linehaul not include this extra version number? It's possible this is entirely unintentional because platform.mac_ver() will omit patch versions sometimes.

Discussion prompted in https://github.com/astral-sh/uv/pull/2509

[wet flax]

I'm 90% sure it's because of it being easier to send all the version segments over.

[formal jolt]

Does it seem problematic for us to omit it?

It's not infeasible it's just not easier 😄

(I see your comment on the pull request now too)

[unreal jewel]

I'm 100% sure 🙂

I did not closely evaluate the data contents of every field

I just grabbed all of the ones that looked reasonably useful

[tribal sedge]

Exciting news https://discuss.python.org/t/pypi-malware-observation-report-outcomes-private-preview/49060

[quaint pivot]

I'm working on it from within the Bytecode Alliance, but I had to get Python 3.13 to support WASI officially first and that just happened this month 😉 My WASI work is on hold while I'm on parental leave, but getting the wheel problem solved is on my todo list (but it will be tricky)

[drifting pollen]

cool, thanks for letting me know!

[wintry cloak]

Hi all, first of all thanks to all the people that make PyPI happen

I've noticed that there seems to be a large build up of support issues within the pypi/support repository (^1), and I'm wondering if the people that usually look after it need help. I'm happy to volunteer some of my time, but of course understand if you'd rather not hand over admin access to someone you don't know and just rocked up to the Discord server (or if you have other policies in place). I'm nhairs on github if that helps.

If this is something of interest feel free to reach out to me directly.

^1 - disclaimer: I noticed this because of my own request. This is not an attempt to sidestep the process though.

[pliant obsidian]

thanks for the offer! like you mention, it requires a high level of trust to give admin access, which is the main blocker. there's been lots of discussions around other solutions, but the good news is that right now the PSF is looking to hire someone to work on this!

please consider applying, or share the job ad! https://fosstodon.org/@ThePSF/112129602456025280

[drifting pollen]

this would be nice to have, as I have a pending organization request since last October

[valid flame]

is there a way to check how many packages are still offering .eggs only?

[tribal sedge]

Check which packages vegans won't eat 🧌

[glossy turret]

are project names reclaimable after deletion?

i.e. if I delete a project, then publish a new version to that name, will that be denied or will it work?

[ancient compass]

As long as it's got a different filename (eg different version)

[glossy turret]

ok, that's what I thought

[ancient compass]

There's a window where someone else can take it though

[glossy turret]

Yup

[old tinsel]

Hi everyone, I hope this is the right place to ask about this... I'm about to release a package and I've found that the most logical name for it is already taken by a abandoned project that seems to me like an invalid, sort of junk project (using the terms "abandoned" and "invalid" as defined in https://peps.python.org/pep-0541/). I've made 3 attempts to contact the author via email trying to get more info, and the homepage of the package appears to be inaccessible.

Is it possible for me to claim this name? If so, how might I get that process started?

[meager umbra]

The procedure is in the document you linked, under "How to request a name transfer".

[old tinsel]

🤦

[rare wadi]

The PEP has a direct link to the ticket template

[old tinsel]

i'm sorry, i read that document so many times but always stopped halfway down the page.

thanks very much

[proud bison]

Hi, it’s almost a year since I submitted a request for the “scverse” pypi org. What’s going on with orgs? Are they still a thing you want to do?

[glossy turret]

I expect they are backlogged similarly to support tickets if I had to guess

[proud bison]

But … nearly a year? I don’t want to be rude but did it make sense to introduce the feature if not enough people exist to support it?

I’m aware of the irony of saying something that could be interpreted as pressure on overworked maintainers after the xz backdoor thing

but my intention is more a “Is it good to introduce a feature that demands more manual work when there’s already too much manual work to handle”?

[serene fern]

Support tickets is one thing, I am quite pissed off with the org thing if I’m being honest. Zero transparency, big words and no delivery. It is bad period.

Multiple high-ups would have already stepped down if this is a public company trying to roll out a supposedly big feature.

[pliant obsidian]

The PSF is hiring someone to work on this stuff full time https://discuss.python.org/t/announcing-a-pypi-support-specialist/49062

[wet flax]

My understanding is that they ran out of funding for the organisation project before it could be delivered, and there hasn't been any availability for someone to drive this forward from the PSF's end. The support specialist role is intended to help with that.

[proud bison]

That's wonderful news, I'm happy there is a solution.

[narrow field]

good morning - i have a problem with pypi ownership of a few packages, a former coworker created, but the handoff never happened

so now tech debt has catched up enough that we need to push some fixes but the package owner is stll the former coworker, however it seems like hes unresponsive

whats the recommended process in such cases

ah - i found https://peps.python.org/pep-0541/#continued-maintenance-of-an-abandoned-project

[pliant obsidian]

yep, that's it. expect a long wait for PEP 541 requests, but do get in the queue so the new Support Specialist can eventually get to it. if you can contact the coworker to handover in any way, that will be much quicker

[latent pier]

Speaking of the new support specialist role, I'm getting the following error from the reCaptcha widget on the PSF PyPI job page, tested on a clean copy of FF on Windows.

[latent pier]

I'm messaged Ee directly about that, as I figured they'd be the best PoC to take a look

[pliant obsidian]

I see the same thing

[tribal sedge]

It might be the domain name, since this link doesn't exhibit the issue, as far as I can tell.
https://pythonsoftwarefoundation.applytojob.com/apply/nyYHuOha9h/PyPI-Support-Specialist

[latent pier]

Thanks! Yup, that's what Ee told me. The working URL is what is posted various places, but the non-working one is what is shown in search engines and elsewhere. I can confirm this is, as I speculated previously, because the page has a link rel="canonical" to the non-working URL, so the latter is shown in place of the former:

    <link rel="canonical" href="http://jobs.pyfound.org/apply/nyYHuOha9h/PyPI-Support-Specialist" />

[grim sleet]

Hi, I can see I'm not the only one on the channel that is waiting for their support ticket. Is there any way to get someone to look at an urgent (for me at least) ticket?
I'm getting a bit desperate, as I'm getting a lot of grief from users who say my package is failing. But I can't deploy a fix until my data limit has increased.

The project in question is PyBoy. The support ticket is here: https://github.com/pypi/support/issues/3757

[meager umbra]

I'm not from PyPI, but if a particular release is broken, can't you yank it for the time being?

[grim sleet]

It's complicated. The newest release works, but because I can't upload the remaining 2 wheels, when users on Windows do pip install pyboy, it defaults to compiling with Cython, but few people have the setup for it. And at the same time, Cython's newest version has a confirmed bug that affects PyBoy... Falling back to the previous version has similar issues with Pillow but on all platforms.
So, yeah.. I'm stuck between the lesser of two evil.

[gentle frost]

I don't know if this new unverified header is useful

it would be nice if the verified part could have a box and then everything outside that would be implicitly unverified rather than make folks suspicious of the links as currently structured

[wet flax]

I think it works to be honest. There's a bunch of packages that use a link to, like, pytorch as one of the non-GitHub links and get 10k stars and such as indicators. This makes it clearer that those bits aren't reliable.

[violet fable]

UX wise, the project links and github statistics look a bit crowded and are harder to click than before.

[glossy turret]

Hmm, I'm seeing some 500s when trying to upload packages, anyone else hitting issues? Definitely could be PEBKAC

[rare wadi]

Can we get some more details?

[glossy turret]

Yeah sure, I am trying to upload a package using the latest twine on Python 3.10. I get 500: Internal Server Error, with no more details. I am re-running with verbose, I'm unsure if that will give more details.

[rare wadi]

No error message at all?

[glossy turret]

Yup, none that I can see

I just get

ERROR: HTTPError: 500 Internal Server Error from https://upload.pypi.org/legacy/
Internal Server Error

Ah

it looks like there is some maintenance happening

[pliant obsidian]

About GitHub's 2FA rollout, they give a shoutout to PyPI

https://github.blog/2024-04-24-securing-millions-of-developers-through-2fa/

Maybe something for PyPI to consider:

We also introduced a 2FA verification checkup that occurs 28 days after 2FA setup, to ensure users have an opportunity to verify their configuration. This checkup was a fail-safe that helped 25% of users successfully reconfigure their accounts if they made a mistake or lost a factor, thereby avoiding account lockout for the user and significantly reducing account recovery support volume for GitHub.

[merry valve]

interesting, I wonder if that means 2FA is always resettable without support intervention within 28 days?

[pliant obsidian]

let's see what this brings:

If you’d like to learn more about how we rolled out 2FA for millions of developers, look forward to a follow-up post that takes a deeper dive into our work in the coming weeks.

[unborn nacelle]

I just wanted to say kudos to the authors of https://inspector.pypi.io, it saves me so much time

[wet flax]

interesting, I wonder if that means 2FA is always resettable without support intervention within 28 days?
I don't think so.

The thing they're talking about is more a reminder of "hey, here's your 2FA setup. Please confirm it's still functional" and I understood the 25% number to mean the %age of users who went from that page to actually reconfigure their 2FA settings for whatever reason.

It works well for GitHub because it's a website you're (probably) logged into ~all the time. This is not the case for PyPI where users log in occasionally, and only to perform priviledged tasks.

[acoustic panther]

@merry valve @tribal sedge Sorry for the ping. Was just wondering about your thoughts on replacing Prism with Monaco for the inspector. I made a POC pull request here: https://github.com/pypi/inspector/pull/165

[formal jolt]

Might make sense to include some context there on why you would want to change it — kind of left guessing otherwise.

[acoustic panther]

added this to the description:

My reasoning for replacing prism with Monaco, is that Monaco provides a more interactive source code analysis experience.

Below are features that prism does not include, and which may be useful for analyzing potentially malicious code:
    - the ability to fold portions of source code.
    - semantic highlighting
    - regex search
    - the ability for the user to set a theme for comfortable viewing.

There are more features exposed by the built-in command palette.

Struggling with discord markdown 😅

[blazing yacht]

Hello, I wanted to say that I should be able to come back to the 3 warehouse PRs I have open, in July, and hopefully get them over the line.
Two are waiting on review, and one is in progress but could do with attention from a maintainer to confirm the approach is appropriate.

• Allow translating from javascript files (needs review) https://github.com/pypi/warehouse/pull/15612
• Make it easier to see the available wheels per package release (needs review, depends on the js translation PR 15612) https://github.com/pypi/warehouse/pull/15087
• Implement Alternate Repository Location for PEP 708 (needs maintainer check to confirm on right track) https://github.com/pypi/warehouse/pull/15716

[valid flame]

just wondering, what was the reasoning behind choosing pyramid as a framework for PyPI? was it the thing to go with when warehouse was being made or does pyramid offer something special over what other frameworks offer?

[unreal jewel]

there's uh

some history

Legacy PyPI predated all modern web frameworks, used it's own custom thing and had basically zero tests.

When I started working on Warehouse we weren't starting from scratch, we wanted to share the existing Database and have both legacy and Warehouse run off the same DB in parallel. I originally started writing it in Django (in fact, the users table looks a lot like a Django user's table still), but quickly ran into issues where the not very well designed Database schema couldn't be represented in Django's ORM (I forget the issue now... I know we had tables with compound primary keys, and some without primary keys at all, so probably that?).

So we had to stop using the Django ORM, which throws away like 90% of the value of using Django.

I wanted something very testable, and most of the other web frameworks I knew (particularly Flask) relied heavily on global state via thread locals, which makes things a lot less testable IMO, so I started writing my own web framework built ontop of werkzeug.

At some point someone (I forget who now) convinced me to that was going to have the same problem we had with legacy PyPI for attracting contributors, so I started searching for another framework that had the properties I wanted.

I did end up porting it to Flask in a branch, but I threw that away because the thread local design infects everything in Flask and was the opposite of what I wanted.

Pyramid ended up having a design that minimized the use of thread locals (they exist, but only as an escape hatch), it was flexible enough to support the existing url structure that we already had, generally worked well with SQLAlchemy, and had a community that was motivated to make changes and add features that we needed or wanted.

Overall, I've been really happy with Pyramid for Warehouse

[valid flame]

thanks

[agile sinew]

Due to storage limitations on the Python package index, the JAX team periodically removes older jaxlib wheels from the releases on http://pypi.org/project/jax. These can still be installed directly via the URLs here.

That’s not actually helpful, is it? Since deleted downloads don’t actually save PyPI storage space?

[unreal jewel]

we don't delete files, but our quotas don't know that

[tribal sedge]

We don't delete files "yet" 😉

[rare wadi]

[proud bison]

I used pyramid back when I did Python web stuff back in the day, and felt like its design was the only thing that came close to being non-opinionated. Really really flexible.

[gusty sapphire]

Hello,
I have a cuda/python package with nanobind as the binder

How can I use scikit build core to compile, upload to pypi so users just pip install and get the wheel according to their OS?.

[glossy turret]

@gusty sapphire probably a question best suited for #scikit-build. They have a getting started tutorial here: https://scikit-build-core.readthedocs.io/en/latest/getting_started.html
Once you have the basic package setup you should take a look at the scientific Python development guide here: https://learn.scientific-python.org/development/guides/packaging-compiled/

[gusty sapphire]

Hello,

I was able to write a TOML that builds my package then I make a wheel and distribute a many Linux wheel

https://github.com/DifferentiableUniverseInitiative/jaxDecomp/blob/push-to-pypi/pyproject.toml

My question is, how can I also provide just source files and then the user can build the package himself.

Can I have two separate installable pypi packages?
One pre-built and the other not

The user has to pip install from a git repo in order to build himself right?

[formal jolt]

👋 Hi! We just ran into project size limits on PyPI for uv — I'm going to open the formal issue and such but this unfortunately means that our latest release was partially uploaded (just two of the wheels). Is my best option to yank that release? :/

[wet flax]

Yea, I'd say so.

[formal jolt]

Oof

[wet flax]

I don't recall if we have un-yanking.

[formal jolt]

There is un-yank

Which... could be interesting later.

[empty ridge]

It is actually even recommended to publish wheels and sdist: https://packaging.python.org/en/latest/glossary/#term-Source-Distribution-or-sdist. Moreover, it's best to build wheels from sdist, not a Git checkout.

[formal jolt]

Why would you not publish both wheels and sdists?

[fair wasp]

Big reason that has come up before is that your project isn't trivial to build, so you don't want end-users having pip fallback to an sdist if there's no matching wheel, and would prefer them to get the sourcedist elsewhere (such as from git) if they are a redistributor.

[wicked wind]

IIRC there was some musings about marking sdists as "non-buildable" for pip on pypackaging-native.github.io

[spare temple]

is there a small app/pre-commit hook somewhere to validate the 'classifiers' (and maybe other bits) of my package locally? So I find about my typo earlier than when I push to pypi and get a 400 response back 🙃

[glossy turret]

the build backend should probably verify those, but that isn't specified anywhere I don't believe

[pliant obsidian]

python -m build gives me:

ValueError: Unknown classifier in field `project.classifiers`: bleep bloop

from ERROR Backend subprocess exited when trying to invoke build_sdist, using Hatchling

[spare temple]

ah, my build backend is setuptools.build_meta which I guess doesn't do that check, 👍 thanks both

[pliant obsidian]

there's a longstanding plan to add this sort of thing to twine check: https://github.com/pypa/twine/issues/430

[cobalt grail]

does anyone know an easy way to see download stats for specific wheel files?

(ideally without a big query account, I tried getting an api key, but gave up... :/ or does the browser UI variant still exist?)

[valid flame]

there is pypi.stats, but IIRC that only shows total downloads, not per file

but maybe you can start from their codebase?

https://github.com/crflynn/pypistats.org

[cobalt grail]

thanks. that's big query again. I don't care that much about the stats to start coding and give google my credit card 🙂

I've filed an issue at least: https://github.com/crflynn/pypistats.org/issues/69

[naive rune]

https://github.com/pypi-data/pypi-json-data might be helpful for avoiding bigquery

[cobalt grail]

the json api doesn't contain download stats afaik (https://pypi.org/pypi/pycairo/json)

[pliant obsidian]

I replied: Yes, this sort of thing is in BigQuery, you can get it using the https://github.com/ofek/pypinfo CLI.

See https://github.com/python-pillow/Pillow/issues/6941 for some examples.

[pliant obsidian]

You do need to get an API key from Google, but don't need to give your credit card, and you get free quota each month

[cobalt grail]

It didn't let me continue otherwise, but let me try again.

[pliant obsidian]

There's some instructions and screenshots on the pypinfo README, although it's possible things have changed by now...

[cobalt grail]

it worked! that was something

[deft igloo]

Hello! I am trying to request an organization. My request was submitted June 12th. Who should I contact to check in on it or make any changes to my request? Thank you!

[obtuse torrent]

Requests as far as late April last year aren't processed so there isn't really anything to check on, it's just in the backlog

[glossy turret]

Is there a site that tracks downloads based on installer/resolver version?

[tribal sedge]

The data in BigQuery may be interesting to you. https://packaging.python.org/en/latest/guides/analyzing-pypi-package-downloads/#data-schema See the details record on the schema - there's a bunch of sub-columns that can be found there, but not all of them will always be present.

[glossy turret]

Oh thanks!

[wintry cloak]

I believe this would be covered by the PyPI support specialist that was being hired a few months back.
#pypi message

From memory applications closed in June? I'm not sure of the current status of it though.

[raw anvil]

Hello all, I was wondering why I cannot register the name due on PyPI. It seems unused, but PyPI prevents me from registering it (403 IIRC). Was there a previous incident with this package name 🤔 ? It doesn't sound like anything I've seen in the standard library or PyPA projects, or anything official/popular.

(I settled on another name, yore for those who are curious, so I'm only interested in learning why due cannot be registered, no need to unlock it or whatever 😄 )

[rare wadi]

Huh, I would think that name would be fine.
Can you try again and confirm the status code and share the entire response body?

[raw anvil]

sure!

Uploading distributions to https://upload.pypi.org/legacy/
Uploading due-0.0.0-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.8/3.8 kB • 00:00 • ?
WARNING  Error during upload. Retry with the --verbose option for more details.
ERROR    HTTPError: 403 Forbidden from https://upload.pypi.org/legacy/
         The user 'pawamoy' isn't allowed to upload to project 'due'. See https://pypi.org/help/#project-name for more information.

lemme add --verbose

[rare wadi]

Does --verbose gi-- yeah

[raw anvil]

INFO     Using configuration from /home/pawamoy/.pypirc
Uploading distributions to https://upload.pypi.org/legacy/
INFO     dist/due-0.0.0-py3-none-any.whl (1.0 KB)
INFO     dist/due-0.0.0.tar.gz (0.9 KB)
INFO     Querying keyring for password
INFO     password set from keyring
INFO     username: __token__
INFO     password: <hidden>
Uploading due-0.0.0-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.8/3.8 kB • 00:00 • ?
INFO     Response from https://upload.pypi.org/legacy/:
         403 The user 'pawamoy' isn't allowed to upload to project 'due'. See https://pypi.org/help/#project-name for more information.
INFO     <html>
          <head>
           <title>403 The user 'pawamoy' isn't allowed to upload to project 'due'. See https://pypi.org/help/#project-name for more information.</title>
          </head>
          <body>
           <h1>403 The user 'pawamoy' isn't allowed to upload to project 'due'. See https://pypi.org/help/#project-name for more information.</h1>
           Access was denied to this resource.<br/><br/>
         The user &#x27;pawamoy&#x27; isn&#x27;t allowed to upload to project &#x27;due&#x27;. See https://pypi.org/help/#project-name for more information.


          </body>
         </html>
ERROR    HTTPError: 403 Forbidden from https://upload.pypi.org/legacy/
         The user 'pawamoy' isn't allowed to upload to project 'due'. See https://pypi.org/help/#project-name for more information.

[unreal jewel]

that looks like the error you get when the name is already registered

[raw anvil]

indeed, but https://pypi.org/project/due/ gives a 404

[rare wadi]

Inspector also says not found

[raw anvil]

maybe it's registered, but doesn't have any release? the 404 would be confusing

[unreal jewel]

they probably deleted all of the files and releases for it

https://pypi.org/simple/due/

doesn't 404

[raw anvil]

oh, OK!

[rare wadi]

oh I didn't know you could do that

I'm over here trying to figure out how to work BigQuery again

[valid flame]

@unreal jewel I'd half expect a better error there. should be doable, right?

might take a look into the code over the weekend

[rare wadi]

yep, one version was previously uploaded

[unreal jewel]

IIRC the reason the /project/FOO/ url 404s but /simple/ doesn't comes down to the fact that the framework we use lets us wire in a "call this code to look up the <thing> that the request is about", and that automatically translates a "couldn't find <thing> into a 404, and the ergonomics of changing that is annoying due to how that /project/FOO/ page works (<thing> is the Release object, not the Project object).

/simple/ doesn't have that same issue because we don't use that "call this code to look up the <thing>" logic, because /simple/'s performance is so important that we've had to very carefully craft the query instead of relying on a generic helper / look up routine.

fixing it is... technically doable? of course, but if my memory is correct it would make the ergonomics of working with the /project/FOO/ view a lot more annoying and manual, but it's been a long time since I've touched that particular thing so I couldbe mis remembering

[rare wadi]

OH
BTW
I was looking for something for Rem last night and I couldn't find it.
Would you be able to be point me to where exactly PyPI rejects PEP 440 non-compliant versions?

Sorry to jump to a completely different thing

[unreal jewel]

there's some indirection here, https://github.com/pypi/warehouse/blob/main/warehouse/forklift/legacy.py#L498 calls https://github.com/pypi/warehouse/blob/main/warehouse/forklift/metadata.py which uses https://packaging.pypa.io/en/stable/metadata.html to verify metadata

and the packaging lib is where it verifies that the Version metadata is valid

we also layer our own additional validations ontop of the packaging lib in https://github.com/pypi/warehouse/blob/main/warehouse/forklift/metadata.py

[rare wadi]

I looked at that, but packaging seems to always return a string

[unreal jewel]

https://github.com/pypi/warehouse/blob/main/warehouse/forklift/metadata.py#L334

it's the Metadata class that does it

https://packaging.pypa.io/en/stable/metadata.html#packaging.metadata.Metadata.version

[raw anvil]

Thanks for the context! The original use-case is to check if a name is available. I think adding a small paragraph somewhere in official docs to mention that one needs to check https://pypi.org/simple/{name} would be enough IMO ^^ I've posted this as an answer on SO: https://stackoverflow.com/a/78711811/3451029

[rare wadi]

ah
I saw https://github.com/pypi/warehouse/blob/main/warehouse/forklift/legacy.py#L675
Which led me to https://github.com/pypa/packaging/blob/main/src/packaging/utils.py#L67-L69
And 4am me was very confused
I can see the error but it just catches it and gives you the string back!!
That makes much more sense, thanks for the information.

[unreal jewel]

:np:

[loud knot]

Hi, what am I missing here ? why the .config files are not the the site-packages folder?

https://github.com/Aviksaikat/nipe_py/blob/main/pyproject.toml

[rare wadi]

This doesn't really have anything to do with PyPI, pinged you in #hatch message

[vapid torrent]

Hi, since last 1hour, our deployments are failing trying to install v1.3.5 of package "install", i just checked on pypi and i see a 404 https://pypi.org/project/install/
Anyone else facing the same issue?

[ancient compass]

There's an entry in wheelodex https://www.wheelodex.org/projects/install/

Looks like it got deleted

You can depend on the wheel directly in your requirements.txt https://files.pythonhosted.org/packages/4d/c8/8cbca135f9e167810756ea2bc34b028501936675fcbd7dadccf752fa4622/install-1.3.5-py3-none-any.whl

[worn pulsar]

sup everyone, im trying to follow https://warehouse.pypa.io/ instalation guide but unfortunetelly im having this issue! does someone knows how to solve it?

[blazing yacht]

Hello! That error looks like make is treating what should be a comment as a command instead.
What version of make do you have?
How was make installed?

[wintry cloak]

~I'm not sure if this is the right place to ask, but I was wondering what the status was on the PSF hiring the Support Specialist to handle the PEP541 requests?~

Edit: Nevermind it appears that I missed this update:
https://pyfound.blogspot.com/2024/07/announcing-our-new-pypi-support.html

[ancient compass]

Is there a reason that maintainers can't setup trusted publishing? Context is vcrpy

[obtuse torrent]

not wanting to publish through GitHub Actions, I guess?

[ancient compass]

I do want to publish through GitHub actions, but I have to publish from my dodgy laptop

[obtuse torrent]

no, I meant that this could be a reason why the maintainers can't/won't setup trusted publishing

[ancient compass]

?

[obtuse torrent]

oh, I see, you are a maintainer of vcrpy

[tribal sedge]

The current ACL allows Maintainers to upload new releases themselves, not manage the project settings like creating a Trusted Publisher - that's currently an Owner permission.
See https://github.com/pypi/warehouse/issues/13366

[ancient compass]

Oh no not that

I want more permissions not less!

[ancient compass]

As an owner on other projects, I'd like to be able to devolve my permissions with the exception of booting me off the project

[tribal sedge]

Feel free to comment on the issue!

[shrewd pelican]

Been getting 502 on upload only since 10PM central on 8/9 nothing else seems amiss though

[glossy turret]

There was an outage around 12PM pacific time for uploads, I wonder if this is related

[pliant obsidian]

There was an incident for uploads lasting about an hour reported at https://status.python.org/, resolved at 19:26 UTC

[shrewd pelican]

Still having the same issue wondering if it's related to this https://github.com/pypi/warehouse/issues/9151

[glossy turret]

that was from a year ago

I would open an issue if you are still getting 502s

[shrewd pelican]

Went ahead and did that didnt even notice how old that issue was, still referenced it though.

[shrewd pelican]

This might be a regional proxy issue I am not seeing others with the same problem. Currently trying to get gh actions to use a fresh runner on my repo but am unsure if that is even possible for me to do on the user side

[shrewd pelican]

So, I suspect that the issue is that example on the core metadata specification for Provides-Dist shows a version format that will not work for warehouse as it uses packaging.requirements.Requirement to validate this header. Checking to see if that is the culprit, kinda weird that twine and auditwheel would both miss that but it really is the only difference between this package and the ones that dont 502 on upload.

[shrewd pelican]

That is the issue.

[wicked wind]

Asking for a friend: if someone is on the receiving end of abusive comments in a GH PyPI support thread (to be clear, not from anyone associated with PyPI), is it appropriate to file a CoC report? Also, what would be the best way to report this to the PyPI support GH admins, so they could potentially lock the issue to avoid further abuse?

[final copper]

Hello -- I'm trying to upload a new project to PyPI and being told I don't have permission (403 Forbidden, not allowed to upload to project etc.), but there doesn't seem to be a project under that name.

The help page I'm linked to suggests it's either a reserved name, or claimed with no releases, but I'm not sure how to distinguish between those. The simple API returns an empty list of versions rather than a 404, so does that mean it's claimed rather than reserved?

It also points at PEP 541, and I started filling in a pypi/support issue, but I'm not sure that's the right place for this seeing as I have no project owner to contact?

[wet flax]

pypi/support is the right place, yes.

[final copper]

Thanks, support issue raised

[lapis meteor]

@unreal jewel so I just came across the blog post from 2021 about how the monthly infrastructure costs for pypi would be around $1.8 million if not for the fact that fastly was donating it.

Do you happen to know the current figure as of 2024?

[unreal jewel]

uh, not off the top of my head. those numbers were always a tad sketch because it was taking the list price of our req/s and bandwidth, which basically nobody spends millions on a service like this and doesn't negotiate a discount lol

let's see what our reqs and bandwidth were last month

Bandwidth: 105460143.30817497 (I think this is GB)
Requests: 18749106.715800002 (I think this is blocks of 10k requests)

[lapis meteor]

105.5 PB in a month, wow. That's quite a bit!

Thanks for the info

Fwiw, at list price that looks like about $12.5 million per month

[unreal jewel]

https://tenor.com/bwaok.gif

I think we technically have a "negotiated" rate now (which then gets discounted to 100%) for tax purposes, but I have no idea what that is

[pliant obsidian]

from PyCon US this year, Fastly said over 10 years:
4,682.1 billion requests and 2.41 exabytes of data
https://www.youtube.com/watch?v=e7h-h-d3FG4&t=2427s

[unreal jewel]

oh I missed that

neato

[final copper]

In the absence of any response to that support ticket (looks like there's quite a backlog there, with most PEP 541 tickets going unanswered), can anyone else answer the remaining questions? Is there anything I can/should be doing to try and find a project contact?

[valid flame]

you might try checking github/gitlab for that project and take contact information from user account or project commits

[final copper]

There is no project info at all that I can see

The PyPI page doesn't exist, and the simple API doesn't expose any info

[rare wadi]

This is you, right? -- https://github.com/pypi/support/issues/4569
staff==0.0.1 was uploaded in 2020
The author probably eventually deleted it

Here's the BQ metadata which includes an email address and a website:

[rare wadi]

It does
That the page exists at all tells you that the project existed at some point
If the project had never existed, you would get a 404
Compare the response of https://pypi.org/simple/staff/ and https://pypi.org/simple/doesnotexist/

[final copper]

That's the request yes

I did note the API discrepancy above

The website doesn't seem to exist but I can try emailing

If it's been deleted, what do they need to do within PyPI if they accept?

[rare wadi]

That I don't know
I don't think they can do it themselves, because they wouldn't have access to it anymore
I assume you'd post their email response on your ticket and the PyPI team would step in and help

[final copper]

Having just downloaded the package based on the URL from the metadata above, it looks like it contains a Python module of a different name, of which the entire code is just a __version__ definition, so not a "usable" package in any practical sense, though I assume the default stance is not to remove/reallocate anything regardless of content

[glossy turret]

(not a PyPi admin, so take with a grain of salt) I believe that would fall under https://peps.python.org/pep-0541/#invalid-projects

[unborn nacelle]

when uploading a file, is there a way to tell whether a file was freshly uploaded or already existed?

i'm implementing an upload function and i'd like to signal this back to the user

[valid flame]

IIRC if a file already exists, it will be rejected

[unborn nacelle]

only if the hash mismatches

if you upload the same file (e.g. a source dist from multiple platform specific build jobs) multiple times, i see a 200 with the same in the html response

[glossy turret]

I would probably just query the API prior to upload

[rare wadi]

Does Test PyPI have Trusted Publishing enabled?

[pliant obsidian]

Yes! I deploy to Test PyPI for merges to main, and prod PyPI for tags. It's nice to test end to end deploying in the same way on both.

[rare wadi]

That's exactly where I was going with this

Do you think the community would be receptive to having https://packaging.python.org/en/latest/tutorials/packaging-projects/ recommend and explain Trusted Publishing first, leaving twine as a fallback?

[pliant obsidian]

I definitely think Trusted Publishing should be the first choice. It's easier to set up, makes things easier to release, and is more secure: short lived tokens instead of long lived ones sitting around on your computer or CI

[tribal sedge]

Signal-boosting this for those that haven't seen it yet. Hopefully it doesn't matter one whit to you, but just in case... https://github.com/pypi/warehouse/issues/16642

[median quest]

Hi everyone, Im seeing some curious behavior trying to upload my first distribution of a project I've (attempted to) call pieces. Namely, even though both https://pypi.org/p/pieces and https://pypi.org/simple/pieces return 404 errors suggesting this project doesn't currently exist I am still seeing a 400 error when I try to upload like:

400 The name 'pieces' isn't allowed. See https://pypi.org/help/#project-name for more information.

I have filed the requisite PEP 541 ticket here 1, but I see there's quite a backlog of PEP 541 issues filed for triage/support.

I'd appreciate any insight here as this particular issue seems abnormal compared to the situations described in PEP 541.

[obtuse torrent]

Project named piece exists so could be typo squatting prevention?

[median quest]

When I searched the name pieces in the web UI I didn't even see a project named piece, but if that does exist then squatting prevention definitely could be the reason I was rejected

[rare wadi]

BQ returns no results either, doesn't look like a deleted package

It's such a generic name you wouldn't think it would be restricted

[median quest]

For me at least it helps explain why this name wasn't taken already 😅

[rare wadi]

What did you drop this for?
https://github.com/AvlWx2014/pieces/commit/41b38ab951af2fa2cfdef5c4691bbe82d4817751
You should keep it.

Also,
1, use the canonical link, don't use an alias
And 2, something neat I just discovered -- you can use string interpolation to add the version number to the link: https://github.com/letsbuilda/crazylibs/blob/main/.github/workflows/python-publish-pypi.yaml#L38

So your runs will link to the specific version that they released

[median quest]

1, use the canonical link, don't use an alias
Ill update the link accordingly

And 2, something neat I just discovered -- you can use string interpolation to add the version number to the link
Using the string interpolation to add the version number is awesome as well so I'm going to do the same!

What did you drop this for?
https://github.com/AvlWx2014/pieces/commit/41b38ab951af2fa2cfdef5c4691bbe82d4817751
You should keep it.
I only dropped the environment usage in trying to eliminate variables while troubleshooting my failed release workflows. It looked at first like I might have uncovered a bug in pdm publish when environments were in use, so I took it out to see if the failures persisted. It wasn't until I tried publishing by hand using an API token that I discovered the underlying cause is the project name being rejected, so I will also go back and revert that change to put environment back

[rare wadi]

👍

Remember to double check your Trusted Publishing config on PyPI to make sure it has the environment listed

[median quest]

Right, I will need to add that back to my pending publisher as well

[median quest]

That is assuming I can get the pieces project name on PyPI 😄

[gentle frost]

is there a process like PEP 541 but for usernames rather than projects?

[rare wadi]

[hasty salmon]

Maybe I can finally have doggo the username

[gentle frost]

how does verification/these new checkmarks work?

[pliant obsidian]

I was curious too so did some git blame to find the docs: https://docs.pypi.org/project_metadata/

[tribal sedge]

Does the "what is this" link above it not work?

[rare wadi]

That's where I got the docs from
It links me to https://docs.pypi.org/project_metadata/#verified-details

[wicked wind]

I just updated my warsaw/pep-694 branch by merging main and resolving conflicts. Now I'm having the following error:

...
alembic.util.exc.CommandError: Multiple head revisions are present for given argument 'head'; please specify a specific target revision, '<branchname>@head' to narrow to a specific head, or 'heads' for all heads
make: *** [runmigrations] Error 1

If I'm reading the SO suggestions correctly (a big IF), it wants me to alembic merge heads but I'm not sure I know how to do that when working on a local branch of warehouse. I couldn't find anything relevant in the warehouse docs about this, so before I burn time digging deeper I wanted to see if anybody has suggestions about how to resolve this problem.

You can see this in the Check Database Consistency job here: https://github.com/pypi/warehouse/actions/runs/10911542782/job/30284554295?pr=16277

[wicked wind]

I think at least one part of the puzzle is: docker compose run --rm web python -m warehouse db merge 0b74ed7d4880 93388f06f5e7 but that still doesn't get me a clean test suite.

[gentle frost]

my bad, I did not notice that 😅

[tribal sedge]

no worries. is there another way we could present that for better clarity?

[gentle frost]

I would recommend that the link be immediately next to the header of the verified fields

[tribal sedge]

instead of the header of the entire section? there's mutliple sub-headers that have verified details

[gentle frost]

instead of the header of the entire

in that case, perhaps get rid of the checkmarks if the entire section is verified?

[tribal sedge]

heard, thanks for the feedback!

[rare wadi]

Maybe one big checkmark at the top, next to the docs link?

Would help draw attention to the docs link

[tribal sedge]

we'll play with it a bit at some point soon, stay tuned

https://github.com/pypi/warehouse/issues/16743#issuecomment-2359230573

[tribal sedge]

This is what was done, and over the next 24 hours as page caches expire, all will be updated to show this format:

[rare wadi]

[gentle frost]

just a heads-up, not sure if there is an open PR, but the docs need to be updated https://docs.pypi.org/project_metadata/#verified-details

[tribal sedge]

There isn't yet, if you wanted to send one!

[analog flame]

I'm trying to publish a package through Github Actions with Trusted Publishing and keep encountering the attached warning and error. I'm following https://docs.pypi.org/trusted-publishers/using-a-publisher/#github-actions and have set up everything as it looks like it's supposed to be in both my .yml (https://github.com/lawnchairmeku/pymember/blob/main/.github/workflows/python-publish.yml), and on pypi (also attached). I feel stumped and could use some help

[rare wadi]

Wow, that's super weird

https://github.com/pypa/gh-action-pypi-publish/commit/27b31702a0e7fc50959f5ad993c78deac1bdfc29 was 3 years ago

I have no idea why GitHub did that

[rare wadi]

Can you merge this and try again please?
https://github.com/lawnchairmeku/pymember/pull/3

And if that fails, rerun with debug logging

[analog flame]

It failed again, here's the run with debug logging: https://github.com/lawnchairmeku/pymember/actions/runs/10948822355/job/30403299934

[rare wadi]

Oh I see

You're doing it to yourself

[rare wadi]

Click the "Workflow file" button to see the file it's using

It's using a three year old commit because you told it to

[rare wadi]

You can't rerun the same action run after making changes
The run will always be executed using the same commit it originally used
You need to trigger the again again after making changes
In this case that means meaking a new tag and a new release

[analog flame]

thank you

[rare wadi]

[analog flame]

🪄

[agile sinew]

#pyproject-metadata isn't releasing. Getting:

Uploading distributions to https://upload.pypi.org/legacy/
Uploading pyproject_metadata-0.9.0b4-py3-none-any.whl
WARNING  Received "502: Bad Gateway"                                            
         Package upload appears to have failed. Retry 1 of 5.                   
Uploading pyproject_metadata-0.9.0b4-py3-none-any.whl
WARNING  Received "502: Bad Gateway"                                            
         Package upload appears to have failed. Retry 2 of 5.                   
Uploading pyproject_metadata-0.9.0b4-py3-none-any.whl
WARNING  Received "502: Bad Gateway"                                            
         Package upload appears to have failed. Retry 3 of 5.                   
Uploading pyproject_metadata-0.9.0b4-py3-none-any.whl
WARNING  Received "502: Bad Gateway"                                            
         Package upload appears to have failed. Retry 4 of 5.                   
Uploading pyproject_metadata-0.9.0b4-py3-none-any.whl
WARNING  Received "502: Bad Gateway"                                            
         Package upload appears to have failed. Retry 5 of 5.                   
WARNING  Error during upload. Retry with the --verbose option for more details. 
ERROR    HTTPError: 502 Bad Gateway from https://upload.pypi.org/legacy/        
         Bad Gateway                             

Is that expected? Don't see any status issues. I don't think anything has changed since I released the last version a few days ago.

I do have attestations: true but I'd assume that doesn't affect anything in this way, even if in beta, and I had it last time.

[agile sinew]

Scikit-build-core released just fine. I don't have attestations: true there yet.

[glossy turret]

This commit was made yesterday to pypi: https://github.com/pypi/warehouse/commit/bdb71dc3c0f4dabedabfe33dcdbb800edd4f4752

possibly related

[agile sinew]

Yes, noticed that other failures seem to have attestations: true. See https://github.com/pypa/gh-action-pypi-publish/issues/263.

[rare wadi]

This could have been much worse had we begun to persist and serve attestations, but that fortunately hasn't happened yet 🙂.

oh

I've been meaning to ask where my attestations have been going

Apparently they just haven't been going anywhere?

[tribal sedge]

https://github.com/pypi/warehouse/blob/30846f6061d63b8c41a3275d1958efe3749524d8/warehouse/forklift/legacy.py#L407-L412

[rare wadi]

Huh, weird

Is it just waiting for the supporting infra to get constructed?

[tribal sedge]

some progress to follow: https://github.com/pypi/warehouse/pull/16623

it's been ... bumpy

[rare wadi]

[ancient compass]

Very ominous that there's no associated GitHub issue

[pliant obsidian]

https://github.com/pypi/warehouse/issues/15871 is the PEP 740 umbrella issue

[valid flame]

Is there any changelog or summary of changes made for each PyPI deploy?

[unreal jewel]

not really

other than git log

[wicked wind]

PR with lots of updates to PEP 694 (upload 2.0) based on DPO discussions: https://github.com/python/peps/pull/3997

[gentle frost]

I'm curious what others think about this https://discuss.python.org/t/is-there-a-process-by-which-a-new-trove-classifier-can-be-added/65252/6

[wicked wind]

PEP 759, External Wheel Hosting: https://discuss.python.org/t/pep-759-external-wheel-hosting/66458

[oak mica]

uv is switching to enabling HTTP/2 by default: https://github.com/astral-sh/uv/pull/8049

I thought I'd mention it here now uv represents a significant amount of the load on PyPI, in case this is a known issue or not with HTTP/2 and PyPI's CDN

[formal jolt]

Definitely let us know if that's a problem!

[tribal sedge]

Thanks for the heads up - I don't think it'll change much on the backend, but will keep an eye out

[restive swift]

Hi! I submitted a PR (#16780) for PyPi Warehouse. Would be happy to learn more about if there is anything I can do to help out regarding details of the PR to make it easier to decide for merging? https://github.com/pypi/warehouse/pull/16780

[honest flame]

I struggle adding a trusted publisher to an existing project. I tried following https://docs.pypi.org/trusted-publishers/adding-a-publisher/ but when I click on "Manage" for a project, then on "Releases", I just see a list of existing releases, no form to configure trusted publishers. But that's an XY problem, I actually want that the github links are listed as trusted. It is strange they are not, given that they come from the package metadata.

[agile sinew]

It's Manage -> Publishing, not Releases

[honest flame]

Ahh! Got confused by the German translations. Thanks!

[tribal sedge]

Add some tests to express the shape of what the response looks like when there are maintainers, since the updated tests only express empty lists.

[wicked wind]

I posted a PR today for "programmatic yanking", essentially a REST-ish API for yanking and unyanking releases. Closes #12708. Hopefully I have included enough context about the design and implementation in the PR description, so I won't repeat myself here. This being my first substantial PR to warehouse, I'd be very happy to chat about it, either in the PR or with more bandwidth here. If this is a direction folks like, I have other plans for REST-ish APIs, including most importantly PEP 694.

https://github.com/pypi/warehouse/pull/16912

[formal jolt]

Ooo I want this.

[wicked wind]

I'd love any feedback on usability you might have!

[formal jolt]

My use-case may have passed, but I'll check it out

[tribal sedge]

I haven't dug into any of the changes yet, but wanted to make you aware of the main prerequisite in my mind to doing more API stuff, which is a necessary change to the macaroon caveats structure. The existing approach is far too broad in permissions, and there isn't a great way to change the approach yet. An idea was started here but still needs work to get out the door before we do more things with PyPI Management API stuff. https://github.com/pypi/warehouse/pull/15240

And I had built in a lot of OpenAPI infrastructure for the Observation reporting API in preparation for other management-oriented interfaces https://github.com/pypi/warehouse/pull/15553

@wicked wind maybe send me a dm with an email I can share an API dev doc I wrote for that initiative.

[vast halo]

@lethal meadow Attempting to explain https://docs.pypi.org/trusted-publishers/ to our GH repository admin (after he expressed surprise at not having to enter any persistent upload credentials at any point in the process), I summarised it like this:

The gist (as I understand it) is that it's a three way link where:

• you just set up the "PyPI <-> GitHub OIDC" part of the link
• I set up the "GitHub Actions <-> GitHub OIDC" part of the link when I added the id-token: write permission in the publish.yml action definition
• pdm publish relies on the other two established links to negotiate a short lived token for the "GitHub Actions <-> PyPI" link when publishing

If that's right (or at least right-enough-to-be-useful), do you think it's enough simpler than the existing explanation to include in the docs in some fashion?

(Edit: I gave up on trying to get Discord to format that self-quote correctly. All 3 bullets should be at the same level, corresponding with the first 3 numbers points in the PyPI docs)

[proud bison]

Are there any news about this?

[ancient compass]

With trusted publishing, what's bootstrap process for the first publish? Do we need to use an unconstrained token to create the package or can we create a token that can only upload a particular package name before it's created?

Oh it's in the docs

https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/

Neato

[lethal meadow]

Yeah that's a little strange that the PyPI docs says: "You don't need to understand OIDC, but here's OIDC". I think the abstraction about a "link" can be useful to folks, linking the PyPI project to a repository. In actuality what is happening is you're creating a policy: "only this GitHub repo and workflow can publish to PyPI"

[vast halo]

Having at least some explanation is good, as I suspect I'm far from alone in preferring to at least vaguely understand the auth mechanisms I'm setting up.

What we currently have is probably still a bit heavy for most readers, though.

[ancient compass]

@empty ridge would you like to take over maintenance of pip-with-requires-python it's a dep of pypa/gh-action-pypi-publish ?

I don't think it will ever need any maintenance, but I don't use it in anything anymore

[empty ridge]

I don't mind, I suppose. I also don't think it's really necessary these days since all non-EOL Python versions come with a modern enough pip..

[wicked wind]

Is there a known delay in updating the simple index HTML (i.e. https://pypi.org/simple) after the first release of a new project?

[rustic venture]

yes, that page is cached for up to 24 hours

since most clients don't use it direclty but rather /simple/{project_name}/

[wicked wind]

Thanks @rustic venture good to know. I was using a library to scan the top-level index, so added a "deep scan" for ones that got missed. I'll try to find a place to document this (AFAICT, the delay isn't documented anywhere, but I might be missing it).

[ancient compass]

https://pypi.org/project/fairground/#history (note the missing numbers)

I'd very much like for warehouse to move the package validation code to something I can run locally

I'm also pretty sure there's already a GitHub issue for it where I promised to extract it

[drifting pollen]

how would I go about finding out the most downloaded reverse dependencies of PyPI projects I maintain?

[honest flame]

The only source to figure out reverse dependencies without downloading and parsing everything yourself is https://www.wheelodex.org/projects/pip/rdepends/ AFAIK, but that's quite slow and probably not intended for automated scans. The download counts can be fetched from https://pypistats.org/api/ without a google bigquery account.

[tribal sedge]

Hiya! I'm not sure I understand what you mean by missing numbers here - do you mean the skips between a7 and a10 ?

[ancient compass]

Yep

Lots of mistakes that twine didn't catch

[tribal sedge]

Aha. A lot of stuff is going into https://pypi.org/project/packaging/ - and I don't think twine depends on that, so that might be an interesting venue to explore

[glossy turret]

I've actually wanted a standalone tool/library that does the same checks PyPI does as well, but I think there are several checks that I would say don't make sense in packaging. Specifically policy decisions such as the set of platform tags that are supported, local build tags being rejected, etc.

[pliant obsidian]

see also https://github.com/pypa/twine/issues/430

[meager umbra]

A standard wheel/sdist validation tool would be great to have.

[valid flame]

A module in packaging would be nice. Would make it easy for packaging tools to integrate without introducing another dependency

[gentle frost]

was there a change recently? https://github.com/pypa/hatch/issues/1786

I thought setuptools backpopulated that field as well

[rustic venture]

Looks like the issue is referencing 2.4. PyPI hasn’t landed 639/Metadata 2.4 yet.

[gentle frost]

it appears that the license expression produces an error upon upload if it's used on a version earlier than 2.4

Hatchling doesn't actually support 2.4 yet

I remember setuptools landing License-Files much earlier than the version of the spec with which it was introduced so I assumed this was fine with the expression as well

[rustic venture]

we merged packaging 24.2 yesterday, so I guess that is the change that caused it to be a validation error now. https://github.com/pypi/warehouse/pull/17055

[gentle frost]

I'm about to craft the fix, does PyPI support metadata 2.4 currently or should I not make that the default yet?

[rustic venture]

pypi does not support 2.4

https://github.com/pypi/warehouse/pull/16949 will probably merge in the next week or so

[glacial cloud]

Hi Team,

What is the process to add Bitbucket Pipelines OIDC provider CI/CD as a new Trusted publisher to allow users select it and set up in PyPI UI?

Our team develop pypi-publish pipe for Bitbucket Pipelines CI/CD integrations and other open-sourced 60+ Bitbucket Pipes.
Also, we develop and maintain the Bitbucket Pipes Toolkit for custom pipes creation.

It would be nice to have Trusted publisher configuration (OIDC integration) in PyPI UI for projects that are stored on Bitbucket Cloud and using Bitbucket Pipelines CI/CD for publishing to PyPI.

Best regards,
Oleksandr Kyrdan
Atlassian Bitbucket Pipes team

[rustic venture]

cc @finite pulsar @merry valve ^

[finite pulsar]

hey @glacial cloud! the process is opening a tracking issue for us to discuss, and then sending a PR once a development roadmap is settled 🙂

you can see a previous example of that process for GitLab: https://github.com/pypi/warehouse/issues/13575

in particular, one big thing that needs to be ensured is that BitBucket's OIDC provider has a suitable set of claims for a Trusted Publisher identity. for GitHub Actions that's something like workflow.yml @ user/repo, and we'd need the equivalent for BitBucket

[glacial cloud]

@finite pulsar @rustic venture Thanks for your help and fast reply! 👍 We'll review the issue and discuss this with our teams. Keep in touch! 🙂

[agile sinew]

PyPI uploads broken? https://github.com/pypi/warehouse/issues/17111

Seems to be reporting Invalid distribution file. WHEEL not found at for uploads.

[rare wadi]

Should be back soon

Does PyPI post the currently deployed commit hash anywhere?

Anything to help confirm when something goes live

[rustic venture]

https://github.com/pypi/warehouse/deployments

[rare wadi]

Does that not get checked until it's actually live?

I'll look at it

[rustic venture]

Correct. The check goes green after that code is live

[teal basin]

is it better to use the simple API or the json API for package metadata?

[valid flame]

Simple JSON API TBH

JSON API (the one that is PyPI-specific) is not guaranteed to be stable and can change.
Simple API defaults to HTML which is not nice to parse
Simple JSON API is the best choice

[rare wadi]

re: the above
@teal basin is asking me about this issue of mine:

Add support for the Simple API
The discussion in #general message reminded me that the legacy JSON API is in fact, legacy, and that I need to work on looking at pulling data via the Simple API.
- https://github.com/letsbuilda/letsbuilda-pypi/issues/33
I... I remember being told that the "Simple" API was standardized, and the JSON API was the "legacy" one
But now I look at https://warehouse.pypa.io/api-reference/legacy.html and see the Simple API marked as "legacy"

It's because uh... The Simple API is "old", but it's standardized so it's not going anywhere; while the JSON API was just copy/pasted from legacy-pypi, and is not standardized and is on the roadmap to be rethought and restructured ...right?

[teal basin]

i think simple API is marked as legacy because its compatible with the legacy?

The “Legacy API” provides feature parity with pypi-legacy, hence the term “legacy”.

the docs in JSON API say

The releases key on this response should be considered deprecated, and projects should shift to using the simple API

so i'm not too sure

[rare wadi]

Yeah, it's standardized in the PEP, so it will always be compatible

[valid flame]

all new standards go towards Simple JSON API

so that is what you should use

[proud bison]

All you need to know is in the comment right above yours: #pypi message

There are three APIs, two of which are JSON, and you’re confusing these two

[rare wadi]

That matches what I said, doesn’t it?
JSON API is subject to change, Simple API (regardless of content type) is standardized

It's weird to me to look at the "Simple JSON API" as it's own entity when it's "just" an alternate content type

Oh you commented on the issue
I really appreciate the help

https://github.com/flying-sheep/PKGBUILDs/blob/main/src/nvcheck/update/sources/pypi.py#L78
lol, you call the PyPI API the legacy one, backwards from the docs labeling the Simple API to be legacy

I'm going to go to https://github.com/pypi/warehouse/issues/16541 and add a chart for people like me

[vast halo]

The simple JSON API isn't just an alternate content type, as new simple API features aren't required to have HTML representations (they're allowed to have them, they're just not required to do so).

So there are two legacy APIs: the non-standardised PyPI-only JSON API (somewhat based on the old pre-warehouse XML-RPC API, IIRC), and the (standardised!) simple HTML API.

The only API that is both standardised and guaranteed to support any new feature additions is the simple JSON API.

[rare wadi]

new simple API features aren't required to have HTML representations
I did not know that
I had assumed that they were kept in parity
Is that listed in the docs anywhere?

[vast halo]

It's somewhat cryptically mentioned in https://packaging.python.org/en/latest/specifications/simple-repository-api/#versioning

Specifically, this bit:

Future versions of the API may add things that can only be represented in a subset of the available serializations of that version. All serializations version numbers, within a major version, SHOULD be kept in sync, but the specifics of how a feature serializes into each format may differ, including whether or not that feature is present at all.

If something can be trivially serialised in the HTML format, it will probably be added (since there's no particular gain in leaving it out), but more complex proposals that add nested JSON data structures (and hence don't map nicely to HTML element attributes) may be JSON-only.

I think the namespace reservation proposal is the only one so far that has considered taking advantage of that clause, though.

[proud bison]

Yeah, that was in @valid flame other message: #pypi message

And that's why it's less confusing to think in terms of three APIs, not two APIs one of which supports two content types.

[finite pulsar]

re: confusion between legacy and non-standard APIs: one of my TODO items is to migrate all of the current PyPI API docs over to https://docs.pypi.org/api/, at which point i'll also fix the confusing "legacy" versus PyPI-specific naming

[wicked wind]

What we really need is a comprehensive REST API. Been thinking about that a lot lately.

[valid flame]

as long as it is a standard for python indices, I am all in. Separate REST API only for PyPI is semi-useless

[wicked wind]

I don't see why it couldn't be, but I'm only personally interested in implementing stuff for PyPI. I'd prefer to think of it as (a little more than) semi-useful 😄

[valid flame]

meaning: we will have tools only for PyPI, while a lot of interest is in having standard for private indices

[wicked wind]

If we write specs first, then implementations, maintainers of alternative index softwares can follow the same specs. I'm not sure if there's a great track record for that though.

[dreamy hawk]

is there a reason my repository link ends up in unverified details? https://pypi.org/project/stringenum/ and https://pypi.org/project/pyanilist

I'm using Trusted Publishing

which should verify it according to the docs here https://docs.pypi.org/project_metadata/#via-trusted-publishing

This seems to only affect 2 out of the 5 projects I used Trusted Publishing on, the other 3 have the Repository link verified

[ancient compass]

Are you publishing attestations?

Do you have the GitHub action pinned?

[dreamy hawk]

Yes https://github.com/Ravencentric/stringenum/blob/e7810b620976e240683cee22ab19c45aba9ce9e4/.github/workflows/release.yml#L64

https://pypi.org/project/stringenum/#stringenum-0.4.0-py3-none-any.whl attestations show up on pypi

[tribal sedge]

I think this exposed a slight mismatch in case sensitivity, thanks for sharing the project and source repos. I wrote this and shared with the original author who may look into it.

[dreamy hawk]

No worries! glad I could be of help. This only affected the two projects I linked (out of the 5 where I enabled Trusted Publishing)

If you don't mind elaborating, where did I introduce a case mismatch?

[tribal sedge]

I don't think you did - I think the way the code handles case sensitive inputs in the publisher is slightly incorrect

[dreamy hawk]

oh, I thought I messed up the casing somewhere

good to know

[rare wadi]

Will Trusted Publishing impact the the ability to add new wheels to old releases later on down the road?

[tribal sedge]

https://github.com/pypi/warehouse/pull/17154 should have fixed it, and it looked like the latest release of stringenum shows correctly

[dreamy hawk]

Yep! Thanks for the speedy response 😁

[teal basin]

what do the core-metadata and data-dist-info-metadata fields mean?

[valid flame]

related to serving package metadata, see PEP 658 and PEP 714

[chrome zealot]

I am sure this is very unlikely to be relevant or help, but I'll mention it on the tiny chance it's helpful for the outage --

I saw strange package yanking behavior around an hour ago I think

Specifically I was in the middle of yanking a release of (rpds-py) and pypi seemed like it went through with the yank, but then I noticed the version wasn't actually yanked and had to do the yank a second time for the same version

(and yes I know we all think we're the ones to break pypi :D, and the above is probably evident in the logs if it is indeed related -- just as I say mentioning on the tiny chance it is relevant! And thank you a million to those of you as usual who keep all our lights on!)

[gentle frost]

is PyPI going through an outage currently? I added a trusted publisher and ran the workflow which succeeded and produced a link but there is a 404

• https://github.com/ofek/binary/actions/runs/12183617622/job/33985493082#step:4:21
• https://pypi.org/project/binary/1.0.1/

[valid flame]

looks like it: https://status.python.org/

[gentle frost]

I wonder if I have to release again after the outage is resolved

[gentle frost]

nope, looks okay now

[gentle frost]

fyi just opened https://github.com/pypi/warehouse/issues/17233

[glossy turret]

I'm interested in splitting out the checks made against uploaded packages in the legacy upload API with the goal of allowing them to also be run against packages locally before upload. Today, users don't have a way to ensure their packages will pass the checks before upload, and not all build backends have the same correctness checks. But before working on refactoring this logic out I wanted to see if this is something the maintainers think is a good idea.

I'm imagining this would be fully offline so it would not check things like permissions, file existance, etc, it would merely check things about the package itself

[wicked wind]

Are you thinking about adding this to an existing project? e.g. perhaps packaging?

[unborn nacelle]

in uv we've had user requests for https://github.com/jwodder/check-wheel-contents

[rustic venture]

Speaking as a maintainer of PyPI, refactoring all those checks into packaging would be more or less welcome. We're already relying on it for metadata parsing, and the benefits for offline use are a big win.

I was in there recently and I think it should be a pretty clean cut with the exception of some of the obvious things like project ownership and name collisions 🙂

there are some idiosyncracies for sure though. i'm not 100% sure packaging is the right project... there will always be some pypi specific logic

[glossy turret]

agreed, I think it probably would best be it's own project

[rustic venture]

FWIW nothing stoping it from being refactored internally to pypi/warehouse and published as a standalone package.

[pliant obsidian]

see also https://github.com/pypa/twine/issues/430

[glossy turret]

I would expect a package check in packaging to be very comprehensive and un-opinionated. PyPI doesn't allow direct references in dependencies or some things that are discouraged which I would expect a packaging tool to allow

[glossy turret]

I think this may be a good path, it can always be broken out into it's own repo if the need arises.

[vast halo]

Even in the standards, we have things where we say "This is allowed in general, but not on PyPI". Emma's direct URL example is one of those, another would be the restrictions on permitted platform tags. Maybe two levels of checking? Pure structural validity checks in packaging and "fit for PyPI" checks in a new package called something like pypi-upload-checks?

[drifting pollen]

what's the best way to get an organization request approved on PyPI? I really need this, and I have a request that's been pending for almost 14 months now

[tribal sedge]

Can you elaborate on why you really need this? What's not having the organization approved preventing? The backlog for approvals is ~8k long

[drifting pollen]

perhaps there are other users with a more pressing need for this than me, but I have a github organization that owns 11 projects and I have just shared ownership with a new collaborator. Without the organization I need to sync the owners manually to each project.

it's not knowing if anything is happening or when to expect approval that's making the wait hard

[valid flame]

I'm not sure if that is something on my end, but for about a week now, I have noticed that when accessing PyPI, a loading screen appears, saying "Fastly verifying your browser" or something similar. It doesn't bother me much, but maybe something has changed in some configs that would make that happen. It's going away too fast to grab a screenshot.

[unborn nacelle]

there's an explanation from ee on dpio: https://discuss.python.org/t/fastly-interfering-with-pypi-search/73597/6

[valid flame]

thanks

I was starting to think it's something on my side and were ready to debug my network/browser 🤣

[wicked wind]

Yep, I had to fix some of my scripts.

[oak mica]

I saw it for the first time today (normally don't use PyPI directly) and the UX is pretty bad, it looks like maybe your browser has been hijacked, especially if you don't know what Fastly is, a "Loading..." Splash screen or something would be much nicer

[proud bison]

It doesn't bother me much,

It does bother me. There’s so much work going into preventing flashing content, and now that has been introduced intentionally

But I guess it only happens once per session, so it’s fine!

[wicked wind]

It sounds like it's not a permanent solution, and I hope that's the case. We still need a viable way of programmatically accessing data and making changes on PyPI. E.g. I have a branch to do programmatic yanking, which would be super helpful for some of our use cases.

[serene fern]

Maybe PyPI should have programmable APIs

[tribal sedge]

There's a bunch of work needed to make tokens more API-useful before most of that work could happen. When they were created, the only use case was project uploads, and that means that most API tokens out there have "too much access" for other API operations

Here is a starter from dstufft https://github.com/pypi/warehouse/pull/15240

[wicked wind]

What's it gonna take to move this along? That PR is still in draft and has lots of comments. I'm personally less interested in the permissions (I just want something I can use), but very interested in the API. Then again, if this is a pre-requisite and it's languishing I wouldn't want it to hold up other progress on APIs.

[tribal sedge]

Generally speaking, priority. Getting auth right for the future API is something we'll likely have to live with for years to come, so we want to get that right before moving forward. And it's not been a priority for me as of late

[wicked wind]

Gotcha. Let's chat about it in the new year. I want to find a way to help.

[raw anvil]

Hello! Are Private :: classifiers only meant to affect PyPI? The original PEP doesn't mention anything about such classifiers. The official docs don't tell if only PyPI should do this, or if other registries should as well. My goal here is to prevent accidental upload of private versions of my projects to PyPI (public registry), but allow users to build and upload to their own private registries.

I myself upload them to a locally-served instance of pypiserver, for example 🙂

I guess I'll have to check the docs of the most common registries (Google stuff, Artifactory, pypiserver, devpi, what else?).

[pliant obsidian]

I think it's PyPI only and not standardised, but other indexes might also implement it.
see https://discuss.python.org/t/pypi-package-name-restricted-namespace-proposal/29007/5 for some history

[raw anvil]

thank you @pliant obsidian!

[vast halo]

Limiting the permitted classifiers is entirely up to the server. The fact PyPI limits them and most private server implementations don't is a large part of the reason the Private :: ... convention developed (since internal distribution is fine, it's inadvertent public uploads that people wanted to avoid)

[restive sapphire]

hi - trying to get warehouse running locally, i've followed https://warehouse.pypa.io/development/getting-started.html# but running make serve exits and gives me this:
Error response from daemon: unable to find user nobody: no matching entries in passwd file
the only reference i'm able to find to this is here: https://github.com/pypi/warehouse/blob/775dc3bb352c6c5a505261b129c940bf0d16b6d3/docker-compose.yml#L84-L86

    environment:
      # Runs as `nobody`, override the home directory
      - PIP_CACHE_DIR=/opt/warehouse/src/dev/.pip-cache
      - PIP_TOOLS_CACHE_DIR=/opt/warehouse/src/dev/.pip-tools-cache

has anybody run into this issue before or can point me towards the right direction?

[restive sapphire]

looks like it's coming from the camo service which has the image pypa/warehouse-camo:2.0.0
it's forked from atmos/camo which has this line:
https://github.com/atmos/camo/blob/master/Dockerfile#L13
but which is missing from pypa/warehouse-camo's Dockerfile so I'm unsure why it's still trying to use that user

[violet fable]

why are you working on this on new years?

I mean, I did do OSS stuff this last Christmas, but like, it's a new year!

[rare wadi]

What are my other options? Go "outside"!? That place is scary!!

[restive sapphire]

😄 don't have much else to do

[tribal sedge]

@restive sapphire can you share some more about the scenario you're hitting? Base OS for your machine, Docker desktop or another virtualization layer?
Maybe a gist of full shell output?

[restive sapphire]

yep, host is Windows running WSL along with Docker for Desktop's WSL integration (like recommended in the getting started warehouse guide)
there's a lot of output from all the other containers, but here's the relavent part:

stripe-1        | stripe-mock 0.162.0
base-1 exited with code 0
files-1         | Serving HTTP on 0.0.0.0 port 9001 (http://0.0.0.0:9001/) ...
opensearch-1    | Disabling OpenSearch Security Plugin
opensearch-1    | OpenSearch Performance Analyzer Plugin does not exist, disable by default
db-1            | COPY 554804
stripe-1        | Routing to 305 path(s) and 452 endpoint(s) with 452 validator(s)
stripe-1        | Listening for HTTP at address: [::]:12111
stripe-1        | Listening for HTTPS at address: [::]:12112
notgithub-1     | INFO:     Started server process [1]
notgithub-1     | INFO:     Waiting for application startup.
notgithub-1     | INFO:     Application startup complete.
notgithub-1     | INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
notdatadog-1    | Listening on udp 0.0.0.0:8125. Displaying metrics: False
db-1            | COPY 32338
static-1        |
static-1        | > watch
static-1        | > webpack --watch
static-1        |
Error response from daemon: unable to find user nobody: no matching entries in passwd file

though it looks like the camo service isn't necessary for the feature i'm working on, so it's not imperative I get this fixed but I'd still like a fully working setup

[tribal sedge]

Apologies for the indirection - the source for the camo used in PyPI dev is https://github.com/pypi/camo/blob/6d05634a52d60d40937c073878c451590097d062/docker/Dockerfile#L17 - not the atmos repo (don't ask, it's a tad messy)
I don't think any of us admins have a Windows machine handy to play with to reproduce.
I am able to execute this command and get a positive result:
docker run --rm --entrypoint="" pypa/warehouse-camo:2.0.0 grep nobody /etc/passwd

[wicked wind]

I finally published my PEP 694 update and restarted a DPO discussion: https://discuss.python.org/t/pep-694-pypi-upload-api-2-0/76316

[unborn nacelle]

i don't know if this is actionable, but i've been looking into the network performance of the uv resolver some more.

the uv resolver mainly needs the list of versions/files, and for that uv has effectively three main modes: cold cache, revalidation requests and warm cache. when using uv lock or uv run , users will usually run the warm cache case (no network requests). when running uv lock --upgrade, uv add, uv pip install or a PEP 723 script (without lockfile), they'll often get the revalidation case: we have a response cached, but it's older than 10min (or we refresh for upgrading), so we want to check if there was a new release/new files since. we send a http request with the usual caching headers and most of the time get a 304 back.

we have to send one revalidation request per locked dependency, and it's both the limiting step on the uv side and we're also firing a lot of requests at pypi. basically, i want to ask pypi "for these n packages, has there been an update since timestamp(s)?" or "here are n package names with an etag each, re they still fresh?" with the least overhead possible. it feels somewhat wasteful to send so many http requests for that.

example visualisations of the requests, same payload but two different scheduling strategies. the x axis is time (330ms and 180ms), y is parallelism and each blue bar is a revalidation request.

the performance of those requests varies a lot, so i'm seeing something between 150ms for 1.1s for my same test project

(i can provide reproduction commands if helpful)

[honest flame]

On pypi side I can imagine that a 304 is quite cheap as it can usually be answered from the CDN or FS metadata cache, but a specialized API that checks N files requires logic on servers side, cannot be reasonably cached and adds quite a lot of overhead. HTTP/2 should also reduce tht overhead of multiple requests to a degree that there is very little actual cost per request. I'm not familiar with the pypi infrastructure but my gut feeling is that such an 'optimization' could end up being a net-negative at scale, because the time a single client needs for revalidation is less important than the server side resources needed to serve a large number of clients.

[mortal silo]

In our case (Qiskit, we submitted the request in Apr 2023) is making our control access messy. Because second factor authentication, we have only one developer that can delete and yank releases, which is something that many times we need to do quickly.

I'm happy to help with the backlog. Is there a process I should do to contribute there?

[merry valve]

is it not possible to add multiple owners to the project? or if you can only have a single owner, add multiple security keys or share the TOTP setup across multiple TOTP authenticators?

[mortal silo]

we are currently using the multi owner strategy. However, it is hard to remove that privilege when a developer is not part of the project any more.

[mortal silo]

What's the process to help with the organization backlog approval? Sounds like something that can be distributed. I'm having some extra time these days. How can I help?

[latent forge]

I submitted a request to create an organization on TestPyPI, how do I change the type of organization from company to community?

[merry valve]

email [e-mail], be sure to include the organization name 🙂

[proud bison]

Don’t hold your breath though. We submitted a request almost 2 years ago and haven’t heard a peep

[faint arch]

Hello - we got kind of caught out by the recent anti-scraping changes. We were using the search functionality to find packages with a specific classifier. Rather than trying to subvert the anti-scraping measures, we decided to rely instead on the BigQuery metadata results. Unfortunately these seem to be missing data - I believe it's the same issue reported in https://github.com/pypi/warehouse/issues/16008 (I also commented there with a specific example).

The XML-RPC browse function does pretty much exactly what I'd like, but the docs all seem pretty discouraging towards using it. Does anyone have thoughts on what we should do?

[wicked wind]

I had to switch some of our scripts to the XMLRPC API. It seemed like the least worst option.

[faint arch]

Thanks - I'm going to propose this at least for the time being.

[frosty chasm]

hi PyPI team. i wondered if someone could help me understand warehouse docs linting setup. I need to lint this pr: https://github.com/pypi/warehouse/pull/17382

i can run make lint
But for instance what would i run to trim trailing whitespace? I know how to do this with precommit but i must be missing docs around how to apply actual lint suggestions. I'm using these docs:

https://warehouse.pypa.io/development/getting-started.html#running-tests-and-linters

➜ make lint
docker compose run --rm base bin/lint
bin/lint: line 7: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
+ find . -name '*.py' -exec python -m pyupgrade --py312-plus '{}' +
+ python -m flake8 .
+ python -m black --check --diff gunicorn-prod.conf.py gunicorn-uploads.conf.py sitecustomize.py warehouse/ tests/
All done! :sparkles: :cake: :sparkles:
780 files would be left unchanged.
+ python -m isort --check gunicorn-prod.conf.py gunicorn-uploads.conf.py sitecustomize.py warehouse/ tests/
Skipped 2 files
+ sphinx-lint --enable=all --disable=line-too-long README.rst CONTRIBUTING.rst docs/dev/ --ignore=docs/dev/_build/
docs/dev/development/getting-started.rst:78: default role used (hint: for inline literals, use double backticks) (default-role)
....
docs/dev/development/getting-started.rst:64: trailing whitespace (trailing-whitespace)
....
make: *** [lint] Error 1

[glossy turret]

Ah yeah I had this issue, I think the dev docs should probably add a note about adding locales.

You should be able to run

sudo locale-gen "en_US.UTF-8"
sudo dpkg-reconfigure locales

If you are on a Debian or Ubuntu system. I think something similar is available via localectl on rpm based systems

[restive sapphire]

make reformat, I believe should make the changes for you via black and isort and such

[merry valve]

unfortunately these are errors found in .rst documents that sphinx-lint is catching, but make reformat won't autoformat, which is annoying

[restive sapphire]

ah, noted

[tribal sedge]

The locale message is a bit of a cargo-cult, and we probably need to remove those from the scripts

[tribal sedge]

@frosty chasm warehouse has an editorconfig file that tells editors with a plugin how to behave, often on file save.

I saw that @merry valve fixed them and merged, thanks!

[frosty chasm]

hi everyone 👋 thank you for all of the help here. that editor config file is really helpful!! @di mentioned setting up my editor locally to atleast handle trailing white space on save. this should be easy enough to implement especially seeing that plugin that i can use. WOW!! It could be helpful for folks to understand they will have to handle linting on their own in the docs and maybe to highlight that plugin tool. i'm happy to help with another small pr if you all are open to it. i didn't know about that plugin!

[tribal sedge]

A section on editor setup might be helpful to folks, sure!

[frosty chasm]

Fantastic. I'll get my own editor set up to see how it works (I think it's quite straightforward!) and will submit a short overview to the docs. thank you @tribal sedge !

[pliant obsidian]

EditorConfig is great, especially being able to have different indents for different file types, and many editors have built in support and there's often a plugins for the others. I found out recently that Trey Hunner came up with the idea for it!

[fallow merlin]

Hi Guys, We are facing issue to access pypi from our organization network, We have included our networking team but they said that the issue is from pypi blocking our ip.
We tried to open issue on github but i think its in queue. so is there any support email or anything with whom we can connect and expedite it.
also please guide me to correct channel if this isn't the correct one

[merry valve]

Can you share the GitHub issue you made?

[fallow merlin]

https://github.com/pypi/support/issues/5466

[merry valve]

You're missing a lot of the commands we ask people to run for those issues, can you try filling out this issue template and running all the listed commands? https://github.com/pypi/support/issues/new?template=access-issues.yml

[fallow merlin]

sure

thank you

[fallow merlin]

https://github.com/pypi/support/issues/5510
a new one

[mortal silo]

I would like to bump this. I really would like to help with the organization request vetting backlog. Sounds like the kind of task that I enjoy plus a bottle neck for the project. Any link on how to make the first steps towards that?

[obtuse torrent]

There was announcement made about it very recently. They're starting to let select companies in and go through community org backlog

https://discuss.python.org/t/state-of-pypi-organizations/33764/25

[dry remnant]

Thank you for sharing this link - I was just looking for a status update on this 🙂

[finite pulsar]

PSA: https://blog.pypi.org/posts/2025-01-30-archival/

[violet fable]

there is an #announcements channel :P (although it's probably role-gated)

[valid flame]

The archival doesn't affect the APIs in any way, do I read that right? (just double-checking)

[merry valve]

nope. we're discussing how best to include the arcihival status in the various APIs, but it won't change the presence of a project in an API

[finite pulsar]

yeah, someone who has the role might need to do it, i can't 😅

[valid flame]

Thanks

[merry valve]

am I missing something or does this channel not work? I'm not super familiar with discord

[violet fable]

Addressed in thread ^. Anyway, this is quite cool!

If your project is configured to do so ...
Is a bit confusing as I thought there wasn't anything to configure to use a README, but then I clicked the link, and that's what it's explaining :)

[unborn nacelle]

nice work! excited to see this in the api, too

[finite pulsar]

Yep! I’m going to open up a PEP for the API side pretty soon!

[vast halo]

Very cool!

/me logs into PyPI and archives all her old projects that are highly unlikely to ever see another release

Heh, one of them was so outdated that the source repo link is dead, and I'm not sure the code ever migrated anywhere else.

[tribal sedge]

@finite pulsar at some point we should follow up with some stats on adoption of projects archived and their ages

[vast halo]

Two of them in fact. I wonder if those links died when Atlassian killed BitBucket's Mercurial support.

[hasty salmon]

It would be really really nice if pip install eventually emitted some sort of warning about archived (and eventually unmaintained ) projects, I’ve seen far too many beginners try to install packages that haven’t had any updates for decades at this point

[pliant obsidian]

yes, that's the next step and is being worked on, see "Where do we go from here?" at https://blog.trailofbits.com/2025/01/30/pypi-now-supports-archiving-projects/

[tribal sedge]

Also a reminder that just because a project hasn't been updated in decades doesn't mean it's not still good and working

(but many are, so I get it)

[vast halo]

Yep, there's a reason contextlib2 (for example) didn't get archived. I'll eventually get around to syncing it with the latest stdlib code, but in the meantime, what's already there all still works fine (the stdlib changes since the last sync have been feature additions rather than bug fixes).

[ancient compass]

There's a bug fix that's needed for 3.8 in ExitStack I think

Oh no it was triggered by contextlib.ExitStack but internal to Exception https://github.com/python/cpython/issues/69968

[vast halo]

Even though the last sync was in 2021, that was still recent enough to be aligned with 3.10. We've veered well off topic for this channel, though 🙂

[hasty salmon]

yeah i agree, but beginners in particular tend to find the broken decades old libraries 😆

[restive sapphire]

would love to get some reviews on pypi/warehouse#17352 😄

[pliant obsidian]

and here's the next step! https://discuss.python.org/t/pre-pep-discussion-project-status-markers-in-the-index-apis/79356

[chrome zealot]

So a topic I have no specific goal in mind in raising -- I was trying to teach new programmers yesterday how to upload a package to PyPI by doing so with TestPyPI, and I was left feeling that that's a bit more challenging than it could be

Specifically what makes it kind of awkward to do with folks is the need for real email (because otherwise you can't finish verification) and worse the need for 2FA

I have 0 qualms that of course these are great things for real PyPI

But I was/am left wondering whether anyone has thought about this a bit recently -- it worked out that I spent ~10 minutes walking people through "irrelevant" bits and then 5 minutes doing the real thing which is "hey run this command to get a package, and now look your thing is on the internet!"

(of course another reasonable response is "this isn't something that's very worth teaching to such new people, so it's fine that that's the distribution of how long it takes")

[merry valve]

these folks didn't already have PyPI accounts, correct? the issue here is "PyPI/TestPyPI's registration process is arduous", not "people can't upload throwaway projects to PyPI", yeah?

I have been thinking about the future of TestPyPI a bit (in the context of introducing "draft" or "staged" uploads: https://github.com/pypi/warehouse/issues/17230#issuecomment-2643839592) but I don't think what I'm proposing there would really solve your problem if that's the case

some ideas in the short term: you could have them upload to your own devpi instance? (not as cool as uploading to PyPI/TestPyPI, I'll admit). or they could all share the same credentials that you prepare in advance, but upload different project names?

[chrome zealot]

Correct! (The former, it's arduous)

I'll have a look at that issue, I forgot to look back here after sending those messages

[chrome zealot]

FWIW I am 100% ok with there being nothing to do in the short term

Not least of which because I won't give this lecture again for.. well I'll give it again in April but then not till September or something 😄

Preparing creds in advance is a great idea actually

[wintry cloak]

Hi all,
I've got a couple of questions around deleted projects.

1. is there a way to know wen a pypi project was deleted?

2. are deleted projects protected from being "revived" without submitting a support request? (I thought they were but maybe they are not, or aybe there is a bug).

Context for those that want more info. For better or for worse I was using the msgspec-python313-pre package. It appears that at some point it was deleted and subsequently revived by a security researcher. This however means that during the period it was deleted it would have been vulnerable to supply chain attacks.
(and in fact if deleted projects can easily be revived many projects would be vulnerable should some common dependency be deleted and go unoticed).
Only place I can currently observer history is https://libraries.io/pypi/msgspec-python313-pre/versions but this only inlcudes uploaded versions.

[pliant obsidian]

2. is answered at https://pypi.org/help/#deletion

Deletion of a project makes it uninstallable, and releases the project name for use by any other PyPI user.

[tribal sedge]

1. is an open question - subscribe and join in on https://github.com/pypi/warehouse/issues/11841

[formal jolt]

I just hit a surprising trusted publishing token error? https://github.com/astral-sh/uv/actions/runs/13707864294/job/38338659361

DEBUG Upload error response: {"message": "Access was denied to this resource.\n\n\nInvalid API Token: OIDC scoped token is not valid for project 'uv-build', project-scoped token is not valid for project: 'uv-build'\n\n", "code": "403 Invalid API Token: OIDC scoped token is not valid for project 'uv-build', project-scoped token is not valid for project: 'uv-build'", "title": "Forbidden"}

It looks configured correctly to me though?

The change is that we're publishing uv-build alongside uv

Suspiciously, the uv publish failed with a 502 immediately after

Perhaps it's because they run concurrently? Otherwise this sounds supported?

Trusted Publishing's design accommodates this use case: maintainers can use the same release.yml workflow for all of their packages, rather than having to split it up by packages.

[merry valve]

what is this doing?

DEBUG Querying the trusted publishing OIDC token from https://pipelinesghubeus11.actions.githubusercontent.com/TWB5keMfCEFP61AxCXHhRpsUPwe13hqsXvfwaVD5qIZiIMY0QA/00000000-0000-0000-0000-000000000000/_apis/distributedtask/hubs/Actions/plans/f9fc176e-7384-4b34-a12e-acd02ee9f12a/jobs/13c06ef3-10ae-5644-64df-f7c0258a361f/idtoken?api-version=2.0&audience=pypi

is that just ACTIONS_ID_TOKEN_REQUEST_URL?

@formal jolt looks like the workflow filename for uv-build's trusted publisher is release.yml but that job run is publish-pypi.yml (and that's what the trusted publisher for uv is also configured to use)

[formal jolt]

That runs as a subworkflow of release.yml

[merry valve]

a reusable workflow? if so: https://github.com/pypi/warehouse/issues/11096

[formal jolt]

I can only presume, I can double check.

[merry valve]

I think I've just never actually seen what these URLs look like in practice 🙂

[formal jolt]

Let me see if that changed...

Sort of running around because this means we partially published uv

So our release is broken rn

[formal jolt]

We were already using a subworkflow https://github.com/astral-sh/uv/pull/11446/files#diff-d4585bdbbb00b46a400628c271f195eb312333d18474299152a0b75fb1a7ec4b

[merry valve]

@formal jolt you might want to output some claims in the debug output here, this is what the PyPA action outputs: https://github.com/pypa/gh-action-pypi-publish/blob/db8f07d3871a0a180efa06b95d467625c19d5d5f/oidc-exchange.py#L165-L184

[formal jolt]

I can look into that next

I am very suspicious of the fact that this broke the other release, unless that's a fluke?

[merry valve]

afaict the 502 is just a fluke

[formal jolt]

Wild timing

Hm

[merry valve]

but it does seem suspicious

[formal jolt]

Anyway, I yanked the release because it was missing the sdist and a bunch of wheels. I'll try again with a sequential publish and see what happens.

Partial publishes are such a nightmare 😄

[valid flame]

couldn't you just add the missing files later?

[formal jolt]

Yes, but in the meantime lots of people's workflows will break

[merry valve]

I suspect changing the publisher from release.yml to publish-pypi.yml for the publisher to match uv might fix this

[formal jolt]

Oh is it different for uv? Will check that

I see it is

Thanks ❤️ I'll try that too

Looks like that's working, appreciate the help @merry valve

Glad it was just a silly user error and not some terrifying bug haha

[valid flame]

hot take: pypi should just have a form to create new release manually, by adding files with drag'n'drop

[merry valve]

np! looks like we had a few bursts of 502s today at the CDN level, so best guess is fastly is experimenting with us 🙂

totally agree that the error message could be improved here

[formal jolt]

The timing of the 502 and the error message definitely sent me down the wrong track

[valid flame]

something to note for the upload api 2.0 I guess

[rustic venture]

uploads don't traverse CDN!

@formal jolt do you have approx timing of your 502s? i'm curious to look closer. uploads endpoint has been behaving fine as far as metrics go.

502 bad gateway is almost certainly our fault 🙂

got timing, taking a look

def had an upload backend go out of commission around the time of your upload. investigating why. it obviously recovered itself... but gonna investigate.

[formal jolt]

I presume you found it, but https://github.com/astral-sh/uv/actions/runs/13707864294/job/38338658887
2025-03-06T21:11:30.5818946Z DEBUG Response code for https://upload.pypi.org/legacy/: 502 Bad Gateway

[rustic venture]

yep, and it lines up on the nose with your 502 😭

we don't like 502s, so i'll put some eyes on how often we're seeing them for uploads. does uv publish not retry?

[formal jolt]

I guess not? I would think it'd use the same retry logic we use elsewhere though

[rustic venture]

heh i also noticed the "uploading..." then immediately "downloading...", what's up with that? (now i'm just genuiney curious)

[formal jolt]

So, reqwest-retry should handle a 5XX but we do something weird? https://github.com/astral-sh/uv/blob/3b83b48fd27fc224c5fd534f88a2c0fb28ffdd06/crates/uv/src/commands/publish.rs#L47-L49 maybe a bug there

[rustic venture]

anywho, i'd suggest making sure uv publish retry at least a lil! twine does and i suspect that papers over a lot of transietns like this on our side.

[formal jolt]

cc @unborn nacelle for when you're online tomorrow

Interestingly, we log that as a transient error too? DEBUG Transient request failure for: https://upload.pypi.org/legacy/

[formal jolt]

Honestly really confused by that too

I think it's just a reporter bug https://github.com/astral-sh/uv/blob/4611690745c2358aaec407bcd620ac1255520255/crates/uv-publish/src/lib.rs#L754

Thanks for pointing that out

[rustic venture]

i assume uv publish is in enough use that its nice to know it hasn't bitten more people 😄

[formal jolt]

The confusing reporter output or the retries thing?

[rustic venture]

Retries. I know upload endpoint is stable, but not assume it will always work stable.

[formal jolt]

Yeah I'm quite surprised by that, it looks like it's never worked to me?

[obtuse torrent]

Can maintainer role on a project yank releases? I am hoping to limit the perms of our core devs to maintainer role so that it's not possible to delete projects but yanking releases is needed.

For clarity, the reason it's not clear for me is because it's not mentioned in the help text on the collaborators page of the project's settings, perhaps it's documented somewhere else but I haven't looked yet.

I'm also unsure whether a project owner (say, an org team set as such) can delete a project when the member is a "Manager" in the org. Similarly, does "Manager" automatically mean that the user can do the things listed on the org collaborators settings help text for all projects?

[pliant obsidian]

no, only an owner can yank. maintainers can't click the "Manage" button where the yanking happens

I guess this section covers it, and yanking could be added to make it explicit: https://pypi.org/help/#collaborator-roles

What collaborator roles are available for a project on PyPI?
There are two possible roles for collaborators:

Maintainer: Can upload releases for a package. Cannot add collaborators. Cannot delete files, releases, or the project.

Owner: Can upload releases. Can add other collaborators. Can delete files, releases, or the entire project.

[pliant obsidian]

org roles are documented here: https://docs.pypi.org/organization-accounts/roles-entities/

Manager: A manager can manage a team, assign permissions for a team and manage a project

I think "manage a project" covers yanking?

[obtuse torrent]

Delete is destructive while yanking isn't so that's why I wasn't sure

[obtuse torrent]

The table explicitly lists that deletions are not allowed so it's unclear if "manage a project" covers yanking, since for individual projects it doesn't

[pliant obsidian]

yeah, would be good to add yanking to the table to make it clear

[merry valve]

@obtuse torrent you may want to comment on https://github.com/pypi/warehouse/issues/12643 !

[obtuse torrent]

Would you be able to tell me if the same problem exists for maintainer role in orgs?

I will look at the issue later as well

[merry valve]

should be the same but I can double check

[agile sinew]

Yes, just saw that when setting up scikit-hep. Org maintainers can't yank. IMO, anyone that is allowed to make a release should be able to yank it, that's bit me with cmake and others before.

I just came here to add a bit of org feedback - when you "move" to an org, you are removed from the collaborators. This isn't too bad if you are owners in both (I still added myself back so I stay in the maintainers list on PyPI's page), but it causes a weird sitatuation if you are a project owner but an org maintainer. Then you lose ownership (including yanking ability, which is the first thing the person who did this noticed) and an org maintainer (me) has to add them back as owner again.

I think at least in the case of mismatching permission levels, moving shouldn't remove the mover from the project.

[merry valve]

yeah, that's https://github.com/pypi/warehouse/issues/13558 🙁

[obtuse torrent]

Can the person added as project owner delete the project if it's owned by an organization and that person isn't an owner of that org?

[agile sinew]

I'd assume so, that's what "owner" means

[obtuse torrent]

Sure but org itself is in some way different since you have to transfer a project to an org.

You can't just add org as owner.

I'm trying to compare between GitHub orgs/repos and PyPI orgs/projects. The way it's shown, ownership model seems different on PyPI though since a project can have multiple owners, there isn't a singular owner entity (org or user) like on GitHub but the fact that you have to transfer suggests that there is some special treatment here, perhaps not permissions-wise though

[merry valve]

the main difference is that moving a project to an org essentially adds all org owners as de-facto project owners. no permissions change for the existing project roles.

[agile sinew]

There doesn't seem to be any sorting for the list of projects in an org.

[merry valve]

I think you're right, can you file an issue?

also, what would you expect: alphabetical ordering or most-recent to least-recent activity?

[agile sinew]

I'd be happy with either 🙂

Most recently added would have been useful in this specific case, since I was having to re-add people who've transfered in and lost ownership. That's why I noticed they weren't sorted via anything obvious

Would that be a bug report or a feature request?

[merry valve]

feature request please!

[dreamy hawk]

did something change? My latest project (uploaded with Trusted Publishing) did not get a "GitHub Statistics" section

An older one for comparison:

[merry valve]

this only happens if you add the repository URL to the project metadata, e.g.: https://inspector.pypi.io/project/atomicwriter/0.2.4/packages/b9/a0/62272b7793f306940b4c994ac05edbfd8a1986705511c678c76842bd14db/atomicwriter-0.2.4.tar.gz/atomicwriter-0.2.4/pyproject.toml#line.26

[dreamy hawk]

Oops, missed that. Thank you!

[quaint ocean]

Were there any packages stolen late last year? As in, randomly transferred from older accounts? Because I know pypi doesn't really reuse package names

Someone on Python Discord wanted to use package they used in the past, but the version is "0.0.1a0" uploaded in December.
They provided readme from github to show that they didn't mess up the name https://github.com/clips/pattern/blob/d25511f9ca7ed9356b801d8663b8b5168464e68f/README.md?plain=1#L55

This is how it's now https://pypi.org/project/pattern/
Owned by https://pypi.org/user/sanepunk05/ - who has only same copy-paste packages (only description changed, readmes all show wrong name), all uploaded the same day.
Inspecting it from Inspector doesn't suggest anything malicious, so I'm confused for the reason.

Because afaik pypi doesn't reuse the names, so that one must've been a transfer... right?

[tribal sedge]

The Pattern project was removed by owner late 2024, making the project name available for reuse.

pypi doesn't reuse the names
PyPI disallows using the same filename - see https://pypi.org/help/#file-name-reuse

[glossy turret]

Is it possible to generate a sigstore bundle with the in-toto predicate then publish it later via trusted publishing? At $dayjob we'd like to adopt trusted publishing and attestation generation. I'd like to generate a sigstore bundle we can use for our own index that doesn't support PEP 740, and use the same bundle to generate the attestation used during upload. Is that possible? And can these be independent jobs (i.e. one signs a wheel, the other converts the attestation and uploads)?

I've read over https://docs.pypi.org/attestations/producing-attestations/, but it's not clear to me if there are any requirements on the relation between the signer and publisher jobs.

[stable hill]

Since it seems like the issue tracker isn't for questions: what exactly determines whether PEP 658 metadata is available for a project? It seems like it isn't automatic for new sdist uploads, even when they contain a conformant PKG-INFO with all static metadata. For example, I was able to get a .metadata file for the most recent Requests wheel, but not for the sdist.

[merry valve]

it's only currently stored for wheels

[merry valve]

sorry for the late reply. there is a validation step that happens where we check that the identity that signed the attestation is the same as the trusted publishing identity. that happens here: https://github.com/pypi/warehouse/blob/35ccde8477a391cac01a35c82b0ece873cb1d4bf/warehouse/attestations/services.py#L160-L176

[glossy turret]

Thank you so much! I'll take a look

[stable hill]

I see, so in practical terms I'd still need to do range requests etc. to resolve dependencies behind an sdist to get at version 2.1+ metadata, even if it is there...

(I don't really plan to support the legacy use cases where it isn't)

[glossy turret]

I will point out that tar files don't support random reading like the zip format. tar files are a sequential concatenation of file entries with no directory entry giving the full listing/offsets. Therefore you may need to read up to the full length of the sdist to actually get to PKG-INFO. You could try to optimize this by calling Tarfile.next(), but I just looked at the sdist for numpy as an example and you'd read pretty much all 20MB of the sdist.

[stable hill]

oh, ouch. Does Pip actually do that? I did see there's a bit of code that tries to iterate backwards with range requests (and I guess search the buffer for the header of a compressed entry?) on the theory that the PKG-INFO will be physically near the end of the archive (since there are documents recommending that build backends arrange for that, IIRC)

but I'd mentally associated that with pre-PEP wheels.

[brittle ravine]

To whom it may concern: Just wanted to note my thanks for approving the organization I requested (RBVI)

[stable hill]

Confused about the "conveyor service" for predictable URLs. As a demo, results truncated:

$ curl -I https://files.pythonhosted.org/packages/py2.py3/r/requests/requests-2.12.0-py2.py3-none-any.whl
HTTP/2 302 
content-type: application/octet-stream
location: https://files.pythonhosted.org/packages/00/93/9c5c04821578c2ee11af83189c5cbd8338724b5e04e1de5dc3643bbc5bbf/requests-2.12.0-py2.py3-none-any.whl
...

$ curl -I https://files.pythonhosted.org/packages/py2.py3/r/requests/requests-2.11.0-py2.py3-none-any.whl
HTTP/2 404 
content-type: application/octet-stream
reason: no file found
...

Is this just only available for certain packages? There should be a such a wheel for 2.11.0 as well, per e.g. https://pypi.org/simple/requests

(Sorry about the Discord linewrapping. The first URL is part of the command in each case; everything after that is output)

[meager umbra]

Looks like the metadata recorded in PyPI for version 2.11.0 is messed up. https://files.pythonhosted.org/packages/2.7/r/requests/requests-2.11.0-py2.py3-none-any.whl works.

[rustic venture]

yeah, indeed the py2.py3 wheel has "python_version" as 2.7

@stable hill i'm always curious when people are using that "API", why that is. it is much better to just query the json api directly to find release files. though in this case (bad metadata) you're probably still gonna have a bad time :/

[stable hill]

I thought it would be simpler to calculate. But given that you apparently have to know the wheel contents to do it that way, I'll just use the JSON API. Thanks for the explanation

(IMO the documentation at https://docs.pypi.org/api/#predictable-urls needs to be clearer about the use case, given this.)

[rustic venture]

clearer about what specifically? i'd love to improve it.

[stable hill]

i.e. that it isn't meant for just translating a name, version and wheel tags into a usable download URL

[rustic venture]

all i can really come up with is adding admonition above it saying what's said in the heading right before "Official guidance

Query PyPI's Index API or JSON API to determine where to download files from."

[stable hill]

or just describing that the python_tag value needs to match what's inside the wheel, not its filename

(as the example code tries to just reuse a passed-in python-tag value)

[rustic venture]

the spookiest part is... who knows what's in the file... since PyPI has always accepted what people sent over the wire in the form

[stable hill]

indeed. Have there been complaints from e.g. Linux maintainers about it not always working?

[rustic venture]

totally!

[stable hill]

Ah, well. I guess this is the place to forewarn them :)

[rustic venture]

more often than not it is changes in capitalization, or bad metadata like this :/

though i think what you've flagged here is actually probably a sublte bug

[stable hill]

case sensitivity is also a real annoyance when trying to support both Linux and Windows, yes. :(

[rustic venture]

https://github.com/pypi/conveyor/blob/de7f427e5a57e97a2097a3698167db4552a50f93/conveyor/views.py#L121-L127

i don't think that if is working as expected

the compat part anyway. it's failing on the file_["python_version"] == python_version

i guess we could just hand over the file if the name matches exactly (basically add the same "or" to the second and

[stable hill]

Ah... I suppose this is an echo of PEP 625, in a way. Except with wheels. Either way the in-artifact metadata isn't matching what the filename advertises. (And Pip will install it anyway, I'm sure....)

[rustic venture]

dunno, conveyor is a classic "built for one purpose and a short duration mission maintained for nearly a decade" situation

😦

it was just supposed to ease the transition from serving off of a file system to serving from object storage

🙃

now everytime it breaks we get yelled at

(not you, you're politely talking)

[stable hill]

It does seem like there are a lot of those things, and they're fundamentally what makes things difficult to improve.

good ol' Hyrum's (? IIRC) Law

(for context: I'm making an installer. I'm essentially starting over, and ignoring the hard use cases initially, and focusing on small modular design. Ideally, making the hard cases work will be plugged in to the base framework.)

[rustic venture]

that's very helpful. i'll say beyond explicitly that you should use the JSON based Simple API

nothing else you're going to find (aside from the HTML Simple API) is specified in a PEP and thus supported in the same way

[stable hill]

Makes sense.

[proud bison]

me too! thank you for making the scverse org, this will help us trememdously with bus factor concerns

[sturdy granite]

hi there

does anyine have a clue how to solve this Python SSL bug on MacOS ?

Looking in indexes: https://pypi.org/simple/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(50, 'Network is down'))': /simple/flask/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(50, 'Network is down'))': /simple/flask/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(50, 'Network is down'))': /simple/flask/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(50, 'Network is down'))': /simple/flask/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError(50, 'Network is down'))': /simple/flask/
ERROR: Could not find a version that satisfies the requirement flask (from versions: none)
ERROR: No matching distribution found for flask

I suddenly destroyed my setup and I cannot do anything about python and is frustrating

[valid flame]

it looks more like your Mac being disconnescted from the internet

[sturdy granite]

I can do everything except installing python dependencies from pypi. I wouldn’t be able to open Discord without internet

HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/ruff/ (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x107e88190>: Failed to resolve 'pypi.org' ([Errno 8] nodename nor servname provided, or not known)

[oak mica]

You're going to have to find a Python or Mac troubleshooting forum, maybe https://discuss.python.org/c/help/7

[violet fable]

could you try pip config debug, you may have a proxy configured system-wide that's inaccessible

(proxy error messages are stupid half of the time)

[sturdy granite]

 pip3 config debug
env_var:
env:
global:
  /Library/Application Support/pip/pip.conf, exists: False
site:
  /Library/Frameworks/Python.framework/Versions/3.11/pip.conf, exists: False
user:
  /Users/insidious/.pip/pip.conf, exists: True
    global.index-url: https://pypi.org/simple/
    global.trusted-host: pypi.org
  /Users/insidious/.config/pip/pip.conf, exists: True
    global.index-url: https://pypi.org/simple/
    global.trusted-host: pypi.org

curl https://pypi.org
curl: (6) Could not resolve host: pypi.org

pip3 install -r requirements.txt returns these errors:
ERROR: Could not find a version that satisfies the requirement flask (from versions: none)
ERROR: No matching distribution found for flask

I'm having hard time understanding why is it looking at simple, but not root pypi.org

[violet fable]

simple URL? that's the simple API that clients like pip use to interact with indices like PyPI

you should look into why curl can't even resolve https://pypi.org. That suggests a larger problem than just pip or even Python.

[sturdy granite]

nslookup pypi.org
Server:         192.168.0.1
Address:        192.168.0.1#53

** server can't find pypi.org: SERVFAIL

[violet fable]

Unfortunately, my troubleshooting knowledge generally stops here. Networking is my not forte.

You could try using a non local DNS resolver, like Cloudflares's (1.1.1.1) although it's weird your local DNS resolver can't resolve pypi.org (enterprise firewall?).

[sturdy granite]

No problem, I will get through everything you have shared, I appreciate for your help!)

[sturdy granite]

home, just trying to do some simple local backend 🤕 can't even run hello world

[ancient compass]

The 503 page seems to link to twitter still

[rustic venture]

thanks! will get that fixed up once things settle