#pip

1 messages · Page 1 of 1 (latest)

limber ore
#

what version did pip start supporting self-referential extras? e.g.

[project]
name = "pkg"

[project.optional-dependencies]
all = ["pkg[a]", "pkg[b]"]
a = ["..."]
b = ["..."]
hidden flame
#

I'm disappointed I don't know the answer to this question off the top of my head as the pip issue tracker stalker I am :p

#

I'll go check.

hoary mist
#

probably with the resolver if I had to guess

hidden flame
#

pip 20.2 and before simply ignore the [self-referential] extras (only testing 20.2, since I'm on macOS 11)
pip 20.3 to 21.1 fail by looking at all older, online versions
pip 21.2 & 21.3 work correctly.
from @willow flicker https://github.com/pypa/pip/issues/10393#issuecomment-941885429

GitHub
Pip doesn’t allow you to self-depend · Issue #10393 · pypa/pip

Description When specifying dependencies, you sometimes want to depend on other extras: [project] name = 'my_pkg' [project.optional-dependencies] test-utils = ['pytest&#...

hidden flame
limber ore
#

thanks!

hidden flame
#

I can't find a changelog entry mentioning self-referential extras so technically it was just the byproduct of some other resolver changes

limber ore
hidden flame
#

good call 👍

mint patrol
#

Hey folks, I'm doing a write-up of how people can test the new truststore feature to give us lots of data points. Unfortunately there's a chicken and egg scenario at play! If someone can't install pip and truststore from PyPI because they have a corporate proxy certificate how do they receive the update safely without having to manually do a bunch of stuff.

lunar gyro
ashen geyser
ashen geyser
#

Quoting myself from work Slack:

pip install -U pip has been the solution of so many installation problems; it’s the “have you tried turning it off and on again” of Python

shy echo
#

:)

foggy forum
shy echo
#

What about it? :P

foggy forum
#

I was surprised to see support was already there! I tried it out and found

shy echo
#

Newer versions of setuptools have it. :)

foggy forum
#

Whaaaat

shy echo
#

Check the end of the issue.

foggy forum
#

https://discuss.python.org/t/help-testing-pep-660-support-in-setuptools/16904?u=abravalheri

Discussions on Python.org
Help testing PEP 660 support in setuptools

Hello all, We have been working for some time with the editable installs according to PEP 660 in setuptools, and I think we are close to the point we can start thinking about a release. If anyone is interested in “beta”-testing and giving feedback, it would be very appreciated! (My idea is to use the same procedure we used for PEP 621: invite ...

foggy forum
shy echo
#

Yes, but it works with setuptools still -- assuming you have a placeholder setup.py?

foggy forum
#

Oh yeah with the placeholder

#

But if you don't have a placeholder it doesn't work

#

What I'm saying is I don't understand what you're trying to say in your tweet

#

Not that you are wrong

shy echo
#

I'm saying other than this setuptools mess, everything works.

foggy forum
#

Yeah that's what I thought you meant

shy echo
#

The only configuration that doesn't work is trying to do an editable install with setuptools without a setup.py file.

#

Using flit / poetry or anything else works.

#

Using setuptools with a placeholder setup.py works.

foggy forum
#

Yeah it all works great dunno why people are so tilted about it

shy echo
#

And, now, with the next setuptools release, even setuptools without a setup.py will work.

#

I don't know.

#

I also don't know why they are complaining that python -m build doesn't generate an editable wheel.

foggy forum
#

I mean maybe you want that in tox

#

Eg tox might want to build an editable wheel once and install it editiably in different places?

#

Probably not though

shy echo
#

In that case, tox is lacking support for modern Python packaging standards, not build. See also tox 4. :)

foggy forum
#

Yeah I mean the usecase is what? build an editable wheel then send it over the network somewhere and install it editiably there?

shy echo
#

It doesn't make sense.

foggy forum
#

I mean it's a cool trick

shy echo
#

Like, it has paths and stuff, from the machine it was built on.

#

It's stupid, and the fact that it's a wheel is purely an implementation detail.

foggy forum
#

Tbh it might work for dask/distributed

#

yeah so you build an editable wheel on one dask worker then install it into the rest of them

shy echo
#

The user will/should never see the wheel.

#

And, by user, I mean anything that's not a packaging tool -- a build frontend or build backend.

shy echo
#

Otherwise, this won't make any sense.

foggy forum
shy echo
#

Again, I'm not sure why you want to even try that.

#

IMO it doesn't make sense to think of "editable wheels" as a thing you want to use ever.

foggy forum
#

Well

shy echo
#

It's a transient file, for pip to get from setuptools/flit whatever to unpack for development workflows.

#

Anything else related to distribution -- use a wheel that's actually meant for distribution -- AKA wheel -- that actually contains all the relevant files.

foggy forum
#

Yeah I'm really just trying to work out what Samuel's usecase is

shy echo
#

🤷🏻‍♂️

I'd rather wait for a response.

#

Anyway, I gotta actually go do work stuff now. :P

foggy forum
#

People are still doing stuff like going all the way to EuroPython and getting up on stage saying urgh packaging what a mess

#

And I don't get it, it's basically all fixed now was my understanding

shy echo
#

People do what they gotta do.

#

And, the maintainers of Python packaging tooling are doing a poor job of communicating things. It's not that our users are dumb -- you can't know something unless someone tells you about it.

foggy forum
#

Lightning talk title: "stop saying packaging is a mess"

#

I think if someone goes out their way to say something is a mess they should at least check if it's still a mess or not

shy echo
#

It's a reputation built over decades. It'll take more than a lightning talk to fix it. :)

#

But, that's a good start.

foggy forum
#

At pyconuk they banned saying other languages are bad

#

And it did a lot of good tbh

shy echo
#

I sneak some of this into my future talks.

foggy forum
#

?

shy echo
#

"Python packaging is not a mess" or something like that.

#

OK, I think I have a blog post somewhere about all this.

#

Oh yes, a local draft with:

# On "Python Packaging is a mess"
foggy forum
#

Honestly since distribute went away it's been fine

mint patrol
shy echo
#

Nice! Thanks for doing that!

foggy forum
#

Is the plan to vendor truststore?

mint patrol
#

Once we're more certain about the implementation working broadly it'd be good to vendor. We don't want to put any pressure on pip to release sooner to get fixes for our early stage library.

foggy forum
#

With truststore in would you consider support for hsts no-user recourse?

#

Eg --use-feature=truststore --trusted-host=whatever seems like a bigger foot gun than just --trusted-host on it's own

little kite
#

I suggested once on the same tracker that setup.cfg is a poor substitute for setup.py and someone asked to see my credentials >.>

lunar gyro
#

There was a time it’s close to impossible to change anything (other than the PyPA specifications) on packaging.python.org; it’s considerably improved recently though

mint patrol
#

Also question about notifying people, wasn't there a distribution channel for UX improvements and experiments? Is that channel still open or available or is there something like it?

hoary mist
#

um

#

there was a mailing list opened for PyPI at some point, that I don't know if anyone ever really used after warehosue launched, and I don't remember if it was specifically pypi or if it was for pypa

#

looks like it was maybe intended for pypi only

frosty pier
#

I don't think that channel is widely used enough to be helpful 🤔

hoary mist
#

Yes and no? It's low cost to use it probably, and the only way a channel becomes widely used is to start using it, but maybe we don't really want to use it ever and that's fine too

zealous cloud
#

Speaking of announcements & discussion, I am surprised to see no comments on this discussion topic: https://discuss.python.org/t/towards-a-pip-audit-subcommand-for-vulnerability-analysis-management/17681

Would appreciate feedback or any statement of support from folks here before I start to loop in a wider audience.

Discussions on Python.org
Towards a `pip audit` subcommand for vulnerability analysis & manag...

Background As part of my day job on Google’s open-source security team, I’ve been working for the past year with a team from Trail of Bits to author, implement and maintain https://pypi.org/p/pip-audit/, a standalone tool for scanning Python environments for packages with known vulnerabilities. This is the first such tool to integrate directly ...

shy echo
zealous cloud
shy echo
#

I actually want all projects to have a per-project blog, on their Sphinx documentation site.

#

That's something I'm working toward, FWIW.

frosty pier
#

Can we have a Packaging-Maintainers and Packaging section in discuss, where new topics can only be opened by PyPA maintainers? 😄 something like that

zealous cloud
#

Why?

shy echo
frosty pier
# zealous cloud Why?

I want to subscribe to get notified of packaging maintainers new topics, but not get spammed with users packaging questions 🙂

dapper laurel
hoary mist
#

I could imagine a per project blog, and then a planet instance that collates them... or just setup a shared blog and let people categorize posts into projects

#

the latter might be nice because some things span multiple projects

mint patrol
#

yeah could use labels to mark posts differently

shy echo
dapper laurel
hoary mist
#

main reason I'd lean towards a shared thing, with tags/categories/whatever is that I think there are two kinds of people:

  • People who want to subscribe to all packaging news
  • People who want to subscribe to some subset of packaging news

and disjoint blogs makes the first one a lot harder to do

#

and I suspect the first one has more people in it

shy echo
#

FWIW, I don't think it's either/or. My mental model is:

  • Have a single "main" blog under packaging.python.org for the first group.
  • Each project gets its own blog for doing whatever it wants, which... should cover the second group.

And the main blog would not be for release announcements, but rather things like communicating deprecations, funding, calls for testing etc.

dapper laurel
#

or, the "main" could collect entries from every blog and just repost them

shy echo
#

I'm gonna be blunt -- I think that completely dilutes the utility of it.

IMO the main blog is best served as a summariser and major announcements channel, doing things like quarterly newletter style summaries for the community (which would be where we link to the individual project's blog posts) and also serve as places where we make broad wide-ranging announcements that affect users "Distutils is going away in Python 3.12 and here's what you need to know" or "Editable installs are implemented all the major backends in the ecosystem".

#

Basically, right now, the PyPA does not have a broad communication channel to users. I want these blogs to serve as that.

#

And, because I'm dumb like that -- I decided to scope this as "Make this possible for the entire Sphinx ecosystem".

hoary mist
#

I guess I personally don't see a ton of value in every random pypa project having a blog, but I'm also not opposed to projects having a blog if they think it would be useful

shy echo
#

Yea, well, they'll have the option to have one -- they don't need to do that.

#

And, I don't think it makes sense for all of them to have it either.

hoary mist
#

I would probably suggest the "main" blog might be better off living under pypi.org somewhere, if only because that's the most visible and well known URL that people have to associate with Python packaging, and it wouldn't be hard to throw up a blog.pypi.org or whatever that just auto deployed from some GH repo

shy echo
hoary mist
#

(I also dislike the packaging.python.org url altogether, so that may just be bias on my part 😛 and I find the difference between what content is on pypa.io and packaging.python.org to basically be a coin toss on whichever spot someone randomly stuck something at some point)

Broadly though, I don't think there's any value to being under python.org really, and for the vast bulk of programmers they'll never interact with packaging.python.org (and those that do, likely won't once they're experienced), but they will almost certainly interact with pypi.org

shy echo
#

Well, that's precisely what I want to change. :P

#

And, making pypa.io / packaging.python.org clearer is also on my TODO list around all this web presence stuff. Part of the problem that I decided that the first step in that is writing a Sphinx theme, and I keep scope-creeping that.

hoary mist
#

I don't think you can change the second half of that? Like there's basically no content you could put on packaging.python.org that would make it something the average python dev is visiting regularly. You can certainly clean up the mess between pypa.io and packaging.p.o though

shy echo
#

Right now, yea, we don't have much.

I do think we will once we start doing a regular aggregated "Python packaging update" and stuff like that.

hoary mist
#

(If I had my druthers, we'd just move packaging.p.o to guide.pypi.org or so)

shy echo
#

And, I would protest very strongly.

#

😛

hoary mist
#

I'm curious why, the way I see it we get basically no value from hanging off of python.org, and it just causes confusion as people bounce through like 17 different locations for things. Like it or not, pypi.org is basically the web presence for python packaging for the vast bulk of Python's users, and anything that directs them off site or lives at a different location tends to feel surprising IME

true sequoia
#

it's always been a bit headspinning though p.p.o makes sense historically

shy echo
#

But, the point stands. :)

hoary mist
#

there is, though it should only be for internal stuff

#

I don't think we've made any public urls using that

shy echo
#

Or, I guess we have not-for-public consumption stuff on {thing}.pypi.io

true sequoia
#

People depend on it publicly as well 😆

hoary mist
#

(well, if my memory is correct, we didn't own pypi.org at first, and started out using pypi.io, so maybe there's still some old nonsense running under pypi.io)

true sequoia
#

First mention on the issue tracker is 4 days old

shy echo
true sequoia
#

(Using the redirects)

#

We learned the hard way with Poetry that if you ever let people depend on something, they will scream when you try to fix the thing they never should have depended on

#

Changing the branch name of poetry-core resulted in multiple people screaming we broke their company's production infrastructure within 30 minutes

shy echo
#

Hmm... @hoary mist we ought to not be serving files to pypi.io?

hoary mist
#

it's relying on the redirect

#

we should probably kill that at some point

true sequoia
#

Because they had hardcoded the branch name despite the fact that's an incredibly foolish thing to do

#

If you do they will scream 😛

shy echo
hoary mist
# 
 curl -I https://pypi.io/packages/source/d/dask/dask-2022.7.1.tar.gz
HTTP/2 301
server: Varnish
retry-after: 0
location: https://pypi.org/packages/source/d/dask/dask-2022.7.1.tar.gz
content-type: text/html; charset=UTF-8
accept-ranges: bytes
date: Tue, 26 Jul 2022 20:07:54 GMT
x-served-by: cache-ewr18170-EWR
x-cache: HIT
x-cache-hits: 0
x-timer: S1658866074.122264,VS0,VE0
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: deny
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-permitted-cross-domain-policies: none
content-length: 122
shy echo
#

Hm... I think we should kill that ASAP.

true sequoia
#

If you're going to kill it I'd suggest escalating brownouts over a period of months with the reason why in the error response

#

Unless you want to tell people relying on it in prod that it's their fault and to adapt 😛

hoary mist
#

yea, we'll probably want to manage the killing of it

shy echo
hoary mist
#

fun fact, we've moved from python.org/pypi to pypi.python.org to pypi.io to pypi.org

#

that's why you still see /pypi in a lot of the urls

shy echo
#

Heh.

shy echo
#

People do all kinds of stupid things, and expect it to work and act entitled when they do things out of contract that break.

#

Yes, I know hyrum's law. :P

#

I see you typing @hoary mist.

hoary mist
#

anyways, somewhat of a diversion. I'm not planning on doing any work on it, so it's not really up to me, but from my POV there's basically a 0% chance PyPI ever moved back under a python.org domain name, for a variety of reasons, but the big one being there's too much garbage on python.org that could be used to attack PyPI.

Given that, and given the fact that for most python devs, the action they visit the web for the most related to Python packaging is to look something up on pypi, I think that for the bulk of people, whatever URL PyPI is at, is their web interface to python packaging whether we like it or not, and we can either embrace that or we can fight it and add additional friction for those users (though obviously not world ending friction or anything).

shy echo
#

Oh yea, I don't definitely want PyPI to move under python.org.

hoary mist
hoary mist
limber ore
hidden flame
#

I love this new version prompt, looks so nice 😄 @shy echo

unknown.png
maiden island
#

So many colours

hidden flame
#

it shouldn't mislead new users when they encounter installation errors now

maiden island
#

Needs more emojis

hidden flame
#

nO.

maiden island
#

🧐

hidden flame
limber ore
#

I use xonsh + starship, love it 🙂

maiden island
#

This old man has no idea what you're both talking bout 😮

limber ore
hidden flame
#

You only use bash? 👀

hoary mist
#

I use starship and zsh, starship is nice

hidden flame
hoary mist
#

it's just a prompt written in rust instead of in a bunch of crazy shell scripts

limber ore
jovial jasper
stuck girder
shy echo
#

Yay! It all works on other people's computers!

#

And, I appreciate the positive feedback. ^.^

stuck girder
#

also looks good in my local (iTerm2 + liquidprompt)

unknown.png
willow flicker
stuck girder
foggy forum
#

Is there any way to hook into "This is likely caused by a bug in x. Report this to its maintainers. Installation Failed"

I want to be able to flag that my sdist won't build because the platform is unsupported

shy echo
#

I don't think there's anything different you can do. If you have suggestions for changes to the generic message that pip's printing, I'd be happy to hear them!

lunar gyro
#

Just wondering loud, perhaps we could use a special string (like heredoc) to communicate “this is an expected failure, just show strings between the markers instead of a full traceback”

shy echo
#

@lunar gyro Anything about pip's sysconfig location logic, that I should not move to installer? I'm thinking of synchronising the two, and start looking into making pip use installer at some point.

lunar gyro
chrome epoch
#

apropos of nothing: does pip make any guarantees around the handling of "yanked" releases between multiple indices? for example, if a third-party index is specified ahead of pypi.org and marks foo==1.2.3 as yanked, will pip honor that marker or will it attempt to install the non-yank-marked version from pypi.org?

hoary mist
#

metadata from index A does not confer to index B

#

pip basically generates a list of files, and each file has metadata attached to it, from the index it is associated with

willow flicker
#

I appreciate pip trying to give a nice error, but it's wrong in this case:

WARNING: Skipping page https://pypi.org/simple/setuptools/ because the GET request got Content-Type: Unknown. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/html
ERROR: Cannot install setuptools>=40.8.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested setuptools>=40.8.0
    The user requested (constraint) setuptools==65.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

This network flakiness is hitting 50% or so of the cibuildwheel Azure CI runs in the last week or so.

I wonder if pip could provide a custom error if no versions were resolved, instead of saying the constraint caused it?

hard thunder
#

is this also a warehouse bug? or some cache eviction problem?

lunar gyro
#

This is a (known?) Warehouse bug but the error message is still off. There are already some special case errors but apparently this is not covered 🙂

hoary mist
shy echo
#

Is there a consistent way to reproduce this error?

willow flicker
#

It was happening fairly often (50% of the time or so), but now it seems to be happening less. Last few PRs didn’t have a problem.

shy echo
#

If you can run pip with max verbosity and with --debug, that might provide more context?

fierce temple
#

What's the most user-friendly place that explains pip installing extras / optional dependencies?

shy echo
glacial meadow
#

Greetings! I'm wondering if the following is supported by --index-url file:///path/to/folder
I have a /path/to/folder where every subfolder is a pip installable directory with setup.py,cfg and a pyproject.toml. Can I tell pip to treat this folder as an on disk index? It seems to search for an index.html

hi @shy echo 🙂

#

For context, I have a monorepo containing both applications and libraries. I want to formalize how the internal libraries are hosted by defining a folder to serve as a index for a flat list of libraries and I want said libraries to refer to each other by name (not by path).

shy echo
#

👋🏻

#

Index URLs (and find-links) expects distributions files (i.e. sdists and wheels) as the references, so... I don't think you can do that.

lunar gyro
#

I seem to recall there’s a hack for that. Try --index-url path/to/folder

glacial meadow
#

yeah, except the folder must have subfolders named after normalized package names, each containing an index.html per PEP503 and all files under the subfolder should be tar.gz and whl

lunar gyro
#

Ah right, I think only find-links can be real flat directories

#

I guess a generator would be useful, static site generator but instead of blog it creates a Simple API index

shy echo
#

I was gonna say: you can probably write a plugin for simpleindex, that does what you want.

lunar gyro
#

The built-in path route should support this

[routes."{project}"]
source = "path"
to = "/path/to/containing/directory/{project}/"
shy echo
#

Well, that won't deal with not-normalised names tho?

lunar gyro
#

Not a problem unless you’re doing something weird. pip normalises the package name when making the request and this is mentioned in PEP 503

Repositories MAY redirect unnormalized URLs to the canonical normalized URL (e.g. /Foobar/ may redirect to /foobar/), however clients MUST NOT rely on this redirection and MUST request the normalized URL.

shy echo
#

Right, but the folders don't have normalised names TP!

#

(I'm guessing)

lunar gyro
#

Just run a script once to create symlinks and get on 🤷‍♂️

shy echo
#

XD

#

@glacial meadow ^

#

:P

lunar gyro
#

But I think if you’re maintaining a repo anyway it’s probably worthwhile to just generate those index.html instead (and maybe use CI or something to keep them in sync)

glacial meadow
#

You see the problem is - I must have .tar.gz or .whl as links in the index.html
I've been trying to avoid it somehow...

#

Maybe a better questions to ask would be - do you have any examples of a Python mono-repo that contains a collection of libraries that may have dependencies on each other and how these dependencies are expressed in metadata

glacial meadow
lunar gyro
#

I must have .tar.gz or .whl as links in the index.html
What’s wrong with href="./mypkg-1.0.tar.gz"?

limber ore
hard thunder
#

the problem is that he doesn't have source distributions generated

#

he just has project folders from which you can build those

#

I'm guessing he would maybe want the index to auto-build those on request? I'm not sure on that

lunar gyro
willow flicker
lunar gyro
#

pip is doing what it’s supposed to do (rejecting a response with incorrect Content-Type) but something is broken in PyPI and it sends an invalid response from time to time

shy echo
#

I wonder if it would be helpful for pip to print more content, like the entire header or something like that.

lunar gyro
#

Adding it to -v is probably a good idea

shy echo
#

Yea, that is what I was thinking.

hoary mist
#

being able to see all the headers an even the response body would be amazing

true sequoia
#

is there any interest in combining --prefix with --platform, --abi, --implementation, and --python-version?

#

I am attempting to use pip to create a venv for a different interpreter version, and one sticking point is that I can only use these versions with --target

#

I can set that to the platlib/purelib of the venv, but I don't get scripts as a result (looks like they get thrown away)

shy echo
#

No, but there's a new --python flag to manage packages associated with a different executable on main.

#

I guess, I shouldn't say no, but no one has expressed interest in diving into that specific can of worms. :)

obtuse lagoon
#

for the same reason as pypa/installer, I'd advise against this

#

IMO passing the correct interpreter is by far the best approach

true sequoia
#

--python does seem like the long term path forward -- I was only looking at released versions of pip and thus did not encounter it

inland dust
#

is it possible use PIP_CACHE on a distributed file system (like, s3 storage on databricks' dbfs) to save on pypi download bandwidth?

limber ore
#

you could use a FUSE mount

dapper laurel
#

When is next pip release planned?

hidden flame
dapper laurel
#

Thanks, missed it somehow

limber ore
shy echo
#

Yea, needs the packaging release.

#

:/

stuck girder
obtuse lagoon
lime gust
#

Did the expectation change about pip supplying setuptools under pep517 if no build backend is indicated?

#

I'm no longer able to build mercurial without setuptools pre-installed.

lunar gyro
#

Can you try 3.9? There’s a bug specific to Homebrew-distributed 3.10 regarding build isolation

lime gust
#

Ah, yes. That works.

#

Thanks.

lunar gyro
lime gust
twilit mantle
#

Are there any plans for pip to support reading certain configuration settings from pyproject.toml? I'm thinking specifically about --index-url, --extra-index-url, --trusted-host, possibly others, and disabling pypi.org.

shy echo
#

Not at the moment, partly because no one has asked for it.

#

If they have, then I haven't seen it yet. 😅

twilit mantle
#

I was thinking along the lines of what PDM does: https://pdm.fming.dev/latest/pyproject/tool-pdm/#specify-other-sources-for-finding-packages
and extrapolating to other tools (e.g. hatch) which require you to configure these as pip environment vars in your pyproject.toml. Rather than [[tool.pdm.source]] which is of course PDM-dependent, it seems like this could be one simple area of standardization. If pip doesn't currently support this, maybe it shouldn't even be [[tool.pip.source]] since it's possible for other mechanisms to be used.

I haven't really thought about this in much depth, but it would be helpful for bringing more things to pyproject.toml. E.g. the envar settings for hatch are less optimal, IMHO.

lunar gyro
#

pip already supports configuration files, I wonder if it makes sense to support $PROJECT_DIR/pyproject.toml where $PROJECT_DIR is user supplied (e.g. pip install --project=.). This same $PROJECT_DIR value could also be used in say, requirements file.

small cove
#

Is there a way to have stable links to a specific version of https://pip.pypa.io ? the switcher only allows stable and latest and trying to guess the url with a version in it also didn't work

past pagoda
small cove
#

that's a pity, thanks anyway

shy echo
#

Any specific reason you want that?

small cove
#

mixture of stable links to doc section headers and a bit of archeology what pip version had which behaviour

shy echo
#

You could build the docs fairly easily, if you're willing to write a script that clones the repo and checks out tags and runs sphinx-build docs/html build/html/{version}/

small cove
#

yep the nox task is also really nice :)

solemn reef
mint patrol
#

Hey folks, regarding truststore: is there any appetite for switching to using the library by default (vendoring and using truststore+certifi on Python 3.10+, on 3.9 or earlier fallback to only certifi)? The library currently is still marked as experimental, but with the go-ahead that we'd be used by pip we would push to make the library "stable" (which from an API-POV it already is)

limber ore
#

I'm hitting a cycle; any docs on how to debug?

docker run --rm -t python:3.8 bash -c "git clone https://github.com/DataDog/integrations-core;cd integrations-core;pip install -e ./datadog_checks_dev[cli]"
limber ore
little kite
#

is there a way to instruct pip to install a universal2 wheel instead of x64/arm?

shy echo
#

Not at this time, no.

little kite
#

alright, thanks

shy echo
#

OK, I guess you could pass in that wheel directly; I'm guessing that's not what you wanted tho. :)

hard thunder
#

Could combine with pip download to do it fully from cli I imagine?

willow flicker
#

@shy echo, any chance we could get a patch release of Pip with https://github.com/pypa/pip/pull/11598? I'm seeing a number of issues on projects about homebrew Python not being able to build anything but setuptools packages. Given I'm heavily involved in trying to provide PEP 517 replacements for setuptools, it's very damaging to have PEP 517 builds broken (but setuptools works). It affects things like hatchling, meson-python, scikit-build, cmake, ninja, pybind11, ... (though it disproportionally affects binary builds of packages, since it's easier to provide a wheel for pure Python to avoid the source build)

Current workaround is to always provide binary packages or use a virtualenv. I don't think homebrew's pip can be patched to include this, because then pip install -U pip would revert to the broken pip.

twilit mantle
#

While we're asking for new pip releases (and I know it's the top of the dependency stack), I would really like to get back to removing old APIs in Python 3.12, but need this to be resolved first: https://github.com/pypa/setuptools/issues/3631

shy echo
#

File an issue? Paul's the RM for this cycle; and he's pretty hesitant to cut a release.

#

(last I checked)

#

Reg 3.12... @twilit mantle I'm not sure what to do to help move that forward TBH.

shy echo
twilit mantle
shy echo
#

FWIW, how bad is waiting for a couple of weeks or so for the pip release at this point?

#

We're gonna cut one in Jan, and I've got a bunch of "ugh, hack" removal plans for the next little while.

twilit mantle
#

Naw it's all good. It's a blocker and yes, I would like to make more progress on the CPython work, but not at the expense of putting unplanned or hacky work on your plate.

lunar gyro
#

Does pip use any of those deprecated APIs, aside from pkg_resources? We are transitioning away from that module entirely and the file is only kept for compatibility. If the usages are all in pkg_resources, we can add a check to ensure the compat code path is never reached on 3.12 to prevent crashes.

lunar gyro
dusty crag
#

how brave it would be to use a pinned extra that locks dependencies so users that want to the exact versions that were using during testing of that package instead of the relaxed ones? Is that doable?

shy echo
#

It's... fine.

#

As long as its not pinned by default. At that point, airflow-style constraints files aren't a bad idea either.

dusty crag
# shy echo It's... fine.

One of my concerns is that current I use pip-compile to keep track of the pinning, as pip-tools is not able to update pyproject.toml file, so I would have to add a feature in pip-tools to be able to update pyproject extra deps... not really a single day job.

shy echo
#

(-c URL)

dusty crag
#

it would be so awesome if python packaging would be able to include lock files.

shy echo
#

Yea, one day. :)

dusty crag
#

yep, I was considering getting them from the URL but the problem is that I do not know the exact version. Dependabot would not be able to update my URL. I would have to do some ugly hacks to parse a lock file, extract pinned version and set an envvar, ones that later I would use inside tox to set the constraint file.

#

At this point the chain start to look as something over-engineered and likely to break.

shy echo
#

I'm not too keen on this living in pip-compile TBH.

dusty crag
hard thunder
twilit mantle
dusty crag
hard thunder
#

setuptools-scm is just used for generating version number for your package, deps don't matter

#

You can just read requirement files in setup.py and declare extra deps dict from there

shy echo
hard thunder
#

(I think #general will probably make more sense since it doesn't seem related to setuptools-scm)

#

(but yeah, should not discuss it here :))

hard thunder
shy echo
#

Tell me why on Mastodon, if you have an account plz. 🙈

hard thunder
#

Will do

shy echo
#

Thank you! ^.^

hard thunder
#

Will save it for later

dapper laurel
#

hey, I am writing a parser for PyPI API, basing it on pips parser. does anyone know how why this parser looks for base tag? it doesn't look like something from Simple Repository API PEP 503. Is it some legacy stuff or am I missing something?

shy echo
#

What line are you looking at?

dapper laurel
#

I mean, I could guess what it is from the code, but this doesn't look like any PEP/spec I have seen, so I don't know if it's me missing something or pip doing something for legacy reasons or sth

shy echo
#

~Legacy reasons for sure.~ I was wrong, it's HTML weirdness.

dapper laurel
#

Great

shy echo
#

#19 is likely a reference to ianb's bitbucket, rather than GH.

#

IIUC, it's for compatibility with the HTML spec.

#

Since HTML supports setting a base tag that changes how URLs are resolved.

#

So, I guess I'll retract what I said earlier -- it's not legacy reasons but "HTML is a bad data exchange format" reasons. :)

hard thunder
#

The warnings here are quite confusing:

Collecting multidict==6.0.4
  Downloading multidict-6.0.4.tar.gz (51 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.3/51.3 kB 717.9 kB/s eta 0:00:00
  Installing build dependencies ... done
  WARNING: Missing build requirements in pyproject.toml for multidict==6.0.4 from https://files.pythonhosted.org/packages/4a/15/bd620f7a6eb9aa5112c4ef93e7031bcd071e0611763d8e17706ef8ba65e0/multidict-6.0.4.tar.gz.
  WARNING: The project does not specify a build backend, and pip cannot fall back to setuptools without 'wheel'.
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: multidict
  Building wheel for multidict (pyproject.toml) ... done
  Created wheel for multidict: filename=multidict-6.0.4-cp310-cp310-linux_x86_64.whl size=113460 sha256=9df21d0f626349e99094ca79f4edb9257de4141c4491220089a1d7fef83f43be
  Stored in directory: /home/ubuntu/.cache/pip/wheels/ae/d2/13/61a3897335dd417ee80bcf70d39fea8eda1f7761d28d5547f5
Successfully built multidict

for a project with this configuration:

[build-system]
requires = ["setuptools >= 40"]

when running pip install multidict==6.0.4 --no-binary multidict in a venv that has latest versions of pip and wheel.

I can report this on the issue tracker but I'm not yet sure what the message is trying to say here.

jovial jasper
#

Pip is trying to tell you to specify a build backend in project.toml so it can do a pep517 build, or to install the 'wheel' package so it can run setup.py bdist_wheel

hard thunder
#

wheel package is present

#

that's what's so confusing about it to me

#

As for specifying the build backend, I do agree it should be there and made according PR to multidict lib, I was just wondering if this warning could be improved in some way.

jovial jasper
#

Hm, indeed. That's confusing because it is building anyway.

hard thunder
# hard thunder wheel package is present

unless it's that rare case of where you would actually need wheel in build-system.requires (assuming that for whatever reason you don't want to specify build-system.build-backend

hard thunder
shy echo
#

Yea, that warning can be removed now.

#

setuptools injects that nowadays and we don't have any semantics to "pick older version" for build backends anyway.

hard thunder
shy echo
#

Perfect.

small cove
#

is there a way to just call pip dependency resolver and get a resolution, without actually installing all the packages? or alternatively, is there a way to get timing numbers just for the resolver? -vvv didn't say anything

shy echo
#

--dry-run + --report

small cove
#

that's perfect, thank you!

lost hawk
#

Am I remembering incorrectly, or is there now a command in pip now that I can run to do a resolve w/o doing the actual download/installation of a project (or at least its dependencies)? I'm asking as I would like to know if I were to say py -m pip install --platform any --python-version 3.11 --implementation py --abi none --only-binary :all: to check if a certain package will only pull in pure Python wheels? (Otherwise I will end up needing to write a custom resolver, I think.)

stuck girder
# 
  --dry-run                   Don't actually install anything, just print what would be. Can be used in combination with --ignore-installed
                              to 'resolve' the requirements.

?

hidden flame
obtuse lagoon
#

sdist support is a bit trickier

#

this package is very early btw, so beware

lost hawk
#

I'm getting some odd errors when trying to just do the resolution: PIP_REQUIRE_VIRTUALENV=0 py -m pip install --dry-run --report --only-binary=:all: --implementation=py packaging says ERROR: When restricting platform and interpreter constraints using --python-version, --platform, --abi, or --implementation, either --no-deps must be set, or --only-binary=:all: must be set and --no-binary must not be set (or must be set to :none:).; Since I set --only-binary I don't see what I'm doing wrong here. And this is after I got odd errors when setting --abi none and --platform any as well.

#

Yeah, something odd is going on. PIP_REQUIRE_VIRTUALENV=0 py -m pip install --dry-run --report --only-binary=:all: markupsafe says it will install MarkupSafe-2.1.1.tar.gz which is not a wheel.

#

(This is with pip 22.3.1 and 3.11.0 .)

remote osprey
# lost hawk Yeah, something odd is going on. `PIP_REQUIRE_VIRTUALENV=0 py -m pip install --d...

FWIW, I attempted to reproduce the above in a fresh Conda environment on Windows with just python=3.11 using the same pip 22.3.1, Python 3.11.0 and invocation, except without --report (and with python instead of py in a Conda env), but I get the desired result:

$ python -m pip install --dry-run --only-binary=:all: markupsafe
ERROR: Could not find a version that satisfies the requirement markupsafe (from versions: none)
ERROR: No matching distribution found for markupsafe
#

However, if I include --report, I get the error ERROR: Could not install packages due to an OSError: [Errno 22] Invalid argument: '--only-binary=:all: . This happens no matter how I try to quote the argument, with either single, double or no quotes around either the value or the whole arg, and occurs on both Git Bash and the default cmd.exe. The -vvv output immediately preceding the error was:

Using cached wheel link: file:///C:/users/c.%20a.%20m.%20gerlach/appdata/local/pip/cache/wheels/96/ee/62/407c247ad088bcb67b530ba3ac1479058c58a651bd6bf09a1f/MarkupSafe-2.1.1-cp311-cp311-win_amd64.whl
Collecting markupsafe
  Using cached MarkupSafe-2.1.1-cp311-cp311-win_amd64.whl
  Added markupsafe from file:///C:/users/c.%20a.%20m.%20gerlach/appdata/local/pip/cache/wheels/96/ee/62/407c247ad088bcb67b530ba3ac1479058c58a651bd6bf09a1f/MarkupSafe-2.1.1-cp311-cp311-win_amd64.whl to build tracker 'C:\\Users\\C. A. M. Gerlach\\AppData\\Local\\Temp\\pip-build-tracker-mo6ff3ks'
  Removed markupsafe from file:///C:/users/c.%20a.%20m.%20gerlach/appdata/local/pip/cache/wheels/96/ee/62/407c247ad088bcb67b530ba3ac1479058c58a651bd6bf09a1f/MarkupSafe-2.1.1-cp311-cp311-win_amd64.whl from build tracker 'C:\\Users\\C. A. M. Gerlach\\AppData\\Local\\Temp\\pip-build-tracker-mo6ff3ks'
Created temporary directory: C:\Users\C. A. M. Gerlach\AppData\Local\Temp\pip-unpack-wc80ve6r
WARNING: --report is currently an experimental option. The output format may change in a future release without prior warning.
ERROR: Could not install packages due to an OSError.
Traceback (most recent call last):
  File "C:\Miniconda3\envs\py311-env\Lib\site-packages\pip\_internal\commands\install.py", line 415, in run
    with open(options.json_report_file, "w", encoding="utf-8") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 22] Invalid argument: '--only-binary=:all:'
lunar gyro
#

Looks like a bug in argument parsing, the code thinks --only-binary does not take an argument (it should)

#

Oh I see the issue; --report also needs an argument:

  --report <file>             Generate a JSON file describing what pip did to install the provided requirements. Can be used in
                              combination with --dry-run and --ignore-installed to 'resolve' the requirements. When - is used as
                              file name it writes to stdout. When writing to stdout, please combine with the --quiet option to
                              avoid mixing pip logging output with JSON output.

So when you do py -m pip install --dry-run --report --only-binary=:all: markupsafe it tries to interpret --only-binary=:all: as the output filename and fails. You need something like py -m pip install --dry-run --report=- --only-binary=:all: markupsafe (dash reports to stdout)

remote osprey
jovial jasper
jovial jasper
remote osprey
lunar gyro
#

I don’t think it’s that worthwhile, if you want to redirect the output to a file you can just do that, if you output to stdout the format isn’t that critical since it’ll very likely be read by humans

jovial jasper
remote osprey
limber ore
shy echo
#

There's an OS level limitation on the length of the shebang, IIRC.

#

Or, maybe that's on some Linux systems?

#

Regardless, I'm 80% sure that's what's happening here.

limber ore
#

oh that's why there's that weird long exec line

shy echo
#

Yeah

limber ore
#

😢

shy echo
limber ore
#

oh it's due to spaces...

shy echo
#

:)

limber ore
#

wow there's so many layers to this particular issue

#

and now I understand the actual reason he says tox works: by default the current directory is used for storage and that user path is unlikely to have spaces on macOS

shy echo
#

Yea, try adding a space to cwd and seeing if that fails.

#

(in docker, or MacOS)

lost hawk
# lunar gyro Oh I see the issue; `--report` also needs an argument: ``` --report <file> ...

Thanks for the help everyone! It looks like py -m pip install --quiet --dry-run --ignore-installed --report=- --target=/tmp --only-binary=:all: --implementation=py --abi=none --abi=any debugpy --python-version="3.11" packaging gets me what I want (a way to easily tell whether a package would even install for WebAssembly use); bonus is the JSON output means I can also log any of the dependencies as safe as well! (Drawback is it means work doesn't need me to write a custom resolver for this which would have gotten me work time for packaging.metadata 😉.)

shy echo
#

One cautionary note: that doesn't handle/affect the marker evaluation environment

#

@lost hawk ^

lost hawk
shy echo
#

It uses the environment markers as inferred from the running interpreter.

lost hawk
shy echo
#

LOL

#

I think we need an environment-specification file for pip and implementing that + deprecating the existing filter flags might be the best mode of doing this -- i.e. this doesn't affect lockfiles but does affect pip's ability to generate certain lockfiles.

lost hawk
# shy echo I think we need an environment-specification file for pip and implementing that ...

That's my next module for mousebender. Once we have packaging.metadata, I want to create a resolver where you pass in the marker/platform details and whether you want the most specific or compatible wheel file (and probably newest or oldest dependency). That way the resolve can be done for any platform on any machine. I have a topic on discuss.python.org where I asked about what platform info is needed specifically for this use case. If we end up wanting to standardize on a JSON format for storing such platform info I'm happy to help with that once I get to that stage

dapper laurel
#

packaging.metadata looks promising

slim eagle
# lost hawk That's my next module for mousebender. Once we have `packaging.metadata`, I want...

sounds like what I'm doing in my Bazel rules. I define target environments, then use those in conjunction with a Poetry or PDM lock file to generate a Bazel build file that uses Bazel's own multiplatform features to select appropriate dependencies at runtime. For example: https://github.com/jvolkman/rules_pycross/blob/main/examples/pdm/BUILD.bazel#L58. And that'll be used to select an appropriate wheel: https://github.com/jvolkman/rules_pycross/blob/main/examples/pdm/example_lock.bzl#L327

obtuse lagoon
#

I can add you to the project

#

or you can create a new one, depending on what exactly you want to do

lunar gyro
#

Looks like resolver also uses marker values inferred from the current platform; maybe that’s a viable feature request?

#

Or I’ve always wanted to be able to do marker-agnostic resolution (all markers expect extras always evaluate to True)

slim eagle
#

isn't that what poetry and pdm do?

obtuse lagoon
idle harness
# lunar gyro Looks like resolver also uses marker values inferred from the current platform; ...

Then you will meet some packages that do really weird thing: https://github.com/ansible/ansible-lint/pull/2712
Is that solvable with resolvelib?

GitHub
Prevent installation on Windows by ssbarnea · Pull Request #2712 · ...

In order to save time on platform that we know it will never work, we add an impossible dependency, named to hint the user about his mistake.
Any attempt to install the package using pip will give ...

lunar gyro
shy echo
idle harness
hard thunder
#

yeah, the hack involves a project with only a single release that has been yanked, the yanked version will actually install fine if you were to try installing it with ==0.1.0

idle harness
dapper laurel
#

is there any way PyPA could align releases of packaging with pip releases? recent issues with new macos tags cause some problems in projects

shy echo
#

Hmm... put that up on the packaging issue tracker?

dapper laurel
#

Sure, I can. I just wanted to check if this is possible first 😄

shy echo
#

I'm not a 100% sure how/what we should do to avoid this sort of problem TBH.

#

Regardless, having it be on not-behind-a-login and on a search-engine-indexed location seems like a much better place to have that discussion. It's not really a throwaway discussion.

dapper laurel
#

sure

shy echo
#

But, 100% agree that we should definitely think + try to do something about this.

fast stump
#

I just confirmed this

#

Using pip itself isn't affected, but get-pip.py is having trouble now :/

shy echo
#

Yea, I see that @frosty pier mentioned Ee.

fast stump
#

Oh indeed

shy echo
#

Thanks for flagging that here!

fast stump
#

No worries!

#

Yipee! It's fixed!

lethal garnet
#

Hey! I hope this is a good place to ask this question.
I would like to create platform specific URL dependencies using the pyproject.toml so that pip picks it up.

Simple example. This works, but the conditional one after doesn't. The example doesn't really makes sense with numpy, but I used it to quickly showcase it.

dependencies = [
    "numpy; sys_platform == 'win32'"
]

dependencies = [
    "numpy @ git+https://github.com/numpy/numpy.git@v1.23.5; sys_platform == 'win32'"
]

How would I go about doing this?

shy echo
#

You need a space after the URL, before the semicolon.

hidden flame
#

trips me up every, single, time

#

packaging 22.0 should raise a better error here AFAIK though

shy echo
#

Yup.

# 
packaging.requirements.InvalidRequirement: Expected end or semicolon (after URL and whitespace)
    package @ https://example.com/; python_version <= '3.2'
              ~~~~~~~~~~~~~~~~~~~~~~^
#

But, still need to get to that upgrade.

lethal garnet
#

Ahh, thank you!

dapper laurel
#

isn't packaging 23 already out?

shy echo
#

It is, but the copy vendored in pip isn't upgraded since it was blocked on setuptools. We're still blocked on a couple of tricky coupling issues.

lunar gyro
#

I wonder if it’d trip people less if the space is always required, i.e. make numpy >= 1.23.5; sys_platform == 'win32' illegal for consistency

dapper laurel
#

I wonder if the space should even matter. I mean, why require the space in the first place?

lunar gyro
#

Semicolon is a valid character in URLs, without the space there would be ambiguity

shy echo
lunar gyro
#

The thing I don’t like with the error message solution is it appears much too late for dependencies specified in pyproject.toml. But maybe that can be solvable if someone implements a pyproject.toml linter in the future.

shy echo
#

Well, ideally, build backends would cover that base.

#

But I'm leaning more and more in the direction of "we have no validation/compliance enforcement in the tooling, and that's bad".

idle harness
balmy shard
#

Hello everyone.

shy echo
#

👋

#

@lunar gyro what do you think we should do with the Debian / Homebrew environment paths mess?

#

I'm inclined to drive the venv approach over the line, but that'll happen in the coming days and not just today.

lunar gyro
#

↑ No prob with that

shy echo
#

So, until then, should we revert the Homebrew-related fix, so that things stay broken exactly as they are?

lunar gyro
#

That PR using get_scheme is on the right track but I don’t know why he wants to also change the distutils code path instead of just the sysconfig one

shy echo
shy echo
#

I'm not fully sure about how redistributors break things TBH, and my hope is venv based isolation will solve that for us.

#

But, I'm not sure what we should do now.

#

With the current logic.

lunar gyro
shy echo
#

👍🏻

#

I'm not sure still, TBH, but happy to defer to you on this.

shy echo
#

Alrighty, one more release made. Holler here if you see something off!

dapper laurel
#

Just wondering (since the milestone in repo is empty) is there any kind of timeline for pyproject.toml-based builds would be default? I think with pip not having it as default, it makes non-pyproject based setups "encouraged" because "why change stuff, it works just fine"

shy echo
#

Once we get rid of all the legacy install mechanisms (eg: setup.py install, setup.py develop), we'll do that.

dapper laurel
#

You mean from pip source code or from community? 🤣

shy echo
#

pip's default install mechanisms; as released.

#

Try pip installing a setuptools + non-pyproject.toml project without wheel in the environment.

lunar gyro
#

I think setup.py install can probably go at this point; the deprecation has been shown for a while and I haven’t seen reported actually related to it. Reports claiming breakage are mostly from confused users.

jovial jasper
#

It's announced for 23.1. We need to decide if we postpone or not.

shy echo
#

I don't think we should.

jovial jasper
#

I tend to agree we should not postpone.

shy echo
jovial jasper
#

To some extent adding --use-deprecated or pip install -U pip<23.1 is a comparable effort for users who'd need time to transition.

#

I suspect to biggest complaints will come from (dependencies on) projetcts which heavily override the setup.py install command and can't build wheels.

shy echo
#

It depends on what setup you're in. From $work and some other institutions, it's easier to get a flag added in to internal builds than to block an upgrade org-wide.

jovial jasper
#

true

shy echo
#

OTOH, these are all cases where the actual fix is elsewhere and this is mostly just... a graceful thing to help people who might need it but will never ask for it. 🤷🏻

#

And, will work around it silently if they don't have it.

#

The main benefit from my PoV is that we give $guyOnTheInternet an actionable thing that's that's different from "do it better" or "listen to us" or "don’t upgrade".

hoary mist
#

I think the big benefit is that pins tend to last forever

#

but flags don't, since it'll break again in the future when pip removes it

shy echo
#

^indeed!

#

Was one of the reasons for the opt-in/opt-out model!

#

Effectively pip moves the slowest to let organisations who move slow deal with changes, with a deadline.

jovial jasper
#

Good. So I'll get back to the question of whether other maintainers feel we need --use-deprecated for setup.py install. 'cause de lazy part of me says we don't 🙂

#

We can discuss on each individual issue.

lunar gyro
#

I am very tempted to say we don’t

shy echo
#

+1 to that, but yea, let's move the rest of this to the issue tracker.

limber ore
shy echo
#

When pip runs setup.py install, I think it's technically possible?

#

I might be wrong tho.

hard thunder
shy echo
#

They're all blockers to switching over.

hard thunder
#

#8368 says:

In a future version, pip will not attempt setup.py install in that case, and fail the installation right away.

#

as opposed to pep 517 build

#

which is why I wondered

shy echo
#

If I wanted to be rude, I'd say "read when setup.py install is run"

#

But more pragmatically, setup.py install is called if setup.py bdist_wheel fails.

#

At that point, we know that package doesn't build; so failing explicitly is a good idea.

hard thunder
#

I guess I get it, switching to legacy hook is more or less equivalent to only calling bdist_wheel and failing if that doesn't work

shy echo
#

Yup!

hard thunder
#

the amount of downvotes suggested something worse 🙃

shy echo
#

Well, that's because it's the URL printed when a package fails to build.

dapper laurel
#

well, there are people who don't go with pyproject.toml even now, and that's by their choice to stick to the ancient ways

hard thunder
#

yeah but they don't have to provide pyproject.toml, they just need to work within an isolated build

lunar gyro
thorny pelican
#

Is it intended that pip should be able to install packages into a different environment? I find that python -m pip install --prefix /path/to/another/venv results in the installed package's console scripts have the sys.prefix of pip and not of the prefix that was specified in their shebang. I also seem to recall that it isn't possible to install into a prefix which is using a fundamentally different Python version / platform tags etc.
Is this expected to work? If is, I will happily distil the problem and raise an issue.

lunar gyro
#

It’s not designed to install things into another environment but allowing a user to do it is “intended” in quotes

thorny pelican
# lunar gyro It’s not designed to install things into another environment but allowing a user...

Thanks. It is a bit surprising then that --prefix is the option for install, but --path is the option for pip list. It was this difference which led me to think that there might have been some intended behaviour wrt. the --prefix flag.

As a follow-up, I remembered that there had been a discussion about having pip as a standalone app. I found it again at https://github.com/pypa/pip/issues/11243. I guess in that approach, pip is still running from the python that it should install into.

GitHub
Issues · pypa/pip

The Python package installer. Contribute to pypa/pip development by creating an account on GitHub.

lunar gyro
thorny pelican
lunar gyro
#

For console_script shebang replacement I seem to recall there’s a specific issue for that (not just for --prefix but allow replacement in general)

thorny pelican
thorny pelican
thorny pelican
hoary mist
#

I suspect --prefix, --target, --root, et al were a mistake tbh

wispy coral
#

Without --scheme they're also somewhat broken on environments like Debian (and possibly Homebrew?)

#

I mean, they won't necessarily do what you want / expect

thorny pelican
#

So there are three options then, I guess:

  1. Do nothing, and live with the status-quo (documenting what you need to do to post-process the result of --prefix to make it likely to work if-and-only-if your --prefix Python is the same version as your pip Python
  2. Deprecate the options
  3. For --prefix at least, let pip take the information from --prefix's Python, not its own

Personally, as you can imagine by the original question, I'm motivated to see #3 - it is how I would consider achieving something like https://github.com/pypa/pip/issues/11243. I'm interested because I want to be able to create a virtual environment which is pip free - my rationale being that I have virtual environments which are shared and read-only, and having pip in the environment is unnecessary / unexpected (since the environment is frozen). I don't want to preclude that environment declaring a dependency (either direct or transative) on pip, so it isn't desirable to pip install <pkg> && pip uninstall pip.

thorny pelican
thorny pelican
jovial jasper
#

If you are looking after pip-free virtual environments, pip --python does the job, because it does not require pip to be present in the target environment. It just needs a pip version that is compatible with the target python version.

thorny pelican
jovial jasper
#

The --python flag even supports installing pip
IIRC, It does not install pip, it "injects" an existing pip into the target environment for the duration of the pip command run.

thorny pelican
jovial jasper
#

My option 3 would have meant that pip itself needn't be source compatible with the python that it is installing for,
If you want something that automatically uses a pip version compatible with the target python, I have hacked together a pip-launcher. https://github.com/sbidoul/pip-launcher (just a hack to play with the idea for now, not intended for general public use)

GitHub
GitHub - sbidoul/pip-launcher: Download and run pip in the current ...

Download and run pip in the current python environment without installing it. - GitHub - sbidoul/pip-launcher: Download and run pip in the current python environment without installing it.

wispy coral
#

I'm getting push-back about including PEP-668 in Debian bookworm, without an override mechanism that a normal user can use. I'll look at preparing a PR, when I get a chance

dapper laurel
#

What override?

shy echo
#

What's the Debian cutoff date?

shy echo
dapper laurel
wispy coral
shy echo
#

That's where I got the name from.

wispy coral
#

Basically, we should get the major pieces in place by 12 Feb

shy echo
#

Is pip a major piece? :P

wispy coral
#

Yeah, ideally I wouldn't want to change it after then, but if we have to, we can

wispy coral
shy echo
#

You might wanna edit that. :P

#

Reviewed, with a minor change suggestion.

wispy coral
#

Thanks!

shy echo
#

FWIW, I'm trying to push for any/all user-facing communication to not reference PEPs and use friendly-to-human names.

#

Hence pyproject.toml-based builds and EXTERNALLY-MANAGED installations; rather that PEP 517 and PEP 668.

shy echo
#

@wispy coral FYI: a test failed and I don't think we can test that in our CI setup as it stands.

wispy coral
#

I fixed that failure (and forgot to commit it with the previous change)

#

Unfortunately few of those existing tests pass for me, locally. So I was doing a lot of testing on CI

wispy coral
#

Oh, and you added a commit, thanks

shy echo
#

I did, yea. 😅

shy echo
shy echo
limber ore
shy echo
#

Did they reinstall after changing metadata?

#

I'm guessing that's what's wrong.

#

Yup, that explains why they're unable to reproduce it.

limber ore
#

nice, good call

spare heath
#

Hi! Is this the right chat to ask questions about using pip to cross-build packages?

dapper laurel
#

#cibuildwheel seems better, since it was made with that exact purpose in mind

spare heath
#

Thank you (also it is the first time I read about it)

#

It seems to use containers and qemu though?

#

my problem at hand is that openwrt builds packages expecting setup.py and it sets all the environment variables so it works. I extended it to support setuptools_rust and maturin, but then I had stumbled upon a package using poetry and calling pip install with the same env did not produce the expected results =/

lunar gyro
#

pip install mostly just calls whatever tool that is handling the build (setuptools, maturin, or poetry) and copy whatever the tool generates into correct locations, so it’s more likely this is a problem with the build tool, not pip

spare heath
#

so I should annoy poetry devs 🙂

#

there isn't any additional env var that might impact pip in this regard

shadow shuttle
#

i have a controlled environment where i'm not allowed to install any third party dependencies. but i have a first party wheel that i'd like to install. pip install --no-deps asdf.whl mostly works great for this, except if the first party wheel happens to have dependencies that don't already exist. is there a way to get an error for that situation?

#

i guess i could pip check after

#

(that's not a perfect solution though, since i do have some other incompatibilities that pip check would complain about. maybe i just roll my own with importlib.metadata)

obtuse lagoon
#

if you roll your own, this might be helpful to you

shy echo
#

I was bored, so...

Screen_Shot_2023-02-20_at_04.35.11.png
hidden flame
#

Is that an UML graph for pip?

shy echo
hidden flame
#

oh that's neat!

shy echo
#

I wrote a Python regex to railroad diagrams thing for this. 🤦🏻‍♂️

hidden flame
#

Well I spent an hour writing a script to merely reverse 36 HTML chunks so I didn't have to do that manually. 😅

shy echo
#

sigh

dapper laurel
#

You know pylint ships with tool for class diagrams, right?

shy echo
#

This isn't a class diagram?

#

Nor is it a UML diagram FWIW. It's a railroad diagram generated from the regular expression for version parsing.

dapper laurel
#

ah, sorry. early morning + somehow I had in my mind UML=class diagram (blame @hidden flame for mentioning UML) 🤣

ashen geyser
hidden flame
cyan cargo
#

given a folder with artifacts from a package build whats the best way to pip install one of those with a set of extras

shy echo
#

What kind are the artifacts?

#

wheels, sdists, mixed?

#

--find-links <path> can work.

cyan cargo
#

mixed, got the dist folder from a previous step an want to install the package in it

shy echo
#

Find links works. You can also have pip install path to <whl>

cyan cargo
#

@shy echo does find links take precedence over pypi?

shy echo
#

I think an explicit path works best?

cyan cargo
#

with explicit paths it seems i cant use extras

shy echo
#

You can?

cyan cargo
#

@shy echo then i might have messed up a glob

fallen scroll
#

A glob won't work if you add extras, because your shell will think that the extras are part of the path (so it won't be able to find any matches).

shy echo
#

^

cyan cargo
#

@fallen scroll im ware, i used some shell expansions on that before with no success

#

@shy echo find-links prefers pypi

#

im now trying pip install "$(echo -n dist/*whl)[toml,test]

shy echo
#

Well, this looks like a gap I didn't know about. 😅

jovial jasper
#

I you know the pkg name,pip install "name[toml,test] @ file://$PWD/dist/name.tgz" shoud work

shy echo
#

Right, I think the issue is that he doesn't know the name of the sdist file already.

#

Though, nothing prevents you from doing it as a two step process FWIW.

#

Figure out the filename and do pip install $filename[foo] or so.

cyan cargo
#

i now have the echo version working, will refine again later

shy echo
#

Awesome

foggy forum
#

It's slightly odd that pip recommends running a pip install --upgrade pip that cannot succeed under EXTERNALLY-MANAGED environments

#

I think it should skip the new version check in those environments

shy echo
#

It should!

foggy forum
shy echo
#

Thanks for flagging!

foggy forum
#

I'll make a ticket?

#

The alternative is that it suggests --break-system-packages !

shy echo
#

Thanks!

#

Yea, well, the option has a clear enough name tho!

wispy coral
#

That means one less patch for us (Debian) to carry, too 🙂

foggy forum
wispy coral
#

Yes. We usually do that, when packages have update notifications. Users of our stable releases are going to frequently have "out of date packages", that's just how it works. So, disabling them saves some annoyance, stops some privacy leaks, and protects some users from breaking their own systems.

foggy forum
#

Ah presumably the user that made the screenshot already ran pip install --upgrade pip

wispy coral
#

Yeah, dunno what happened there

ashen geyser
#

Famously sometimes it makes people

foggy forum
shy echo
#

Debian rewriting INSTALLER is a good idea overall anyway.

foggy forum
#

It's a bit unusual to get into a situation where check_externally_managed() raises but was_installed_by_pip() returns true

#

Because the user already broke their system packages presumably they don't care about breaking them further

#

So I think it might be most correct for the update notice to suggest using --break-system-packages where necessary

foggy forum
shy echo
#

I think this was a case of a user manually upgrading over the existing Debian stuff, and then having that workflow break coz their OS added the marker file and the newer version of pip respects it.

#

I don't think there's an easy answer for this case. :/

foggy forum
#

Maybe print the upgrade notice but don't print the upgrade command?

shy echo
#

That's a non-actionable message tho.

foggy forum
#

Yeah there's no correct action the user can take

#

They need to do a reinstall

#

Maybe the upgrade notice should be replaced with a "you broke your system packages" notice

shy echo
#

@wispy coral do you think it's possible that the DPKGs generated by Debian would patch the INSTALLER file in the .dist-info directory? That'll remove the need for this patch.

foggy forum
#

You'd need to swap some lines in pip too

#

Currently it calls get_remote_version() before was_installed_by_pip()

shy echo
#

Yea, the thing they don't want is the upgrade prompt.

foggy forum
#

and the http call

shy echo
#

But, yea, we should swap those lines.

foggy forum
shy echo
#

LMAO

#

Nah, it's correct on unstable.

#

I wouldn't bother changing/improving things on old Debian versions, and am happy to trust them on this front.

wispy coral
#

It looks like right now, we simply don't provide INSTALLER files at all, in most cases (but there are so many packages, that tooling varies)

#

It would be simple enough to add one

shy echo
#

This isn't to indicate that we shouldn't modify things. It's to indicate that it's intended to be installed via apt

#

Or rather has been.

wispy coral
#

Yeah

shy echo
#

In this context, it's that pip's upgrade prompt logic is gated on whether it's installed by itself.

#

TBH, just setting this in pip's build system's rules file is also good-enough.

wispy coral
#

Anyway, it's too late now to set INSTALLER for this Debian release. We can do it for the next release

shy echo
#

Yo, no hurries.

wispy coral
shy echo
#

Thanks!

balmy shard
#

In a virtual environment when I do the following python3 -m pip download --no-binary :all: --require-hashes --no-deps --dest /tmp/tmp0x_u81ay --requirement /home/debian/repropy/ret.txt' it says that it is Installing build dependencies ... done for each requirement. Can anyone please tell me where exactly it is installing those build dependencies? Asking as the actual virtualenv does not get those build dependencies installed.

shy echo
#

You can increase verbosity and see the details.

obtuse lagoon
#

FYI, this is mentioned in the --verbose description but it's easy to miss if you're like me, but verbosity has 3 levels (apart from no verbosity) which can be enabled with -v, -vv, and -vvv

shy echo
#

IIRC, there's only 2 levels.

obtuse lagoon
#

yeah, I think I vaguely recall that, IIRC the last level doesn't actually add anything, it's just there for future use

#

not sure though

shy echo
#

You can do -vvvvvvvvvvvvvvvvvv if you want.

#

Doesn't change behaviours. :)

obtuse lagoon
#

ah gotcha

shy echo
#

Blame optparse

#

🙃

obtuse lagoon
#

I was under the impression it was limited to -vvv 🤣

fallen scroll
#

FWIW, I believe the direct answer is "in a temporary virtual environment that is deleted after pip is done".

lunar gyro
foggy forum
lunar gyro
#

You don’t seem to use pip directly and this comes from conda mambabuild? I suspect they are using pip in an unsupported way. Should ask them instead.

foggy forum
#

I suspect they are using pip in an unsupported way.
yeah I got that feeling too, but I'm not sure where pip is getting called here. Asked them here https://matrix.to/#/!sAfhhwsNChQnBSbjQJ:gitter.im/$HmK-Uitk6RhElKsp4_YJ-rEFQuBFG8720XpTePnbRCs?via=gitter.im&via=matrix.org

Matrix - Decentralised and secure communication

You're invited to talk on Matrix. If you don't already have a client this link will help you pick one, and join the conversation. If you already have one, this link will help you join the conversation

lunar gyro
#

Where? The only pip call I see is pip check and that doesn’t build anything

foggy forum
#

script: {{ PYTHON }} -m pip install . -vv --no-deps

lunar gyro
#

Oh OK, took a while to find that

foggy forum
#

if I cd into the (rather unusual) prefix directory and run that command manually it works fine

lunar gyro
#

I suspect the --no-build-isolation flag is somewhat set for the invocation (maybe via environ?)

foggy forum
#

does no-build-isolation ignore build-system requires?

lunar gyro
#

It does, without build isolation you are implying you want to set up the environment on your own so pip stays out of the process

#

If you have build isolation on there should be a line Getting requirements to build wheel before Running command Preparing metadata (pyproject.toml)

foggy forum
#

ah I see

#

what's the env var for --no-build-isolation?

#

thanks!

foggy forum
#

putting the .whl in the source repo and doing script: {{ PYTHON }} -m pip install --no-deps ./versioneer-0.28-py3-none-any.whl && {{PYTHON}} -m pip install . -vv --no-deps && pip uninstall --yes versioneer works

#

that's probably not what we want though xD

lunar gyro
#

Yes. I wonder this can be overridden when you actually call pip though?

#

Not familiar with the build tool

graceful meadow
#

Hello. I'm trying to do a cross-platform test and validation of a requireements.txt by using pip. I tried this on a linux system:

python -m pip install --dry-run --ignore-installed --platform win32 --only-binary=:all: -t win32dir "psycopg2 ; sys_platform == 'win32'

But it seems that pip does not take the --platform switch into account when parsing the requirements but only to choose which binary to download. The above command leads to Ignoring psycopg2: markers 'sys_platform == "win32"' don't match your environment.
On the other hand, by removing the sys_platform specifier, pip downloads the right psycopg2-2.9.5-cp310-cp310-win32.whl package.
Should I consider that a bug or am I wrong and it's just that it was not designed for that purpose ?

lunar gyro
graceful meadow
#

Thanks for the answer, I was not aware of this issue.

limber ore
shy echo
#

What about it?

limber ore
#

just a reminder that hopefully we can get it in the next release

#

(I'm not sure what the release schedule is)

shy echo
#

Well, there's a tracking issue for that.

#

And, no one has seemingly had the time to tackle that.

lunar gyro
#

From what I can tell the possibility of this getting into the next release is close to none unless a new contributor pops out from nowhere

shy echo
obtuse lagoon
#

friendly ping

#

it'd be good to have this for the next release

shy echo
#

Merged.

obtuse lagoon
#

whops and I just noticed a typo in the new entry 😅

#

dammit dyslexia

shy echo
#

Follow up PR?

obtuse lagoon
#

yeah

#

doing it rn

shy echo
#

Thank you!

obtuse lagoon
#

will link here for better visibility

hidden flame
#

I did not mean to write a mini essay on pip's issue tracker, but the mention of mypyc caught my attention :)

jovial jasper
jovial jasper
#

This seems to fail on MacOS with python 3.7 and 3.8 only.

shy echo
#

Huh.

#

I'm guessing that sys.path is messed up somehow in the build environment's subprocesses? The platform module is on the standard library paths.

fallen scroll
#

All environment variables of the calling process are erased, and there's probably something critical in there that's required for Python to work.

shy echo
#

I'd be surprised, but that's possible!

cyan cargo
#

with modern pip, can i assume thatthe pip-egg-info folderx are no longer used since a while?

lunar gyro
cyan cargo
lunar gyro
cyan cargo
foggy forum
cyan cargo
#

legacy setup py install based out of tree builds

foggy forum
#

the thing in build/../project.egg-info/ ?

shy echo
#

No, a folder literally named pip-egg-info IIRC.

foggy forum
shy echo
#

The answer to your question is since 2020, @cyan cargo

lunar gyro
#

Hey so I’m not too far off

foggy forum
#

setuptools still makes a project.egg-info in the source directory though

cyan cargo
#

I dropped the code for it from setuptools_scm

jovial jasper
#

Eh I was sure to have worked on that but could not find it right away 🙂 Thanks @shy echo !

#

BTW, anyone with a mac willing to look at the failing test?

shy echo
#

I tried, but couldn't. :/

#

Let's xfail it for now, and file an issue for fixing it?

lunar gyro
#

It’s so difficult to look into the build process

shy echo
#

100%

#

I want to switch it away from subprocess calls to in-process calls.

lunar gyro
#

It’d be slightly viable if we have a mode that disables build isolation but still populates the environment

#

That way we could inject into the site-packages setuptools

shy echo
#

Why is that necessary?

#

FWIW, my motivation is to reduce the subprocess stack that we end up having in our runs, which make attaching a debugger and diagnosing issues ~impossible.

shy echo
shy echo
stuck girder
#

now resolved:

We’ve identified an infrastructure change that has been rolled back and we are monitoring recovery.

shy echo
#

Yea, fun.

shy echo
#

@lunar gyro @wheat pollen So... do y'all remember pyrax==1.9.8?

#

It's the one that Poetry's author had shared with us and it caught a bunch of issues in the resolver and it was ridiculously difficult to resolve correctly and all that...

#

With the current main, I got a resolution result in 26s.

#

13s with a primed cache.

wheat pollen
#

I don't remember it, but that's awesome.

shy echo
#

I'm excited about the backjumping change. 😅

cyan zephyr
#

Any schedules on 23.1 release? I hope pip-tools would have time to adapt to pip internal changes before the release.

shy echo
#

Early April is the plan, I think.

cyan zephyr
foggy forum
#

Is 23.1 switching to pep517 by default?

shy echo
#

If you don't have setuptools or wheel installed.

#

In the environment.

#

Which is a lot of people coz venv doesn't install wheel.

foggy forum
#

You mean setuptools and wheel?

shy echo
#

yea

foggy forum
#

Hmm probably not enough for flake8 pyproject.toml then

shy echo
#

LMAO

#

I don't think the flake8 thing is going to change unless Anthony can convince himself to remove all the other configuration methods.

limber ore
#

just use Ruff

polar nova
#

ruff > flake8

#

depending on the repo size, you can run ruff over 50 times in the time it takes flake8 to run once

dapper laurel
#

and now, with ruff having most of flake8 plugins implemented, flake8 makes little sense to use

mint patrol
#

Is the ramdisk setup failing typical for pip CI? Happened twice in a row for me :/

shy echo
#

It's not, no.

#

Hmm, I wonder if Github broke RAM disks again. 🙈

mint patrol
#

@shy echo also hope your truststore issue is resolved now! Was just working on getting the trustore+pip PR ready to review

shy echo
#

I haven't tried since. The only place I really can test out truststore is on $work stuff, and I only saw that you made a ridiculously quick fix on the weekend.

And, I'd prefer to not need to open $work stuff on the weekend if I can. (been quite successful so far!)

#

But thank you for the quick fix! It's appreciated! ^.^

mint patrol
#

We got lucky in that David and I were both online at the right time 😉 We're both happy to see it getting used and want to unblock our early users 🚀

polar nova
shy echo
fringe sand
#

Hi, just joined the server following a question I have asked on Stack Overflow (https://stackoverflow.com/q/75907427/21537524), and a user pointed me towards here. My question was the following, I would like to know how to clone and compile C++ code into binaries within the package during pip install? I don't know if I should paste the whole post over here or not, don't want to feel like invading in any way

Stack Overflow
How to clone and compile C++ code into binaries within the package ...

I need to make a Python package openface, that will act as an API of some existing C++ research code OpenFace, that I don't own.
I have forked the existing code base into openface-backend, and rewr...

shy echo
#

This is not really a pip question though, but a question about build backends.

#

OP posted on https://discuss.python.org/t/how-to-clone-and-compile-c-code-into-binaries-within-the-package-during-pip-install/25394?u=pradyunsg as well. Let's leave the rest of the discussion there

Discussions on Python.org
How to clone and compile C++ code into binaries within the package ...

(I have this question yesterday on Stack Overflow, How to clone and compile C++ code into binaries within the package during pip install? - Stack Overflow, and an answer suggested me to ask it here instead) I need to make a Python package openface, that will act as an API of some existing C++ research code OpenFace, that I don’t own. I have ...

shy echo
#

@jovial jasper Are you around on Discord?

jovial jasper
#

Hi @shy echo I see your last posts. Indeed patching packaging is not a good idea. Especially since it emits spurious warnings about LegacyVersions when normalizing for comparison or something.

shy echo
#

I'm honestly not sure how difficult the suggestions I've made are TBH!

jovial jasper
#

Not sure either. There are quite a few places where packaging.version.parse is used. All these spots need some work if we want to warn with relevant context.

#

I've not looked at detecting LegacySpecifiers from pip-land.

#

All in all, that looks like too much for me to do in time for 23.1.

shy echo
#

No worries. I'll be spending some cycles on this in the next couple of days.

limber ore
#

is there a way to download and have --find-links take precedence? currently pip download -f dist --no-cache-dir pkg_name will always download unless I force the built version to be different in which case nothing will be found and the link directory is finally utilized

#

(I understand this may be a fringe use case since we're trying to download something that already exists in a directory)

#

I feel like there may be some magic command line incantation but I can't figure it out

lunar gyro
#

No

lunar gyro
#

Urgh, why is RAMDisk failing so often lately in CI. Something seems off.

limber ore
#

how much is the CI time impacted without that?

shy echo
#

One of us should file an issue in actions/virtual-environments.

shy echo
fallen scroll
#

Has anyone looked into using SeparateBodyFileCache for pip's downloads? I was wondering why it seemed to take a second for pip to find cached wheels, and I discovered that it currently (indirectly, through cachecontrol) loads the entire cached file into memory before using it.

shy echo
#

I don't think so, not yet.

idle harness
# fallen scroll Has anyone looked into using `SeparateBodyFileCache` for pip's downloads? I was ...

PDM also suffers from this: https://github.com/pdm-project/pdm/pull/1498
It is blocked by the upstream cachecontrol project. Does anyone know the maintainer here?
It makes the cache not working on, for example, pytorch private index.

GitHub
Use split-body cache to make cached wheels disk-stable by tgolsson ...

Hey!
This relates to #1496 since it's one of the things that caused files wheels to "change" when they really were just modifying headers.
Doing a bit of an ugly PR here just to get a...

little kite
#

Would it be possible to upload some metadata alongside the pip zipapp? e.g. if I were to cache the zipapp I wouldn't want to redownload it unless the version's changed

shy echo
#

Technically, yes. The piece missing is someone deciding how we want the folder structure to look like for the zipapp.

FWIW, regular content caching should be set up on that, no?

little kite
#

You mean by doing a head request and checking the etag for instance? Yeah, I suppose that'd probably be sufficient, unless we wanted to know what the actual version of pip in the zipapp is

fierce temple
#

I can't trick/force the dependency resolver into thinking one package is another right

#

foo depends on bar, for some strange reason the developers of bar release their prereleases to bar-dev

shy echo
#

No.

fierce temple
#

But that doesn't help for installing foo, as I can't simply install foo and bar-dev

shy echo
#

Or, at least not at the moment.

fierce temple
#

K, figured I'd have seen it at this point if so. Thanks

lunar gyro
#

That’s one thing I’ve been thinking we should do. There’s already spec for this (Provides-Dist), we just need an implementation

shy echo
#

And a story for how that works in a world with uncurated repositories.

#

The last time that this came up, the social part was the blocking concern IIRC.

fierce temple
#

Meaning what sorry

#

(the uncurated bit)

#

Ah as in you don't want some random malware package claiming it provides django or whatever?

shy echo
#

Yea.

fierce temple
#

Is there any extra danger there than said malware package being installed itself? (I assume you're about to explain that to me 🙂

shy echo
#

And, also less malicious code but unfriendly ones, eg flask claiming it provides django, and deciding on what the story needs to be (is it that we don't care about giving tools to deal with this, or is it that we do care?)

fierce temple
#

Right ok. Or I guess you exclude it from anything but user-facing tools

shy echo
#

Not sure what you mean by that.

fierce temple
#

(I'm not familiar with how Provides-Dist works so I could be saying nonense as usual). But what I mean is such that a user can say "when something says it wants foo I want foo provided by bar", but a package baz cannot say "I need foo and the foo I want is bar"

#

(So exclude the functionality from dependencies, but allow frontends to do it or whatever)

#

And if you install bar as part of installing baz (because it directly depends on it) you forget the fact that it claims to provide foo. OK I'm gonna stop talking since it seems like these are not so insightful ideas 😄 so probably if this is reasonable someone who thinks about it mor ethan my 3 seconds will come up with better ones

lunar gyro
#

I do have some ideas around it, not very structured but I think should work with some polish (which is the bulk of the work of course)

hoary mist
#

I think Provides-Dist is fine as long as you don't take it into account unless you've already decided to install the thing providing it

limber ore
#

can someone please point me to the code that creates Windows executables for CLI entry points?

lunar gyro
limber ore
#

thank you

limber ore
#

is MUSL not yet supported?

❯ pip download --only-binary=:all: --platform=musllinux_1_1 --python-version=3.10.9 --implementation=cp pillow
ERROR: Could not find a version that satisfies the requirement pillow (from versions: none)
ERROR: No matching distribution found for pillow

https://pypi.org/project/Pillow/#files

shy echo
#

Using those tag options is problematic; coz they don't work quite as gracefully.

limber ore
limber ore
foggy forum
shy echo
#

Ask the setuptools folks. They might not know about this.

#

Also, why does this matter?

foggy forum
#

regular installs behave the same between pyproject.toml and setup.py it's only -e that causes a problem here

#

the issue is on distributed contributors are having trouble with our new pyproject.toml editable installs

#

eg usually you have a project checked out like:

graingert
└── projects
    ├── dask
    │   ├── dask
    │   └── pyproject.toml
    ├── demo.ipynb
    └── distributed
        ├── distributed
        └── pyproject.toml

5 directories, 3 files

and when you're editing graingert/projects/demo.ipynb you expect to have the editable packages graingert/projects/dask/dask and graingert/projects/distributed/distributed on the sys.path

#

but you end up with graingert/projects/dask and graingert/projects/distributed

limber ore
foggy forum
limber ore
foggy forum
lunar gyro
#

I seem to recall @shy echo wanted me to do something during the sprint

shy echo
maiden island
#

To save me time, did pip >= 23.x move to using the PEP691 JSON Simple API?

maiden island
#

Checked change log - Can't see anything that stands out other than direct_url.json changes ... But that seems to respect the hashes dict in metadata ...

#

Does pip use direct_url.json when a mirror is in use?

shy echo
#

We moved in an earlier release.

maiden island
#

22.2? - I didn't interpret that from the NEWS, but ok cool.

shy echo
#

Support PEP 691. (#11158)

maiden island
#

I think that half explains the bug old mate is getting. I think he just has either:

  1. Old generated index.v1_json without black2b_256 and the JSON metadata from pypi does
  2. Serving HTML index from the mirror and messing pip up
maiden island
shy echo
#

Oh, 691 is using content negotiation and also says use JSON by default IIRC.

maiden island
#

I don't recall the PEP stating JSON by default if that's what you mean

#

But, not important. See what they come back with and go from there ... But I might look more to see if I can repro - I just don't have any mirrors handy anymore ... I don't even run bandersnatch or have access to PyPI's anymore

hoary mist
#

pip should still work fully with a mirror that doesn't have PEP 691 json

shy echo
#

^ that's what the content negotiation is for

maiden island
#

No idea

#

But I don't think pip does that from memory

shy echo
#

There is some weirdness around blake2b and warehouse, based on a conversation I had today.

maiden island
#

It uses the mirror + it's simple API

hoary mist
#

pip just says "hey give me this url, I'll accept PEP 691 json, PEP 691 html, or regular html, but I prefer PEP 691 json", and the server goes "okay cool, here's the ersponse in X format"

maiden island
#

When did blake2b come around?

hoary mist
#

pip does not use the pypi api, it only uses the simple api

#

er pypi json api*

maiden island
shy echo
#

We need better names still.

#

🙈

hoary mist
#

/pypi/*/json

#

lol

#

listen

#

we're good at naming things

#

that's why we have a library called packaging

maiden island
hoary mist
#

which is definitely never confusing

shy echo
#

Never, correct.

maiden island
#

No idea where to report that

hoary mist
#

that's probably the correct place

shy echo
#

pypi/infra seems correct.

hoary mist
#

blake2b has been used in pypi since warehouse launched iirc

#

I'm not sure if we always exposed in the json api

maiden island
#

Really, I never recalled it - But I don't pay that much attention

lunar gyro
limber ore
#

I'm so happy this exists today

shy echo
#

:)

ashen geyser
#

haha which usecase? system without internet access where you want to sideload pip.pyz and a wheelhouse to?

shy echo
#

Oh, yea, there's a --python flag too.

limber ore
limber ore
native obsidian
dusty crag
#

Is it normal for pip to not consider macos wheels with versions like 12_6? Look at https://sourceforge.net/p/ruamel-yaml-clib/tickets/20/ -- basically a wheel such ruamel.yaml.clib-0.2.7-cp311-cp311-macosx_12_6_arm64.whl is ignored because pip seems to recognize only 12_0 , 13_0,... but ironically for 10_ it did support minor versions.

#

Very easy to spot running python -m pip debug --verbose|grep py311.

lunar gyro
#

TL;DR macOS 11 and up are semver-ish so build backends are supposed to normalise 12.6 to macosx_12_0. The 10.x line is different because bumping the second segment for that line denotes ABI breakage.

shy echo
#

Yea, MacOS changed their ABI promises so the expectations changed.

dusty crag
#

Maybe putting some comment son https://sourceforge.net/p/ruamel-yaml-clib/tickets/20/ would help Anthon acknowledge that his builds are not really usable due to the way he buit them (likely with outdated tools as python -m build seems to do the right thing locally for me).

lunar gyro
#

What build backend is ruamel.yaml using? Maybe we should fix that backend instead, urging the package to upgrade the backend should be a much easier sell

shy echo
willow flicker
limber ore
#

it says it's supposed to but I cannot get that to work on any platform, whether using the file in the current directory, a relative path, nor an absolute path

hard thunder
limber ore
hard thunder
#

pip docs are inconsistent, the first command is without -m, second is with

#

It should be without

shy echo
#

Oh, there shouldn't be a -m!

dapper laurel
#

"good first issue" PR candidate!

hard thunder
dusty crag
#

is there a way to detect if my package dependencies are not met? Basically I wonder if I can call from python pip check in order to see if there are any conflicts reported. Apparently some downstream packagers forgot to package the right dependencies in.

lunar gyro
shy echo
#

I think build has built out similar logic in that way, but I don't know if it's public API or if it can be adapted.

dusty crag
#

I would prefer to use some relatively stable api for that. importlib/packaging are even better than pip. Maybe I can spot something similar already implemented.

#

i do only want to check version of one package but i would prefer to check against the dependency I have declared in my own pyprojecty.toml in order to avoid putting the version condition in two places (metadata and code)

limber ore
limber ore
#

are we going to release a version that drops 3.7 soon?

stuck girder
lunar gyro
#

Also nothing in the code base really benefits from 3.8+ so it’s not practically meaningful

limber ore
#

okay thanks! I was asking because I'm about to release a new version of #pyapp today and if we were going to release a version soon I was going to hardcode the final supported version of pip for 3.7

#

side note: I think even if we wouldn't directly benefit in code we should be aggressive about dropping EOL versions

remote osprey
# limber ore side note: I think even if we wouldn't directly benefit in code we should be agg...

One problem with doing that arbitrarily for packaging projects "low in the stack", especially pure-Python and where there's little other practical benefit, means it could needlessly delay adoption of the up to date standards those projects implement, given it is typically much easier for users to upgrade package versions than their Python runtime version, and practically speaking 3.7 is tied with 3.8 for the most widely deployed Python version in terms of PyPI package downloads (though presumably biased by automated downloads running on RHEL and Debian, I would guess).

limber ore
#

that's an interesting point that I hadn't thought about however I wonder if that is actually true in practice e.g. if a distribution ships pip X.Y.Z does the same distribution increment Y or are they pinned to only patch upgrades? if the latter then adoption of standards is not impacted

remote osprey
limber ore
mint patrol
#

Hey folks, I see that security@python.org is listed as the security contact for pip. There are some changes to the PSRT that I'm working on, who is the best contact to involve in those discussions from the pip POV?

shy echo
#

Me?

#

I'm on the PSRT for pip, so like... There's that.

mint patrol
#

Cool, I'll send you an email then Pradyun 🙂

sharp pebble
#

RE: My previous discussion on AV triggering on Python ensurepip invocation.
@rare umbra CC'ing for visibility (though I believe this is out of your realm, just closing the circle there.)
It appears at least RAV is triggering for 'suspicious arguments', which tracks with Steve's initial comment that some command line arguments are probably the source of the detection.

image.png
shy echo
dapper laurel
#

great choice

thin ruin
hard thunder
#

We fixed a bug that meant pip could not use PEP 658 metadata served by a package index. As a result pip will now use the metadata served by PyPI, hopefully improving download speeds significantly for projects with large wheels.
Cool!

shadow shuttle
#

hello! https://github.com/pypa/pip/issues/8076 / https://discuss.python.org/t/proposal-overrides-for-installers/23666 bites me often enough that i'd like to solve it, even if only just in my own fork of pip.

...but now comes my sketchy question. reading resolvelib, it looks like if a resolvelib.Provider were to simply ignore incompatibilities in find_matches and return candidates filtered to the override, it would do exactly what i want. any idea if that's true?

yes, the Provider interface documents the following: All incompatibilities *must* be excluded from the return value. yes, this makes sense for regular usage where you don't want incompatibilities, but in this case i want to ignore incompatibilities! did i read the code wrong and actually resolvelib will break horribly if this doesn't hold?

i guess i'm about to find out either way, but any insight would be welcome

ashen geyser
#

Regarding UX design in the face of a messy reality, I really think yarn is one of the best package managers. It has this: https://yarnpkg.com/configuration/manifest#resolutions

I think that design is best: You don’t just globally relax rules (leading to cargo culting rules that nobody knows the rationale for anymore) but basically patch a dependency’s rules.

Probably doesn’t help with your question, sorry.

Manifest fields

List of all the supported fields for a Yarn project manifest (package.json files)

dapper laurel
#

I think PDM has something simillar

limber ore
#

Cargo does that as well but I think it's not as feasible in Python because our environments must have exactly one version of a package whereas with Rust and JavaScript there can be multiple

ashen geyser
#

That comes with its own problems. Rust people often re-export dependencies to reduce the problem of “I’m returning an object from lib1v1 and trying to pass it to an API expecting the version from lib1v2.

For us it’d be more like “lib1 depends on lib2<v2 while in reality it handles lib2v2 just fine” or “every single version of lib1 failed to specify a requires-python<3.11”.

For both use cases, this is a useful feature though.

shy echo
#

That way, if you end up with that candidate in incompatibilities, you have no solution.

#

FWIW, returning something that is in incompatibilities will/can end up putting the resolver in bad state. I forget if we had an explicit check for that case?

mint patrol
#

Has there been any discussion on SBOM generation support for pip / other installers? I've been doing some experiments with SBOM generation from packaging metadata/PEP drafts. Obviously can be a separate tool (a-la pip-audit) but wanted to kick off a discussion on that.

dapper laurel
#

On Poetry side, if some standard will be made, we are open to discussion and implementation

mint patrol
#

When you say standard, do you mean a standard for transforming existing packaging metadata that is standard compliant into SBOMs (SPDX, CycloneDX, etc)

dapper laurel
#

I mean standard in terms of installers/packaging standards

#

Prefferably something backed up with PEP

mint patrol
#

Gotcha, yeah that one that has my eyes right now is PEP 710, you should review that one for feasibility in poetry 🙂

#

That one makes SBOMs possible at all from a pre-installed environment.

#

I could see the utility in a PEP that basically says, hey if you're going to offer SBOM functionality here's the list of things you take, from where, and where you put them, for each SBOM format. Would mean conformity across packaging tools that end up offering that information.

shy echo
#

I think there's lukewarm interest in SBOMs, because ~all of the parties who care are institutional, and most of the maintainers of installers are not working on them in an institutional capacity. 😅

#

You could in theory be generating fully populated SBOMs today, with pip install's report functionality. It does move the onus for doing that to a wrapper layer around pip tho.

#

That's got all the information that one could want when populating from an empty slate, to know what's going to be available.

mint patrol
#

ack on pip install report, I got told about that functionality elsewhere too so will look at it for sure.

hoary mist
#

the big difference is pip install reports require you to know up ofront you want that information, and 710 lets you decide you want it later on

mint patrol
#

(When you say "that information", can you clarify @hoary mist ?)

shy echo
#

Yes, and most people who care about supply chain security probably also care about controlling build-time stuff. 😅

hoary mist
#

the provenance information

shy echo
#

Or, I guess they should unless it's somewhere in the gradient of security theatre IMO. 😅

hoary mist
shy echo
mint patrol
#

Totally understand the lack of interest in SBOMs since they're for compliance but provenance, reproducibility, etc seem useful to most/all software consumers imo?

shy echo
#

Yup, I never said otherwise. 😉

The question was about SBOMs, which is what I was responding to.

mint patrol
#

Gotcha!

#

I think Donald was replying to the cases of "where X came from" as provenance info, not SBOMs. Might be wrong there?

hoary mist
#

Yea I was mostly just riffing on pip install reports vs pep 710 provenance

shy echo
#

^which I agree about the difference of.

hoary mist
#

though I defintely see use cases for SBOMs where youcan't control the pip install