#pip

1 messages · Page 8 of 1

shy echo
#

We might need like a "file database" later, because of pkgutil-style namespace packages.

#

It might also be that we can never actually do this -- I don't know yet -- but if we do end up going that route, it might make sense for installer to update to reflect whatever the resolution of things would be.

#

It's easier to relax later in installer than to make pip stricter-than-feasible.

shy echo
lunar gyro
#

Nothing except someone to review it

shy echo
#

Neato. I'll do that today then. :)

lunar gyro
#

When do we plan to release 22.1? Maybe I can add a --use-feature option for it

#

(Likely in a separate PR)

shy echo
#

I'm thinking sometime next week, but I wouldn't hurry up about it -- I'm A-OK to get this in, as part of a patch release if need be, as long as we flip the switch in this to 3.11+ instead of 3.10+.

#

Oh, this is just off unless there's an env-var -- ignore what I said above then. :P

high stone
#

hi! i installed python 3.8.6 x86_64 (python.org 64bit installer) on macos 12.3.1 arm64. i found that when using virtualenv, the venv'd python pip will not accept 10_9 wheel tags anymore and demands >= 12_0.

> /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 -m pip debug --verbose | grep cp38-cp38-macosx_10_9
  cp38-cp38-macosx_10_9_x86_64
  cp38-cp38-macosx_10_9_intel
  cp38-cp38-macosx_10_9_fat64
  cp38-cp38-macosx_10_9_fat32
  cp38-cp38-macosx_10_9_universal2
  cp38-cp38-macosx_10_9_universal

> ll venv-py3/bin/python
[...] venv-py3/bin/python@ -> /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8

> venv-py3/bin/python3.8 -m pip debug --verbose | grep cp38-cp38-macosx_10_9
[nothing]

i'm utterly mystified by this. what is going on?!

high stone
#

ok, i upgraded the installation to 3.8.10 and now it seems to work. whatever that was...

lunar gyro
#

A major version bump in OS version (e.g. 10 -> 11 -> 12) usually means there are incompatibilities, so the toolchain by default rejects a library compiled against v10 on v11 (semantic versioning). But Apple decided to not follow this version logic, so the toolchain needed to implement a special case for it. This special logic was added around late 2020 (when it’s announced with the introduction of the M1 chips) in the middle of the Python 3.8 timeline.

high stone
#

ah, good to know. thanks 🙂 I'll simply make sure to use 3.8.10 as a baseline

cyan cargo
#

anyone aware of how the pip tests used to use path.py vs pathlib vs pytests old py.path.local, as im porting the setuptools_scm testsuite over to pathlib, im seeing a lot of pain points and i wonder if pip has something nice to extract/republish

shy echo
#

We have our own path implementation, that we've modified incrementally to match pathlib.

#

And it's mostly been a drop in replacement.

#

That said, we still haven't made the switch. 🙈

cyan cargo
#

unless you have type hints already in place, its a daunting task ^^

shy echo
#

Oh, monkeytype is a great for that.

#

I'm not a 100% sure if it annotated tests, but try it. Run the test suite once, and it should be able to extract all the types for the test fixtures.

#

And dump them inline.

lunar gyro
#

I should start a quest converting everything to pathlib

shy echo
#

I mean, for the test suite, we're almost there.

#

I'm inclined to suggest starting there.

frosty pier
little kite
#

I don't know what shell they are using but for the var to be visible to pip they need to export it, in Bash that would be done with export foo=bar

shy echo
#

Yea, it might just be missing the export.

cyan cargo
#

anyone aware of tols surrounding pip that help catching missed upgrades to libraries due to lifted minimal python versions

obtuse lagoon
#

not around pip

#

but it's outside pip

cyan cargo
willow flicker
#

Is there a way to get pip to not print out the "run as root" warning? (for running in manylinux via cibuildwheel)

limber ore
#

no, I've tried

willow flicker
stuck girder
#

I seem to remember there was an issue to make all the warnings configurable/silenceable?

willow flicker
stuck girder
#

yeah, there's a few. I think the idea for a general solution came from one of those specific ones, but in the end a specific one was made because it was easier/quicker to implement 🙂

little kite
#

it's been done, just hasn't been released yet

#

the wording is a bit clumsy, would have suggested --no-warn-running-as-root

shy echo
#

That's been shortened since.

#

You can check what it's called in the current beta release. :)

chrome epoch
#

@shy echo sorry to dump that possible regression on you, i'm also available to help debug synchronously if it'll help 🙂

shy echo
#

Heh, I'm only seeing this now @chrome epoch. 🙈

Appreciate the offer! It turned out to be not that complicated, once I had a reproducer. I'm really happy that --debug exists now though!

willow flicker
#

FYI, if a package has releases that are non pre-releases but have all been yanked, and the only non-yanked releases are pre-releases, pip won't install the prereleases automatically like it would if the yanked releases didn't exist. Weird corner case, but someone saw it happen.

#

fastjet had 0.0.1-0.0.3, all yanked, and has had betas or RCs (didn't check) since then, but won't pip install fastjet without a --pre flag due to the old yanked releases.

shy echo
#

Is this true with the latest version?

#

And what about with the previous version?

willow flicker
#

They've just fully deleted 0.0.1-0.0.3 to make it work, so now I can't test it with different pip versions. 😦

#

Probably not the very latest pip? Wildly guessing though.

shy echo
#

Okie. I'll set up a local test, and try this.

#

There were fixes around yanked releases.

#

(in 22.1)

willow flicker
#

The failure was observed on 21.0.1. By the time I tried it on 22.1, the old versions had been deleted.

shy echo
#

Okie. Good to know!

willow flicker
#

What's the ETA for the pip with the opt-in build dep verification? All conda recipes with cmake & ninja are broken (since they don't install the actual Python package), most recipes with numpy are broken (since oldest-supported-numpy pins the oldest PyPI version, but conda often builds with a higher minimum version). We are also holding off on the #cibuildwheel release (with 3.11 beta 1 as opt-in!!!) until we get the new pip, since it will likely break a few cibuildwheel users (hypothetical in that case).

#

Going to try to get pip 21.1.0 marked as broken in conda, excluding 22.1.0 on every recipe is not scaling well.

shy echo
#

There isn't really an ETA -- I've been stuck AFK for the last few days, which is why I haven't been able to cut the release.

#

It's basically blocked until "Pradyun gets 30 free minutes with his laptop".

willow flicker
#

"sort of coming soon" is fine. 🙂

keen summit
#

Just to confirm: pip's per-user wheel/http cache are intended to be safe to use from multiple pip invocations at once, right?

shy echo
#

Read, yes?

#

Write, I think so?

#

I don't know off the top of my head.

hoary mist
#

I answered this elsewhere, but for posterity:

  • HTTP cache should be fully concurrent execution safe, it uses file locking to coordination concurrent access.
  • Wheel cache will build in a temporary directory, then shutil.move the file over, so it depends (tm), on the same filesystem it'll use os.rename, but tempdirs are often not on the same filesystem, so it'll use shutil.copy2, which maybe has a race condition if it's doing the copy in userspace, but newer python versions use kernel/os level copy file APIs that may or may not have that data race
foggy forum
foggy forum
keen summit
hoary mist
mint patrol
#

Writing up the documentation for the truststore feature, are ya'll cool with sending users having issues with --use-feature=truststore to the truststore repo or would you prefer keeping them in pypa/pip?

shy echo
#

I'm a-OK with directing users to the truststore repository.

mint patrol
#

Cool! Also, do we have a method/connection to elevate answers on StackOverflow? There are a bunch that would have a much better answer to "corporate proxies certs" than to disable verification now.

shy echo
#

Write them, and then wait for 27 years. XD

mint patrol
#

hehehe

foggy forum
#

What do folks think about pip dropping support for unsupported versions of openssl?

hoary mist
#

does pip even use openssl besides indirectly through requests/urllib3/stdlib socket

foggy forum
#

maybe sha3

foggy forum
hoary mist
#

oh this is urllib3 wanting to drop support for it

foggy forum
#

Yep and SecureTransport

#

Or instead of dropping support pip could adopt the same support policy of openssl

hoary mist
#

BigQuery can tell you what versions of OpenSSL are being used to hit PyPI, but personally I wouldn't care if it dropped support

foggy forum
#

Yeah I ran out of credit xD

#

could we get an ssl version in the environment markers?

lunar gyro
#

urllib3 plans to transition to a new implementation called truststore to provide cert validation, we’ve been working with them and will likely drop openssl even before urllib3 does (besides on linux where the system ssl is openssl in the first place)

foggy forum
#

It's still a subclass of ssl.SSLContext, truststore is 3.10 only so it's always 1.1.1+

lunar gyro
#

If pip ever do decide to drop open openssl it will follow a similar strategy as well. We won’t just say an old openssl stops working, it’ll likely be bounded by Python version, and since pip drops a Python version when it goes EOL those old versions gets discarded gradually

foggy forum
#

No I mean dropping openssl <1.1.1 not dropping openssl support outright

lunar gyro
#

I don’t think pip should be actively enforcing that. CPython already does, and that’s good enough.

young nacelle
#

Hello guys, I am currently working on the editable installation for setuptools and I got myself thinking about the following:

  • I understand that right now there is no clear way on how a backend can pass messages to the frontend (in this case pip), this piece of UX is not covered by the existing PEPs.
    However, I was under the impression that warnings would be displayed (if nothing else). Is this assumption incorrect?

    When I tried to use a warning to display a note to the user, I could not see any warning. My idea to bring attention to the directory hosting the off-band link tree used for strict installs, so it is not removed accidentally.

lunar gyro
#

IIRC right now pip doesn’t show any stdout or stderr from the backend, unless there’s a build failure or in verbose mode

young nacelle
cyan cargo
#

hmm, i'm wondering, has any experiment in the direction of using tools like mypyc to create "speed up" builds of pip been done

stuck girder
hidden flame
#

if you want more details (since I didn't cover the infrastructure changes much) just ask!

#

I'm happy to lend a helping hand.

mint patrol
#

Hey all! We're deciding which version of OpenSSL we should support for urllib3 v2.0 (we're favoring 1.1.1+) and wanted to drop this issue here as there are some special concerns for pip and users of the unbundled downstream Linux pip packages: https://github.com/urllib3/urllib3/issues/2168#issuecomment-1162249750

GitHub
[v2] Drop support for OpenSSL<1.0.2 · Issue #2168 · urllib3/urllib3

Context The TLS situation in Python has considerably improved since the early years of urllib3, thanks to the hard work of persons like Christian Heimes and Cory Benfield. urllib3 took advantage of...

hoary mist
#

if you're using pip to upgrade system packages, you get to keep both pieces

mint patrol
#

haha, I know it's not recommended but also know you likely get reports anyways.

cyan cargo
#

would it make sense for pips build isolation to sometimes ignore --no-binary :all: - setuptools-scm 7.x hadbeen a dsaster for homebrew as there was a loop between setuptools_scm and importlib_metadata when using non-binary builds

little kite
#

I don't think that would matter for Homebrew, they don't use pip to perform dependency resolution (i.e. they pass --no-deps), they just give it a list of sdists to build and install

lunar gyro
#

I think it does if there’s “sdist inception” i.e. if an sdist requests a build dependency that needs to be built from sdist with a build dependency and so on. no-deps does not apply to this (because you just can’t). Maybe this is exactly where pip needs to draw the line? A build dependency’s build dependency must have binary distributions available.

limber ore
#

A build dependency’s build dependency must have binary distributions available.

quite reasonable 👍

little kite
#

Is that meaningfully different from a runtime dependency chain?

lunar gyro
#

--no-deps doesn’t apply to build dependencies because without dependencies the project won’t build

little kite
#

If no deps weren’t passed, would the same then be done for transitive runtime sdist dependencies?

lunar gyro
#

Runtime dependencies only “need” each other at runtime, so the dependency loop can be easily broken (the resolver can simply record “this exists”). But build dependencies in a loop need each other to build themselves, and neither package can be built without the other one being built first, resulting in a dead knot.

little kite
#

well, if we're only looking to break the dependency loop, why would we require that second-level dependencies have binary distributions?

#

pip already has the ability to detect loops I thought (the fork bomb protection mechanism) so could fall back on wheels selectively

lunar gyro
#

Because it’s much easier? 😛 We cannot use the existing detection because build dependencies of each build dependency are in a dedicated isolated environment, and the existing detection does not work as-is (isolated environments can have different versions of a same package, which is not the case for runtime dependencies).

shy echo
#

I believe we allow nested sdist-needs-sdist build logic.

#

There's actually fork-bomb protection logic in pip, for precisely this case.

fickle shell
stuck girder
fickle shell
# stuck girder in fact, I contributed a fix to disable the pip version check in this action (be...

It will be EOL for me only in 2026 🙂 https://jugmac00.github.io/blog/ubuntu-versions-cheat-sheet/

Ubuntu Versions Cheat Sheet

about versions, names and versions
version code name python go debhelper end of standard support eol 14.04 trusty tahr 2.7.5 + 3.4 ? 9 04/2019 04/2024 16.04 xenial xerus 3.5 ? 9 04/2021 04/2026 18.04 bionic beaver 3.6 ? 11 04/2023 04/2028 20.04 focal fossa 3.8 1.13 + 1.14 12 04/2025 04/2030 22.04 jammy jellyfish 3.

stuck girder
#

I'm sorry! I bet a warning showing as an error but not erroring is the least of your worries! 😄

lunar gyro
foggy forum
#

What should package publishers do about packages that can't be installed anymore? Eg because of ez_setup/2to3?

#

Go back and yank them all?

young nacelle
#

I suppose that if someone is interested in supporting these packages they could create a shim to add backward compatibility. Other than that there is not much that can be done right? The deprecation period of such features is there so people have time to think about a strategy. This includes contributing to packages they care about... (people also can try to claim abandoned packages if they don't have a response from the maintainers when proposing changes that are important like these ones).

lunar gyro
#

I think the thing here is not about transition but more about what to do with those old versions after you’ve upgraded the build toolchain

young nacelle
# lunar gyro I think the thing here is not about transition but more about what to do with th...

I see... Python doesn't have a good story for this right? I believe it is related to the problem described by @willow flicker in his article about upper bound constraints https://iscinumpy.dev/post/bound-version-constraints/.

Right now there is no way we can tell PyPI/pip that some versions stopped working with the "uncapped" dependencies it once listed.

I don't know, maybe the long term solution for the community would be adding the caps, but having an automatic system to increase the cap once a new dependency version is released and tested...

willow flicker
#

having an automatic system to increase the cap

That doesn't scale. That requires no one else caps you, and it requires that releases happen on arbitrary schedules (based on your dependencies, rather than what you are doing). And it requires that you are still around to make releases, which is what caps were supposed to "help" with in the first place!

#

The primary problem is you are exposed to those old versions because someone else is capping somewhere.

hoary mist
#

it's a hard problem because fixing it directly would require replacing the contents of old packages

#

Maybe pip needs some way (if it doesn't have one) to add additional constraints to the build environment itself

lunar gyro
#

We could, but the problem is what constraints to add for what packages. Those packages are u constrained to begin with, and we certainly don’t want to cap all unconstrained packages.

hoary mist
#

yea

#

there's always the escape of building a wheel manually and storing it locally

obtuse lagoon
#

has anyone thought of allowing providing PyPI with some metadata of supported package versions?

#

that way pip could prioritize maintained versions, which could somewhat mitigate some of these issues

#

it would big a big change though, but it could be rolled out like the resolver

hoary mist
#

so that alternative installers don't have to reimplement

shy echo
#

A separate library sounds about right to me!

hoary mist
#

quick, someone think of a name that's better than pypi-user-agent

shy echo
#

LOL

limber ore
shy echo
#

linehaul-user-agent?

hoary mist
#

oh linehaul-user-agent might be the better one

#

since it's really linehaul that this works for

#

I dunno

shy echo
#

You typed my inner monologue.

hoary mist
#

I plan to throw this up under pypa/ on GH unless someone objects

#

maybe it should be pypi/ hmm

#

dunno

#

guess I'll throw it on my personal for now

foggy forum
hoary mist
#

I dunno anything about Jython

shy echo
#

FWIW, I think it should be pypi/?

#

Although, pypa/ also makes sense and is where most of the reusable code lives.

lunar gyro
#

PyPI feels like a more visible? place since it’s mostly useful for people interacting with PyPI, not any index

shy echo
#

That's is on a machine with a fairly high performance SSD + single core CPU scores.

I'll try and see if the numbers look more favourable on something with slower IO systems and slower single core perf -- NFS on a shared server with a Xeon.

foggy forum
shy echo
#

Sure, but I'm not sure that is particularly different/useful?

foggy forum
#

Shorter number of lines

#

Probably fewer bytes written to disk?

shy echo
#

I don't think a few bytes of disk space + I/O is worth optimising, given that we're going to be doing subprocesses and networked I/O in the script.

foggy forum
#

Also I don't trust sys.meta_path...

#

(also you should enable the encoding warnings on pip)

shy echo
#

Why is trust involved here?

foggy forum
#

Eg set PYTHONWARNDEFAULTENCODING

foggy forum
foggy forum
jovial jasper
#

pip 22.2 is out! https://discuss.python.org/t/announcement-pip-22-2-release/17543

Discussions on Python.org
Announcement: pip 22.2 release!

On behalf of the PyPA, I am pleased to announce that the pip team has just released pip 22.2! Highlights Query indexes using the PEP 691 JSON API. This works by sending a new Accept header, with supporting indexes responding with the new JSON format, while indexes which do not support it can continue to respond with PEP 503-compliant html. ...