#pip

1 messages · Page 7 of 1

hidden flame
#

I did find a number of last minute issues (including crashes) with the feature. Some of them seem like show stoppers (or at least, we'll need to disclaim more unsupported features on release).

#

I was aiming to get my release blog post ready for tomorrow morning, but it is nearly 1 AM right now and I don't want to stay up too much later. I will be around tomorrow morning and I will finish my post tomorrow so it's available soon after the release if you decide to release then.

#

I'll note that no one has yet replied to your question about reverting one of Pradyun's PRs. I also haven't gotten the chance to look at it.

jovial jasper
jovial jasper
hidden flame
#

@jovial jasper I'm taking a look at the editable requirements with download/wheel FYI.

hidden flame
# 
WARNING: Using pylock.toml as a requirements source is an experimental feature. It may be removed/changed in a future release without prior warning.
Obtaining file:///home/ichard26/dev/flower/ (from pylock.toml)
ERROR: The editable requirement file:///home/ichard26/dev/flower/ (from pylock.toml) cannot be installed when requiring hashes, because there is no single file to hash

I am running into this error when locking a local project as an editable. I guess lock doesn't support mixing local directories with hash-locked packages.

jovial jasper
#

So for now, you need to remove the editable from the lock file and do a two-step install. pip install -r pylock.toml ; pip install -e . --no-deps. That was already like that with requirements.txt with hashes.

hidden flame
#

Yeah. I should've remembered that issue you opened earlier. My bad 👍

jovial jasper
#

The release PR is up in case anyone wants to chime in on the changelog. I'll push the button later tonight.

hidden flame
#

I left a pretty long comment on the -r pylock.toml issue. I'm going to step away from the keyboard since I've been here for long enough, but I'll be back to publish my 26.1 post (once we decide what to do with -r pylock.toml for pip 26.1).

finite perch
#

Somebody enabled GHSA on pip repo? Looks like an AI slop report

hidden flame
#

I did at the request of Seth

#

I've asked Seth to send us an email on why/how we're supposed to use GHSA on pip for pip security reports.

stuck girder
hidden flame
#

Gotta love that the pylock discussion is split across like 3 different issues 🙃

verbal bronze
finite perch
verbal bronze
finite perch
#

I think the pip maintainers and PSRT should discuss that on a mailing list convo, and we can agree on what the correct process is, right now I'm leaning towards turning off GHSA against pip until a process can be agreed on

verbal bronze
hidden flame
#

Sorry, @verbal bronze , are you part of the CPython Core Team?

#

Actually, I think you're part of the PSRT, too. If so, please follow up with Seth. He's our main contact between the PSRT and pip.

hidden flame
verbal bronze
hidden flame
#

I was just wondering why you were asking about our security process. I thought you were an outsider.

verbal bronze
#

I’m also on the PSRT, so I get notifications for all GHSA and didn’t see any for pip which is why I noticed.

#

In general I feel we should probably split out CPython and pip security, and have a separate security-pip or the like.

hidden flame
#

@finite perch I hope you aren't too bummed that I didn't include dependency resolution improvements in my latest release post (which I'll share shortly, assuming no one yells at me).

finite perch
#

Well, I might use it as an oppertunity to write my own blog post 😉

hidden flame
#

Yup :P

finite perch
#

(probably not, I barely have time to write this)

hidden flame
#

hey, I wrote my post also on short notice LOL

#

Although I have lots of free time right now, so it's not comparable.

finite perch
#

Not sure I have much to say about it other than "is better - because of weird backtracking heuristics"

hidden flame
#

That's why I didn't include it because I'd be even worse at explaining it.

#

It's magic, don't worry. 🪄

finite perch
#

Wait until y'all see the "secret" project I'm working on, taking the magic up a couple of orders of magnitude 🥲

hidden flame
finite perch
#

There shall be no asking "why" something works, you just look at the 600 scenarios I have it benchmarking against

hidden flame
#

You should write that whenever you do blog about it.

hidden flame
finite perch
mint patrol
#

pip 26.1 is an amazing release, thank you everyone!!! 💜 I'm planning to blog about relative dependency cooldowns 😄

mint patrol
mint patrol
#

It would take a bunch of effort to create a POC for it, so figured someone might know off the top of their head.

finite perch
#

Is this saying that because URLs are still accessible for deleted distributions that lock files will still use them even if they no longer exist on PyPI? Almost certainly

dapper laurel
finite perch
#

That said we do get the resolver involved currently, so maybe it would produce an error, but that's seen as a deficiency in the current design and will likely be removed at some point

jovial jasper
#

But I doubt it would catch that case, though.

#

Because it will not query the index if the artifact is pre-constrained by the lock file.

timid stag
#

Such packages could be quickly removed from PyPI, but the related GitHub repositories are not always identified. If the threat actor uses different Git hosting, the situation is even more complex, as they tend to have limited code search functionality. If the malicious repository uses uv.lock to pin dependencies, the malicious code stays active even after the removal from the PyPI!

I would tend to think of this as pypi's problem to solve rather than the installer's... ? since fundamentally the problem is that a malicious file exists on pythonhosted.org ?

finite perch
#

Right, this is a design issue, lock files are url based and assume the index will only serve safe distributions and stop serving them if deemed not safe.

PyPI assumes the installer will check the index API, but that assumption goes out of the window with lock files

timid stag
#

a lock file could, for that matter, contain URLs for private indices.

#

I suppose an installer could implement a separate "security scan" using the other artifact metadata to check if the URLs are still listed... but of course that takes time, and kinda defeats the purpose of recording those URLs

#

(I guess there's value in checking that the lockfile's recorded hashes match what the index claims)

cosmic pebble
#

Couldn’t pypi adopt a policy that says for projects that are created and then immediately deleted, it doesn’t commit to retaining those distributions?

#

to give security scanners time to catch malicious projects

hidden flame
#

Doesn't uv's pylock implementation have the same problem? If you don't trust the URLs, then you need to add an extra step where you query additional information from the index (or a third party vulnerability database or similar).

#

I would find the relevant issue on GH, but GH is having yet another issue 🙃

image.png
hidden flame
past pagoda
timid stag
#

(or maybe they just fixed it while I was trying?)

stuck girder
chrome epoch
#

uv's lockfile has the same issue (i think all lockfiles do); we're looking at adding a relatively lightweight malware check against OSV for the lockfile to catch any locked URLs that point to malware

#

PyPI's policy is one thing but this is unfortunately probably present regardless, since we can't force arbitrary third-party hosts whose file URLs are locked to take them down when they're malware

hoary mist
#

I don't think PyPI commits to hosting deleted those files forever FWIW. The fact the urls continue to exist after deletion is mostly just because it's easier to not delete things than to delete things, and it's useful to keep them around sometimes, and the incremental cost is pretty low to doing that

jovial jasper
#

Things seem to be quiet so far. Only one issue with pylock and Python 3.15 alpha. I feel this does not warrant a bugfix release but I'm all ears if there are other opinions.

#

If nothing else shows up, I plan to close the 26.1 cycle this weekend. Please keep main quiet until then.

jovial jasper
#

Looks like we are getting a bunch of spam PRs on pypa/get-pip. I don't seem to have permissions to block it. Anyone available to do that?

shy echo
#

There's a 24 hour restriction to limit to only prior contributors.

jovial jasper
hoary mist
#

It's been a bit since I've looked at what can be done to speed up a pip install, is there anything that I'm missing from this:

pip --disable-pip-version-check install --no-deps --only-binary :all: -r requirements.txt

Where the requirements.txt is a list of things like:

attrs==26.1.0 \
    --hash=sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309 \
    --hash=sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32

technically speaking, there's multiple requirements files all being passed into pip install if that matters?

I don't see any other options that appear like they'd affect the performance of pip install, but figured I'd check here to see if I missed anything.

shy echo
#

fast-deps?

finite perch
#

I don't remember if hashes are involved if it would matter, as the full wheel is downloaded regardless right?

hoary mist
#

--use-feature=fast-deps right?

#

does not appear to have an effect

#

there's a good bit of stuff being installed, so it might just be that FWIW, trying to see if I can get make build to be faster in warehouse when one of the requirements files change (log - https://gist.github.com/dstufft/f65ba7e51b1d1a504939f5d01d5f2450), and just trying to make sure I'm not missing anything on pip's side 🙂

finite perch
#

Hard to guess where the time is being spent without a full reproducer. I suspect though it's downloading and unzipping.

dapper laurel
#

@hoary mist any reason not to use uv if you need big perf? (I know pip is the OG, but uv is undeniably faster)

finite perch
#

All the metadata parsing and processing, and resolver loop have been very well optimized, the only thing uv does over pip in this kind of scenario is download and unzip in parallel, and there are scenarios where that slows things down instead of speed things up

shy echo
#

@hoary mist are you thinking of cold cache or hot cache runs?

dreamy sandal
shy echo
finite perch
#

So that optimization doesn't do anything

shy echo
#

I have a note somewhere to run pip on 3.15 with Tachyon, and start poking at perf there. I welcome anyone willing to look into that before me. 😅

finite perch
#

I plan to do that some time soon, but I'm mostly focused on complex resolutions, and in that scenario projects aren't supporting 3.15 yet

jovial jasper
hoary mist
hoary mist
hoary mist
#

pylock.toml didn't save any noticeable amount of time 🙁 Once it's stable though it would probably make sense for warehouse so building warehouse doesn't rely on warehouse being up 😅 (just fastly and the object stores).

I did end up cutting the pip install time in almost half by using --no-compile (well I think that's what saved the most time, it's the only thing that I did to pip itself), took the ~70s down to ~38s, then did python -m compileall /path/to/virtualenv/lib -j 0 (which added 7s back, so --no-compile saved about 32s, but python -m compileall knocked that down to 25s saved).

Did some other changes and ended up saving ~40s (out of 107s I started with) overall.

There's another 10s (not related to pip) on the table but not sure if it makes sense yet or not (I think it does, but thinking about it!)

https://github.com/pypi/warehouse/pull/20005 if folks are interested, but I think the big thing for pip was just not compiling the pyc files. The fact that python -m compileall is so much faster might mean pip could get some speed ups by just waiting till the end and calling that? lol

finite perch
#

Oh yeah, I keep forgetting the biggest performance difference is uv doesn't compile by default

#

python -m compileall runs multiprocess, I think by default? (if not you can turn that on for further speed up)

#

I benchmarked this in the past, and it was faster for a lot of files, but much slower for only a few files

hoary mist
#

yea I did -j 0 to do multiple processes

finite perch
#

Running pip itself compiles some of the standard library if you're not using a distribution that has pre-compile some of the standard library, whereas uv doesn't do that because it's not Python, that can make a performance difference if you're running python commands after pip

hoary mist
#

yea I'm doing python -m compileall -j 0 against the stdlib basically first thing, before I ever invoke pip

#

then I run pip install, then compileall against that too lol

#

uv pip is still an order of magnitude faster for my test case, but I think it might be a worst-ish case for pip?

Basically it boils down to installing a whole lot of stuff (to warm the cache up), updating 1 dependency in my requirements.txt, which invalidates that docker layer (but not the cache), then installing a whole lot of stuff again (the same whole lot of stuff, with 1 dep upgraded).

So everything is cached sans that 1 dep, and I think that means it boils down to who can get stuff out of the cache and onto the disk faster?

finite perch
hoary mist
#

I'm using a cache mount in both cases yea

finite perch
#

So I think in this case uv is doing a multi-threaded direct copy, and pip is doing a single threaded copy and unzip

hoary mist
#

yea that was my guess

#

well something along those lines

finite perch
#

pip might implement what uv is doing here one day, but I don't think anyone is working on caching layer architecture right now

hoary mist
#

Yea that makes sense. I'm pretty happy with a taking 107s down to 67s, and if someone wants it faster locally I set it up so they can carry a 2-3 line local patch to drop it down to ~35s

#

the static files build still takes 5ever, but I have almost no idea what the tooling looks like to make that faster 🙁

inland creek
hoary mist
#

Hmm, is there any option to pylock.toml to have it understand it's updating an existing pylock.toml and only make the minimal amount of changes?

#

like how pip-compile will handle updating an existing requirements.txt from a requirements.in

finite perch
#

No, we don't have an equivalent to a compile or sync command yet. This scenario would be good to have written out to extend the lock command or have a compile command.

#

I would say though there is some confusion about what pip-compile does with an existing requirements.txt , it's defined as an output file but then is used as an input, it broke my expectations when I first started using it in my workflow.

#

So I would want to tweak the UX if we had something similar

hoary mist
#

yea makes sense, just thinking through what converting warehouse to pylock would look like 🙂

naive fractal
#

Bernat has a PR draft up to add a pip-lock command to pip-tools -- which he's still working on and I haven't gotten to look at almost at all yet -- but I don't think it treats the pylock.toml as part of the inputs. It's "just" multi-run resolution with different markers to get the various results.

#

I would say that the output file being part of the inputs passively/automagically is a source of confusion. It trips people up on how -o - > reqs.txt acts differently from -o reqs.txt. If we're building that workflow again, I'd think about how to make it opt-in or more explicit.

stuck girder
jovial jasper
willow flicker
chrome epoch
#

hey pip folks! i've noticed that it's been hard (through no fault of anybody) to coordinate disclosures between pip and uv, in part because (1) the PSRT/security@ volume can be very large, and (2) we often individual reports of the same bug from the same reporter without linking the two of us together.

with that in mind, do you all have opinions about establishing a kind of clearing-house (a private channel here, a mailing list, something else?) to help us coordinate between the two? my thinking is that we can use it as a lower-volume venue that gets shunted to once a PSRT topic gets confirmed as affecting both.

timid stag
#

Myself and others have been noticing fairly frequent Cache entry deserialization failed warnings emitted by pip since 26.1. Was there an explicit change to how cache is handled (e.g. removing some deprecated legacy support, or explicitly warning about a failure that used to be ignored because it's just cache)? I don't see any thing obvious in the changelog, but I've been blind before

#

(it's benign enough, and not yet clearly reproducible, to the point where I couldn't really file a bug report)

cosmic pebble
#

I’ve seen and ignored those messages as well

jovial jasper
shy echo
stiff tusk
# shy echo FYI: I was discussing pip's error message with <@265297524129005568> who has exp...

Hey, so no idea if @shy echo is still here or not. Probably not but anyways - question: should I use the existing .link attribute to reference the documentation, or should I add a new .details_link specifically for the documentation link?
So it looks like the InstallationError has a .link attribute that gets populated by the _populate_link() method on its exception class. Since it doesn't inherit from DiagnosticPipError, it shouldn't overlap, but there is also the MetadataGenerationFailed and InstallationSubprocessError which inherit from both classes where the .link attribute could already be populated.

stiff tusk
hidden flame
#

the dangers of a month-long vacation 🙃

image.png
#

I'm still on vacation for 5 more days! after that... it'll be a LOT of catching up on notifications

maiden island
#

Remember this is volunteer work 🙂

shy echo
#

Can't hurt to have this!

frosty pier
shy echo
#

Hellooo

leaden delta
#

o/

fierce temple
#

whee, pip's CVE is still making my builds fail

#

probably I should look more carefully at that, since I suppose a CPython release isn't necessarily happening as-soon-as-tomorrow, even though I see the fix is merged: https://bugs.python.org/issue43993

shy echo
#

What CVE?

fierce temple
#

@shy echo the urllib3 one

#

CVE-2021-28363

#

as far as I can tell, at the minute, if you create a virtualenv using the latest CPython, the virtualenv will fail safety, because even if your host Python's pip is new enough, the vendored CPython pip is vulnerable

#

(and safety will complain with that CVE)

frosty pier
#

I think you wanted to say virtual environment, note venv does not create virtualenv, neither does virtualenv for what it's worth

fierce temple
#

@frosty pier yes, unfortunately it's going to take lots of campaigning to fix my muscle memory (calling them the same) :), I know they're different though

shy echo
#

virtualenv can be run once with --upgrade-embed-wheels. For venv, it should be possible to do --upgrade-deps on each call.

#

Both of those will get you a "safe" version of pip IIUC.

fierce temple
#

nods

#

And how does one tell tox to do those things

#

And should it do them by default?

#

(Or do it by default when the current version of the embedded deps isn't safe)

shy echo
#

Well, that's a thing for @frosty pier to deal with.

fierce temple
#

nods

#

(I found the answer for tox, but yeah seems to my naive eyes, unless someone's careful, it's a bit too easy to run vulnerable versions of embedded deps. But definitely not a pip issue, so good job me, first use of the pip discord channel is offtopic :C)

shy echo
#

It's all good. :)

frosty pier
#

tox does not seed package handling, delegates everything to virtualenv

#

so if your virtualenv does the right thing your'e good

#

you can always instruct virtualenv to pool the latest by setting the env var VIRTUALENV_PIP=20.1.1

#

could even do that via tox setenv 🙂

fierce temple
#

I think it's hard as an end user to think of things that way

#

E.g. I think end users think tox is responsible for creating the virtual environment that each test env runs in

frosty pier
#

virtualenv will periodically update you automatically 🙂

frosty pier
fierce temple
#

If you e.g. asked "does tox execute venv or virtualenv to create each one" I suspect many don't know

frosty pier
#

and you shouldn't until you care what versions get seeded 🙂 at which point you can learn about it

fierce temple
#

I think I can agree with that yeah

#

My point though was "somethign in the stack should be making sure users who aren't paying attention here aren't using versions that are known to be insecure"

frosty pier
#

there's no way to differentiate between insecure, breaks builds, and good release 🙂

#

so depending on which you're optimizing you might want different things to happen

fierce temple
#

Well there is, no, it's just slightly annoying

frosty pier
#

note if you want to always force virtualenv to pull latest release you can set the download flag in your virtualenv.ini 🙂

fierce temple
#

Specifically: "check for CVE on creation, and if present, run pip install -U"

#

(otherwise do nothing)

#

(the first part being the annoying part)

frosty pier
#

you're assuming CVE database is accessible, in many cases it's not, e.g. behind firewalls

fierce temple
#

right 🙂

#

I know this isn't trivial, but I'm saying I think to me that's the behavior one'd expect

frosty pier
#

so that's not really a working solution, might catch some small number of cases, but will significantly decrease performance

#

I wouldn't personally 🙂 would find odd if virtualenv would reach out to the internet at every run to check if a new pip release is secure or not

fierce temple
#

You wouldn't have to do it every time

#

You can do it once

#

Well no I guess you can't, never mind

frosty pier
#

also would potentially be breaking CIs constantly, because a build that was working yesterday, is today breaking without you changing anything and you pinned everything

fierce temple
#

But basically at this very moment, wouldn't you guess 99%+ of people, even ones who think they're fully up to date, are using a virtual environment where the pip inside of it is vulnerable?

#

I would -- unless folks have trained themselves to immediately run pip install -U pip setuptools right after creating the virtualenv, which I know people used to do

frosty pier
fierce temple
#

No no I'm not asking what behavior people expect when creating a virtualenv

#

Forget about that 🙂

#

I'm just saying practically speaking right now, security-wise

frosty pier
#

I don't think they'd think that either 🙂

fierce temple
#

Everyone in the world who's created a virtual environment in the past few weeks is getting a version that's insecure, yes?

frosty pier
#

if you expect to not pull in always the latest you'd expect to not have the latest security fixes

#

you trade security for stability here, and many are happy with that

fierce temple
#

I'll think more. But personally, I think you're assuming people are more aware that the pip they get is embedded than I think people actually are.

#

And I don't think I'm comfortable personally with "people don't care about security", that's why we're supposed to 🙂

#

To me, embedding deps is a perf optimization, and folks probably don't realize that just because you ran pip install -U pip on the Python you used to create the virtualenv, that has 0 effect on your virtual environments.

frosty pier
#

I'm not saying they don't care about security, but some people care more about stability. E.g. behind corporate firewalls bad security is not that bad, because the firewall already protects you; but you're much more impacted by stability breakages

fierce temple
#

Right I understand those cases don't want to hit the internet -- I'm not saying ignore them, they just seem like they should be solvable. E.g. those cases already have flags for telling them not to right? E.g. pip has --no-index

frosty pier
#

not true 🙂

#

they might just be hitting the internal index servers 🙂

#

that are security reviewed 🙂

fierce temple
#

Right, so can't we follow those same indexes for versions of the embedded deps?

frosty pier
#

?

#

What I'm saying here just concludes that there's no good solution here to make happy both the security favored people and the stability favored people. The current existing solution is the best compromise I could come up with until now. If you have better proposals feel free to propose them, but keep in mind that there's many cases stability+performance will trump security concerns. The system today does allow you to configure what you favor though (via the disable autoupdate/force download/etc) . But that's on you to enable. The default is a middle ground that gives some times for things to stabilize but forces an update after that time duration.

fierce temple
#

Yup totally fair. I think I'm not proposing something fully concrete at the minute. My initial comment was quite simplistic, it's simply -- "I have a sneaky suspicion that at this very moment, an absolutely huge number of people creating virtual environments are getting insecure versions and don't realize that's the case, because they think they've protected themself by keeping CPython up to date, and perhaps additionally, by keeping pip up to date on that global CPython"

#

I don't know what a concrete proposal looks like but yeah thanks for the discussion, if I think of something will definitely reahc out.

frosty pier
#

Yeah, but you could also turn that around and when a broken pip/setuptools comes out the case might be "I have a sneaky suspicion that at this very moment, an absolutely huge number of people are annoyed and blocked because their virtual environments is created in a broken state" 🤷

willow flicker
#

Question on pip that likely has come up before. Adding a pyproject.toml causes a massive number of lines (multiple thousands, usually) about looking for and finding links to be added to the simple verbose output (a single -v). For things like compiling code, where the verbose output is useful in logs and that's the only way to see it, this is a big problem (it used to overflow travis' limits) - I've had least one person refuse to use pyproject.toml because of the extra output in the past. Would it be possible to hide that output under a larger number of -v's? Guessing this has been discussed before, but thought I'd check.

pallid egret
#

and if the latter, does anyone know of some packages that have that metadata?

lunar gyro
pallid egret
#

but is it ever provided via the API, or only by local metadata?

#

API == PyPI API

#

warehouse

limber ore
little kite
#

It's a little surprising how every folder you put on sys.path is interpreted as a site directory by pip - if I put a folder on path with a dist-info folder in it, pip will think that distribution is installed

lunar gyro
#

There is actually no such thing as “site-packages”, they are just directories. Python doesn’t offer any mechanisms to distinguish a site-packages directory

obtuse lagoon
#

the site module does

#

you can get a list of the site packages directories

lunar gyro
#

site only works in very restricted situations, basically only reliable on interpreter startup and in default configurations (which is what it’s intended for, so that’s fine, but you shouldn’t build things around it).

hoary mist
#

I think there's also things like Debian having multiple directories that make this more difficult as well

#

also platlib vs purelib

fierce temple
#

the CI I run for my dotfiles has started taking 8 hours and timing out github actions

#

the cause appears to be pip taking multiple hours resolving requirements

#

I can try to diganose why, but first I figured I'd share that awful sentence (the first one, not the second one)

#

(meanwhile pinning python-language-server>=0.36.2 seems to somehow fix it)

limber ore
ebon sinew
#

I had that too a few weeks ago. There was a google form to fill somewhere, I filled it and never found out what it had become

shy echo
#

It's something that I need to look into, but... well, I'm one tiny human and need to look into that.

shy echo
lunar gyro
shy echo
#

For the user's usecase, how do you feel about me suggesting they use local versions on their wheels to have them install preferentially?

#

(whelp, I accidentally posted the comment)

lunar gyro
#

Yeah local versions is reasonable. Or wheel build tags. And I already did mentioned that. I think the main issue I have is the user is not describing the use case very well, the description jumps around constantly, when I listed out all the possible solutions upfront the response it “nan none of them works for me because [blah blah]”, and when I pointed out at least one of the solutions does the response becomes “nah we are not looking for that because [something not mentioned previously]”

shy echo
#

Yea, I saw that.

lunar gyro
#

Maybe I’m too frustrated at this point but I’m quite sure they’ll respond something similar to your local version suggestion

shy echo
#

It was only when I saw "compute canada" that I want, ah, I know what they're struggling with.

#

Cool cool, well, hopefully I can shield you from further frustration here.

#

OR, we can be frustrated together. :P

foggy forum
thorny torrent
#

Can you try using /celery-wheelhouse/repo/simple as the index URL rather than just /celery-wheelhouse/repo? The page at /celery-wheelhouse/repo doesn't look like it's formatted per PEP 503

foggy forum
thorny torrent
#

You tried "/simple/pycurl/", I mean just "/simple"

foggy forum
foggy forum
#

wonder why the /json stuff didn't work

foggy forum
#

Does pip need the JSON API severed with the correct content types?

thorny torrent
#

I don't think pip uses the JSON API at all, at least not for getting downloads links

foggy forum
#

Hmm ok

lunar gyro
#

The JSON API is not standardised and is missing functionalities currently so pip can’t use it (there’s work toward standardising it but it’s not close to finished)

fierce temple
#

Is showing resolvelib messages at a higher level than INFO something that was discussed somewhere

#

I'm referring specifically to the "This is taking longer than usual" backtracking message

#

And very specifically, to it not showing up when running with --quiet

#

Is asking for that message to be shown a reasonable ticket (and/or does it already exist somewhere -- my github search doesn't find it)?

lunar gyro
#

My reading comprehension is off (probably since it’s 2am here), do you want the resolver to display more or less?

fierce temple
#

@lunar gyro all good 🙂 more

#

I want it to say "I am stuck" even when running in --quiet mode

lunar gyro
#

Sounds similar to https://github.com/pypa/pip/issues/10210 (except the --quiet part; I think this does not show on quite mode (because the level is INFO). I haven’t had time to review it, but the things we need to work out are 1. what exactly the message should look like, 2. what context we need to keep so the message can be formatted, and finally 3. how the implementation should look like. Would be nice if you could join the thread in that issue.

GitHub
when going into complex version resolving, output conflicting versi...

when pip outputs this message: INFO: pip is looking at multiple versions of webencodings to determine which version is compatible with other requirements. This could take a while. It does not give ...

foggy forum
#

@frosty pier currently if you do pip install --require-hashes -c constraints.txt .tox/dist/.../ham.whl then pip fails because it can't find the hash for the built asset

#

Ultimately I'm looking to use a hypermodern workflow but use tox/flit instead of nox/poetry

frosty pier
#

but there's no hash to be found for the whl

#

feels to me like you want --require-hashes not globally but only within the constraint file

#

in which case that flag should be within the constraint file 🙂

#

to only apply to that part

#

there's no way AFAIK for tox to provide the hash for the wheel on the CLI (which would be the alternative) 🙂

#

you could also make the argument that pip should not apply require hashes to the wheel because there's no way for the user to provide that on the CLI

#

ultimately that would be something to take up with pip maintainers 🙂 @foggy forum

foggy forum
foggy forum
frosty pier
#

that would imply mapping the package to a requirement file 🙂

foggy forum
#

Mapping?

frosty pier
#

tox builds an sdist/wheel; it passes this to the installer (pip); to provide the hash as a tmpfile you'd need to generate a requirement file and append to that the hash of the generated build, and now don't pass the package itself but the requirement file

#

I assume when you said tmpfile you meant a requirement file that's temporary

#

(or constraint file)

foggy forum
frosty pier
#

are you?

foggy forum
#

I thought so

#

At least

#

It parses the command line from tox then bundles it into a requirements file with hashes

#

And does some hacks to handle extras

#

Then passes that to pip

frosty pier
#

ah wait 😄

#

can you share tox_install_command 😄

#

oh it's below there

foggy forum
#

I linked directly to the line that did it

frosty pier
#

so what's the hoal here?

#

to force require-hashes globally?

#

for all command runs

#

any benefit for forcing for the package we just build?

foggy forum
#

It doesn't work otherwise

#

You can't install extras from a package unless you pass the built asset

#

Ideally I could tell pip to ignore file:/// deps for hashes

#

Eg It doesn't require hashes for -r deps: a .txt from disk is just as powerful as a .whl from disk

foggy forum
frosty pier
#

You'll need to define clearly what hypermodern means

#

And I don't follow clearly what you're trying to achieve here and what's the gap

foggy forum
#

ah the magic is in nox-poetry

#

what hypermodern does that I'm interested in, is create a constraints.txt (with hashes in). That constraints file is passed to nox which is passed to pip

I want tox to use that constraints.txt such that tox can use in tox bootstrap, and in installing any other toxenv

#

this way I can get dependabot to update my contraints file from a setup.cfg or flit pyproject.toml

#

@frosty pier if that makes sense

frosty pier
#

I don't follow entirely to be honest 🙂

#

But I'm happy to provide a way in tox for it, if you can explain step by step what needs to happen at each step in tox, and what are the feature gaps @foggy forum 🙂

foggy forum
#

And optionally include --hash=sha256:... flags

foggy forum
lunar gyro
#

The official answer for needing to exclude some packages for has requirement is you need to split your requirements into two commands, one with hashes and one without (and use --no-deps on the second)

frosty pier
frosty pier
frosty pier
foggy forum
#

I'm installing modernize[test] @ .tox/dist/...modernize-py3...whl

frosty pier
#

Can you explain what is a hypermodern Python? The article you've linked above is more like use tool a, b, c and you'll get hypermodern python; but doesn't explains what traits make it hypermodern; so I find hard to understand what you're trying to achieve here exactly

foggy forum
#

Currently for me I see the hypermodern workflow as using nox and poetry to create securely pinned dependency lists that can be used to later run multiple different test environments with subsets of those dependencies

frosty pier
#

how do you acquire a subset of dependency? 🙂

#

we don't have today lock files defined - see PEP 665, so I assume you're talking in you case about using pips non standardized requirements.txt/constraints.txt files?

#

when do you create/update these set of dependencies?

#

what do you mean when you say run multiple different test environments? can you provide an example?

foggy forum
#

currently I'm using dephell to make a contraints.txt

foggy forum
#

also it's -crequirements.lock because that's what dephell calls it. But actually it's a -cconstraints.txt

foggy forum
foggy forum
#

but uses a separate requirements.txt for each toxenv

#

I would be using one contraints.txt for all toxenvs

frosty pier
#

@foggy forum can you open an issue for tox and explain all the above in there too? Thanks, sounds like a feature request for tox 4

foggy forum
foggy forum
foggy forum
foggy forum
#

@shy echo why does a file:/// dep need hashes in --require-hashes mode, but a -r requirements.txt does not?

#

The assets are equally powerful and have equal provenance

shy echo
#

Those requirements need to have hashes though, if my intuition is correct.

foggy forum
shy echo
#

Because it's a wheel file -- AKA thing pip-will-install?

foggy forum
#

Right but it's got the same power and provenance as a .txt

shy echo
#

Okay. I see what you mean.

#

Basically, the format is "requirements" + "options".

foggy forum
#

It's just a different file extension and the deps are in a metadata file

shy echo
#

-r requirements.txt is an option.

#

file:///....whl is a requirement.

foggy forum
#

Sure but requirements.txt is more powerful than file:///...whl

shy echo
#

And requires-hashes says "make sure every requirement has hashes".

foggy forum
#

Right what I'm saying is that policy doesn't make sense

shy echo
#

🤷‍♂️

#

I'm not gonna be driving any changes to the requirements.txt format -- there's PEP 665 which is much better for the lockfile case.

foggy forum
#

Oh I'm suggesting the opposite

#

I'm not suggesting that requirements.txt itself needs to be hash pinned

shy echo
#

?

foggy forum
#

Just that file:///....whl has no need to have hash enforcement applied

shy echo
#

Allowing local files not not have hashes? Bleh.

#

I won't block such a change, but touching the requirements.txt format is touching blue paper.

foggy forum
#

Blue paper?

shy echo
foggy forum
#

As in blueprints or blue touch paper?

#

Oh I see

shy echo
#

"To do something that instigates or initiates some intense or widespread action or reaction."

foggy forum
shy echo
foggy forum
# shy echo I won't block such a change, but touching the requirements.txt format is touchin...

Can you reply as such to https://discuss.python.org/t/some-way-to-pass-hash-on-the-cli-or-to-ignore-file-uris-when-using-require-hashes/5256 and I'll try and get a PR

Discussions on Python.org
Some way to pass "--hash" on the CLI or to ignore file:/// uris whe...

Currently I’m working around this with a custom install command, that hashes any file requirement arg and puts it into a requirements file before passing it onto pip here: https://github.com/PyCQA/modernize/pull/228/files#diff-256be86b218458267e29f38e19906417R72 I’m also proposing to use this work-around in tox https://github.com/tox-dev/tox/is...

foggy forum
foggy forum
foggy forum
shy echo
#

Huh?

foggy forum
#

For an http client to permit insecure connections to an hsts domain it must require a rebuild from source

shy echo
#

rebuild of?

foggy forum
#

Pip

shy echo
#

I'm very confused.

foggy forum
#

hsts forbids a user facing invalid certificate bypass

#

That's presumably why pypi.org has the hsts header set

shy echo
foggy forum
#

It's set for all http clients

shy echo
#

Well, in any case, I'm not the best person to ask about pip/PyPI's network security posture.

foggy forum
#

It's re --trusted-host

#

It shouldn't be supported for hsts hosts

shy echo
#

Ah.

foggy forum
#

I can see why it currently works like it does

#

But the spec is clear

#

Coming into compliance would obviously take a few years ofc

shy echo
#

Could you bring this up on the issue or on discuss.python.org? I'm, honestly, not the best person to discuss the details of handling of TLS stuff with.

foggy forum
#

Why are there so many forums?

shy echo
#

issue trackers -- project-specific stuff.
discuss.python.org -- multiproject stuff.
IRC / Discord -- ephemeral conversations.

#

What else is there? XD

hoary mist
#

I don't think it's true that the RFC is even designed for UAs that aren't browsers

#

Also the parts of the RFC that say that their should be no user recourse are non normative, and thus just suggestions

foggy forum
hoary mist
#

@foggy forum to be clear, I'm not opposed to the idea of improving pip in this area 🙂 I just want to make sure that, if we do that, we should evalute it on it's own merits

tranquil notch
#

how does pip install -e . work

#

what does it actually do

foggy forum
#

Ah hsts specifically

#

Yeah I assume you'd want to allowlist pypi.org for a while after landing support for it

little kite
#

what's the easiest way to see which installed packages depend on another, installed package using pip?

stuck girder
# 
$ pipdeptree --packages black
[snip warnings]
black==21.9b0
  - click [required: >=7.1.2, installed: 8.0.1]
  - mypy-extensions [required: >=0.4.3, installed: 0.4.3]
  - pathspec [required: >=0.9.0,<1, installed: 0.9.0]
  - platformdirs [required: >=2, installed: 2.3.0]
  - regex [required: >=2020.1.8, installed: 2020.11.13]
  - tomli [required: >=0.2.6,<2.0.0, installed: 1.0.4]
  - typing-extensions [required: >=3.10.0.0, installed: 3.10.0.1]
#

And reverse:

$ pipdeptree --packages platformdirs --reverse
[snip warnings]
platformdirs==2.3.0
  - black==21.9b0 [requires: platformdirs>=2]
    - blacken-docs==1.9.1 [requires: black>=19.3b0]
    - darker==1.3.0 [requires: black>=21.5b1]
  - norwegianblue==0.2.1.dev9 [requires: platformdirs]
  - pypistats==0.13.0 [requires: platformdirs]
  - virtualenv==20.7.2 [requires: platformdirs>=2,<3]
    - nox==2021.6.12 [requires: virtualenv>=14.0.0]
    - pipenv==2020.11.15 [requires: virtualenv]
    - poetry==1.1.8 [requires: virtualenv>=20.0.26,<21.0.0]
    - pre-commit==2.15.0 [requires: virtualenv>=20.0.8]
    - tox==3.24.3 [requires: virtualenv>=16.0.0,!=20.0.7,!=20.0.6,!=20.0.5,!=20.0.4,!=20.0.3,!=20.0.2,!=20.0.1,!=20.0.0]
little kite
#

that's neat, thanks!

wispy coral
willow flicker
#

What's the estimated date for pip 21.3? Planning around the in-tree-build feature, which I assume still is coming in 21.3.

stuck girder
lunar gyro
#

The most important feature would be PEP 660 (standardised editable install), but that is currently still in review. The expectation is to release sometime in October. (And yes in-tree-build is still expected to be included as well.)

willow flicker
#

Roughly late October is fine. Thanks!

frosty pier
#

In future would be nice to move it to September? 😄 So that it can make it into the October 5th new Python version release 😄

foggy forum
#

could pip default to --use-pep517 if setuptools is not present?

foggy forum
#

or attempt the legacy install and fall forward to an isolated install?

shy echo
#

Huh?

#

Ah, file an issue against pip.

foggy forum
#

are there any packages on pypi that currently fail to install with a missing setuptools but don't succeed with pep517 ?

#

runtime dependency on setuptools

@shy echo what do you mean by this?

shy echo
#

You need setuptools, for pip to use with the legacy build mechanism -- which happens when pip runs -- which makes it a runtime dependency.

shy echo
#

There's a long d.p.o discussion on PEP 517 and packages that can't be built into wheels.

foggy forum
#

What's the mechanism called when there's no pyproject.toml file but there is a setup.cfg?

shy echo
#

legacy

foggy forum
#

oh that's invalid

shy echo
foggy forum
#

well there's pip install --use-pep517 .

shy echo
#

And...?

foggy forum
#

that works without a pyproject.toml file

#

I get Building wheel for foo (PEP 517) ... done

#

The --use-pep517 flag (and corresponding environment variable: PIP_USE_PEP517) can be used to force all packages to build using the pyproject.toml based build system interface. There is no way to force the use of the legacy build system interface.

#

but it doesn't work - echo '[metadata]\nname=foo' > setup.cfg && pip install --use-pep517 . fails

$ echo '[metadata]\nname=foo' > setup.cfg && pip install --use-pep517 .
ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
little kite
#

yes, a legacy source tree as defined by PEP 517 must have a setup.py

#

it was quote, unquote decided that a setup.py-less legacy source tree would be novel and should not be encouraged. If you can delete setup.py then you are also able to create a pyproject.toml

foggy forum
#

actually it seems that echo '[metadata]\nname=foo' > setup.cfg && pip install --use-pep517 file:///$(pwd) works

little kite
#

is this an exercise in how to do the wrong thing? :P

foggy forum
#

If the pyproject.toml file is absent, or the build-backend key is missing, the source tree is not using this specification, and tools should revert to the legacy behaviour of running setup.py (either directly, or by implicitly invoking the setuptools.build_meta:legacy backend).

#

well I accidentally copied my root file system to /tmp when I did this: echo '[metadata]\nname=foo' > setup.cfg && pip install --use-pep517 file:///$CWD

#

oh I did it again

little kite
#

👀

foggy forum
#

yeah it's important to use ( and not {

#

anyway it seems pep517 is suggesting implicitly invoking setuptools.build_meta:__legacy__

#

which pip does!

#

but not for file paths, only for file URIs

#

paths without a pyproject.toml or setup.py are valid trees under --use-pep517

little kite
foggy forum
#

I'm on pip 21.2.4 from /home/graingert/.virtualenvs/testing39/lib/python3.9/site-packages/pip (python 3.9)

#

it seems it didn't stick

#

echo '[metadata]\nname=foo' > setup.cfg && pip install --use-pep517 file:///$(pwd) works for me

little kite
#

it did stick, the URI's just following a different code path...

foggy forum
#

ah so that's a bug?

#

also works if I tar it:

pip wheel ../foo.tar.xz --use-pep517
Processing /home/graingert/projects/foo.tar.xz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: foo
  Building wheel for foo (PEP 517) ... done
  Created wheel for foo: filename=foo-0.0.0-py3-none-any.whl size=933 sha256=476cb8f57fcb4c090725cc5278a6e78628e91ada9912d2d52182e1097521ca1f
  Stored in directory: /tmp/pip-ephem-wheel-cache-oq0wuqpu/wheels/bb/a9/b8/adb7c9529b31805cbc6c6442ba0fde207978500afb2c0ba972
Successfully built foo
little kite
#

that should probably be rejected because it's not gzipped, in addition to not having a setup.py or pyproject.toml

foggy forum
#

isn't tar.xz valid as an sdist?

#

nautilus didn't let me make a tar.gz

little kite
#

PEP 517 requires sdists to be gzipped tarballs

foggy forum
#

that's the output format of build_sdist

#

not the input format

#

seems the input format is unspecified

#

yeah it's clear: it's the sdist from the backend: The new restrictions for sdists built by PEP 517 backends are:

#

the frontend can do whatever it wants to get a source tree

little kite
#

that's why I said "probably" - it's probably a vestige of legacy installs, not an alternative input format that pip was specifically designed to support, like a git repo

foggy forum
#

one day --use-pep517 is going to default to True, when that's the case I think it's fine to treat empty pyproject.toml and absent pyproject.toml the same

#

The intention is to switch to using the pyproject.toml build system interface unconditionally and to drop support for the legacy build system interface at some point in the future.

what's gating this?

#

PEP 453 says setuptools has got to go, so either that happens or the pep should be updated to move the goalposts to when the legacy install is removed

foggy forum
#

@shy echo it seems to me as though there are three build systems

#

the legacy "call setup.py directly" then the call setuptools.meta.__legacy__ one, and then the call whatever build-system says to do

#

it's not really clear which one is the "legacy" one

#

this morning I assumed that --use-pep517 opted into the modern build system - which will call setuptools.meta.__legacy__ if there's no pyproject.toml and --no-use-pep517 opted into the legacy build system which was calling setup.py directly

shy echo
#

setuptools.build_meta.__legacy__ is a PEP 517 build backend, that's used exactly as any other pyproject.toml-based build backends are used.

#

It's the default if none are specified in the pyproject.toml file; or if the file doesn't exist and you use --use-pep517.

little kite
#

If it's correct that opting into PEP 517 should mean invoking the legacy backend without a setup.py, then build has been doing it wrong this whole time

shy echo
#

WAIT, what do you mean by "legacy backend"?

little kite
#

setuptools.build_meta.__legacy__

shy echo
#

That term is unclear to me -- do you mean setup.py bdist_wheel or setuptools.build_meta.__legacy__?

#

ACK

#

What does build call if there's no build-backend in pyproject.toml?

little kite
#

It calls setuptools.build_meta.__legacy__ iff there's a setup.py in the project root

#

It's a little more nuanced actually, it'll inject setuptools.build_meta:__legacy__ if there's a pyproject.toml without a build-system table, or if build-backend is missing, or if there's a setup.py but no pyproject.toml, but it'll error if the project contains neither a pyproject.toml nor a setup.py. It seems that pip doesn't check if setup.py exists, so it'll inject setuptools.build_meta:__legacy__ even if both files are missing

obtuse lagoon
#

I thought I removed support for that

#

which was added without discussion to fix something else

#

(without discussion on that particular behavior)

little kite
obtuse lagoon
#

ah, I misunderstood

#

so it is behaving as intended

#

it is erroring out

little kite
#

it's erroring if it were to call setup.py directly, but not with PEP 517

obtuse lagoon
#

just a bug with file://

little kite
#

there's some disagreement/confusion about what it should do when --use-pep517 is passed

foggy forum
#

Imho pip should eventually not care about setup.py at all

#

And treat an absent and empty pyproject.toml the exact same way

#

And I think pep 517 should be updated

little kite
#

The fallback's there to help with the transition from the setuptools way of doing things to PEP 517 - we don't want to elevate one specific backend above the rest for all eternity. If there's no setup.py and build-backend is absent, pip (and build) should error. The wording's a bit muddled but we should be able to agree on the intent.

#

This keeps coming up, so yeah, we ought to update the PEP

foggy forum
#

Or maybe 3

#

Depending on what files are there, how the path is specified, and if use_pep517 is True False or None

shy echo
#

You're getting unexpected weird behaviours when you pass --[no-]use-pep517?

#

If you're trying these things out, can you put all this into some sort of table? 🙈

#

I'm pretty sure that the whole thing is just... weirdly broken.

foggy forum
#

@shy echo I'll work on that table shortly

shy echo
#

<3

obtuse lagoon
obtuse lagoon
foggy forum
obtuse lagoon
#

and potentially succeed

foggy forum
obtuse lagoon
#

that is not a source of truth, but rather just user-friendly documention

#

IMO the project should either have a setup.py or a build-system table in pyproject.toml

foggy forum
#

setup.py based Legacy interface, that we’re working to migrate users away from.

To me means pip will one day not care about setup.py

foggy forum
hoary mist
#

I'm pretty sure the intention is that at some point pip will no longer directly call setup.py

#

it may end up being called via PEP 517

foggy forum
#

via setuptools.build_meta:__legacy__

#

Which works today with --use-pep517

foggy forum
hoary mist
#

Someone either has to maintain support for setup.py or we need to declare it no longer supported by modern tools. It makes a lot more sense to implement that logic in setuptools than it does to implement it in every installer

little kite
#

it would simplify things a lot in build if we could rely on setuptools to error out if there's no setup.py when the legacy backend is invoked. The fallback logic is fairly convoluted and we're having trouble even remembering how things are supposed to work

foggy forum
#

Imho setup.cfg should also be allowed

#

Or maybe setuptools should refuse to build the UNKNOWN package

#

Or pip could deny the UNKNOWN wheel

little kite
#

it makes no sense invoking the legacy backend for setup.cfg without a setup.py, what you're proposing is to default to using setuptools when there's a setup.cfg, which I'm strongly against

foggy forum
#

No I'm suggesting default to --use-pep517 and follow the current file:///path interpretation of pep517 - invoke setuptools.build_meta:__legacy__

#

pip would ignore setup.py completely

#

Is Paul in here?

little kite
#

the legacy backend's purpose is to offer a way to call setup.py through PEP 517 machinery. If there's no setup.py then you should specify the regular, non-legacy setuptools backend and setuptools should preferably refuse to process setup.cfg without it.

foggy forum
#

Who's it harming if a valid setup.cfg works under --use-pep517?

#

The legacy backend is different from the legacy interface right?

#

Also what's the difference between the legacy backend and the legacy interface?

little kite
#

it's harming PEP 517, the whole point of this endeavour is to democratise the build backend whatever, not to just run setuptools with a build frontend slapped on top of it. The legacy backend is the PEP 517 implementation in setuptools which tries to be compatible with calling setup.py the old way. The legacy interface is calling setup.py the old way.

foggy forum
#

I mean what's the externally visible difference?

#

Is there any point in having legacy backend and legacy interface?

#

Oh it's the extra virtual environment, eg if you need numpy already installed to be able to build the current thing

little kite
#

pip doesn't actually use virtual envs for isolation but yeah, some things don't work with PEP 517, or they might only do if you pass --no-build-isolation (to confuse things even further).

foggy forum
#

What is used for isolation?

little kite
#

sitecustomize

foggy forum
#

Ah so what's the difference between --no-build-isolation --use-pep517 and --no-use-pep517 ?

little kite
#

--no-use-pep517 runs setup.py, --use-pep517 --no-build-isolation invokes the PEP 517 backend without isolation

obtuse lagoon
#

my main objection against this is that setup.cfg-only projects are a new thing

#

dealing with setup.py projects is understandable as that's leagcy, and how things used to work

#

but that's not the case for setup.cfg-only

#

setuptools does not have an alternative entrypoint, and in fact, that was proposed and they refused to add

#

so I think it's very reasonable to have to specify a backend in pyproject.toml for setup.cfg-only

little kite
obtuse lagoon
#

otherwise we are kinda giving preferred treatment to setuptools, apart from what we really need to have

#

which is bad, and should be avoided IMO

#

but that's subjective, and my opinion 😅

shy echo
#

Man, I hate this mess.

little kite
#

the trick's to use the passive voice

shy echo
#

LOL

foggy forum
#

That's gonna need a big warning every time it's fallen back to

foggy forum
#

omg when I pip installed / the other day it iterated through all my keybase folders and sent everyone a notification

obtuse lagoon
#

to make things easier

#

if there's no setup.py, it should not be called

#

that's my interpretation of the PEP anyway

foggy forum
obtuse lagoon
#

yes

foggy forum
#

Are you saying it's invalid to explicitly list it in a pyproject.toml ?

obtuse lagoon
#

no, but you probably don't want to do that

foggy forum
#

@shy echo peps are afaik living documents - eg PEP517 changed a few times

shy echo
#

PEP 517 was provisionally accepted, not final'd

foggy forum
#

ah I see

#

I still think deleting the setuptools damocles is probably for the best

#

there's a history of cpython saying stuff will be removed and then not doing it

#

eg asyncio loop kwarg, collections.abc

#

so maybe it's fine?

shy echo
#

Yo, I'm not touching this. This is purely CPython process shit.

#

Im fine as long as we don't end up changing status quo wherein pip maintainers make the call for when to remove setuptools from pip's install mechanisms which will happen when legacy buils mechanism gets removed

foggy forum
#

Could use-pep517 be moved into the use-feature/legacy-feature flags?

#

Eg --use-legacy-feature=setuppy-invoke ?

lunar gyro
#

The problem is not where the flag can go, it's having the behaviour behind any kind of flag (instead of the default) is too disrupting

foggy forum
#

add a 1 second sleep each time setup.py is invoked, doubling every month 😉

foggy forum
#

@shy echo which can did you kick down which road due to which already open PR?

little kite
#

the PR they commented on? they reassigned it to 22.0

hidden flame
#

Got a question: was there a bug in older pip that would cause it to install a x86_64 wheel on an arm64 architecture on Mac?

#

I'm curious because psf/black is getting multiple reports of regex being installed with the wrong arch and we haven't been able to diagnose the root problem yet :/

#

Thanks in advance!

lunar gyro
#

I don't recall seeing such reports, but this can happen if you install the package with Rosetta and then run it without. I think Homebrew went through some sort of transition phase on this so perhaps that's what's going on?

shy echo
#

Well, 3.8 will use Rosetta. 3.9 supports M1, but it depends on however that person installed things.

polar nova
#

Is there some way to see what packages would be installed from a requirements.txt file?

#

eg fastapi requires starlette, so starlette would also be installed

little kite
#

nope

obtuse lagoon
foggy forum
#

pip and pip-tools use different resolvers and so you might get different results

#

Also new packages can be uploaded getting newer even more different results

#

Eg you might get anyio or not depending on FastAPI version

#

pip-compile and poetry take the guesswork out by pinning a valid resolution in your source code

willow flicker
#

(not related to a bug or no bug in pip with installing the wrong wheel)

wise mesa
polar nova
#

ah

#

I was trying to enhance pip-licenses to be able to work from a requirements.txt

wise mesa
obtuse lagoon
#

but only supports wheels rn

wise mesa
#

Can it produce partial trees when some dependency fails to build?

obtuse lagoon
#

there is no building because it only supports wheels

wise mesa
#

but it dowloads them and unpacks, right?

obtuse lagoon
#

if necessary to check the metadata, yes

#

but it does not install anything

foggy forum
#

It doesn't do the fancy partial download?

#

Like use-feature=fast-deps

obtuse lagoon
#

no

#

it's a very basic implementation of a python package resolver

foggy forum
#

Looks cool though, I wish pip-compile had a better resolver

hidden flame
lunar gyro
foggy forum
#

I thought it was safe 😦

lunar gyro
#

It’s OK if you are ready to nuke and rebuild the virtual environment from scratch at a minute’s notice

#

Homebrew Python installations (all its libraries really) are very volatile compared to anything else

little kite
#

it's especially annoying if you install pipx with homebrew

foggy forum
#

oh right yeah I nuke them often

#

thank goodness for pre-built wheels

willow flicker
#

Me too, I'm always happy to rebuild a venv. I'd consider them about like a build directory, something ephemeral

lunar gyro
#

Yeah, for project management that's the best approach. But usages like pipx are exactly where you shouldn't use Homebrew Python against because those environments are more of a persisted nature. I've lamented for a while that pipx really shouldn't be distributed on Homebrew (or at least should not use Homebrew Python as the base interpreter by default), but some Homebrew diehards insist this is not a problem. Redistributors are gonna redistribute and there's nothing you can do about them 🤷‍♂️

little kite
#

I use nix (the package manager) personally and it has the same problem, if the pipx derivation changes, all the venvs break. I don't know what I'd recommend for installing pipx on Mac honestly.

frosty pier
#

Pyenv 👍

little kite
#

Yeah that's probably the easiest but it's another thing you have to install/manage on top of your package manager

frosty pier
#

Definitely worth it and easy installation

#

Brew python is for brew python projects, not for development 😬

lunar gyro
#

There’s a command in pyenv called python-build that basically a bunch of shell scripts that compile Python in one command. I just use it to compile Python to an arbitrary location and ln it to my PATH

hidden flame
#

Is there a way to remove all cached files for a specific package w/ pip?

#

I just tried python -m pip cache remove black and it seems like it only deletes the local wheel cache leaving the http cache intact 😦

#

for context I'm publishing a pep 503 index (using github pages :p) for some experimental wheels for psf/black and I'd rather not have the cached wheels cause issues down the line. although to be fair I don't know how pip treats packages with a local version segment

hidden flame
#

oh hmm, so it seems like pip first checks the index before using the HTTP cache (and github pages' max-age is ~10 minutes) .... alright then this shouldn't cause issues :)

lunar gyro
#

If you just want pip to never use cache on a certain machine, it's easier to set the environment variable PIP_NO_CACHE_DIR (or write a config file)

foggy forum
hidden flame
foggy forum
#

Neat

shy echo
lunar gyro
#

Did we talk about enabling Discussions?

#

This feels like exactly the kind of thing that should go there

shy echo
#

I don't know if we did -- but I think that's probably worthwhile.

#

I kinda wanna do what I've done with Furo, wherein we direct users to the discussions for first comments/triage.

#

The main concern I have is that we'll get flooded with a lot of support questions + people not understanding what GitHub discussions are.

#

I mean, it's kinda the opposite direction of why I wanted us to use a issue forms -- it pushes users to write more "complete" issues. There's absolutely no structure to a discussions post. 🤷‍♂️

lunar gyro
#

we'll get flooded with a lot of support questions
We don't have to answer them 😜

young nacelle
#

Hi guys, I am working in some integration tests for setuptools that focus on running pip to build from sdists for some popular packages in the ecosystem (https://github.com/pypa/setuptools/pull/2863)

Does anyone knows if it is possible to use some kind of "pseudo-isolation"? By that I mean run everything inside a venv as a normal PEP517 build, but force the version of setuptools to be taken from a specific path (the idea is to test the version of setuptools under development, instead of downloading and installing the latest stable version on PyPI).

I did manage to implement the test using --no-build-isolation and installing the build dependencies myself, but if possible, it would be nice to not have to use this option (and replicate the most common user workflow).

GitHub
Add integration tests focusing on installing sdists via pip by abra...

Motivation
As shown in #2849, sometimes it is tricky to identify when a change introduces backward compatibility problems, specially considering that a lot o packages still rely on distutils.
Integ...

obtuse lagoon
#

I think you would be better off using the pypa/build API

#

we already implement virtual environments

#

you can create one and install the build dependencies, and then the setuptools you are targeting

#

and use that to build the project

#

the interaction between these APIs is not amazing

#

yet

#

we are working on it

#

so please pin the minor version until we reach 1.0.0

#

example usage

young nacelle
#

Thanks! I will have a look on it.
Ideally if I can rely on build to create the virtualenv with the dependencies, then force the version of setuptools under development, before proceeding with the build, that will be ideal

obtuse lagoon
#

yes, that is perfectly doable

frosty pier
#

You can also use virtualenv to create a virtual environment with a locally built setuptools

little kite
#

the virtualenv path is probably less boilerplate

obtuse lagoon
#

true, our API is still a little bit rough 😅

young nacelle
#

Thanks guys, currently I am parsing the TOML and installing the build-time dependencies using pip -t just to minimise the overhead.

I think Jaraco's main concern here is to not diverge from the regular user workflow which does not include --no-build-isolation.

Even if I use virtualenv, I would need to use that flag, correct? (and install the build-time dependencies myself)

Maybe the final solution would be having 2 sets of tests. One very similar to the one I am using right now checking against pip, and another one checking against build as suggested by Filipe

little kite
#

you might wanna try using a constraints file, IIRC PIP_CONSTRAINS works for build requires

frosty pier
obtuse lagoon
#

do you have any link to the relevant issue or PR?

young nacelle
# frosty pier That's not enough you need to also install the wheel/sdist build requires FYI

Yeap, I am parsing the TOML installing the build-time dependencies using pip -t, exporting PYTHON_PATH, and then installing the sdist file using pip -t again.

But yes, this thing with the pip -t was just some trick to avoid overhead...

I agree that creating the virtualenv would be better.

I will investigate the build and PIP_CONTRAINTS proposed here, they sound promising...

merry wolf
#

In this thread from March 2020, Dustin and @lunar gyro say that pip doesn't use the simple api from pep 503. But Bret says he will. But Pep 503 was written in 2015 to describe what pip already did. So am I having a bad brain day or is there something not adding up?

young nacelle
# obtuse lagoon do you have any link to the relevant issue or PR?

issue: https://github.com/pypa/setuptools/issues/2862
pr: https://github.com/pypa/setuptools/pull/2863

GitHub
[FR] Add integration tests to catch breaking changes in the API · I...

What's the problem this feature will solve? It would be nice to have integration tests focusing on the usage of setuptools "public API" by some popular packages in the com...

GitHub
Add integration tests focusing on installing sdists via pip by abra...

Motivation
As shown in #2849, sometimes it is tricky to identify when a change introduces backward compatibility problems, specially considering that a lot o packages still rely on distutils.
Integ...

obtuse lagoon
#

thanks!

lunar gyro
merry wolf
lunar gyro
#

This thread is talking about the root URL page specifically, not PEP 503 in general.

merry wolf
#

ohhhhh, dang thanks

lunar gyro
#

IIRC you’re not the first one to misread the thread either, it’s surprisingly confusing for some reason

merry wolf
#

phew!

scenic perch
#

Why does pip export deps with edited - as _ or _ as -?

lunar gyro
#

pip exports package names according to what the package author wants to package to be called, not how they are specified in another package’s dependency list.

scenic perch
#

right, on some places, it is said that pip converts - to _ and one some places it is saying it converts _ to -. A user would always call it with _. So why would it convert it to -?

obtuse lagoon
#

users can call it with either - or _, the names are normalized

#

like said above, pip exports the actual name of the package, as specified by the authors

scenic perch
#

so why does pip export convert them, if both of the characters work?

#

like face_recognition is exported as face-recognition

lunar gyro
#

Because the author of face-recognition calls the project face-recognition, not face_recognition

scenic perch
lunar gyro
#

Again, pip does not convert anything, it simply displays whatever the author provides

foggy forum
#

What do you mean by export deps?

#

Like pip freeze?

scenic perch
#

yeah

foggy forum
#

Yeah I like to canonicalize names in my .in files

stuck girder
#

about the pip cache and GitHub Actions:

Often a key like ${{ matrix.os }}-${{ matrix.python-version }} is used so that a unique pip cache is created and restored for each OS x Python version combination

Would it be ok to only use ${{ matrix.os }}, so the same cache is shared between all Python versions on the same OS? (This is what happens on my own computer.) Or would it be important to isolate by Python version too on the CI?

#

I'm asking because actions/setup-python has added built-in pip caching! this is really good news, much less config => easier to set up => more caching used

instead of one of these complicated things: https://github.com/actions/cache/blob/main/examples.md#python---pip, can just add cache: pip and (optionally cache-dependency-path: foo to specify something other than requirements.txt to actions/setup-python

however, it only uses the OS and not Python version for the cache key. is this a problem?

lunar gyro
#

As long as you don’t have funky build-from-source stuff (e.g. different binaries but same wheel name) it should work. pip’s caching system is platform-agnostic, and how wheels are named effectively handles platform differences automatically

#

The advantage of the “usual” approach is to make cache-busting is easier (when you need to bust you don’t need to bust cache for all your jobs)

stuck girder
#

sounds good!

foggy forum
#

what's the status of https://github.com/pypa/pip/issues/9020#issuecomment-713103977 ?
is it still a bug or is it now a design choice - poetry devs seem to think it's a design choice but the last I see is @pfmoore saying it's a bug?
https://github.com/python-poetry/poetry/issues/3472#issuecomment-979866861

GitHub
pip 20.2.4 with 2020 resolver does not use pinned version in the co...

What did you want to do? Consider this requirements.txt: -c constraints.txt Django==3.1 \ --hash=sha256:1a63f5bb6ff4d7c42f62a519edc2adbb37f9b78068a5a862beff858b68e3dc8b \ --hash=sha256:2d390268a13c...

GitHub
Poetry fails in CI/CD with ERROR: In --require-hashes mode, all req...

I am on the latest Poetry version. I have searched the issues of this repo and believe that this is not a duplicate. If an exception occurs when executing a command, I executed it again in debug mo...

lunar gyro
#

The tracking issue contains more context and how to fix it. https://github.com/pypa/pip/issues/9243
There’s currently no-one working on this, as far as I know. It seems that people just don’t care enough to actually do anything (except blueyed who made some progress).

GitHub
[regression] resolvelib-based dependency processing of constraints ...

What did you want to do? Use pip-tools generated requirements.txt + constraints.txt pair with hashes via pip install -r requirements.txt -c constraints.txt. It seems to be unintentionally broken wi...

shy echo
#

FWIW, I don't think the poetry maintainers are even involved in the discussion.

#

It seems like OP posted a giant closing note there.

lunar gyro
#

I mentioned also in the thread, the fix should not be difficult. I don’t plan to work on it since I made a policy for myself to not fix bugs in the resolver myself, but only to help contributors. Paul, Pradyun and I were like the only three[1] people on the planet that know how the resolver works and I don’t want it to continue being the case.

[1]: There’s now a fourth, a folk with username notatallshaw

foggy forum
shy echo
foggy forum
lunar gyro
#

I’m not Pradyun but I think this one is the last feature to bring the new resolver to cover all use cases of the legacy one (except for “features” we “redefined” as bugs)

shy echo
#

Yea, pretty much. That, and the fact that we'd need another round of communication around our resolver removal that none of us have had the time to do in quite a while.

foggy forum
#

Is there a ticket for coordinating the removal of the legacy resolver?

cyan cargo
#

anyone aware which versionsof pip still have from pip._internal.utils.misc import get_installed_distributions pipdeptree is a bit of a bad actor

lunar gyro
cyan cargo
late mortar
#

I've run into an issue with "platformdirs" failing on Android (the package vendored in pip's). any guidance on who / where I should go to report the issue ?

shy echo
#

On the platformdirs issue tracker?

late mortar
shy echo
#

Yes.

cyan cargo
#

is there any acceptable way to interact with wheels from the pip cache for importing in temporary processes?

context is wanting to create a new type of bootstrapper for execnet, and considering where to put/use wheels to use for it

execnet is what pytest-xdist uses under the hood for multi-process and mutli-host testing

shy echo
#

Not really. The only promise there is you can delete subsets of the cache, and it'll still work.

And, there's no promises about the structure.

cyan cargo
#

ok, so to get a wheel, the best idea would be pip wheel or pip install -t --no-deps

lunar gyro
#

You can also do pip cache list --format=abspath and copy things out I guess

shy echo
#

I think so. I guess you can also use the pip cache command to list files for a package

lunar gyro
#

But that only works for the wheel cache, not the http cache

shy echo
#

Yep.

#

But IIUC, that's all he needs.

lunar gyro
#

I don’t quite remember if pip wheel uses the wheel cache. If it does then it’s the best solution

shy echo
#

It does, as I remember.

cyan cargo
#

is there any way t use pips dependency resolver, but completely avoid installing anything

i have this outstanding task of implementing a shared wheel cache that i'd love to make work - but i dont want to fork pip to create the poc, and i don't want to go around instal lthen replace

shy echo
#

#53 -- not yet, but it basically needs someone to figure out what the output format should be.

cyan cargo
shy echo
#

Not really, since we could also have selected sadists for installing

#

sadists 😂

cyan cargo
#

hmm, reads right

#

@shy echo so the pep does not support locking in the way resolution might search/find things? unforutnate in terms of resolving i suppose

shy echo
#

Yea, that's not the intended use case. That's something that the pinning thing that @lunar gyro proposed in a separate thread would do.

cyan cargo
#

i see

#

hmm, i would be fine if the output was a requirements file ^^

shy echo
#

But a requirements file doesn't capture all aspects of the output.

cyan cargo
#

im ware of that, currently what i do is makea venv, pip install all the things, then grep filter a freeze to get a constraints file

#

my context is about shipping about 70 (soon more) testsuites that share baseline external dependencies

little kite
#

would anybody know why pip does that?

shy echo
#

Not quite sure, to be honest.

little kite
shy echo
#

Yeaaaa

little kite
#

Is there a way to instruct pip to install the build requires from the build-system table without triggering a build, into the current environment?

cyan cargo
#

it seems there isnt a uiltin

shy echo
limber ore
#

I'm experiencing a weird issue.

basically, I have a plugin for a pyproject.toml-based build backend that does subprocess.run([sys.executable, ...]). this subprocess appears to not have access to build reqs even though the pip install -v . output shows them being installed. python -m build -w works fine.

any ideas?

little kite
#

I'm guessing the subprocess is not inheriting pip's sitecustomize, pip doesn't use venvs for build isolation

limber ore
#

oh, how can I ensure that?

little kite
#

are you passing env to the subprocess?

limber ore
#

no

little kite
#

is something manipulating the Python path?

#

or if you could show where this is happening, I can take a look

limber ore
#

pip's last command:

  Installing backend dependencies ... done
  Running command 'C:\Users\ofek\AppData\Local\Programs\Python\Python38\python.exe' 'C:\Users\ofek\AppData\Local\Programs\Python\Python38\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py' prepare_metadata_for_build_wheel 'C:\Users\ofek\AppData\Local\Temp\tmpuqjqofiz'
little kite
#

I ran python -m pip wheel --no-deps . on that Pydantic fork and it failed with AttributeError: module 'hatchling.build' has no attribute 'prepare_metadata_for_build_wheel'

limber ore
#

that's expected, should be main error right after

little kite
#

ah, I see, it falls back on building a wheel and then it can't call mypyc

little kite
#

well, it can obviously import mypy, right? otherwise setup wouldn't run at all

#

so I think it has something to do with how mypy is reading imports

#

or maybe something that mypycify is doing

#

meaning it only looks in sys.prefix and sys.exec_prefix and since there's no venv, that'll be the un-isolated site packages

#

which also explains why it works with build

limber ore
#

oh darn, good find! what can I do about this?

little kite
#

you could monkeypatch mypy in hatch-mypyc I guess

limber ore
little kite
#

👍

jolly forum
#

greetings, i have a question regarding pip and setuptools, i'm not sure if this is the correct channel tho.
but, when i create a package (a wheel or any other packages supported by setuptools)
the package seems to be correct, but, when i try to install it, the install folder only has the pre compiled files and it's missing all images, fonts and other stuff that my package needs.

shy echo
#

Hiya! We usually don't do user support here, but I've got a bit of time at hand so I'm happy to help as a one-off. :)

Can you share your setup.py file?

jolly forum
#

yes one moment.

shy echo
#

Can you paste the code here in a code block? Put a line with the following three characters before and after it: ```

#

(I'm on mobile, so can't really see .py files easily)

jolly forum
#

its too long for that, it automatically converst it as a file

#

but i can do it on parts, one moment

shy echo
#

Ah, interesting.

#

No, it's fine. :)

jolly forum
#

This is the setup section.

if __name__ == "__main__":
    # Static strings are in setup.cfg
    resources :list = Find_Resources(File_extentions)
    with open("resources.txt", "w") as file_:
        file_.write(json.dumps(glob("BaseFramework/*"), indent=4))
    write_version_info()
    setup(
        version=get_version(),
        packages=find_packages(
            include=["BaseFramework", "BaseFramework.*"], exclude=["BaseFramework.tools.release"]
        ),
        package_dir={"BaseFramework": "BaseFramework"},
        package_data={
            "BaseFramework": [
                *Find_Resources(File_extentions),
                "resources/themes/*.json",
            ]
        },
        data_files=[
        ],
        include_package_data=True,
        extras_require={
            "dev": [
                "pre-commit",
                "black",
                "isort[pyproject]",
                "flake8",
                "pytest",
                "pytest-cov",
                "pytest_asyncio",
                "pytest-timeout",
                "coveralls",
                "pyinstaller[hook_testing]",
            ],
            "docs": [
                "sphinx",
                "sphinx-autoapi==1.4.0",
                "sphinx_rtd_theme",
                "sphinx-notfound-page",
            ],
        },
        install_requires=[
            "kivy>=2.0.0",
            "pillow",
            "kivymd @ https://github.com/podraco/KivyMD/archive/refs/heads/ColorUpdate-Buttons.zip",
            "kivydg",
            "twisted",
            "colorama",
            "websocket",
        ],
        setup_requires=[],
        python_requires=">=3.9",  # Due to there is list[str] in common.py that is only available in python >= 3.9
        # entry_points={
        #     "pyinstaller40": [
        #         "hook-dirs = BaseFramework.tools.packaging.pyinstaller:"
        #         "get_hook_dirs",
        #         "tests = BaseFramework.tools.packaging.pyinstaller:"
        #         "get_pyinstaller_tests",
        #     ]
        # },
    )
#

these are the methods used

import os
import re
import subprocess
import sys
from datetime import datetime
from time import time
from pathlib import Path
import json
from setuptools import find_packages, setup
from glob import glob

assert sys.version_info >= (3, 6, 0), "BaseFramework requires Python 3.6+"

G_package_name = "BaseFramework"
G_root = os.path.join(os.path.dirname(__file__), "BaseFramework")
File_extentions = [
    ".png",
    ".jpg",
    ".jpeg",
    ".ttf",
    ".otf",
    ".avi",
    ".mp4",
    ".json",
    ".ini",
    "kv",
]


def Find_Resources(path_patterns: list = [], data_files: bool = False):
    if not path_patterns:
        print("SETUP: Find_Resources :: Unable to fetch resources. the pattern is empty.")
        sys.exit(1)
    resources_list: list = []
    resource_dict: dict = {}
    for Container, SubDirs, Folder in os.walk(G_root):
        print(
            f"OS WALK RETRIEVED:"
            f"Container   :::   {Container}\n\t"
            f"SubDirs     :::   {SubDirs}\n\t"
            f"Folder      :::   {Folder}\n\t"
        )
        filename: str = ''
        for filename in Folder:
            for suffix in path_patterns:
                if filename.endswith(suffix):
                    filepath = os.path.join(str(Path(*Path(Container).parts[1:])), filename)
                    resources_list.append(filepath)
        if data_files is True:
            resource_dict[Container] = resources_list

    return resources_list


def get_version() -> str:
    """
    Get the current version of the package (__init__.py).
    """
    version_file = os.path.join(
        os.path.dirname(__file__), G_package_name, "__init__.py"
    )
    version_file_data = open(version_file, "rt", encoding="utf-8").read()
    version_regex = r"(?<=^__version__ = ['\"])[^'\"]+(?=['\"]$)"
    try:
        version = re.findall(version_regex, version_file_data, re.M)[0]
        return version
    except IndexError:
        raise ValueError(f"Unable to find version string in {version_file}.")


def write_version_info():
    """Create _version.py file with git revision and date."""
    filename = os.path.join(os.path.dirname(__file__), G_package_name, "_version.py")
    version = get_version()
    epoch = int(os.environ.get("SOURCE_DATE_EPOCH", time()))
    date = datetime.utcfromtimestamp(epoch).strftime("%Y-%m-%d")
    try:
        git_revision = (
            subprocess.check_output(["git", "rev-parse", "HEAD"])
            .strip()
            .decode("ascii")
        )
    except (
        subprocess.CalledProcessError,
        OSError,
        IOError,
        FileNotFoundError,
    ) as e:
        # CalledProcessError has no errno
        errno = getattr(e, "errno", None)
        if errno != 2 and "CalledProcessError" not in repr(e):
            raise
        git_revision = "Unknown"

    version_info = (
        f"# THIS FILE IS GENERATED FROM BaseFramework SETUP.PY\n"
        f'__version__ = "{version}"\n'
        f'__hash__ = "{git_revision}"\n'
        f'__short_hash__ = "{git_revision[:7]}"\n'
        f'__date__ = "{date}"\n'
    )

    open(filename, "wt", encoding="utf-8").write(version_info)
shy echo
#

This is a package with only Python code, and no C/C++ code, correct?

jolly forum
#

yes

#

it's pure python and some images and needed files, a small UI App.

shy echo
#

Hmm... May I suggest changing things up a bit, instead of trying to get things to work with setuptools?

jolly forum
#

how so?

shy echo
#

Using flit to build things instead of setuptools.

#

Your __version__ is in BaseFramework/__init__.py, right?

#

Place a new file named pyproject.toml next to the setup.py file, with the following contents:

[build-system]
requires = ["flit_core >=3.6.0,<4"]
build-backend = "flit_core.buildapi"

[project]
name = "BaseFramework"
description = "A small UI app"
requires-python = ">=3.9"
dynamic = [
    "version",
]

dependencies = [
    "kivy>=2.0.0",
    "pillow",
    "kivymd @ https://github.com/podraco/KivyMD/archive/refs/heads/ColorUpdate-Buttons.zip",
    "kivydg",
    "twisted",
    "colorama",
    "websocket",
]

[project.optional-dependencies]
dev = [
    "pre-commit",
    "black",
    "isort[pyproject]",
    "flake8",
    "pytest",
    "pytest-cov",
    "pytest_asyncio",
    "pytest-timeout",
    "coveralls",
    "pyinstaller[hook_testing]",
]
docs = [
    "sphinx",
    "sphinx-autoapi==1.4.0",
    "sphinx_rtd_theme",
    "sphinx-notfound-page",
]
#

And then run git add .

#

Then, try generating a wheel using pip wheel .. You can open the generated wheel like a zip file, by renaming the .whl to .zip. Check the contents of that -- that should have all the resources correctly added in.

shy echo
jolly forum
#

well, version is generated, but i have the major version like that

#

all tho i know that the wheel and other packages has the packaged data

#

the issue is that it's not installing those files inside the evnrionement.

#

all tho, i'm going to try that method and see what happens 😄

shy echo
#

Ah, interesting! If the wheel has the data, then that is getting installed.

#

What might be happening is that the code is making an assumption about where the files are, that might be wrong?

jolly forum
#

thats the thing.
this is the structure that appears on the environment

tree C:\USERS\USER\ENVX\LIB\SITE-PACKAGES\BaseFramework
├───include
│   ├───includes
│   │   ├───standards
│   │   │   └───__pycache__
│   │   └───__pycache__
│   ├───kv
│   │   └───__pycache__
│   └───__pycache__
├───resources
│   ├───audio
│   │   └───__pycache__
│   ├───font
│   │   ├───TheFont1
│   │   │   ├───OTF
│   │   │   ├───Web
│   │   │   └───__MACOSX
│   │   │       ├───OTF
│   │   │       └───Web
│   │   ├───Roboto_Condensed
│   │   ├───Roboto_Mono
│   │   └───__pycache__
│   ├───image
│   │   └───__pycache__
│   ├───themes
│   ├───video
│   │   └───__pycache__
│   └───__pycache__
├───server
│   └───__pycache__
├───offuscate
│   ├───includes
│   │   └───__pycache__
│   └───__pycache__
└───__pycache__
#

it's empty except of the pre compiled python files.

shy echo
#

HUH.

#

Is there a BaseFramework-{something}.dist-info folder in the site-packages?

#

If so, can you check what the RECORD file in that folder contains?

jolly forum
#

let me see.

#

i dont understand what happened but with the pyproject.toml is the information is now there.

#

Thanks For your help, you really helped me a lot today SlimeRainbow

shy echo
#

Happy to!

#

It's not being used anymore.

jolly forum
#

thanks :D, now i have to replicate the issue and try to find the root cause.

#

have a nice day 😄

shy echo
shy echo
jolly forum
#

ho, yes, but that's usually what i do, if i find a strange behavior, i report that to the github source, if it's open source, if the issue was on my end, then i would have to make ammends 😄

shy echo
#

:)

fickle shell
shy echo
#

There's no docs of the environment variables. They're computed from the command line options, and exposed that way.

shy echo
#

Can I get some 👀 to review the code in https://github.com/pypa/pip/pull/10795 ? I'd realllly like to get this out the door this month, but... I'll admit it's a bit too close for our release cycle. 🙈

149500510-5c36a3b3-dce6-4ae3-93bc-658bca67970e.png
#

Better error message FTW!

little kite
#

That looks great, well done

shy echo
#

Thanks!

stuck girder
#

yes does look better! I've been using the dev version with the progress bars since it was merged and I really like it and am looking forward to more 🙂

foggy forum
#

Could it say "this is a problem with {package_name} not pip"?

#

Rather than alluding to "package named above"

shy echo
#

We may not have a package name; and if we do, it'd literally be mentioned on the line above.

foggy forum
#

Oh the from file:///... bit

shy echo
#

Yup.

foggy forum
#

Have you seen the new spec for exception group formatting?

#

Sort of reminds me of this output

shy echo
#

Not yet, no.

foggy forum
shy echo
#

That's... very different. But I guess I see what you might be thinking?

hidden flame
#

I'd guess this is an explicit part of the design but I find the duplication of the exact error message to be confusing in this case (see below, the old image I attached wasn't graet). It's great when there's an exact error message above (say subprocess-error) and then the more user friendly one (say metadata-generation-failed). | cc @shy echo

#

I think why it feels confusing to me is because it makes it appear two separate things failed when really there was one issue. It's also not as clear which package / file / source idk was problematic compared to the netadata-generation-failed example, you have look a bit up to see the "discarding ..." message. It mayy be better to print something like:

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

for the second final error message if they're identical. It'd at least help with the visual clutter which could be scary.

hidden flame
#

Haven't been able to break it though so it appears your cookies will stay safe for another day 🍪

Fantastic work overall!

shy echo
#

Although, I have a fun Windows-only failure in a test for logic I haven't touched. So... That's great.

verbal lotus
#

I was told on other server's internals-and-peps that this is for when a package can have a particular version replaced without causing a version bump (pip checks if build tag is higher). I have hashes pinned anyway, so I don't benefit from these checks over the network that pip is doing. Is there a way for me to disable pip's networking assuming I have 100% cache hits? My CI is spending a minute of Collecting... and Using cached..., presumably because using a cache still involves a lot of network requests

shy echo
#

I don't think there's any way to do that.

#

If you want, you can move to a two step process, where you first create a wheelhouse and then install from that.

#

The "wheelhouse" is basically what the documentation calls the output directory of pip wheel.

#

Then, only the first step hits the cache/network. The second step uses the wheelhouse generated by the first step and needs absolutely nothing else.

small cove
#

Does pip implement any locking to prevent parallel pip invocations from corrupting the pip cache or the venv? I already found that pip install numpy & pip install numpy & pip install numpy & pip install numpy & pip install numpy & pip install numpy & pip install numpy & pip install numpy reliably produces file not found errors for me

shy echo
#

There's no protections, no.

small cove
#

ok, thanks for the answer

young nacelle
#

Maybe someone can help me with a question: given a wheel file, is it possible to tell pip to install it with some given extras without knowing the name of the package?

e.g. let's say I have a script with:

wheel_file="build-0.7.0-py3-none-any.whl"
extras="virtualenv"

can I do something like the following?

pip install "$wheel_file" --extras "$extras"

(The alternative would be opening up the zip file, looking for a METADATA member, parsing it and extracting the name and reconstructing a string with "$name[$extras] @ file://$(realpath $wheel_file)"... which is a very convoluted process...)

shy echo
#

If you have the wheel, you can derive the package name out of it -- the section before the first -

#

So... You can't end up in a situation where you have a wheel file without knowing the name of the package.

little kite
#

You can also just install $wheel_file[$extras], without having to parse the wheel filename

shy echo
#

I'm not sure that works?

young nacelle
#

Thank you very much @shy echo and @little kite.

@little kite suggestion does seem to work as long as CWD contains the wheel file directly itself

little kite
#

You can provide an absolute or relative path, the wheel file can be anywhere

#

If you're actually writing this in Bash remember to quote $wheel_file[$extras]

young nacelle
#

Thank you, my bad. I was very sure that I had tried that before from a different directory and it not work, but I was wrong, it does seem to work fine

shy echo
#

Wow, I didn't realise the entire ecosystem of Python packaging that wasn't PyPI itself, depended on non-HTML5 index pages. 🤦‍♂️

hidden flame
#

wait what, they depend on non HTML5 index pages? 👀

shy echo
hidden flame
#

looks fun

#

and my stress level already increased a lot when black had one issue opened about our unofficial APIs being more strict (since black is now compiled with mypyc which needs types to be not violated). The type annotation of black.Mode's target_version argument is Set while people were using list and other collections (which broke after we shipped mypyc by default)

shy echo
#

Ahahaha

#

Yea... "don't use unofficial APIs" is not a fun one to discuss with people.

hidden flame
#

it's amazing how much Hyrum's Law truly finds its way into everything 🙃

hidden flame
#

so 69 comments later... basically everything $corporate is generating non PEP 503 index pages apparently? I feel like I should be surprised but I am also not

shy echo
#

Yea... I feel like I should've anticipated this. I swear the one thing I remember from discussions w/ @hoary mist from like, 2017, was that Hyrum's law is just... always a thing to worry about when making changes.

hoary mist
#

lol

#

if you're not explicitly disallowing it, someone, somewhere out there is relying on it

#

100%

#

I saw the discussion going down

#

Might be worth adding a checklist to try and run against the major implementations

shy echo
#

In this case, the someone is Artifactory, Google, AWS, et al -- and all their users are very happy about the whole thing. 🙃

hoary mist
#

and/or asking them to donate resources to handle it

#

(also fine to say it doesn't happen often enough to bother if they're not doing it on their own accord)

shy echo
#

I want the vendors to step up, instead of doing their work for them.

hoary mist
#

yea, fair enough

limber ore
shy echo
#

Yep.

There may be any other HTML elements on the API pages as long as the required anchor elements exist.

limber ore
#

thank you 🙇‍♂️

hidden flame
maiden island
willow flicker
#

Is there a 20.0.x coming soon? Conda-forge has to modified to remove Python 3.6 from the pip recipe before a new release, or the automated release system will break Python 3.6 again, after we patched it for the last three releases. Just trying to get a feel for how hard I should push to get the patch in soon. 😉

shy echo
#

I'll cut it today or later this week.

#

I'm not feeling a 100% today, so... It might be later in the week.

shy echo
maiden island
shy echo
#

No, looks like bad eating habits.

hidden flame
#

I dislike fatty junk food more and more as times passes, I still love sweets but yeah junk food is named for good reason 😅

willow flicker
#

Conda-forge is patched, pip can release any time and the auto-update will work correctly. 🙂

shy echo
#

Good to know!

willow flicker
#

Now I have to fix @lime gust's zipp, same issue...

#

Conda-forge really needs to have some sort of test for this...

shy echo
#

resists urge to pick up yet another side quest in OSS

shy echo
#

Well.. no pip release tonight. GH just went down. 😆

maiden island
#

LoL, really? I just did a release of one of my tools ~10 mins ago

hidden flame
#

I heard that pradyunsg uses codespaces which is down at the moment

unknown.png
#

also yeaaa issues and pulls aren't doing so great either

shy echo
#

Everything is down. Hurray.

#

I play with Codespaces for all the projects that I don't think are important enough to clone. :P

stuck girder
#

sorry!

shy echo
#

Still no pip release, I'm playing TABS now.

lunar gyro
#

@shy echo Should we update ensurepip to 22.0? (Or did you already?)

shy echo
#

We should.

#

I haven't.

shy echo
lunar gyro
shy echo
#

Okie!

#

Wait, you also have the commit bit tho. Can someone without the commit bit confirm? 🙈

#

I have a feeling that they'll get redirected to the issue form, but I wanna check.

little kite
shy echo
#

👍

#

That's what I was hoping it won't do, but then again, I guess that's a good thing.

little kite
#

There's an option there "(maintainers only) Blank issue" but when I click on it nothing happens

shy echo
#

Correct, as the name suggests, it's maintainers only. :P

little kite
#

Yeah, just wasn't sure if it was meant to be visible

shy echo
wispy coral
#

I was asking in #setuptools (#setuptools message), but this is probably a better question for here: Is there any reasonable way for an isolated build in pip to know anything about the environment it's being installed into?
The reason I'm asking is that in Debian we replace the default sysconfig scheme to install to /usr/local (away from anything package-managed), when we're doing system-wide installs. But people prefer not to have a local tree inside their virtualenvs, so you need to know what the target is....

Discord
Discord - A New Way to Chat with Friends & Communities

Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

shy echo
#

Not right now, no.

#

Got a link to a fuller description of the issue?

wispy coral
#

No, I'll have to file one.

#

Although, now thinking about it, I get the right behavior with SETUPTOOLS_USE_DISTUTILS=stdlib, so I think I'm missing something that's not the issue I asked about here

shy echo
lunar gyro
#

I mostly need to think through the implications of 1. removing backtracking and 2. making the Criterion structure public. I may be able to find some time for it toward this weekend.

shy echo
#

IIUC, we already have Criterion exposed in one of the other hooks.

#

(through causes)

true parrot
lunar gyro
true parrot
shy echo
#

@pylang that issue is in the milestone for dropping the legacy resolver.

foggy forum
#

why is:

Python 3.11.0a5 (main, Feb 12 2022, 17:11:59) [GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from packaging.markers import Marker
>>> Marker("python_full_version > '3.6.2'")
<Marker('python_full_version > "3.6.2"')>
>>> Marker("python_full_version > '3.6.2'").evaluate({"python_full_version": "3.11.0a5"})
False
>>> Marker("python_full_version > '3.6.2'").evaluate({"python_full_version": "3.11"})
True
>>>

?

#

poetry puts that ; python_full_version >= '3.6.2' there because black has a requires_python >= 3.6.2

shy echo
#

Because it's a pre-release version. Look at how markers is comparing versions.

foggy forum
#

so it's impossible to define a marker that matches ">= 3.6.2" ?

foggy forum
# shy echo Because it's a pre-release version. Look at how markers is comparing versions.

it looks like Version comparison behaves as I expect - but not packaging.markers.Markers evaluation:

>>> packaging.version.Version("3.11.0a5") >= packaging.version.Version("3.6")
True
>>> packaging.version.Version("3.11.0a5") >= packaging.version.Version("3.6.2")
True
>>> packaging.version.Version("3.11.0a5") > packaging.version.Version("3.6.2")
True
>>> import packaging.markers
>>> packaging.markers.Marker("python_full_version > '3.6.2'").evaluate({"python_full_version": "3.11.0a5"})
False
>>> packaging.markers.Marker("python_full_version > '3.6.2'").evaluate({"python_full_version": "3.11.0"})
True
>>>
#

(I expected all of these to be True)

#

oh it doesn't use version comparison

>>> import packaging.specifiers
>>> packaging.specifiers.Specifier(">3.6.2").contains("3.11.0")
True
>>> packaging.specifiers.Specifier(">3.6.2").contains("3.11.0a5")
False
shy echo
#

File an issue and PR?

#

I agree that this is a bug in packaging, FWIW. :)

foggy forum
#

can I get a CI aproval

#

or configure "Require approval for first-time contributors who are new to GitHub "

shy echo
#

Just triggered it. :)

foggy forum
foggy forum
#

ah Brett Cannon is not on this disc

foggy forum
#

@shy echo what's the process to get packaging vendored?

hidden flame
foggy forum
hidden flame
#

that's just the escaped id based mention

#

I didn't want to ping him

lunar gyro
#

TallSnarkyCDN will work as well

hidden flame
#

oh yeah, I'm dumb :)

lost hawk
#

The @ mention worked; I'm on the server, just not this channel

#

And I have updated my nickname on the server

hidden flame
#

wait what, that id mention worked .......... sigh so that was extra dumb 🙃

#

@hidden flame

#

whelp that did mention me.

final crypt
#

Hi, I currently created a fork of a project and published to pypi with a different name, but when I install both they don't complain. Even though the package name is different, they provide the same python module. Shouldn't pip just abort when such conflict happens?

young nacelle
shy echo
#

This doesn't fail, search for overwrite in pip's issue tracker -- there's an open issue for this behavior.

final crypt
#

I think I don't have Provides-Dist on the metadata, but you are saying that pip doesn't check for that?

young nacelle
final crypt
obtuse lagoon
#

@shy echo is there any point on working on it right now, or would it be better to just wait until pip switches to installer

shy echo