#general

1 messages ¡ Page 1 of 1 (latest)

hearty depot
noble socket
lean lake
#

@lyric hedge gets access where ever I have it basically 😛

distant briar
#

So you come in two for one package deal :)

misty dew
lean lake
#

God bless the queen.

blazing lantern
lyric hedge
lean lake
#

That’s my man

soft glen
high stone
blazing lantern
#

I m the Canadian he s referring to

wheat crag
tidal escarp
still forum
silver vapor
hard trail
keen wadi
civic compass
loud merlin
obtuse holly
#

What is linehaul?

spiral urchin
obtuse holly
#

ah 😅

#

should the url in the packaging guide be switched to new repository?

spiral urchin
#

yeah feel free to do a pr

high stone
#

That's the link in the guide as of right now. :)

obtuse holly
high stone
#

Oh nice, you're quick. :P

#

And so is Brian! Thanks for the quick fix on that. :)

finite lynx
finite lynx
elfin shuttle
elfin shuttle
#

Which channel is best to ask a BanderSnatch Question?

#

I don't know if I'm experience a bug or user error

fallen shuttle
elfin shuttle
#

lol, yeah I would have thought that but I see a bunch of automated messages in there

#

I didn't want it drowned out

fallen shuttle
#

You can also have a look on the existing issues, and do a quick search on https://discuss.python.org to see if the question was ever made before

elfin shuttle
#

I'll check once more. How embarassing for it to be there

fallen shuttle
elfin shuttle
twin zealot
lusty quarry
distant briar
#

He would want to replace current config with pyproject.toml rather than handle both of them which is fair because it adds to the complexity. But if he does that, suddenly some projects are forced to migrate to pep 517 build system because that's what presence of pyproject.toml requires. I think setuptools has a separate legacy build system (build_meta.legacy?) that's pep 517 compatible but actually works the same unlike the regular build_meta? I don't think it's flake8 limitation, just his personal opinion.

inland prawn
lusty quarry
#

that's a bummer

inland prawn
#

Yeah

lusty quarry
#

Flake8-pyproject also has bad manners and force-feeds Flake8 the spam it so despises.

😂 I might use it just due to that sentence

distant briar
#

I don't really understand the sentence lol

#

What spam does it force-feed

fallen shuttle
distant briar
#

Is anyone else a little bummed out about flake8 dropping support for user-wide configuration

inland prawn
#

Nah

#

Project-wise makes more sense

distant briar
#

My problem is mainly with file exclusion

#

Like, I want to globally exclude venv directories...

mortal shore
#

I think the long term plan in pip is to remove the old way of building

#

So everything goes through the py project path

#

Albeit with a default py project

#

If my memory is correct, once that happens it seems like the other blocker on that thread is gone

inland prawn
distant briar
#

Would be cool to have statistics on projects that still use non-pep 517 build process

mortal shore
#

I assume because black got some blow back when they were a very early adopter of pyproject and people got confused by the fact using it to configure your black settings opted you into an unrelated change in how your project was built

distant briar
#

Even actively maintained projects use those because they either didn't have to update or maybe because of the "packaging is a mess" dogma

#

Idk

mortal shore
#

That was a messy time back in the day

distant briar
#

towncrier was the first tool that made us use pyproject.toml

inland prawn
#

Flake is probably the only project that actively refuses to use pyproject

distant briar
#

Even though we used black earlier than that, just didn't set the options in pyproject.toml at the time

#

flake8 and nose2 seem like most notable projects that are in the considering category on there

#

And pydocstyle

soft glen
obtuse holly
#

pip still has different behaviour based on the presence of pyproject.toml
is this still true?

distant briar
#

Technically legacy backend is supposed to do the same thing (imitate pre-pep 517 behavior) though but presence of pyproject.toml does in fact cause pip to go through different code paths and some discrepancies could occur I imagine

#

build isolation might work somewhat differently?

#

Since that's pep 517's thing

mortal shore
#

the big thing is pyproject.toml opted a project into build isolation

#

was the main breaking point

#

not sure if it still does

#

or rather, nto sure if non pyproject.toml still uses not build isolation

manic belfry
#

and setup.py without pyproject.toml means no build isolation

inland prawn
manic belfry
#

Well... I think there were loong discussions about that. I've not investigated to dig out the reasons but my instinct says this ship has sailed.

spiral urchin
#

It’s extremely unlikely since the general goal is the other way around, to enable isolation for all projects

distant briar
#

PEP 517 was written in such a way that the frontend (pip) is free to choose between invoking setup.py (as used to be the case before the PEP) and using the setuptools.build_meta:__legacy__ PEP 517 backend. Though that was actually a change made to the PEP after it was implemented in pip.

#

The discrepancy seems non-ideal but if the project is already being worked on (which clearly it is if you want to start using pyproject.toml) and someone wants to use pyproject.toml then it makes sense to force them to move on and use the new way of doing things instead of that legacy thing that we would love to just go away 😄

#

There was a brief time (well, about a year) when pyproject.toml existed (PEP 518) and PEP 517 didn't so someone might have gotten opted into that solution when they didn't want to but that was probably not that many users and kind of early adopters :P

tidal kiln
#

pip does not follow the PEP though

#

the legacy backend should be equivalent to invoking setup.py

#

just makes it unnecessary to have specific code to do the setup.py invokation

#

pyproject.toml isn't tied to PEP 517, PEP 517 is only triggered when you specify the build system table

#

all other cases default to the previous behavior of invoking setup.py, even if it might be through the __legacy__ backend

#

I know this can be a bit confusing

mortal shore
#

eh

#

the problem comes from build isolation

#

and neither PEP requires build isolation, or even says much about it

#

the closest it comes is a non normative suggestion in 518 to use isolated build environments

tidal kiln
#

yep

distant briar
#

517 does recommend using an isolated env

#

SHOULD not MUST

#

so if that recommendation is followed then running __legacy__ backend does cause it to use that isolation

mortal shore
#

@distant briar that's in a non normative section

#

which means it's not actually part of the spec, it's just recommendations

distant briar
#

Yeah, I did say it's a recommendation

#

not a requirement

golden falcon
inland prawn
#

😐

indigo token
#

Can we get PyCQA to detox? It’s always a bad sign when people don’t engage with polite good-faith arguments.

#

If someone gets so paranoid they see sealioning and bad faith arguments everywhere, they should probably take a break from maintaining software

obtuse holly
golden falcon
#

The GitHub org

lyric hedge
#

apparently there's some bad history between black and PyCQA or something ¯_(ツ)_/¯

#

don't ask me for more details because I don't know, I've just noticed mentions of them in our core team chat were never positive

tidal kiln
#

I don't understand why we continue to reward his behavior

lusty quarry
inland prawn
lusty quarry
#

to be fair to him, Twitter is broadly crazy lol

blazing lantern
tidal kiln
#

he was added as a PSF fellow not so long ago

#

last time I tried to point out his bad behavior he made fun of my boundaries, which coupled to everything else I was dealing with at the time, lead to me taking a break from open source work

#

largely everyone else in this community is actually pretty great, but his involvement makes me feel unsafe

lusty quarry
#

"unsafe" is quite strong, we should try to avoid classifying speech or text as harmful/violent. but I feel ya 👍

tidal kiln
#

I mean, this is the accurate description for how I feel, but I acknowledge I may be in a particularly vulnerable place

#

and to be clear, the issue here is not one particular action, but rather that is it very much a pattern, and he doesn't even try to understand how his actions impact other people or take accountability

mortal shore
#

A pile on doesn’t seem super helpful, but if someone makes you feel unsafe or you think they’re creating a hostile environment then a good option would be to bring up those behaviors to the relevant CoC reporting addresses if those spaces have them

#

It looks like pycqa has one

#

(This is not any sort of official thing ofc, just my opinion of what I’ve seen work for bad behaviors in the past!)

noble matrix
#

Hi all, finally found my way to this Discord

distant briar
#

hello o/

mortal shore
#

o7

lusty quarry
#

hey there 🙂

noble matrix
#

Noticing the topic of discussion, FWIW Anthony is the creator and maintainer of some of my very favorite Python tools and I really wanted to like him, but unlike anyone else I've interacted with in the Python community, every single interaction I have had with him was negative. I'm not someone who says this lightly as I don't think I've had to say it about anyone else I've met in open source (or maybe even in general), but Anthony is the one person in this field whom I actively avoid any interaction with due to how badly things have gone in the past, and how interacting with him invariably harms my motivation and well being.

I thought I was the only one, but I've both seen firsthand and heard secondhand from a number of different people recently their experiences with him that were a lot worse than what I went through...legal threats, intimidation, insults, harassment, the whole nine yards. I hesitated to say anything because I also see that Anthony does a lot for the community in terms of his work as a maintainer, but it seems like things are just getting worse and worse and it is harming more people, the community and honestly himself too. I would be supportive of whatever is the best way forward to constructively handle this situation and willing to help in that regard.

blazing lantern
# tidal kiln he was added as a PSF fellow not so long ago

That group of "we" quite possibly is unaware of what people are bringing up here. Remember that most of the community isn't on Twitter, don't congregate in the relatively small circle of proactive Python project contributors that we do, etc.

#

The key thing is to always report bad behaviour to the appropriate conduct group.

lusty quarry
lusty quarry
#

ok, just emailed about flake8 🤞

indigo token
indigo token
# lusty quarry "unsafe" is quite strong, we should try to avoid classifying speech or text as h...

Why not? I understand that we should give the benefit of doubt, and shouldn’t postulate our subjective opinion as truth, but @tidal kiln simply talked about how something made them feel, and didn’t claim to know an universal truth about how to classify what happened.

And yeah, according to what everyone said here, this seems to be a pattern and not an isolated incident.

I don’t want a witch hunt either, but I do want a safe community, so yeah: The CoC is there to help people figure out what to do in such a situation.

tidal kiln
#

FYI arch folks already reported him to the pytest CoC and nothing really came of it

indigo token
#

Welp, I was about to write “but when that doesn’t help, other solutions need to be found”, but thought I shouldn’t be so pessimistic. I should have been pessimistic. My mistake was ignoring the fact that nothing has happened so far, so my assumption that the processes might work was of course flawed.

I hate situations like this. Every community has those long time problematic members that drove away countless others who tried the official channels and then left quietly when nothing happened because the problematic member is friends with people or has clout or something. I call that person the “missing stair”: https://en.wikipedia.org/wiki/Missing_stair

tidal kiln
#

I think the PSF CoC for eg might be much better than this one, but I haven't gotten around to gather myself to do anything, and am not keen on being targeted by his community

indigo token
#

I wish one could approach those teams without the fear of it becoming a witch hunt of the victim outweighing the hope that it might actually fix something. But I fully understand that you feel the way you do. If you need an advocate at any time, hmu.

lusty quarry
noble matrix
#

this would be off topic but my point is

noble matrix
tidal kiln
#

okay, I think I should just go ahead and do it... if anyone wants to share your experiences with anthony (good and bad) feel free to write something down and send it to me

#

that way I can attached that to the mail

#

and hopefully put this behind us, whatever comes out of it comes out of it

distant briar
#

I misclicked the like button on Pradyun's tweet (which I tend to on mobile and then I remove it after few seconds) to Anthony and apparently got blocked (I assume that's the reason anyhow, I didn't have recent interactions with him otherwise) which made me kinda sad because I was actually following them on Twitter and wanted to continue to see his posts on my feed blobpain

torpid veldt
distant briar
torpid veldt
high stone
torpid veldt
#

You may have luck going via sigmavirus24 than PSF?

distant briar
#

Maybe something else contributed to it than just me liking their post then, wasn't really my intention to pile on

torpid veldt
#

I certainly don't want to tell anyone not to report conduct to a conduct committee, but there's positive movement in this space but not going anywhere fast

#

What is the pip behavior with an empty pyproject.toml? And what is it supposed to be?

#

Oh you spoke about it way up there in the chat log

torpid veldt
# golden falcon that's not too bad, I've been banned from PyCQA because I suggested that the Pyt...

https://github.com/PyCQA/meta/issues/54 can you post to this thread or are you still banned? (Can you even see it?) Is there something I could write here for you?

GitHub
Do not ban poeple from participating in PyCQA as a whole because of...

I can understand if a discussion get heated in a repository and you ban a person from the repository you maintain. But please do not ban someone from the whole organization for something minor. Two...

#

I'm conscious I may have fallen into the trap of telling you how not to step on the missing step

torpid veldt
golden falcon
torpid veldt
#

Maybe @high stone could write something there because as far as I can tell everyone likes them

golden falcon
#

I don't really want to be involved with these people more than I already have, my last interaction with them was on a Twine issue where they wanted to ban a user for being a bit shall we say slow on the uptake. Clearly that's a bannable offence

torpid veldt
#

sigmavirus24 or Anthony?

golden falcon
#

The former

torpid veldt
#

Urgh

golden falcon
#

They are basically just feeding off each other

#

a banquet of meanness

inland prawn
torpid veldt
#

Maybe talking to The-Compiler could help?

torpid veldt
inland prawn
torpid veldt
#

But yeah the whole permabanning people thing was brought to my attention by one of the banned people emailing me for help, I told them to try The-Compiler and Pierre

spiral urchin
#

Maybe it’s time we enforce the fallback behaviour globally for projects without a pyproject.toml (not build isolation, just switch to always use PEP 517 to build wheels). This is what we want to do anyway and it makes the flake8 opposition moot (since the presence of pyproject.toml no longer affects anything)

I’m only 10% sarcastic

mortal shore
#

I'm pretty sure build isolation is the behavior that flake8 is worried about

spiral urchin
#

So the presence of a pyproject.toml enables build isolation, which flake8 doesn’t like? Oh that’s more complicated then (although I think I mentioned that on flake8’s issue tracker I fail to follow the reasoning and nobody has come up with any explaination)

inland prawn
#

Honestly, I don’t understand why flake8 opposes pyproject.toml. Flake has nothing to do with builds or dependency management…

mortal shore
#

afaict the rationale is:

  • The presence of pyproject.toml affects how a project is built, primarily build isolation.
  • If flake8 uses pyproject.toml to handle configuration, then their uses will be forced to opt into those changes, regardless of whether they want to or not.
spiral urchin
#

Which is not true, as a lot of people have tried to explain, but they just insist it is the case without any concrete examples

inland prawn
mortal shore
#

I think both of those things are true still?

woven plover
#

A full clear and believable case has been made ages ago, it's pretty much the same reason why pytest only has stopgap support for config via pyproject.toml since years

spiral urchin
#

Sorry, I meant the second point never is true. The first is, but… why does it matter?

mortal shore
#

@spiral urchin I think flake8 wants to reduce the number of configuration files it supports to just (1)

spiral urchin
woven plover
#

IMHO People are not entitled to force maintainers Into repeating fruitless discussions again and again

spiral urchin
#

In that case I’m actually OK with their decisions on both the config format and to tell people to f off

mortal shore
#

FTR, I think at this point it's probably reasonable for tools to just assume that projects are going to be willing to opt into pyproject.toml semantics, so if it were me running flake8, I would just do that... but the decision not to do that seems reasonable?

spiral urchin
woven plover
mortal shore
#

It's entirely possible that there's a secret third portion of rationale which is:

  • The authors of this project don't like TOML and/or the PEPs that introduced pyproject.toml etc.

But I have no idea if that's true, and even if it were true... it's reasonable to say you don't like a format and don't want to support it.

mortal shore
#

Unfortunately it's impossible to stop people from wanting a tool to support X, and for them to all individually ask tool to support X

inland prawn
woven plover
#

I'd love to eventually provide an library that solves it (config +options via cli /env/ini/pyproject.toml with a sound type system

mortal shore
#

Not liking TOML as a format is a reasonable opinion.

#

all config file formats are awful

woven plover
#

Wrt flake8 My understanding is that the ini details leak deeply into the plugin api, and they won't slap on sloppy toml support

torpid veldt
#

I think the issue is really about the permabanning of people not necessarily technicalities in configuration

mortal shore
#

I wouldn't personally lock the pyproject.toml issue then get upset when people don't follow instructions and open up new issues about pyproject.toml, as I think at some point you're just setting yourself up for more frustration.

It's hard to say, certainly projects have a right to set their own boundaries, and enforce those boundaries, but at some point that can cross into a dark pattern. I'm not sure wrt flake8/Anthony as I don't follow that project closely and I can't see any of the tweets linked as apparently anthony blocked me though I don't remember ever interacting with them (which is fine, idc if someone blocks me, I run my mouth enough :P).

At the end of the day, if the keyholders of the PyCQA and the community disagree on what's reasonable, there's not much that can be done except for the community to stop using PyCQA projects and create their own alternatives, and stop providing space for them.

high stone
torpid veldt
#

Oh I'm blocked now

high stone
#

Well, I'll put a moderator hat on and say that any further discussions on the flake8 / PyCQA / pyproject.toml support etc should move to #off-topic. Most of the discussion here isn't something that's packaging specific.

heady herald
lyric hedge
#

Is there a small library or self-contained code snippet for parsing the [project].python-requires pyproject.toml field? Black would like to provide a smarter default for --target-version if PEP 621 metadata is available. https://github.com/psf/black/issues/3124

high stone
#

packaging's specifierset?

lyric hedge
#

Sounds about right. Not super keen on adding packaging as a dependency of Black, but I'd imagine it's the best option for parsing this field.

#

Thanks!

inland prawn
distant briar
#

lol, I was looking through my downloads folder and was wondering why would I have an invoice from PSF for 50$ :O And then remembered the org user testing lol...

lyric hedge
#

oh! what's the status on pypi organisations?

#

I forgot that a) they were being worked on, and b) it's a paid feature(?)

tidal kiln
#

I think free for oss projects, paid for companies

distant briar
#

Ye, that

#

User testing included both OSS and company orgs

distant briar
#

The features are being worked on and each round needed you to try to do different things (as they get implemented I imagine)

#

All links are there

#

Including minutes from user testing

lyric hedge
#

nice to see funding for such a requested feature 😄

blazing lantern
woven plover
indigo token
mortal shore
#

right, I meant that I don't think it's directly invoking setup.py vs going through the legacy build hook that flake8 is worried about, but that build isolation is either on or not based on that file

distant briar
#

Now that editable support is in, is there any situation when pip still needs to directly invoke setup.py over using legacy build hook?

#

Barring the build isolation

spiral urchin
#

That depends on whether setuptools does bdist_wheel/develop and PEP 517/610 differently. I believe there are not hard blockers if the two behave the same. (I don’t think they do though.)

fallen shuttle
spiral urchin
#

I meant 660

obtuse holly
mortal shore
#

eh

#

it has trade offs

#

like everything

noble matrix
#

The actual issue is that build isolation is keyed off the presence of that file, which contains a number of specified and unspecified contents, whereas the precense of the build-system table within that file.

mortal shore
#

well originally that file was really only intended for build tools 🙂 I don't think we were explicit about that though

#

and the thought was keying off the presence of that file would be more consistent

woven plover
#

Valid content of the section would have been a better key in retrospect

spiral urchin
#

The decision to use it to opt into build isolation came first, and everyone wants to put their tool configurations in that file afterward. So arguably it’s the people who use the file for non-build-tools are “wrong”

mortal shore
#

also which valid content

noble matrix
spiral urchin
#

It’s changed after everyone and their dogs started putting [tool.black] in the file

mortal shore
#

yea 🙂

#

like I said, originally it just said "tool", where the people mostly working on it mentally filled in the gap to be "build tool"

#

and then black came along and said "hey this is cool"

noble matrix
#

Yeah, should have checked the blame first then commented vs. the other way around, but yeah that was changed a little over 3 years ago. The original was less explicit, though it seems reasonable to not interpret it to be exclusive to build tools, since it wasn't actually specified anywhere

The [tool] table is where tools can have users specify configuration data

mortal shore
#

and we expanded on it afte the fact

#

yea I don't think anyone did anything wrong

noble matrix
#

The blessing and the curse of packaging PEPs being chronically underspecified, IMO

mortal shore
#

I just think the folks working on it had one thing in mind, and were insufficiently clear in the PEP, then other people interpreted it a different way, and eventually we just expanded it

#

I wrote like 8000 words about version numbers and that still wasn't enough to remove all the edge cases

noble matrix
#

Was the practice of tools using setup.cfg for configuration established by that point, or not? That would have been the main precedent I would have looked to at the time

mortal shore
#

I think there was a handful of things that could use it, like pytest and such

spiral urchin
#

And the unfortunate historical tendency of tool authors preferring to guess instead of asking for clarifications (it’s getting better these days)

noble matrix
#

Since like it or not, the community was likely to (and did) see pyproject.toml as that latter file's replacement

mortal shore
#

but most projects used their own format as their primary

#

I actually think it's kidn of nice to have it used for anything

noble matrix
#

To be fair, in a lot of cases the interpretation might seem obvious enough to some readers that they wouldn't think to question it

mortal shore
#

yea I don't think it was ambigious in a noticeable way tbh

noble matrix
#

Even @fallen shuttle interpreted PEP 621's spec of the license.file field very differently then intended and didn't realize till I pointed it out, and the only reason I knew was that I had the same misconception about it for like a year that I only got corrected talking to Brett about it despite re-reading the PEP many times and being the author of PEP 639 on licensing of all things

#

Honestly I'd be very surprised if we're the only ones

lusty quarry
golden falcon
#

what was the license field misconception that you had? Just out of curiosity

noble matrix
#

That license.file was the equivalent of license_file/license_files rather than license = file:

lusty quarry
#

is pep 639 accepted yet?

noble matrix
#

Actual examples showing the resulting core metadata output for the example input of each field in [project] would have gone a long way toward clarifying that and several other misconceptions people have had. I've learned the value for that firsthand in PEP 639. When I get the chance I'll propose a PR adding them

#

Not yet, I need to do one last round of changes splitting up the ancillary stuff into separate documents to make it way shorter (and making the terminology section into a glossary and linking terms there and to the PyPUG one, to reduce some confusion), and then do what is hopefully a final round of review in a new thread.

golden falcon
noble matrix
# golden falcon erm, so IIRC, the difference between these two is that `license = file:` goes in...

Yes, specifically the contents of the file specified by license.file gets injected in the License Core Metadata field, whereas the file specified by license_files (or the deprecated license_file ) gets dumped as files into the .dist-info directory (with several major caveats fixed). With PEP 639, the top level license will be a SPDX license ID/expression while license-files will basically be an enhanced license_files without its problems, and will also store the license file paths in core metadata (as Setuptools already does based on an earlier draft of the PEP)

#

Of course, PEP 621 doesn't actually say any of that and leaves it essentially completely implict

golden falcon
#

then why is license.file mutually exclusive with license.text?

noble matrix
#

Because they both get dumped in the same (to be deprecated) core metadata field, License

#

It wasn't so obvious though initially, since a lot of other packaging systems only allow specifying a license file or license text for other reasons

golden falcon
#

It makes sense if you believe that the License field from CM should contain the full text of the license but CM provides no such direction

mortal shore
#

the License field can't actually contain the full text

#

withuot relying on non standard behavior

#

at least, it can't in the general case

indigo token
#

I’m completely lost and just gave up on specifying licensing metadata correctly for the time being. Is there some overview document that tells me

  • which combinations of fields are recommended
  • which ones are deprecated
  • and what exactly they all do
lusty quarry
#

just use Hatchling that supports pep 639

#

license = "MIT"

blazing lantern
indigo token
#

Ah, I thought that was already accepted. OK cool so assuming this section won't see any changes: https://peps.python.org/pep-0639/#project-source-metadata

I can use hatch right now and specify

  • license = 'GPL-3.0-or-later' for my own, license = 'LicenseRef-Proprietary' for my company's projects
  • Don't specify license-files.globs as it defaults to a list that captures LICENSE

The to-be-deprecated behavior is to specify a license classifier in classifiers or to specify a table as value for license.

Did i get that right?

PEP 639 – Improving License Clarity with Better Package Metadata | ...

Python Enhancement Proposals (PEPs)

lusty quarry
#

yup!

noble matrix
# mortal shore at least, it can't in the general case

Oh, because of limitations in the RFC *822 file format, because of the semantic "Text indicating the license covering the distribution where the license is not a selection from the “License” Trove classifiers," or something else? How does that intersect with license.file, then? Would be very interested to know more.

noble matrix
# indigo token I’m completely lost and just gave up on specifying licensing metadata correctly ...

For PEP 639, I have https://peps.python.org/pep-0639/#appendix-user-scenarios (which will be moved to its own document, and hopefully to a PyPUG Guide or similar if/when the PEP is accepted giving) giving a high-level overview and https://peps.python.org/pep-0639/#appendix-examples which gives both basic and complete examples. For the legacy approach, my current guidance is just "Use license classifiers if it precisely specifies your license (see https://peps.python.org/pep-0639/#mapping-license-classifiers-to-spdx-identifiers for details, and use a SPDX identifier/expression in license.text"

You've summed it up right, and avoiding license-files if you don't need it (as the specified default globs should capture most first-party license cases; the ones I specified in the PEP are the same ones I added to conda and wheel some time ago, which spread to Setuptools and elsewhere) will insure you against any changes there. license seems pretty unlikely to change at this point since there was clear consensus on the current approach; after careful consideration I am solid on the current design for license-files and there haven't been any major objections, but its not inconceivable Brett or others could want a change, e.g. going with just a top-level key and either the paths semantics https://peps.python.org/pep-0639/#only-accept-verbatim-paths, the globs semantics https://peps.python.org/pep-0639/#only-accept-glob-patterns or some (un)happy medium https://peps.python.org/pep-0639/#infer-whether-paths-or-globs I don't expect the default behavior to change, though, since it is what most if not almost all current tools do now, so it should be safe to rely upon.

PEP 639 – Improving License Clarity with Better Package Metadata | ...

Python Enhancement Proposals (PEPs)

mortal shore
#

you can sort of fake it by using the RFCs case folding

#

but if you unfold the case folding, then you lose the new lines completely

lyric hedge
#

For those who work on packaging, would it be fine to update Specifier.filter and SpecifierSet.filter's return type annotation to use Iterator instead of Iterable? It's causing mypy to think next isn't supported on the return result of those methods as the Iterable protocol only requires __iter__.

#

I was about to submit a PR, but I thought "hmm, maybe this type annotation is purposefully lax so the API (specifically what it returns) can be changed later on." 😄

#

Hmm, actually it's probably better to not change the return type annotation, I don't want to make changes to packaging even more annoying :)

#

wrapping the method in a iter(...) call isn't that annoying

high stone
#

The return type of the filter builtin is Iterator, so it's likely sensible to do that. Wanna file a PR?

torpid veldt
lyric hedge
#

I'm not sure how that's any better than Iterator

torpid veldt
#

Also a bit of an odd interface

#

Maybe you want a Callable[[Version | str], bool]

#

Then people pass it to whatever filter they want

lyric hedge
#

what...? why would you want a callable anywhere in Specifier.filter ?

#

It's filtering an iterable of versions (either str or packaging.versions.Version) using the specifier value of the instance, currently returning an iterable

spiral urchin
#

Matching return type of filter makes sense. Side note, this makes me kind of want to be able to write filter(specs.matcher(prereleases), versions) but it’s probably me being weird

torpid veldt
#

People would use it with:

[x for x in items if specifier.specified(x)]

mortal shore
#

how is that different than __contains__

#

[x for x in items if x in specifier]

torpid veldt
#

wait does that actually work? @mortal shore ?

mortal shore
#

it should

#

there's also .contains()

#

.filter() is smarter than contains though

# 
This method is smarter than just ``filter(Specifier().contains, [...])``
        because it implements the rule from :pep:`440` that a prerelease item
        SHOULD be accepted if no other versions match the given specifier.
torpid veldt
#

Maybe just pick the most efficient implementation and expose that as the only implementation

mortal shore
#

well it depends on what question you're asking exactly

spiral urchin
#

Both are awkward with prereleases=True (functools.partial is cumbersome)

fallen shuttle
#

I was going through PEP 517 again tying to understand the metadata_directory for prepare_metadata_for_build_wheel...

Does anyone know if there is any guarantees about metadata_directory not containing any previous files at the prepare_metadata_for_build_wheel stage? Is it valid to pass metadata_directory="."?

I can see that the backend can add other files to this directory, but there seems to be no restriction about it being pre-populated before the first time the backend is presented to it.

tidal kiln
#

metadata_directory must be a directory generated by the prepare hook

#

the goal is to allow people to look at the metadata without building

fallen shuttle
#

In my mind this should be simply an empty directory (possible temporary), which then would be used by the prepare hook

#

But, this does not seem to be guaranteed by the PEP... so apparently it could be any directory?

tidal kiln
#

ah sorry, I misread

#

yes it can be any directory

#

there are no guarantees there

fallen shuttle
#

Ok, thank you very much for the clarification

tidal kiln
#

IMO the build backend should just do its own thing and let errors raise in case of conflicts etc

#

but in practice, I think practically everyone is supplying an empty directory

#

otherwise you risk issues 😅

fallen shuttle
#

It seems that . is also being used, which may contain .tox/.venv

#

Well, time to revert some refactoring I did last time

tidal kiln
#

hum, interesting

#

I don't think you should use that directory for anything other than actually writing out the metadata

fallen shuttle
#

Yeah, it is more about a check that I changed, although the PEP does seem to allow other files to be written there (that are not metadata, e.g. could be some sort of cache)

#

Anyway, thank you very much Filipe

spiral urchin
#

Yes the PEP only says the backend must put metadata files there, as long as you do that, other files are not forbidden

#

Actually IIRC there were discussions to use that as a cache directory to make rebuilds faster. The discussion didn’t go anywhere but at least arbitrary files are not unexpected

soft glen
spiral urchin
blazing lantern
# lyric hedge For those who work on packaging, would it be fine to update `Specifier.filter` a...

Changing it to Iterator on its own isn't accurate thanks to https://github.com/pypa/packaging/blob/main/packaging/specifiers.py#L915 which returns a list which itself is not an iterator. So to switch to Iterator will also require updating other code to return an iterator directly (which I'm personally not opposed to, just wanted to point out it's not a simple find-and-replace change)

GitHub
packaging/specifiers.py at main ¡ pypa/packaging

Core utilities for Python packages. Contribute to pypa/packaging development by creating an account on GitHub.

lyric hedge
#

I already noticed, I haven't filed the PR though ¯_(ツ)_/¯

#

thanks for pointing it either way! I reworded my PR description to note this just so we are making more informed decisions

jovial venture
elder kettle
vernal plover
drowsy basalt
#

What is the correct way to do codegen in the context of modern python packaging? I've tried searching the internets but nothing came up, though I might be using the wrong keywords.

By codegen I mean have data files and compile them to python files during build / packaging, so that on user systems the project can delegate the loading to python rather than need to parse the data files then convert that to python datastructures.

The impression I have from PEP 517 is that I should develop a local / custom build backend, do my codegen there, then delegate back to some other build backend, but I don't know that that's the right solution. Also are there frontends/backends to avoid for such a task (aka which just don't support it)?

And finally, old-style packaging (setup.py) allowed custom commands, is that still supported or should that be moved to external scripts / utilities / ...?

spiral urchin
#

develop a local / custom build backend, do my codegen there, then delegate back to some other build backend
Yes
Also are there frontends/backends to avoid for such a task (aka which just don't support it)
The project-building API is entirely Python, so all backends should have at least some support for being called as a Python module (otherwise it can’t work as a backend. All frontends should also support this since frontends treat all backends the same
old-style packaging (setup.py) allowed custom commands
I believe you can still write those if you want to, assuming you use setuptools as your build backend. But yes generally external scripts are more preferred

drowsy basalt
green canopy
drowsy basalt
# spiral urchin > develop a local / custom build backend, do my codegen there, then delegate bac...

By the way is there a known pattern / convention for making the delegation dynamic? I assume a local backend would just be hardcoded against the project's selected "real" build backend but how should a tool like cython or similar handle delegating after it's done its preprocessing? I assume it would be using a key in its own tools section but is there a customary name for the name of that key? And are there standard APIs for discovering backends the way frontends handle build-backend? Pep517HookCaller?

spiral urchin
drowsy basalt
# spiral urchin Since the PEP 517 API is just Python, you can just call the Python functions dir...

Yah but I meant if I create / provide a backend which only does codegen, it'll need to delegate to a "real" backend for the actual building, but I probably don't want to impose a backend on downstream users.

This means my codegen backend needs a way to configure and then call the real backend (or even chain into a second codegen / preprocessing backend).

So I was wondering if there was already a customary way to do that, or even direct support in foundational libraries.

spiral urchin
# drowsy basalt Yah but I meant if I create / provide a backend which *only* does codegen, it'll...

No there’s not an established way for that, as far as I know, so configuring it under your own tool section is probably the most straightforward way.
There’s a discussion on providing a cleaner way to hook directly into the build tool though (the most obvious use case is for C modules but essentially it covers any kind of “compiling” including codegen) but there’s not much work done on it yet (let me find the link to that)

drowsy basalt
#

Ok cool! Thanks a bunch for the helpful answers.

spiral urchin
keen rover
keen rover
#

general

blazing lantern
drowsy basalt
woven plover
blazing lantern
lusty quarry
drowsy basalt
strange knot
#

In case you missed it, PyPI was subject to a phishing attack this morning: https://twitter.com/pypi/status/1562442188285308929

Python Package Index (@pypi)

Today we received reports of a phishing campaign targeting PyPI users. This is the first known phishing attack against PyPI.

We’re publishing the details here to raise awareness of what is likely an ongoing threat.

Twitter ¡
lyric hedge
#

d'oh that's unfortunate :(

wicked canyon
distant briar
#

2fa is already forced on top 1% most downloaded packages

#

hide personal emails
emails can only be exposed by a package distribution so the project author can simply not include the email in package authors

#

provide a forwarding proxy email (same way github does it)
GitHub doesn't forward emails from their proxy emails

#

or however you want to call them

#

detect those packages and don't allow them to be posted in the first place
that's a noble goal but how do you propose this should get done? and someone has to work on this so another question is how do you think funding could be secured for this?

#

I know what you meant

#

as I said, it doesn't forward emails sent to that email

#

either way, it doesn't really seem like an issue if you can just not put your email if you don't want to be contacted about the package

#

but the thing is, the maintainers often want to be contactable

#

then they're signing up for it

#

I actually have no idea what you're asking to be done. How would you detect these packages?

#
  • force 2fa on packages that over 1000 downloads or packages that other packages depend on
    That I somewhat agree with I guess? There's been a bunch of people discussing why 2fa wasn't just forced for everyone rather than just top 1% back when that happened on Twitter I think. I guess this would be sort of different since it wouldn't increase the barrier to making your first package (but you would very soon need to enable 2fa anyway so the barrier is still probably there as soon as you try publishing a second version).
  • require other maintainers approval before posting new release (same way github does it)
    This might be something that will be doable with draft release functionality? Not sure, you might want to look for that.
  • hide personal emails and provide a forwarding proxy email (same way github does it)
    I don't think making whole email forwarding thing is something that's really PyPI's job. And the only source of personal emails are the packages themselves so this is part of the distribution, it's not something that PyPI can just hide, it's literally part of each distribution if the project's maintainer added their email there. The email you have set up on your PyPI account is not publicly available on its own if you don't publish it yourself.
  • detect those packages and don't allow them to be posted in the first place
    already talked about it above, not really anything else to add
  • detect those attacks (typos names, scraping of users emails, etc...) and block them along side with ip's of attackers (this won't stop them, but if you can't beat them slow them)
    Scraping of users emails doesn't seem very detectable. Detecting typo names limits the available name choices to real packages a lot, I don't think it would work great unless you combine it with something else that would suggest that the package may be malicious.
#

​

Detecting typo names limits the available name choices to real packages a lot
Case somewhat in point, there's PyPI, there's PyPy, there's mypy (probably something else I'm forgetting, Python community sure seems to love these kind of names :)) which you could say are typo names

#

lol

spiral urchin
#

btw publishing my new package mypi

distant briar
#

unless you want to file a request for removal

#

but it looks like a real package, not a squat

#

let's put up their 2fa to a test

#

(no)

spiral urchin
#

published in 2014 so technically mypy is the typo package!

distant briar
#

damn

spiral urchin
#

Oh wait Mypy actually has a previous life that goes back to 2009

distant briar
#

was it name squat

spiral urchin
#

No it got several releases, pretty legit to me

distant briar
#

huh, a wsgi framework

#

it's part of the project endpoint

#
  • you can just download the project distribution itself and get the same info
#

you can also just google for a pypi mirror and find one that doesn't implement any of this special handling

noble matrix
#

<@&815744082823348274> what about a PyPI channel?

mortal shore
#

what's the difference between a pypi channel and the warehouse channel

#

maybe warehouse channel should be renamed pypi

noble matrix
#

Not so smart people like me might actually find it, which has both pros and cons 😆

strange knot
#

I'd be in favor of renaming it

high stone
#

OK, #warehouse has been renamed to #pypi

lusty quarry
inland prawn
#

CMakeLists.txt is compilation instruction. My guess is that with ninja and cmake in [build-system] it compiles automagically

distant briar
#

yeah, they're using scikit-build

#

which wraps setuptools but does all the cmake build stuff

inland prawn
#

btw aside from using setuptools, is there any project that is gonna replace distutils for python 3.12+?

#

also, why is there no installer channel?

lusty quarry
#

oh totally missed pyproject.toml, thank you both

blazing lantern
blazing lantern
inland prawn
noble matrix
lusty quarry
#

set of utils helpful in for example building C Extensions etc

#extensionlib eventually

woven plover
#

Im wondering, what would be a good way to force in overrides for pyproject.toml settings on ci/other builds

Context is adding override tools to setuptools_scm/it's successor to ease stuff like test pypi uploads (which fail on dev versions with local tags) as well as easing builds for downstreams

Cc @lusty quarry @fallen shuttle

#

While we are at it, it would be nice to have a common standard for entrypoints of dynamic Metadata providers,

noble matrix
# woven plover Im wondering, what would be a good way to force in overrides for pyproject.toml ...

I'm not a tool expert like those two and maybe I'm misunderstanding the question, but from a standards perspective, the possible answers varies depending on the table (build-system, project and tool) you're interested in. TL;DR: There are other ways to dynamically override some settings in some tables downstream of pyproject.toml, but AFAIK the only general solution is to either have the CI/etc modify the pyproject.toml programmatically, or generate the pyproject.toml with another tool/backend at or before build time (which I believe exist). More specifically:

For [project], anything not explicitly listed as dynamic cannot be modified anywhere downstream of pyproject.toml, so in that case, the CI must modify the pyproject.toml directly (or have it be generated pre-build time). For dynamic settings, that depends entirely on the backend.

For [build-system], requires can be added to (but not, AFAIK, subtracted) by the get_requires_for_build_* PEP 517 hooks of the backend, which some backends allow you to control directly (e.g. setup_requires with Setuptools, which can be dynamic in a setup.py). I'm not aware of a way to modify the other items without just editing the file.

For [tool], it entirely depends on the tool involved, other than editing pyproject.toml directly.

woven plover
noble matrix
woven plover
lusty quarry
#

no need for agreement between tools. just have a plugin per thing that you care about that wraps the tool you maintain

woven plover
#

Btw, is there any recommendations for renaming / superseding a pypa package?

lusty quarry
#

I'd just emit warning on import

woven plover
#

There will be new pypi names, deprecations , backwards compat concerns, so load of fun

lusty quarry
#

json/toml in a env var

it'd be nicer for ppl if your new tool supported various options as env vars rather than one that is a file dump, if you have time

woven plover
#

But right now im not going to invent one

lusty quarry
#

I mean isn't it just foo = os.environ.get('PREFIX_FOO', options.get('foo', 'default'))?

woven plover
#

There is multiple possible prefixes, handling types and validation, im not going to write that out or invent a good automation for it right now now, the rabbit hole doesn't fit my time constraints

lyric hedge
inland prawn
lyric hedge
#

I'll defer to @high stone for whether an installer channel makes sense, but that sounds reasonable.

#

I didn't know installer has become a PyPA project, TIL.

high stone
#

Heh.

inland prawn
#

❤️

high stone
#

Merged it. Usually @-mention me on the PR, on a weekend?

inland prawn
#

thanks @high stone

high stone
#

Or Fridays.

inland prawn
high stone
#

Stuff slips, if I see it during the week day, but I usually do end up looking at my GH notifications some time during the weekend.

lyric hedge
#

That reminds me I really need to clean up my github notifications

#

perhaps I should just mark all as done though. I used to save certain notifications for things that needed attention later on, but I do not have the time or energy to deal with that many things.

lusty quarry
#

I open tabs then cull periodically

queen hornet
kind pasture
lean lake
#

Anyone been able to tune dependabot's emailing? I get an email for every rebase etc. - Feel that's a little spamming. If so, got examples? Is it config driven or GUI clicking if so?

distant briar
#

have you considered switching to something else :)

#

renovatebot maybe

#

or something based on GH Actions with cron

lean lake
#

For the most part it's fine and I get it weekly. Just it's notifications are spammy and quick google wasn't fruitful so took lazy approach

distant briar
#

You could also just disable automatic rebases in Dependabot

lean lake
#

That might be a way - good call

distant briar
#

because dependabot PRs are just standard PRs so you get same notifications as with any other PR

#

alternative is setting email filter for "@Dependabot pushed 1 commit" :)

drowsy basalt
fallen shuttle
fallen shuttle
woven plover
fallen shuttle
#

In that case, maybe what you seem to be looking for is indeed an interop PEP. I am again very happy to contribute on that, but I won't be able to lead the effort.

#

In https://github.com/pypa/setuptools/issues/3415 I expressed some ideas on what I think could be useful. This could be one way of doing it, but since the response was not very enthusiastic, I assume it is not what tool maintainers are looking for.

We probably need some ping-pong between what the tools are looking for and what the backends can provide. Based on the previous feedback, I think the best action for setuptools now is to take a step back and ask what the tool maintainers are looking for.

Then we can check if an implementation is viable, and if not, propose an alternative.

woven plover
lusty quarry
#

I don't have time to think about ^ for a while. too much work & Hatch stuff on my plate

noble garden
mild dew
low drift
high stone
kind pasture
valid sorrel
cobalt mason
lyric hedge
final pier
oak depot
#

Is there a specification of the minimal set of names/attributes/things that are supported in pyproject.toml? In particular, none of the tutorials that I've found mention how to define the 'long description' of a package. (If this is the wrong place for this question, a pointer to the right one would be appreciated.)

lyric hedge
oak depot
#

Thanks!

mortal shore
#

you probably want that ^

lyric hedge
#

is that page new(-ish)?

#

to be honest, I've seen enough talk about how the PyPUG needs a lot of work and is pretty outdated specifications-wise. So I haven't looked at the PyPUG in a while, assuming it wouldn't be particularly useful.

mortal shore
#

not sure how new it is

lyric hedge
#

I don't mean to discredit the recent work put into the PyPUG, just my perspective on the optics

mortal shore
#

I think people have been trying to get the PEPs all into packaging.p.o

lyric hedge
#

that's good 👍

distant briar
#

Whenever someone asks me, I always link to packaging.python.org over whatever PEP introduced the thing when possible. packaging.python.org is great for the most part, it's just hard to find because it just doesn't show up high in Google (or at all)

#

There's a bunch of documents in packaging.python.org that just link to the PEPs rather than describe it directly on the page and there may be some things that aren't there at all but still, most of the things that less experienced people are looking for already are available in packaging.python.org

calm marten
lyric hedge
#

welcome!

calm marten
#

Good day famil

calm marten
#

I want to be a contritor, just getting 🌟 with open source development

lyric hedge
#

what interests you packaging-wise?

rocky mauve
inland prawn
#

Is having dev dependencies (linters or packages for docs building) defined as extras a bad practise or just a code smell? I see it in packages sometimes, is there any official information about it?

spiral urchin
#

I don’t think there’s any “official” recommendations (or any recommendations really), but it’s not best practice IMO. Dev dependencies should generally be pinned down (==) for reproducibility and extras are not suitable for the job

distant briar
#

test and doc extras specifically actually have a defined meaning in core metadata spec

#

whether it's a good practice is up for debate I guess, I think it's alright

#

if it's a dedicated extra then pining in it isn't really bad

spiral urchin
#

Yeah the problem is not that using them is bad (it’s not), but more about it’s difficult to maintain them (compared to say lock file or pre-commit)

#

Maybe bad practice is too strong; not optimal, more like

manic belfry
#

I usually declare them in extras and pin them in separate requirement files (requirements-test.txt, requirements-doc.txt). For linters I tend to use pre-commit which has its own pinning mechanism.

mossy pulsar
#

I never use requirement files for development dependencies or pin them in extras and in 12 years of using python maybe had 3 times issues because of it, when excluding the broken release fixed the issue 🤷‍♂️ so I don't know from where @spiral urchin statement comes from, especially considering Python does not have an official lock file.

#

IMHO pinning is overrated 😅 as long as you run your CI before you cut a release and you have an exhaustive test suite there's no real need for it 🤷‍♂️ IMHO.

#

The only place I pin is linters and type checkers 👌

#

So I think we can agree that it's not a code smell in either way, if you have pinned dependencies, or if you use requirement files or extras of not 🚫👍 do what works for your pipeline

woven plover
#

I wish the lock file pep had more adoption

For me pins primarily come as constraint files to ensure many people use the same versions

spiral urchin
#

One thing I lament is people have become too unimaginative about locking 🙂 Lock files come in many forms, npm/cargo/etc. style is but one way to approach it. You don’t need to make that one format fit all use cases, and locking is usually still beneficial for a use case if you look past that particular format.

mossy pulsar
#

Dunno locking by design pins you in to run older versions of packages all the time which for OSS is bad for security. And having to deal with update lock file ever so often in general and my experience is more hassle than benefit 🤷‍♂️

lusty quarry
#

I wish the lock file pep had more adoption
me too 😢 after locking Hatch would be close to feature-complete

mossy pulsar
#

The pep was never accepted so would be pointless to adopt it 🤔

spiral urchin
#

I mean the fact __pypackages__ not being accepted didn’t stop people

#

Not that I recommend this personally

lusty quarry
lyric hedge
#

I'm surprised someone (almost) without any Python experience is tackling this, but my goodness has it been too long. I'm happy to something is happening :)

merry rune
#

I'm curious; I'm looking for a new home for scikit-hep/cookie, scikit-hep/repo-review, and the Scikit-HEP developer pages. They've become more general than Scikit-HEP with it's 11 backends, etc. I'm thinking I could merge them into a single repo (or maybe 2). Would that be something useful to propose to the PyPA? PyPA/quickstart or something? Otherwise I'll be targeting scientific-python project, which is fine. Just thought I'd see if there's interest here, or if people think they are too opinionated or specific, etc.

golden falcon
#

These overlap with the PUG and the sampleproject repo to some extent and I wouldn't want the scikit-hep pages to be "sullied" by the kind of pushback we've seen on modernising their PyPA equivalents. If you think you can maintain "editorial control" in the PyPA then why not, but I think people will start worrying about how these might or might not represent PyPA's position on various (rather dull) matters that the scientific community wouldn't have to concern itself with

civic compass
merry rune
#

Agreed, I'd not want it on packaging.python.org or some other "official" location, but "quickstart.pypa.io" or something similar doesn't seem too bad. It supports 11 backends, including all PyPA builders. But happy to continue toward putting it in scientific-python, just wanted to judge interest.

indigo token
queen hornet
#

although there's an argument for pinning and using Renovate (or dependabot) to create update PRs specifically for dep changes, then you can see failures in isolation

nimble rain
lusty quarry
distant briar
#

lol

tidal kiln
#

kind of a nitpick, but the pypa org on github has the location set to USA, wouldn't it make more sense to remove it?

nimble rain
west drift
#

basically, i install every package once globally and then transparently hook them into the project using a path finder that reads pyproject.toml/poetry.lock or requirements.txt (doing dependency resolution through vendored poetry if required)

#

Usage is monotrail run python <normal python args>. You can use -p 3.x to select a python version, it will install the version you requested through python-build-standalone

#

I've also made an interactive mode for jupyter notebooks (monotrail.interactive(numpy="^1.21", pandas="^1")), a pipx clone (monotrail ppipx --extras jupyter black .) and a tox clone (monotrail run -p 3.8 -p 3.9 -p 3.10 command pytest). There's either a single standalone binary to download or a pip installable package, depending on how you want to run it.

#

Incidentally, i've also written a wheel installer that's faster than pip's singlethreaded (e.g. 5.8s vs. 7.6s for plotly on my laptop) and can additionally be multithreaded, if anybody is interested in reusing that

#

The idea is that you have one binary and can just run any project without having to use any environment management, not even installing python yourself

inland prawn
#

Or you can use PEP582 compliant package manager like PDM

west drift
#

as far as i understood pep582 still installs every wheel again for every project and python and you still need to manually control the content of __pypackages__, right?

inland prawn
#

Well, yes. Global packages make no sense since you can have one project use package X version 1.2.3 and another project using the same package in version 2.5

spiral urchin
#

install every package once globally
What does globally mean in this context? Are they installed to the global site-packages?

west drift
#

no on the contrary; it's installed to .cache/monotrail, each wheel in a separate folder

#

the path finder has the resolved set of packages for your project incl selected extras and makes only those packages importable

tidal kiln
#

I have thought about doing the same thing

strange knot
tidal kiln
#

anyone got a source for PyPy wheel of several versions?

spiral urchin
tidal kiln
#

yes that's what I have been using but only has older wheels AFAICT

#

most recent is Python 3.6

mossy pulsar
# 
>>> Version('4.0.0b2.dev1') >= Version('4.0.0b2')
False
>>> Version('4.0.0b2.dev1') <= Version('4.0.0b2')
True
#

why is this? 😄

#

isn't .dev1 one step ahead of the base version? 😄

mortal shore
#

that's the first development release of the second beta of version 4.0

#

not sure why it would go the other way?

mossy pulsar
#

because generally you don't keep releasing dev versions 😄 and then one day you release the non dev variant

#

and it's how setuptools-scm also works 🙂 it adds dev versions on top of the last release

#

so if you tag a commit with 4.0.0b2 and do a release and then add a commit, the version will be 4.0.0b2.dev1 so in theory 4.0.0b2.dev1 is after 4.0.0b2

#

so the version comp should evaluate as such? cc @woven plover

mortal shore
#

that sounds like setuptools-scm is producing illogical version numbers

#

the above comparisons are as expected with PEP 440

mossy pulsar
#

duno, makes sense to me 😂

mortal shore
#

it works the same as any other pre release tag

#

4.0.0b2 is the second beta release of version 4.0.0

#

4.0.0.dev1 is the first dev release of version 4.0.0

mossy pulsar
#

that makes sense

mortal shore
#

the only thing that's really different about it is you can make a dev release of another pre-release

#

which is kind of weird and esoteric TBH

mossy pulsar
#

but if you combine those:
4.0.0b2 is the first release of beta 2 (implied zero for dev)
4.0.0b2.dev1 is the second release of beta 2, no?

#

be that esotheric feels like should compare the other way 🤔

mortal shore
#

that would require special casing the order of dev releases when used in conjunction with other pre-release tags, which I don't recall anyone bringing that up during PEP 440 discussions TBH, so I won't say that we really thought about doing that specific thing in that specific combination other than PEP 440 explicitly calls it out as something to be avoided in most cases becuse it creates versions numbers that are "hard for a human to parse"

#

We probably would have still done it the way we did because I don't think tht particular edge case is worth making a special case for, and one of the goals of PEP 440 was to minimize differences in how versions parsed/sorted from the existing pkg_resources code, and pkg_resources sorted it that way too

#

but maybe?

#

in either case, that's as specified in PEP 440

mossy pulsar
#
  1. latest tag (with a version number)
  2. the distance to this tag (e.g. number of revisions since latest tag)
  3. workdir state (e.g. uncommitted changes since latest tag)
mortal shore
#

in abstract the mechanisms that setuptools-scm is using make sense

mossy pulsar
#

this generates version that are reasonably simple to read (at least for me) '4.0.0b2.dev27+g6a097123.d20220910 so that's 27 commits past the 4.0.0b2 release and has a marker for dirty dir

mortal shore
#

they're just not producing a logically ordered version version using PEP 440 versions

mossy pulsar
#

yeah 😄 hence me getting surprised here 😄 and wondered why would Version('4.0.0b2.dev1') >= Version('4.0.0b2') evaluate to False, but I see what you mean

woven plover
#

@mossy pulsar setuptools_scm adds 1 to the last number by default, so the dev version is for the next version

foo.dev comes before foo

woven plover
#

So increment(tag).dev ensures that we get a bigger version

proven quartz
high stone
#

@woven plover on your question in a now-deleted channel, about having packaging versions created without parsing: please file a new issue on the packaging repo?

There's currently no way to do what you want to do.

woven plover
#

thanks

golden falcon
alpine tendon
#

Does anyone know of a software distro that ships venvs instead of packages for applications?

#

With more and more things assuming venvs (narrow version ranges, lots of major bumps) it's becoming hard to package apps in distros with 300+ packages while keeping pip check happy. So I'm wondering if anyone has found a way to handle that.

#

kinda like an anti-debian approach if you will

golden falcon
#

doesn't homebrew do something like that?

distant briar
#

Nix should keep applications isolated but it probably doesn't use venvs for this

golden falcon
#

nix doesn't use venvs, no

#

homebrew example

spiral urchin
#

Homebrew does not ship venvs but creates a venv for each application (if the formula specifies it)

distant briar
#

the goal is probably isolation so I figured it's probably worth mentioning it anyway

spiral urchin
#

It’s probably the closest you can get to shipping venvs

alpine tendon
#

with venv I mainly meant isolated/vendored deps yeah

#

thanks, I'll have a look at homebrew

woven plover
merry rune
modest anvil
sick crescent
lusty quarry
distant briar
#

Huh

#

until PEP 639 revamps license fields in package metadata, I guess I'll just depend on the classifier

lusty quarry
distant briar
#

Hmm, I was trying to figure out what things need to be done before packaging 22.0 could be released. I see that it adds packaging.metadata but there's a WIP overhaul of that in https://github.com/pypa/packaging/pull/574 which I guess 22.0 may be somewhat dependent on (either by waiting for #574 or by dropping metadata support in 22.0 and delaying it until #574 is done)? Since you probably wouldn't want to release packaging with metadata support in an unfinished state.
Looking through the repository, switching to flit and adding some release automation may be some other things that you would want to be done before cutting a release?

distant briar
green shadow
nimble rain
blazing lantern
# lusty quarry > I wish the lock file pep had more adoption me too 😢 after locking Hatch would...

FYI I have not given up on solving this problem. Work on it for me is currently blocked on https://github.com/pypa/packaging/issues/570 which is blocked on https://github.com/pypa/packaging/issues/570 (ping @mortal shore ).

GitHub
Plans for `packaging.metadata` ¡ Issue #570 ¡ pypa/packaging

Implement a from_email_headers() class method that does strict parsing for wheels 1 Implement a to_email_headers() method for wheels 2 Evaluate where from_email_headers() could potentially be tweak...

lusty quarry
blazing lantern
lusty quarry
#

oh I thought another format would be proposed

blazing lantern
# lusty quarry oh I thought another format would be proposed

If by "another format" you mean "something other than what PEP 665 proposed" then it will be different (or at least tweaked). Going to start opinionated and see where that takes things by making it all very concrete. I have an idea on the "sdist locking problem", but I want to get wheels working first since it's easier

lusty quarry
#

I'd strongly recommend the next PEP focus heavily on use cases, then detail the format

blazing lantern
#

That's the plan. I am going to lean hard on my use-cases and then see where I get pushback to control scope creep

lusty quarry
civic compass
spiral urchin
hearty girder
#

When using hatch publish with an API token, hatch attempts to use the same cached token for all projects, because the repository user is always _token_. Is there a way to cache API token per-project?

woven plover
inland prawn
distant briar
#

Either way, just enabled it on 10 more projects 😄

warm isle
#

I wonder if there's a library capable of understanding requirements files format, or is this something I should go ahead and reinvent?

mossy pulsar
#

tox 4 has some logic for this, inspired by what pop does

#

Looked at a few libs but neither were good enough

#

So perhaps check what tox 4 has 🤷‍♂️

#

Could be extracted to a lib

distant briar
#

Depends on what you need this for, if this is just for your own use then simply parsing each line with packaging.requirements.Requirement after ignoring lines with sth like if not line.strip() or line.strip().startswith("#") could be enough for you

wheat crag
#

Poetry has a very exhaustive implementation

#

But it's an implementation detail of Poetry and not really designed for external use (it's specifically structured for our usage)

inland prawn
queen hornet
warm isle
strange knot
#

Since when can you enable 2fa

lusty quarry
spiral urchin
#

Libraries do not need lock files is true in its purest form: distributing a lock file with a library is always wrong, but it’s very easy for people to interpret library to mean development of a library and talk about dev dependencies—and they are also correct to an extent, since there is definitely benefit to be able to lock certain development tools such as Black and Mypy for consistent results. This is probably the worst thing about discussions mentioning the lib-app distinction: there’s always people in a discussion messing up what those actually mean and the whole conversation gets derailed quickly.

blazing lantern
#

It's also messy when considering formatters and linters with e.g. pytest. Formatters shouldn't be changing much, so getting bugfixes and faster perf is nice when you don't pin. Linters can lead to new lints, but maybe you want that? But pytest is an API for most and so you should be specifying at least a minimum version. Basically there's reasons for pinning and not pinning when doing library development and neither side is more right than the other. This is why VS Code tries to support both approaches, although admittedly our historical stance of pushing people to install per environment has led to consistent push back (people really don't like having multiple copies of the same tool installed), and thus we are changing our position.

spiral urchin
#

our historical stance of pushing people to install per environment has led to consistent push back
It’s kind of interesting though that pre-commit has been gaining popularity recently, which installs all those dev tools in separate environments behind the scenes. Or maybe those people pushing back also refuse to use pre-commit as well? I have no idea.

mystic latch
obsidian cave
blazing echo
obsidian cave
obsidian cave
obsidian cave
uncut vine
severe finch
mellow sequoia
obtuse holly
#

how is it possible to deploy on rtd, hide behind a cdn, and have rtd redirect to the custom domain?

#

or does rtd use fastly as their cdn?

#

(also not sure why it says fast.ly, they're branded as fastly and don't own the fast.ly domain)

wanton raptor
unborn heart
wanton raptor
wanton raptor
sinful bison
#

Hi all! Anyone know if it's possible to use musllinux as an environment marker? I've been trying to pip install checkov under alpine, but fails since it depends on pyston and pyston-autoload for certain platforms, and wheels have only been built for manylinux. https://github.com/bridgecrewio/checkov/blob/master/setup.py#L72-L73

GitHub
checkov/setup.py at master ¡ bridgecrewio/checkov

Prevent cloud misconfigurations and find vulnerabilities during build-time in infrastructure as code, container images and open source packages with Checkov by Bridgecrew. - checkov/setup.py at mas...

spiral urchin
sinful bison
blazing lantern
blazing lantern
obtuse holly
#

interesting

bitter mantle
versed lily
spiral urchin
blissful grove
#

Question that should probably be obvious, but I can't find it mentioned anywhere: in a METADATA file, are the keys (Requires-Dist etc.) case-sensitive, or case-insensitive?

mortal shore
#

rfc 822 says case insensitive

scenic abyss
wanton raptor
pure tartan
wanton raptor
sick crescent
#

Hello everyone I m still working on the

distant briar
#

Dependabot now supports updates to Python dependencies for pyproject.toml files that follow the PEP 621 standard for our supported Python package managers.

merry rune
#

I hope this also means GitHub dependency graph starts working with PEP 621 packages?

uneven karma
tidal kiln
#

so it really only works for dependabot 😕

inland prawn
#

I don’t really get why people need „used by” on GH…

spiral urchin
#

Vanity is a very important motivation

sleek haven
queen hornet
queen hornet
#

back up and running now 🎉

coarse mason
lime ocean
minor rune
lavish night
fair surge
coarse iris
hallow echo
flint phoenix
echo ruin
#

@glass sand changes to zipfile have to go to zipp first before going to CPython?

#

I assume a PR which adds some missing pathlib methods to zipfile.Path is uncontroversial hopefully right? Should/can I send that there then and then it'll make its way downstream to CPython?

glass sand
#

@echo ruin The don't necessarily have to go to zipp first, but that's preferable because the workflow is simpler in that direction.

#

It's also beneficial to release in zipp first, get some feedback from the public, fix any unexpected bugs, then sync into CPython.

#

And yes, I don't expect any controversy in adding pathlib methods.

echo ruin
#

Great thanks, PR incoming in an hourish.

kindred iris
echo ruin
kindred iris
#

Hey there @echo ruin , been a minute

echo ruin
#

Just a few 🙂

modest estuary
hallow kiln
boreal field
#

Hey, I just found a malicious package on pypi, posted 15 hours ago.
May I name it here or is it better to just go through security@pypi.org ?

queen hornet
#

best to email

boreal field
#

got it. ty!

obtuse holly
#

using pep 621 and pep 508 specs, is this a valid dependencies array in pyproject.toml?

[project]
dependencies = [
  'aiohttp>=3.7.0,<4.0; python_version < "3.11"',
  'aiohttp>=3.8.2,<4.0; python_version >= "3.11"'
]
#

i'm not quite sure it is... i know this is valid for a requirements.txt file tho

lusty quarry
#

yup

#

but I'd invert quotes

bright parcel
rapid glacier
#

First time I saw this construct.

Is it just me or does it feels somewhat weird/unintuitive to have a toml file and then have this 'list' stored as a semi-colon delimited string(especially with those nested quotes)?

merry rune
#

It is a normal TOML list, it's just holding PEP 508 strings (which can contain semicolons). It pretty much looks exactly like it would inside setup.py, actually.

inland prawn
#

Yeah. IIRC there was a counter proposal PEP that suggested the approach using full TOML syntax (like Poetry does)

soft glen
#

Yes there was a counter proposal: https://peps.python.org/pep-0633/ At the end (sadly) the PEP 508 syntax was chosen because a PEP for it already exists and is used to define build-system requirements according to PEP 518. Arguments against it like hard to write, hard to teach, hard to parse and probably hard to extend wasn't weighted in the amount I would do it.

lusty quarry
tidal kiln
#

I am leaning towards sysconfig TBH

#

but want to be careful about adding new APIs

spiral urchin
#

I assume by “this logic” you mean choosing between native 32-bit, native 64-bit, 32-bit on 64-bit, cross the ARM/x86 split?

tidal kiln
#

yes

spiral urchin
#

A part of me kind of feel it should’ve been in sys or platform TBH, but if that’s not possible at least sysconfig

tidal kiln
#

if you build a native module on a 64-bit platform with a 32-bit interpreter, build backends need to use a 32-bit tag

#

so they basically need to copy that code

#

well, platform already provides the information needed to figure it out, so I am not leading to much in that direction

#

sys doesn't have a full platform key anyway

#

so sysconfig seems a bit better to me

#

but this is very much arguable

spiral urchin
#

Putting it in packaging is not a bad idea since this (in theory) needs to be future-compatible. Say if we bake this into platform, a 64-bit interpreter won’t be able to infer its own platform name on a 128-bit CPU; putting this in packaging allows for the possibility (just upgrade packaging)

tidal kiln
#

true but that is an issue already

charred swift
#

Their failure doesn't relate to any possible runtime error

#

as far as I can tell

high stone
#

(you've replied outside the thread :P)

charred swift
woven yarrow
#

If I have a setup.py, no matter how minimal, I can do python setup.py --version to see the version of my project (as well as a number of other bits of metadata).  What's the equivalent if I have a pyproject.toml-only package?

spiral urchin
#

Don’t think there’s an out-of-the-box solution for that, I wonder if it makes sense to add this to #build

woven yarrow
charred swift
#

@woven yarrow hatch version does that

uneven karma
#

What about other backends?

woven yarrow
uneven karma
queen hornet
charred swift
#

@queen hornet hatch project metadata | jq -r .name or hatch project metadata name

uneven karma
distant briar
#

I believe that this should give you the path to the dist-info folder:

import build

metadata_path = build.ProjectBuilder("path/to/project").metadata_path("path/to/project/dist")

Then for name and version specifically you can just parse the name of that folder (as it's just {name}-{version}.dist-info) and for everything else you would have to open metadata_path/METADATA file and parse it with email parser lib (or maybe importlib.metadata lib if that can parse external files, not sure)

#

Seems that importlib.metadata doesn't have any public API for parsing external files so yeah, you would have to just use email.message_from_string on contents of the METADATA directly.

distant briar
woven yarrow
distant briar
#

yeah, there's hatchling which is just the build backend and hatch which is whole project management tool

distant briar
#

I didn't know build has public API

#

Ohhh... There's build.util.project_wheel_metadata, I missed that...

#

yeah, you should use that 😬

#

Because I completely missed that what I suggested wouldn't create an isolated env which is what this util will also do (unless you choose to disable it with isolated arg).
Whichever way you choose, note that some backends could not implement the hook that allows you to get just the metadata without building the wheel as well in which case the process might be slower than you'd like but on the bright side, all major backends seem to be implementing it (to my knowledge, hatch was the outlier until some time ago but now I don't think there is any)

#

Would be nice if the util had an option to not fallback and an option to call build_sdist instead of build_wheel for performance reasons though I'm sure the latter could definitely create some discrepancies in some cases but I feel like it could sometimes not matter to the user of that API, idk.

spiral urchin
blazing lantern
# spiral urchin In this case not backend, but frontend. Hatch and most other workflow tools don’...

But it's also hard when everyone asks for their workflow to be supported. If we mandated static version numbers in pyproject.toml then you could just parse that to get the number, but then the setuptools_scm folks would be upset, so we created the concept of dynamic fields, which complicates things. Apply this to the various facets of packaging and you end up in our current situation of trying to meet people where they are with what they expect from setuptools while not locking people into setuptools

spiral urchin
distant briar
#

yeah, build.util.project_wheel_metadata is that interface

golden falcon
#

I'd personally rather have something that spits out JSON you can pipe into jq (using the transformation from PEP 566) than setuptools-esque CLI flags. I'm not sure if that something should be build

distant briar
#

it returns importlib.metadata.PackageMetadata instance

#

which aside from fields that can be defined multiple times is just a dictionary

#

it has a json field too

golden falcon
#

Yeah, the json method does the PEP 566 thing

#

so all we really need is a CLI to surface import build.util; print(build.util.project_wheel_metadata(".").json)

#

and, well, it should probably implement a cache... generating wheel metadata takes time

spiral urchin
lusty quarry
blazing lantern
manic belfry
lapis nest
pliant junco
finite kayak
heady lake
distant saddle
lime raft
low quarry
woven yarrow
lapis nest
distant briar
lusty quarry
#

I would recommend choosing Hatchling (as the official tutorial defaults to) especially for beginners

#

some more:

  • uses Git-style glob patterns rather than stdlib globs to better match user expectations
  • easy and explicit control of what gets shipped in the sdist target whereas with setuptools it's a mix of conditional wheel options and a MANIFEST.in file which can be quite confusing
queen hornet
#

disclaimer: ofek is the author of Hatchling 😉

I'm a user and have been switching over from setuptools/setup.* to hatchling/pyproject.toml and it's gone just fine

uneven karma
#

And... How complicated are your builds?
Do you do anything fancy?

queen hornet
#

for some I did a hop to setuptools/pyproject.toml as a stepping stone. so far for only simple pure Python libraries/CLI

woven yarrow
distant briar
#

personally in my migration I mostly have been concerned with tool.setuptools stuff being in beta as I would like to avoid distributing sdists that could quickly stop working

#

so I went with using setup.py for those things instead

woven yarrow
#

About Hatchling: its readme is uninspiring: https://pypi.org/project/hatchling/, and hatch feels like more than I want. Will there be an effort to make hatchling seem more real all by itself?

lusty quarry
uneven karma
# woven yarrow I'm looking for feedback on a draft blog post https://nedbatchelder.com/blog/202...

firH
Hello!
I'm going to point out a few things, but they're mostly pretty nitpicky. Overall it all seems pretty well put together, nice job!

I don't really have any comments about the blog post except -- what's this "writing docs" thing? Is that the name of one of your projects??

About the repo itself...

My biggest complaint is the inclusion of non-related file like an EditorConfig and a Makefile.
You're marketing this repo as a "just what you need to get started, nothing more", but then you're including, well, more.
When teaching, it's really best to leave out all the optional and/or non-related stuff, because newcomers will get confused.
Your Makefile is actually a perfect example of this. You're talking about "the simplest possible introduction to Python packaging", and then your repo comes up as "51% Makefile".
Preocts the Egg uses Makefiles too, and I sometimes use their repos as good examples of how to do packaging, and a couple of times I've almost accidently scared beginners away because they "didn't realize you need "so much "make" stuff" just to "make" a package".

Why make the version dynamic? It would be a lot less confusing to just leave it as a static field.

Having a dev-requirments.txt in addition to a pyproject.toml has you installing dependencies two different ways for the same package!
Since you already have a make target for it, I would just delete the dev-requirments.txt file outright.
But I would actually reccomend adding a "dev" extra to the pyproject.toml, which lets you do pip install --editable .[dev], which installs all of your normal dependencies and your dev dependencies all at once.
I stole that from Preocts as well and I've been liking it.

I would also mention the py for Windows, since most of your target audience will likely be using Windows.

woven yarrow
#

I haven't used [dev] extras, but I can see the appeal.

distant briar
#

I like the Makefile but it is definitely not clear that it wouldn't work on Windows and I imagine most people aren't already familiar with make. I thought about Windows but only really looked into whether twine commands will work without shell support for wildcards and as it turns out... They will!

lusty quarry
#

yeah another benefit of Hatch envs/scripts is Windows support

#

(I use Windows)

uneven karma
distant briar
#

I found it easier to quickly test Linux things on Windows than Windows things on Linux because man, Windows is heavy

lusty quarry
#

I felt "stuck" on Windows for a long time since I need assistive technology things but Windows is awesome now

uneven karma
inland prawn
woven yarrow
inland prawn
#

Like in CLI tools for —version flag

high stone
#

installer PR 147

spiral urchin
#

I wonder if there’re people attempting to use pre-commit for this. Instead of trying anything to deduplicate, it’s possible to simply duplicate but use tools to keep everything in sync.

inland prawn
#

That would add complexity. That tool would have to understand all build tools there are and hope they don’t change they way they work. Other way would be to build package on each commit, but this sounds like a lot of wasted time and resources

spiral urchin
#

Deduplicating version declaration adds complexity, you just push it into the build backend.

#

I can argue the exact same point for every build backend adding code to populate the version number either direction

woven yarrow
#

surely there are fewer build backends than there are repos that would need pre-commit tooling? and pre-commit tooling doesn't replicate with the repo, so is easy to break.

spiral urchin
#

Not sure what the difference is. Pre-commit tools can be shared as well, I’m probably missing something from the argument against it

woven yarrow
#

When I clone your repo, i don't get a pre-commit hook enable. I have to take a manual step to enable it. But you're proposing that a thousand devs install a precommit hook so that a half-dozen build backends don't have to add code?

#

(thousand is a low estimate 🙂 )

spiral urchin
#

No, only the devs that need to bump the versions need that hook installed. Everyone else just read the duplicated version strings from either pyproject.toml or __init__.py or wherever

woven yarrow
#

there are many many devs that update version numbers....

spiral urchin
#

… and those same number of devs also build the packages after bumping… still installing the logic from their chosen build backend when doing so

woven yarrow
#

i must be misunderstanding something. I thought you were saying that it's not reasonable for every build backend to implement this feature. if the top five build backends implement it, then only five dev teams have to think about it. your alternative is for every project to have a pre-commit hook for their release people to make it work. It seems like an odd tradeoff.

spiral urchin
#

First of all, I never said it’s not reasonable for every build backend to implement the feature. All I did was wondering whether the same can be done with a pre-commit hook. Not sure where the negativity came from.

With build tools, every dev building every project (which is usually a part of the release process) needs to install a build backend with extra logic that pulls version from code into package metadata. With a pre-comit hook, every dev doing version-bumping for a project (which is also usually a part of the release process) needs to install one extra thing to bump versions. The hook is developed once, or at most once for a backend, and that backend with a pre-commit hook available can then drop that special logic and rely solely on a static pyproject.toml. The two approaches seem to be on the exact same level of complexity to me.

lusty quarry
#

I very much dislike pre-commit hooks

charred swift
#

@woven yarrow If you'd like more feedback on your approach, I too have recently switched away from setuptools in favor of hatchling and couldn't be happier for the reasons already highlighted above. I made the switch after noticing that the official packaging tutorial had been updated with hatchling as the default

golden falcon
mortal shore
inland prawn
#

sounds like an overengineered approach... I honestly don't get the hate on pre-commit... (and maybe we should take it to #off-topic ?)

golden falcon
#

probably 😛

woven yarrow
woven yarrow
#

(and btw, sorry if I have hijacked this discord/channel/thread/etc!)

charred swift
#

I don't know so much about the roadmap or what direction it's going, but I can say that it's been perfectly adequate for my needs. It seems to get out of the way so I can focus on my actual code instead of packaging it

lusty quarry
# woven yarrow (and btw, sorry if I have hijacked this discord/channel/thread/etc!)

suggestions?

on https://hatch.pypa.io/latest/ the very first thing says:

Standardized build system with reproducible builds by default

"build system" links to https://hatch.pypa.io/latest/build/#packaging-ecosystem & it's first line says:

Hatch complies with modern Python packaging specs and therefore your projects can be used by other tools with Hatch serving as just the build backend.

"complies" links to the PEP 517 configuration

I don't know how to make that more prominent

woven yarrow
#

Thanks. My question was about hatchling, not hatch. When I look at hatch, it seems to do a lot of things, most of which I already have solutions for, so my inclination is, "not for me." Someone recommend hatchling as a build backend, so I go to read about it: https://pypi.org/project/hatchling/ A one-sentence readme doesn't help me adopt hatchling as a tool, and I am back to trying to understand hatch, whether hatchling will somehow hatch-ify my project, etc.

lusty quarry
#

everything I linked is about Hatchling config

woven yarrow
lusty quarry
#

I will improve the readme ty 🙏

woven yarrow
#

Fabulous, thank you so much.

lusty quarry
woven yarrow
#

btw, I count the hatchling readme on pypi as "a hatchling page"

#

so maybe we are in agreement

golden falcon
#

in your makefile you have python -m build --sdist --wheel - this will build the sdist and wheel independently of each other. By default build will build a wheel from the unpacked sdist to make sure you've packed everything correctly

woven yarrow
#

Just to be clear: I love everything you all are doing to improve the packaging world. I want to help educate and smooth the rough edges, and learn how to make better packages myself.

lusty quarry
charred swift
blazing lantern
charred swift
spiral urchin
inland prawn
civic compass
charred swift
charred swift
# spiral urchin I didn’t read much of it, but why is `whl_scheme` needed? Entries in `RECORD` ar...

The way I've written it, the package doesn't need to be importable to uninstall it. I don't even look at sys.path actually. The RECORD file locating I've got in there (today) uses pretty much the same logic that installer uses to figure out where to install it (where of course the package doesn't need to importable to install it). I have to do some more testing to check how important whl_scheme is (this may be a real drawback to my approach though). installer uses whl_scheme here https://github.com/pypa/installer/blob/main/src/installer/__main__.py#L67 so I used it in the same way. Unfortunately in my project (today) the user has to supply this info whereas installer just looks it up from the wheel: https://github.com/pypa/installer/blob/0b72640fcdd95c80a73bb94eaa3377381ef5b8e0/src/installer/_core.py#L29-L33

#

so, for example, you could uninstall a package from a venv that's not active if you use the right --root

#

or you can activate the venv and not have to use --root

charred swift
unborn harbor
bright parcel
#

Is it just me, or does anybody else consistently typo [tools.foo.blah] ? 😆

inland prawn
#

where typo?

bright parcel
#

pyproject.toml

distant briar
#

took me a while to realize the typo lol

#

I don't know if I do it consistently but I definitely did it before

inland prawn
#

well, to be that person, it isn't technically a typo (I don't think people validate pyproject.toml contents that strictly), it just wouldn't work lol

bright parcel
distant briar
#

if you type [tools.setuptools] when you meant to write [tool.setuptools], you made a typo

#

or I guess more generally a mistake if you didn't know that it's wrong

bright parcel
#

I'm an existence proof that sometimes you're eyes just glaze over that typo

distant briar
#

whether something is a typo or not depends on the intentions of the author :)

inland prawn
#

but I don't think there is a "universal validator" since it could only validate if tables' names are correct

queen hornet
# 
Invalid file: pyproject.toml
[ERROR] `data` must not contain {'tools'} properties
blazing lantern
woven yarrow
#

when i dug into TOML syntax recently, I was dismayed to see a few aspects that were neither Obvious nor Minimal 😦

uneven karma
#

Like what?

inland prawn
#

Also interested

high stone
#

TOML syntax

civic compass
queen hornet
#

I don't know

lusty quarry
#

@noble matrix how's PEP 639 going?

golden falcon
spiral urchin
golden falcon
#

hang onto the original value and validate it against format: "date-time" (or any of the other datetime format options)

#

but yes, the validator wouldn't be able to tell if the value should be of type string or a datetime

noble matrix
# lusty quarry <@333125359811952640> how's PEP 639 going?

Thanks for the ping; I didn't want to reply until I had the PR up so I wasn't overpromising and underdelivering again, but I've been actively working through the final set of changes before starting a third and hopefully final round of review. Its mostly just adding the glossary and updating various places in the text to use it, though that ended up being a lot more comprehensive than I thought it would be (though also more of an improvement to the overall clarity and unambiguity).

I initially was just going to convert the existing terminology section into a glossary and use the PyPA glossary for the rest, but I ended up doing a complete rewrite/rework of the whole section to make it more focused, comprehensive and easier to navigate, and ensure the definitions were accurate, up to date, cross-referenced and more concise. That's done, and I've now been working through the document adding them and clarifying the corresponding language, which has ended up also incorporating other clarifications, cross-references and other textual refinements at the same time—I'm about halfway through right now.

Once that's done hopefully in the next couple days, and address one other key point (related to the current "Tools" terminology and guidance, which I need to re-evaluate to align it with other PyPA standards and ensure is reasonable, clear and not over-complicated) I'll open a PR, and after that it's just the simple mechanical task of sticking the non-core parts in separate documents. To note, none of the changes should have meaningful normative impact on the content, as least isofar as you've implemented it thus far. I also want to make some non-user-visible formatting and technical changes (SemBr, using Intersphinx references instead of hardcoded URLs, etc) but I'll do that in the background after I've finished the others and start a new thread for a final round of discussion, so as to not block further substantive progress.

wicked canyon
#

So... how is codegen supposed to work in modern python packaging?

I've an old library for which most of the behaviour is encoded as data files. For convenience & performance, these data files are converted to Python during packaging (or on demand for unpackaged use), currently that's achieved by overriding (hopefully) all the right distutils/setuptools cmdclass, but it's not entirely clear to me how that's supposed to work (even at a high level) following PEP 517 / PEP 518.

Is there something that's built-system independent, or is a custom build backend (?) necessary, or is this something which should be done by picking a build system / backend (hopefully one which supports this sort of things) then using its bespoke features to achieve? Is there a guide somewhere?

inland prawn
#

this should be done by the build backend of your choice (the one that supports this kind of operations)

spiral urchin
#

I think you can still use setuptools

wicked canyon
inland prawn
#

I don't think there is a list, but as you mentioned setuptools, I would just stay with it until it doesn't fulfill your needs. Keep in mind most backends have none or very limited support for scripting via setup.py-like file

wicked canyon
#

Shame. I'm using setuptools because I inherited it (and it was basically the only option for non-trivial stuff in setup.py anyway), so I figure if I'm going to redo packaging anyway I might as well redo the entire thing to a modern standard, seems a shame to just punt a bit.

inland prawn
#

didn't see the code, so it might be wrong, but maybe it would be a good idea to change the approach in code/data loading so you don't rely on such features?

distant briar
#

setuptools supports modern standards

#

I imagine the main alternative that supports this kind of shenanigans would be hatch(ling)

#

Poetry can have plugins but it enforces an opinionated workflow on you, flit doesn't have plugins

#

There's also PDM which seems to support plugins too

#

But yeah, unless you're looking to switch, I'm not sure you really need to, setuptools supports modern stuff.

#

One argument I found for hatchling over setuptools is how editable installs work in setuptools after they started supporting pep 660 - not well with static analysis tools (though you can work around it)

#

But I personally use setuptools or flit for my things

wicked canyon
# distant briar setuptools supports modern standards

I'm not sure that's very helpful as codegen doesn't seem to be part of any standard, so surely dropping setup.py would require finding a different way to perform the codegen step in setuptools anyway? It seems to be a bit of an open issue: https://github.com/pypa/setuptools/issues/2591. Apparently declarative cmdclass was added to setup.cfg https://github.com/pypa/setuptools/pull/2571 but it's not 100% clear that it will also work in pyproject.toml and the official doc say tools.setuptools is beta: https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#setuptools-specific-configuration

lusty quarry
distant briar
#

You were saying that you want to redo your packaging to a modern standard and I'm saying that setuptools supports those modern standards so you don't have to move away from it

wicked canyon
lusty quarry
wicked canyon
fallen shuttle
wicked canyon
# fallen shuttle If you need to do something very custom you have to write some scripting or crea...

I honestly have no idea. The entire ecosystem seems to be moving away from setup.py and towards a more declarative project management system so I figured taking a look might be a good idea in case it would make maintenance easier / simpler.

You might ditch setup.py if you use something like hatch, but you still will need some python file. At that point, does it really matter if your custom code is defined in a setup.py or path/to/your/customization_file.py?

Hopefully it would reduce the amount of code and the number of hooks to plug into, but I've no idea that it does. From the outside it seems a bit like the baseline is being churned and everything's off the rails, but I'll readily admit I've not had much of a look. Maybe its more to the benefit of tooling and having a one-stop shop? It seems like a great deal of improvements concern that to avoid the ever-increasing multiplication of bespoke configuration files.

distant briar
#

The general recommendation is to use declarative metadata wherever you can which is supported by both setuptools and hatch.
In your call to setup() you would only be declaring the things that you're doing dynamically while declaring project name, dependencies and whatever other metadata directly in pyproject.toml.

#

So it's really a question of how much code you need to have to implement the thing you need with hatch vs setuptools but you definitely wouldn't have to declare any static metadata in py file in either.

wicked canyon
#

Thanks to you all.

distant briar
#

It is, yes.

#

If it's an attribute that can be specified directly in the [project] table inpyproject.toml, you'll have to add it to the dynamic list but if you don't, you'll notice that immediately after trying to build since it will error and tell you that :)

inland prawn
distant briar
#

it's probably not a bad idea to check whether build still works immediately after doing this as this causes pip to use build isolation and that could potentially break something depending on what custom stuff you're doing

soft glen
#

This is a little bit surprising:

import email.utils
email.utils.parseaddr("Me [Some Department] <me@example.com>")
('', 'Me')

I would expect ('Me [Some Department]', 'me@example.com') or some kind of error if the input is invalid 🤔

spiral urchin
#

Parsing of the originator fields is known to be very intricate. The rules are not very normalised (mostly since email address has very permissive rules)

distant briar
#

I'm not sure if you control the input here but if you do then one thing you can do is quote the name

#

the problem is with the square brackets and it actually also occurs with email address too, I'm not sure how you can fix it there though

#

I once ran into it with an address in the form of 12345678+githubbot[bot]@users.noreply.github.com

soft glen
#

Yes, I know this is very intricate. So I was hoping that email.utils is doing the hard part for me 😅 Quoting works (if I use "!) 👍

https://stackoverflow.com/a/24958244/9750706 seems to extract the rules quite good. So I still wonder if there's a package out there, that implement these rules already.

Stack Overflow
Validation/Format of display-name in from header

I need to know what are the rules for validation/format from(name-addr) field in the email.
In rfc explained the format of name-addr, but goes into detail about the display-name.
Like this:
From: J...

noble matrix
#

To secure his favor, I present an offering of a holy <>

fallen shuttle
# soft glen Yes, I know this is very intricate. So I was hoping that `email.utils` is doing ...

You might have some luck if you are willing to go a bit lower level and do some manual checks:

>>> from email.headerregistry import AddressHeader
>>> parsed = AddressHeader.value_parser("Me [Some Department] <me@example.com>")
>>> len(parsed.all_defects) > 0
True

>>> parsed = AddressHeader.value_parser('"Me [Some Department]" <me@example.com>')
>>> parsed.all_defects
[]
>>> parsed.mailboxes[0].addr_spec
'me@example.com'
>>> parsed.mailboxes[0].display_name
'Me [Some Department]'
soft glen
#

whooh, looks promising 👍 Will play around a bit.

bright parcel
#

@soft glen is there a ticket open on this? It's been a long while since I looked at the RFCs or code. RDM often gets to these things sooner than I do these days. If there is a ticket, @-me (even though. I am on the email team)

soft glen
#

@bright parcel No, I haven't opened a ticket for this until now. Is it worth it? Where should I open it.

noble matrix
soft glen
noble matrix
coarse mason
# wicked canyon Oh so it's possible to have some of the bits in setup.py and some in pyproject.t...

Absolutely. I have at least one project set up this way (albeit with setup.cfg for now, until setuptools+project.toml is out of beta), and will be migrating others over time.

I (1) dynamically exec() the version number out of a module in the package, and (2) replace the version of Read the Docs that any docs links point to (I want README to point to latest when shown on GitHub, but to a specific release/version when shown on PyPI).

lusty quarry
coarse mason
noble matrix
# coarse mason Absolutely. I have at least one project set up this way (albeit with setup.cfg f...

(1) dynamically exec() the version number out of a module in the package

Why not just the standard

[metadata]
version = attr: sphobjinv.__version__

in setup.cfg, which is stable and has been around for many years, or

[project]
dynamic = ["version"]

[tool.setuptools.dynamic]
version = {attr = "sphobjinv.__version__"}

in pyproject.toml? Then you don't need a seperate version.py to exec either.

(2) replace the version of Read the Docs that any docs links point to (I want README to point to latest when shown on GitHub, but to a specific release/version when shown on PyPI).

That would still require a plugin or dynamic config in Setuptools, but that Hatch plugin @lusty quarry mentioned is specifically designed for that sort of use case, IIRC.

noble matrix
#

Actually, I think I just discovered it the other day looking for tools to programmatically extract Sphinx object inventory data; I just didn't make the connection until now since the impression I got of its main functionality looking at the docs vs. the readme was very different.

Skimming the docs (and not seeing the readme) I had no idea that its flagship functionality was interactive use searching for objects to reference via a CLI, and had assumed that it was mostly focused on use as a library (which is what I was primarily looking for at the time), since interactive use via the CLI is barely mentioned there. It was only checking out the Readme above that I realized it also contains core functionality for searching local or remote Sphinx object inventories in interactive use via a CLI, which also seems super useful to me.

bright parcel
#

I really like hatch’s ability to call a function to get the version dynamically. Then I use importlib.metadata.version() to extract the version number for display

coarse mason
#

Really interesting project BTW the name

#

1 dynamically exec the version number

#

Actually I think I just discovered it

coarse mason
echo ruin
#

I think the pkginfo release that just went out might have broken twine and gh-action-pypi-publish

grizzled spade
inland prawn
#

@high stone any chance for installer release? (mostly for types detection via py.typed marker)

lapis nest
high stone
#

And, yea, I'll cut a release. :)

distant briar
#

Both the old and new work on hand-written parser in packaging has been educative for me, cool work

inland prawn
#

what parser?

distant briar
#

packaging package has a hand-written parser for marker and requirement strings

#

just leaving a note of appreciation for that work here 😄

west drift
#

In PEP 508, it says:

specification = wsp* ( url_req | name_req ) wsp*
url_req       = name wsp* extras? wsp* urlspec wsp+ quoted_marker?

Shouldn't it say (wsp+ quoted_marker?)?, like in the parsley grammar it also says (wsp+ | end)? Otherwise "pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686" (no whitespace after the url) were an invalid requirement and that clearly should be allowed

high stone
#

I spotted that too yesterday -- not sure if we should be updating that PEP or just copying it over to packaging.python.org and updating there.

west drift
#

if there's consensus to merge fixes, more examples and text clarifications i'm willing to port PEP 440 and PEP 508 over to packaging.python.org

blazing lantern
high stone
mossy pulsar
#

tox 4.0.0rc3 is now out, this will soon become the stable release unless someone reports any more release blockers, help us test it with your project if you can! Thanks! https://tox.wiki/en/latest/changelog.html#v4-0-0rc3-2022-12-05 (tox 4 should be mostly backwards compatible with v3 minus some features long deprecated in v3 and now using isolated builds by default; as shiny new features I think the biggest are built-in wheel support instead of sdist for packaging, editable wheel support and fancy colors for reporting!)

west drift
# blazing lantern My vote is to just update at packaging.python.org

i've made a PR to move PEP 508 to packaging.python.org, currently nearly verbatim without any content changes: https://github.com/pypa/packaging.python.org/pull/1177

GitHub
Add PEP 508 by konstin ¡ Pull Request #1177 ¡ pypa/packaging.python...

This moves PEP 508 to packaging.python.org as discussed on discord
Diff between PEP 508 and dependency-specifiers.rst: https://gist.github.com/konstin/059c0e72ff77049e0eb2ce0d738990fd

regal bronze
civic compass
high stone
#

Both the old and new work on hand

high stone
#

Please feel welcome to poke at the parser in this PR and try to break it. PR reviews are also appreciated, as is feedback on potential improvements to the error messages coming out of it. :)

https://github.com/pypa/packaging/pull/624

dire vortex
#

Hi, I don't see a channel for pipenv, so I'm posting here