general

[echo ruin]

sounds like fun. enjoy.

[woven plover]

Btw, has anyone ever experimented with cythonizing most things, i wonder about the impacts of it on tools like pytest, pluggy, zokusei

[tidal kiln]

I'd be more tempted to try to use mypyc

[woven plover]

@tidal kiln maybe thats something to take a look now, does it support dataclasses/attrs?

[tidal kiln]

honestly idk 😛

[woven plover]

hmm, i wish dataclasses/attrs wasnt using a codegen

[lusty quarry]

@echo ruin @mossy pulsar mind if I make a refactor PR? mostly for perf, esp. importing

[echo ruin]

Please do

[mossy pulsar]

👍

[echo ruin]

FWIW the "controversial"-y change I care about personally is https://github.com/ActiveState/appdirs/pull/137

I don't think that PR is right (which is why I didn't merge it)

The behavior I'd probably want (which probably has been proposed by someone) is just "use $XDG_* on all OSes if they're set in the environment explicitly

But probably we should discuss that one.

[lusty quarry]

for https://github.com/platformdirs/platformdirs/blob/788ce4ce6d782fe7170012d98b67b052c0f0bd70/platformdirs.py#L140 does expanduser do anything without a tilde?

[spiral urchin]

expanduser is no-op unless path starts with a tilde

[lusty quarry]

cool, I'll remove that call then

[tidal kiln]

@glass sand could you explain the purpose of test_resource_opener and test_resource_path tests? https://github.com/python/importlib_resources/blob/a024adf2c5235218aeb6d90817fe93b4e63496b9/importlib_resources/tests/util.py#L115

they are building a package and then trying to access a file that isn't there

and this is supposed to work

I am voice if you want to join, or you can just reply here

they are the only thing that breaks when I change the {open,read}_binary implementation to the files API

[woven plover]

@tidal kiln i recall it's related to a regression where subdirectories where allowed, i can check once i get back to the computer after dinner

[tidal kiln]

thanks

[lusty quarry]

https://github.com/platformdirs/platformdirs/pull/1

[woven plover]

@tidal kiln i'm no longer finding the issue

[tidal kiln]

I looked for context in the commit log but didn't find anything

the issue is that my CompatibilityFiles implementation in https://github.com/python/importlib_resources/pull/221 does not let you open resources that are not in ResourceReader.contents, even if ResourceReader.open_resource would return some data

[woven plover]

i recall a strange regression a few years back, where we used the paths api to get directories, that somehow broke telling it needed files

maybe the issue was on bitbucket

[tidal kiln]

I think the tests abuse this implementation detail in the legacy documentation

so, I am gonna add the file they are trying to open to the package contents

which should fix things

well

there are still issues

sigh

if ResourceReader.open_resource raises FileNotFoundError, the legacy API will still succeed

or not

agh

I'd really like to understand what these tests are trying to do, because I can't figure it out 🙃

they now fail essentially because we call ResourceReader.contents before ResourceReader.open_resource

on the first test because the requested file isn't in contents, and the second one because the test ResourceReader.contents implementation will raise FileNotFoundError if there is no reader path

the only reason I can come with up for those tests is to make sure the situations I described work

which seems weird

[woven plover]

sorry i was distracted afk, unfrotunately without the oriiginal issues i recall, i have no idea what happened back then in detail

[lusty quarry]

refactor went well for Windows 😄

[echo ruin]

Nice! Well done.

[high stone]

@tidal kiln Whenever you're around, ping me. I wanna chat about the whole destdir thing, because I'm still not sure if that needs support in the default destination.

[high stone]

There's likely folks here who'd be interested in https://discuss.python.org/t/8745. :)

[tidal kiln]

destdir should definitely be supported in the destination IMO

but let's have a chat when I finish lunch

@high stone I am voice if you want to join

[high stone]

Eating lunch myself. I'll join after.

[tidal kiln]

okay 🙂

[woven plover]

@tidal kiln https://github.com/python/importlib_resources/issues/85 - found it by sheer luck - someone pinged - the move from BB got me

[tidal kiln]

I just stumbled upon that issue when looking into a bug report 😛

I don't think anything there warrants those tests

[high stone]

Heyo folks! Just wanted to let y'all know that I posted a list of rules for this server on #rules, modelled after the Python Discord.

Make sure to read them!

[idle egret]

@high stone I do not have sufficient permissions to read the #rules 😬 😅

[high stone]

How about now?

^@idle egret

[idle egret]

I have access now, but I see no rules.

[high stone]

And, how about now?

Lemme know if you can react in the message, if you see it!

[idle egret]

No, cannot see a message. Wanna join some video for a min to resolve this issue? Idk about discord - or I could share a jitsi instance @high stone

[high stone]

Hmm... Discord permissions probably.

I'll explore in ~5 hours. Gotta do $DayJob now. 🙈

Here's a repost though, until I can figure that out:

We have a small but strict set of rules. Please read over them and take them on board. If you don't understand a rule or need to report an incident, please reach out to [e-mail].

Rule 1
Follow the Discord Community Guidelines and Terms Of Service.

Rule 2
Follow the PyPA Code of Conduct.

Rule 3
This is an English-speaking server, so please speak English to the best of your ability.

Rule 4
Keep discussions relevant to channel topics and guidelines.

[idle egret]

Have fun. Here it is after work time already.

[golden falcon]

I can see them in the #rules channel and I don't think I have any special permissions

[lean lake]

Sweet - If we can bridge this to IRC that would be grand. I feel this with cpython dev channels too would make sense

What do people think the separation benefits?

I liked watching #python-dev on IRC from time to time. WIsh more cpython chat happened there tho

[echo ruin]

bridging would be great for me too I think -- the issue usually I think is just mapping, since IRC is not many-to-many, so each channel would need a matching IRC channel

and yeah #python-dev is usually a dead zone 😄

[lean lake]

Yeah, bots only

[blazing lantern]

There's also where to do the IRC since Freenode is apparently in the middle of collapsing

[lean lake]

Yeah dstuft and pradyunsg seem keen on libra chat

[blazing lantern]

there's a conversation going on about that right now to see if we can convince folks to rally around a single platform

[lean lake]

@blazing lantern - Sweet - on disscussions ?

[lean lake]

welcome @lyric hedge !

[lyric hedge]

heya it's the bandersnatch docs person here

[lean lake]

I'd also like to discuss getting @lyric hedge commit bit to bandersnatch

It's that on dstuft + ernest etc.

*only

Seems I have repo access but not org access.

jaraco is an owner

and @high stone 😄

[high stone]

@ewdurbin is Ee now.

Oh, can you not add them yourself @lean lake ? Lemme fix that.

[lean lake]

Can't add people to PyPA org to then add to bandersnatch repo no. That would be great 😄

[lyric hedge]

@lean lake only maintain perms?

[lean lake]

I'll give you as much as you want.

[lyric hedge]

wait no

I meant that you only have maintain perms for bandersnatch?

[lean lake]

Ya, only there. That's all I have 😄

[lyric hedge]

Wow, that was actually the same thing with ambv and black.

I think, it sounded like it, although I don't remember well

[lean lake]

Yeah, neither of us are leading PyPA / psf people, so it makes sense. I like to only have access I need.

e.g. I once had root to every server + network device @ some big company I've worked for. I didn't "need" that the access controls when I joined sucked. I didn't ever let my laptop out of my sight or it was locked in my safe when I went out. lol

This has since been tightened a lot thankfully.

[high stone]

Invited ichard26 on Github to bandersnatch!

[lyric hedge]

thanks @high stone !

@high stone I think you messed up, the invite you sent is for the whole PyPA GH org, I don't need that 😅 bandersnatch outsider collaborator is good enough

[high stone]

Yea, well, you're getting the commit bit, so you're gonna be a PyPA member now. :P

[lyric hedge]

Oh, I-did-not-know-that

whoops :p

[high stone]

IIUC, that's what @lean lake asked/wanted me to do. :)

[lyric hedge]

I assumed that since cooperlees is marked as a "contributor" on bandersnatch, they aren't a member so hence, "whelp I'm going to be come an outside collaborator for bandersnatch now"

IDK, I don't understand why their badge on that repo is not "collaborator "

though

[lean lake]

It's fine - embrace it. 😄

[lean lake]

^^ slid into our DMs

[woven plover]

just for my personal understanding, even tho python likes to normalize versions, are slightly non-normalized versions ok (for setuptools_scm calver support i want to enable always double digit days/months/)

[spiral urchin]

The new server icon on the top left (not sure about Discord terminology) doesn't look too good. Maybe we should crop the image?

[mossy pulsar]

yeah, 🙂 need a 16:9 variant of it, only had at hand the 4:3 one

[lean lake]

I would personally always try to match pep440 somehow to go to adding your standard - just easier for future and current tools to understand etc.

[woven plover]

the versions would be valid pep404, just not completely normalized (to have the dates roll along with each other better=

[lean lake]

Pep not found? Haha

[woven plover]

whops

[lean lake]

I got it was typo

I am the typo king

[woven plover]

Btw i somehow ended up with a matrix channel for pypa, what should I do about it

[lean lake]

Seems fine to me. I like date versions …

[woven plover]

That's going into setuptools_scm 7

[lean lake]

Cool. black uses it. We can use black to test anything if it helps.

[woven plover]

Btw what specify variant does black use, i want to add more calver schemes

[lean lake]

If interested, our CI release to PyPI works nice with our “real versions” too surprisingly

[woven plover]

At work we use calver by date, which is yy.mm.dd.patch

[lean lake]

We just cut a version via GitHub releases, it tags and then setuptools_scan versions correctly

[woven plover]

I plan to add something where you can press a butt to do all release stuff

[lean lake]

YY.MR - where R is a release number - today we use box

[lyric hedge]

Black uses yy.mbN (I have no idea how to write this properly) roughly for now (e.g. 21.5b0 or 19.10b1)

[lean lake]

I don’t recall what standard it is

There you go - ichard is smarter

[lyric hedge]

Once we go stable, we'll drop the bN part

[woven plover]

Isn't b just a beta marker

[lean lake]

Yeah

[woven plover]

Ok, funky, not sure if this can be easy

But it's required I think, will be fun to figure

My goal is to provide building blocks to tag releases by approval of auto created merge requests, using towncrier

[lean lake]

https://github.com/psf/black/blob/main/.github/workflows/pypi_upload.yml - we just use GitHub release UI. Yeah our changelog/news isn’t auto magic which would be nice but the releases sometimes fixes up chnahelog wording. Hard to automate that.

[tidal kiln]

@spiral urchin @high stone is resolvelib.AbstractProvider.is_safisfied_by guaranteed to be called on candidates the resolver tries to choose?

my use-case is, I have an expensive operation that needs to be done to make sure if the candidate is valid

doing it on find_matches seems bad because I will need to do it on all candidates

so, what I want to do is return possible candidates, not candidates that are required to be compatible

and then when the resolver chooses what it want, it would call is_satisfied_by to make sure the candidate really is valid

maybe this is exactly the purpose of that function, but I wanted to confirm 😆

I just implemented it like that and it seems to work fine

nevermind, it does not work, it raises resolvelib.resolvers.InconsistentCandidate

opened https://github.com/sarugaku/resolvelib/issues/79

[high stone]

You can make it into a generator, to iteratively do it as spew it out.

[tidal kiln]

ah

no, that doesn't work because the resolver consumes everything

nevermind

no

it still doesn't work

the resolver consumes the whole iterator

but maybe that has something to so with the algorithm?

[tidal kiln]

it's probably the algorithm

starting()
  adding_requirement(build[virtualenv], None)
    consuming build-0.3.1.post1-py2.py3-none-any.whl (expensive!)
  adding_requirement(build, None)
starting_round(0)
    consuming build-0.3.1-py2.py3-none-any.whl (expensive!)
    consuming build-0.3.0-py2.py3-none-any.whl (expensive!)
    consuming build-0.2.1-py2.py3-none-any.whl (expensive!)
    consuming build-0.2.0-py2.py3-none-any.whl (expensive!)
    consuming build-0.1.0-py2.py3-none-any.whl (expensive!)
    consuming build-0.0.4-py2.py3-none-any.whl (expensive!)
    consuming build-0.0.3.1-py2.py3-none-any.whl (expensive!)
    consuming build-0.0.2-py2.py3-none-any.whl (expensive!)
    consuming build-0.0.1-py2.py3-none-any.whl (expensive!)
  adding_requirement(build==0.3.1.post1, Candidate(name=build, version=0.3.1.post1, extras=(virtualenv)))
  adding_requirement(virtualenv>=20.0.35; extra == "virtualenv", Candidate(name=build, version=0.3.1.post1, extras=(virtualenv)))
  pinning(Candidate(name=build, version=0.3.1.post1, extras=(virtualenv)))

I don't understand why it needs to consume everything here

but the logic is super complicated

so

[woven plover]

i recall pip faced the same issue,

[tidal kiln]

I got it to work!

I was misusing the api

the example in repo is not very efficiently implemented 😅

https://github.com/FFY00/python-resolver

[spiral urchin]

Sorry wasn’t around yesterday. I think you’ve already figured it out but the resolver only consumes things when it needs to, exactly because how Python packaging works. is_satisfied_by is therefore not called for every candidate, only those the resolver must "visit" (check if it can be pinned)

[trim haven]

just changed it to a 16:9 version 🙂

[lyric hedge]

It looks so much better

[mossy pulsar]

Million thanks for that

[zealous swan]

omg just noticed the platypus in the top left, so cute!!! 😍

[blazing lantern]

python-committers and discuss.python.org

[high stone]

https://discuss.python.org/t/vote-on-core-dev-chat-choice/8870 -- I like what I see!

[lean lake]

Yeah looking like a good chance for Discord

[merry rune]

Does anyone need to be pinged for packaging? https://github.com/pypa/packaging/pull/430 is a typing fix that would be nice to have eventually.

[high stone]

@merry rune me! :)

I'll take a look after $DayJob.

[blazing lantern]

I added the two of us as reviewers

[merry rune]

Great, thanks (to both of you, I can't reply to two at once)! No rush, just pinging on it once a week. 🙂

[zealous swan]

Is there any precedent to an unresponsive maintainer of an important package (PySocks) for another maintainer to be added to the project in order to do maintenance?

Emailed Anorov multiple times over the past month without a response.

[mossy pulsar]

Dunno, but requests is in the same boat, and I've been trying to get added without success 😬 so if you manage do let me know 😂

[zealous swan]

Requests has two maintainers that both respond?

[mossy pulsar]

Not true, at least in my case

I've reached out to both of them and got no response, my pr from December is still without answer... https://github.com/psf/requests/pull/5707

[zealous swan]

That PR is on our radar fwiw

[mossy pulsar]

One of the maintainers have their GitHub status as stepping away from open source...

[zealous swan]

I think that status is out of date

[mossy pulsar]

Eh

[zealous swan]

We've discussed it over email in the past week

[mossy pulsar]

Took a while then, but at least would be nice to give a comment on the pr 🤷‍♂️

[zealous swan]

Sure thing, I can get something in there. Sorry it's taken so long, it is an important fix.

[zealous swan]

Going to reply to my original question.

[mossy pulsar]

Sorry for hijacking the topic 😬

[zealous swan]

No problem!

[mossy pulsar]

I think we've already missed pip June edition, but maybe can get it in for the September/October release...

[blazing lantern]

Not that I'm specifically aware of, but I would assume the process would be https://www.python.org/dev/peps/pep-0541/#how-to-request-a-name-transfer

[high stone]

Yea, an issue on pypa/pypi-support should be the thing you need to do here.

@blazing lantern should we be doing that for toml, or should we have another round of emails to @uiri instead to get access to the package name on PyPI?

[zealous swan]

Thanks @blazing lantern, I'll start that process.

[lean lake]

@high stone / @blazing lantern - is this for toml in stdlib or upgrade pip’s support? We got this complaining about black toml support and we’d like to do what ever pip does https://github.com/psf/black/issues/2280

[obtuse holly]

hello kosayoda

[dull aspen]

ARU you got a lot of people ic

[obtuse holly]

Lmao yeah

[dull aspen]

another one caught, hello chris

[lyric hedge]

Welcome to PyPA Discord y'all!

[obtuse holly]

I def didn't share the invite in python discord or anything 👀

[lyric hedge]

👀

[dull aspen]

haha lol, he shared it in the meta, indirectly thro a bot

[obtuse holly]

It's whitelisted now 🥳

[merry rune]

#setuptools-scm: @woven plover Why is the example <package>/version.py? Wouldn't it be a little better to have the example <package>/_version.py? Then users could do from ._version import version as __version__, or whatever, and not be affected by changes to the output template as much.

I'm asking since I'd normalized on using version.py everywhere (Scikit-HEP developer docs, scikit-hep/cookie, and lost of our packages) since it was the way the example had it, but now another developer is unhappy that it's public. - link: https://scikit-hep.org/developer/packaging#git-tags-official-pypa-method

[blazing lantern]

Probably should send another email, else we think about taking over or create simpletoml (like simplejson)

[high stone]

Orthogonally related to both. 😅

[visual epoch]

Hi

[warm isle]

hey, so I've been digging around the legacy PyPI codebase trying to find the first time the JSON API was introduced. And I can't find it, any pointers?

[lean lake]

https://github.com/pypa/pip/issues/10034

[high stone]

Yea, I noticed.

[lean lake]

What does knowing the date help you work out? How accurate you hoping to be?

[warm isle]

Just wanted to mention when JSON API was first introduced. A year is good enough.

The earliest I found was I think ~2014

[woven plover]

Good point, let's change it

[woven plover]

@merry rune can you open a issue /pr so I don't loose it, im outside atm

[lean lake]

To get most accurate, probably a discuss/email question asking Donald, Ee, Dustin would prob know or where to look. The old code is in a repo somewhere. I just don't know where

You say "digging around" - You been commit message hunting already tho?

[merry rune]

Of course, I'll start with an issue.

[warm isle]

I couldn't find the JSON API related code in the legacy codebase. Found some issues copied from bitbucket but not the actual pull requests.

[lean lake]

Yeah, I've never looked at it. Only touched warehouse

[high stone]

It's not on the legacy PyPI -- only in the new Warehouse codebase.

[warm isle]

X-PyPI-Last-Serial - what are the expectations of this header?

My understanding is that the serial is a monotonically increasing integer sequence that is unique for each pypi.org project and that changes every time the project is updated - a new version is published or a new release file is added to a version? It can be used for caching purposes.

Am I missing anything?

[lean lake]

Did legacy include it from warehouse?

[high stone]

No, it just didn't have it.

[high stone]

I'm not sure there's any guarantees around X- headers?

[lean lake]

Na. That's it. based of pgsql IDs I believe under the covers. But just needs to be unique and incrementing

[lean lake]

We're trying to write the PEP that changes that 🙂

[high stone]

I mean... We don't have to encode all the implementation details.

Just the smallest useful subset.

[lean lake]

I agree. But clients use this header I'm pretty syre.

*sure

[high stone]

I mean, the simple API doesn't say anything about the HTTP headers, just the content. I don't see how the JSON API would be significantly different.

But, hey, happy to be wrong.

[warm isle]

last_serial is part JSON API response - whether the header is included in the documentation, the property of the response should be described in a way that will be sufficient for external indexes to implement

[obtuse holly]

Hi :D

[lean lake]

Amazon AWS savvy people (I live in a $big company world) is there a free tier to test shit with or will I have to pay? I want to test setting up and fixing bugs on https://github.com/pypa/bandersnatch/pull/886 (Adding s3 support to bandersnatch)

I know I could probably ask the FB External Cloud team (we have some DR and crap on AWS), but trying to see if I can not abuse work first

I'd guess all I really need is a test s3 bucket and an instance to run bandersnatch from.

[spiral urchin]

don’t know about free tier but there are tools to spin up a fake AWS locally for testing

[lean lake]

How does on some that? I’ll google.

*one do some of that

[keen yew]

Look up LocalStack, MinIO and moto

[golden falcon]

@high stone I'm wondering, with installer, how does one know which scheme to pass to SchemeDictionaryDestination? The documentation shows sysconfig.get_paths(), but I'm thinking it should always be the distutils one, since the wheel spec is built on that? I think the major difference is that distutils has headers iso of includes so the installation of any wheel which includes ( :) headers in its data folder would fail with sysconfig. Also, should installer verify that the destination paths are valid before commencing the installation?

P.S. Thanks for all your work on installer, it's great!

[tidal kiln]

don't use distutils

it's going away

and distutils should now use the sysconfig paths anyway

https://github.com/python/cpython/pull/24549

😅

[golden falcon]

But includes in wheels are placed in a folder called headers mirroring the distutils scheme. installer keys the scheme dictionary using the folder name. If you use sysconfig, which does not have a headers, it'll key error. Try installing something with headers in it, e.g. greenlet to see what I mean

[tidal kiln]

probably because they used to use distutils

doesn't look like they use it anymore

the equivalent scheme would be include IIRC

maybe @glass sand could provide more context

[golden falcon]

If by "they" you mean greenlet, I don't think it's got anything to do with them, it's in the wheel spec.

[tidal kiln]

the spec specifically calls out include

https://www.python.org/dev/peps/pep-0427/#file-contents

it only says the schemes were initially taken from distutils

[golden falcon]

Interesting, further down it says: "Move each subtree of distribution-1.0.data/ onto its destination path. Each subdirectory of distribution-1.0.data/ is a key into a dict of destination directories, such as distribution-1.0.data/(purelib|platlib|headers|scripts|data). The initially supported paths are taken from distutils.command.install."

Anyway, the wheel package is producing wheels with "headers", not "include".

[tidal kiln]

given that distutils is deprecated

I'd say open a issue with the upstream

if the scheme isn't in sysconfig, you could optionally also look to distutils and emit a warning

[golden falcon]

See this very old wheel issue: https://github.com/pypa/wheel/issues/53

Apparently wheel did use sysconfig originally

[glass sand]

I have very little context to add, unfortunately. But I welcome and encourage to unearth and resolve issues like these early in the deprecation period.

[obtuse holly]

Why is it going away and what's replacing it?

[tidal kiln]

setuptools

https://www.python.org/dev/peps/pep-0632/

[mental bridge]

interesting server, but what is the point?

[idle egret]

packaging 😄

[obtuse holly]

Is that in the stdlib?

[tidal kiln]

no

see the PEP I linked, it explains the motivation

[obtuse holly]

hmmm

if distutils is removed, how can setuptools be installed

will have to read it

[tidal kiln]

it uses itself

[obtuse holly]

Very confused, okay

[high stone]

@obtuse holly it's kinda similar in concept to how gcc compiles.

[tidal kiln]

you run setuptools from source to install itself

[warm isle]

Hi all, I opened a discourse thread for the JSON API PEP proposal: https://discuss.python.org/t/pep-rfc-python-package-index-warehouse-json-api-v1/9205
I'd appreciate your input!

[lean lake]

I second we need help here. I tried with some others back in 2018 to implement a new API and offer that, move tools then deprecate the current JSON API and that failed all due to trying to get the maintainers of PyPI to agree / say yes to something.

So this time trying the PEP, then implement route. So please get around it. Also trying to move the current JSON into a schema and version it calling it 1.0 so people who need can pin there. Then we start making changes ...

[visual epoch]

Hi

[visual epoch]

Yes unfortunately there is an inconsistency in the PEP.

[golden falcon]

I suppose installers will just have to duplicate the "include" key at "headers" post-distutils, should probably make a note of it somewhere

[visual epoch]

we were supposed to maybe add more paths later but that hasn't been figured out.

[visual epoch]

We could talk editables here instead of in off-topic

https://github.com/dholth/setuptools_pep660/blob/master/tests/00_package_dir/setup.py "solves" the importing your package before it's installed problem by putting "b" in the "a" directory

Recall that a package is a directory with python files and a module is the individual python file and it's too much to remember the distinction between packages modules and distributions all the time.

not surprisingly setup.py develop doesn't support 'a' in 'b' mode, it always finds the best root directory to point a .pth to - and it seems reasonable to encourage sane package layouts

Next time I want to distribute some bare .py files though I'll definitely provide the "" package (root) and put them in a src directory, and avoid the annoying py_modules parameter entirely.

[golden falcon]

Okay, but package_dir does not distinguish between a = b and a = b/a. The latter does not change the name of the package, and it's the more obvious way to map a package in "src" to a package in the distribution, e.g. {"foo": "src/foo"}

[visual epoch]

Yes 00_package_dir is obviously wrong but distutils supports it.

[golden falcon]

Well, it's a terrible idea to allow renaming the package but I wouldn't say it's wrong

[visual epoch]

I agree with that. It is not a moral question.

Do we care if 'improved' editable mode supports it?

[golden falcon]

I think we should support sourcing packages from multiple directories. We could error if the package names are different, there's no way to make that work with a .pth file anyway (sans import hackery). We definitely shouldn't put a folder on path when the names don't match like develop does now.

[visual epoch]

Some people used to have collections of all the setup.py's on pypi plus extracted metadata...

It would be nice to see the range of values for packages and package_dir arguments.

[visual epoch]

@layday I got the prototype working at https://github.com/dholth/setuptools_pep660/blob/master/src/setuptools_pep660/editable_wheel.py

[lean lake]

😮

@mortal shore in the house

[woeful wharf]

The Warehouse crew is stepping in, it seems 😄 I'm thrilled to join the monotreme discord club ❤️

[lusty quarry]

one step closer to finishing Hatch v1 (rewrite): publish command completed, no longer using Twine 😄

[lean lake]

wonders if we'll get Ee

That's not annoying ...

[visual epoch]

@lusty quarry tell me about pep660 in hatch

[lusty quarry]

@visual epoch here's a minimal config:

[build-system]
requires = ['hatchling']
build-backend = 'hatchling.build'

[project]
name = 'foo'
dynamic = ['version']

[tool.hatch.version]
path = 'src/foo/__about__.py'

# The settings include/exclude/packages may be a
# comma-separated string or an array of strings
# 
# - include/exclude are Git-style glob patterns
# - packages are relative paths to Python packages
#   and the distribution path will be collapsed to
#   only include the final component
#
# If all 3 are omitted then every target has its
# own guessing logic
[tool.hatch.build.targets.sdist]
exclude = [
  '.github/',
]

[tool.hatch.build.targets.wheel]
packages = ['src/foo']

[visual epoch]

Neat

[lusty quarry]

just upgrade pip to the PoC https://github.com/pypa/pip/pull/8212

[visual epoch]

I know that part already 🤣

[woven plover]

@high stone @mortal shore question for packaging - are there any considerations for providing a mypyc based speed up version of it? i was wondering about minimized load times for the version parsing and general usage (the less expensive, the more easy it would be to standardize on having the version metadata written out by setuptools_scm be based on actual version objects, alltho its a general pain that there is no way to serialize/deserialize the normalized metadata (and to have some intentional de-normalization for aligning date based calver)

[high stone]

None yet -- Could you file an issue for this? This sort of stuff should really be archived in the issue tracker, where it's possible to search / reference it easily. :)

[woven plover]

https://github.com/pypa/packaging/issues/441
https://github.com/pypa/packaging/issues/442
https://github.com/pypa/packaging/issues/443

[high stone]

Thanks! I'll respond today / over the weekend. :)

[lyric hedge]

If anyone with triage perms for packaging.python.org could close this issue that would be awesome 😄 https://github.com/pypa/packaging.python.org/issues/617

[high stone]

Done!

[fresh cove]

Hiya!

koobs from FreeBSD here

Python Team

[fresh cove]

Would like recommendations/guide on the simplest way to directly run a python invocation (using build / pep 517 features) that would return a grouped list of dependencies for a package

I imagine this would be something like, or somewhere build uses packaging to process them (dependencies)

[lusty quarry]

@fresh cove just direct or transient too?

[fresh cove]

direct is fine

we would run it for every python package/port in freebsd

we want to be able to quickly qa what a package declares (with specifiers) and a port *_depends on

we're in a sad state where many maintainers dont correctly, precisely or completely declare depends or their specifiers

[lusty quarry]

https://www.python.org/dev/peps/pep-0631/

[fresh cove]

yeh, i understand the peps/meta, just trying to figureout the 'correct build invocation' to output them

from an extracxted sdist (which is what we use) or repo checkout of a package

[lusty quarry]

just use a toml lib

[fresh cove]

I got close to it with setuptools with this: https://bsd.to/DBng/raw

ofek needs to support any python package (whether they use pyproject, setup.py or otherwise)

which is why i need it at (i assume) the 'build' (package) level

as it supports the 'legacy' setup.py, and pyproject (it depends on toml and packaging)

[lusty quarry]

ah, then I'd install first in a virtualenv

[fresh cove]

um

i dont think its really related to venvs or not

im just looking to run a python -m <something) or python script command (ive been told the 'build' package is the future) to output a list of dependencies and their version specifiers, ideally grouped by dependency type

[golden falcon]

There's no command that'll do that for you but you can use build to inspect a package's dependencies, either by building a wheel or 'preparing' its metadata

https://pypa-build.readthedocs.io/en/stable/api.html#build.ProjectBuilder.metadata_path

[lusty quarry]

@fresh cove to read wheel:

import email
import zipfile

def get_metadata_from_wheel(path):
    with zipfile.ZipFile(path, 'r') as zip_archive:
        dist_info_dir = ''
        for path in zipfile.Path(zip_archive).iterdir():
            if path.name.endswith('.dist-info'):
                dist_info_dir = path.name
                break
        else:
            raise Exception(f'Could not find the `.dist-info` directory in wheel: {path}')

        try:
            with zip_archive.open(f'{dist_info_dir}/METADATA') as zip_file:
                metadata_file_contents = zip_file.read().decode('utf-8')
        except KeyError:
            raise Exception(f'Could not find a `METADATA` file in the `{dist_info_dir}` directory')
        else:
            return email.message_from_string(metadata_file_contents)

[fresh cove]

@lusty quarry appreciate that. definitely aware of metadata (post build stuff), need to grab it from source (sdist / repo source)

however that is defined (which is why id need to use 'build' (which uses 'packaging')

im sure theres a get_deps() like method somewhere

ill trawl the sources

like, im sure, that something out there (maybe pip) uses 'build' methods to determine the deps it needs

just need to identify what that is

[lusty quarry]

@fresh cove the issue is setup.py is dynamic so you'll need to build the thing or mock setuptools.setup

[fresh cove]

by build methods i mean the 'build' package

@lusty quarry yeh, so i have been just told about 'build' which provides a unified interface to this, with setup.py as the legacy option

@lusty quarry i understand that in certain cases, since at least for setup.py, that its dynamic, that certain things may not be derivable

@lusty quarry but again, i got close using setuptools.distribution in https://bsd.to/DBng/raw

[tidal kiln]

do you only need the dependency strings, or also need resolution?

[fresh cove]

but 'build' is the 'new way'

define resolution?

as in, some string that 'satisfies' it ?

i just need the declarations

[tidal kiln]

yes

[fresh cove]

like

[tidal kiln]

okay

[fresh cove]

test:

foo:<specifier>

bar:<specifier>

setup:

[tidal kiln]

yeah

[fresh cove]

baz:<specifier>

thats it

ideally specifiers and markers

our goal is to be able to do make checkdeps in all python ports

to check our *_DEPENDS against python ones

[tidal kiln]

hum

[fresh cove]

for name matching, and version spec matching

where make checkdeps runs some python command

happy to have that depend on build or packaging, or whatever

[tidal kiln]

https://github.com/pypa/build/issues/301#issuecomment-863276790

[fresh cove]

https://github.com/pypa/build/issues/181

would seem to be the same thing

perhaps with a bit more smarts (derivation/resolution), where we dont necessarily/strictly need it

"This is up for discussion. People have been using pep517.meta.load for this, so it could be just that exact behavior. "

mmmmm

[tidal kiln]

yeah, there's no reliable way

[fresh cove]

o_O

[tidal kiln]

you can read pyproject.toml but that only works if the backend implements PEP 621

which is optional

https://www.python.org/dev/peps/pep-0621/

[fresh cove]

"I can also see a good use case of this for a project like https://github.com/Arkq/flake8-requirements so it can get dependencies of a project using a single common source, regardless of whether the project uses setuptools, poetry, flit, PEP621, etc."

i understand the sources of the 'no reliable way'

[tidal kiln]

the proper way to would be to ask the backend for the actual metadata

[fresh cove]

but presumably build / packaging / pep517 et al aims to uh ...

[tidal kiln]

which is what the link I sent above does

but this involves invoking the backend

[fresh cove]

i understand that each backend might have its own format

but if they all use build (which build aims to be)

how are/can these not be provided/inspected/seen by build?

[tidal kiln]

it's not just that, dependencies could be dynamically injected

by the backend

[fresh cove]

what things are intended to use build now or in the future?

presumably pip and future tools, no?

[tidal kiln]

I'd say so, yes

[golden falcon]

is building a wheel or preparing its metadata to inspect a package's dependencies a showstopper for you?

[fresh cove]

no

but, ideally not

c extensions are a problem, whole deps chains

[golden falcon]

then have you tried metadata_path?

[fresh cove]

sdist / repo sources

we wouldnt mind if this misses 'dynamic deps, or whatever'

so supplememntal, what are the classes of deps that can be done dynamically, any all?

[tidal kiln]

note that metadata_path will only perform the build if the backend does not implement the metadata generation hook

setuptools does for eg

so you won't actually build anything

we will just ask it for the metadata

we only build if the backend does not provide this interface

[fresh cove]

so if pip et al is supposed to or going to use build and builds goal is what its documented as, that means that either pip will need to have support for every build plugin/backend to grok its deps to do stuff with them, or not

and if it needs to grok all plugin formats, / whats build for?

[tidal kiln]

why do you think pip needs special handling?

like, where and for what

there are two things here! 1) build dependencies, and 2) runtime dependencies

which ones are you looking for? I was talking about 2)

[golden falcon]

wheel metadata is standardised, pip does not need to have knowledge of any backend

[fresh cove]

something is going to fetch deps, wherever/however theyre declared yeh?

[lusty quarry]

@fresh cove build is a "frontend", it merely invokes whatever the backend is defined as in build-system.build-backend

[fresh cove]

either pip via build is going to do it

or pips going to do it

and build builds wheels (alone right?)

without pip

[tidal kiln]

I think you are talking about 1)

[fresh cove]

so how does build build wheels without processing and knowing about all deps required to do so?

[tidal kiln]

those are simpler

and do not require any build

[fresh cove]

so whats the distinction in terminology here

re deps

im talkingt setup_requires, install_requires, tests_require, extras_require (whatever theyre called now)

to the extent they 'can' be knowsn (dynamic foo aside)

[tidal kiln]

there are the deps that you need to build the package, and the ones you need to run the package

okay

[fresh cove]

where setup -> BUILD_DEPENDS, install -> RUN_DEPENDS, tests -> TEST_DEPENDS and extras is optional ones

[tidal kiln]

let's take a step backwards

[fresh cove]

yep, lets 🙂

[tidal kiln]

what are you trying to achieve?

[fresh cove]

1. freebsd builds python packages from sdists or repo sources

2. we create packages from these

3. we need to declare depends

4. deps are specified in a billion different ways by each upstream

[tidal kiln]

okay

[fresh cove]

5. we want to after fetching sources, compare our declared *_DEPENDS with what the upstream declares

without having to know about 50 billion formats and ways (requirements, setup.cfg, build, toml, setup.py, pbr, something else, poetry)

[lusty quarry]

[build-system]
requires = ['backend']  # build deps
build-backend = 'backend.build_api'

[project]
name = 'foo'
dependencies = []  # runtime deps

[fresh cove]

6. i assume build / packaging / toml / et al from pyca is useful / designed in this regard

[tidal kiln]

so, all that is standardized

[fresh cove]

it is in spec

i get that packages declare deps in toml's, or other 'standard' ways

[tidal kiln]

right

[fresh cove]

not everything uses tomls, not everything uses pbr, not everything uses poetry, not everything uses setup.py not everything uses setup.cfg not everything uses setuptools, not everything uses distutils

so build, as its been described to me, and per its docs

aims to supplant all that distutils/setuptools/setup.py mess

or at least in terms of 'the unified interface thing'

and add support for backends, so it doesnt need to be the 'one tool that does everything'

version specifiers, markers, et al are now 'standardized' (additionally in the sense that no matter what tool is used these days, these are fairly standard in all)

because they end up being processed by pip (or setuptools or whatever), so they have to be correct

so, these packages have standard dependency declarations, but declared in a bajillion different ways (read lines from requirements.txt, whatever)

im still on the same page yeh?

[tidal kiln]

yes

so

the way you get the build dependencies is as done in https://github.com/pypa/build/blob/main/src/build/__main__.py#L80-L83

[fresh cove]

awesome

ok, question

[tidal kiln]

and the runtime dependencies, https://github.com/pypa/build/issues/301#issuecomment-863276790

[fresh cove]

i read build supports plugins, but also supports 'legacy setup.py' things yeh ?

if a build system is not 'specified'

can/will build 'autodetect' build systems based on the presence of certain files and install build plugins (from pypi) accordingly?

[tidal kiln]

if a build system is not specific, it will fallback to invoking setuptools

[fresh cove]

or will it need to be 'explicit'

ok, sweet, this is consistent with my understanding

following that previous link

return set(self._build_system['requires'])

build.build_system['requires'] ?

i think i need to get my terms updated

build_system_requires = setup_requires ?

[tidal kiln]

well

no

kinda

[golden falcon]

Briefly:

• BUILD_DEPENDS -> ProjectBuilder.build_system_requires | ProjectBuilder.get_requires_for_build("wheel")
• RUN_DEPENDS -> ProjectBuilder.metadata_path(tempdir) and use importlib.metadata to read the dependencies
• TEST_DEPENDS -> no substitute

[fresh cove]

get_requires_for_build is the backend additional set, ok

extras?

[tidal kiln]

build_system_requires are the dependencies to run the backend

[fresh cove]

yep, gotcha

in adittion to build_system_requires

[tidal kiln]

get_requires_for_build are the dependencies to run the build, which may need extra stuff

setuptools for eg needs wheel

to build wheels 😛

[fresh cove]

right

but say setup setuptools:sdist

it doesnt, so its an extra,

(declared as such or otherwise)

So, what of extras_require: name:foo[bar] deps ?

as far as the future goes

[tidal kiln]

you get the metadata via the second method

[fresh cove]

right, so the presence of a 'name'

indicates an extra?

[tidal kiln]

and look at the dependency strings declared

[fresh cove]

or is there a specific group/class of them?

a dict/list whatever i mean (the data type indicates the "run" depends type) ?

[tidal kiln]

it will say something like Requires-Dist: some-package ; extra == 'some-extra'

[fresh cove]

right

so this seems like it leds to a helper would be AMAZING

[tidal kiln]

the interface is not great

but yeah

[fresh cove]

if not existing in build, where would be a good palce for it

[golden falcon]

should note that build also has a function for verifying that dependencies are satisfied in the environment

[fresh cove]

with respect to the scope of build / packaging / other packages

yep i saw that

[golden falcon]

https://github.com/pypa/build/blob/5a3c5243a2135236f207da3a4b6401e1561f952d/src/build/__init__.py#L93

[fresh cove]

@golden falcon im delaying assessment of this because venv and system-site packages considerations for us

@golden falcon we need to test packages against dependencies installed by us (in a clean room), but not in a python clean room (or in addition to)

[tidal kiln]

same as we do in arch

[fresh cove]

yup

good to know youre involved in downstream

same problems

[tidal kiln]

yeah 😅

[fresh cove]

that emoji exactly

always.

:]

im forever 7 levels deep yak shaving python foo

[tidal kiln]

ah yeah

[fresh cove]

so, for context

the motivator for this

[tidal kiln]

guess how pypa/build started 😛

[fresh cove]

is we currently use a setuptools wrapped setup.py invocation to build/install sdists

and now, some projects are coming out with no setup.py's

and we have no answer yet

we were considering pip, etc (since they wrap a lot of the underlying mess)

but yeh, hopefully, with build et al

there's a better unified way

@tidal kiln yay.

[golden falcon]

I think arch was using dephell to generate setup.py's for a while

not recommended

[fresh cove]

i think a helper using terms that people are familiar with would help the transition immensely

oh wow.

[tidal kiln]

yes, very much not happy about that

but what can you do

some people are adamant

[fresh cove]

i think literally, and as close as possible (even if its not perfect),. im dreaming of python -m build.helpers.deplist(old-dep-type-name)

🤤

that would be amazing.

even if old-dep-type-names are just aliases to the real/future ones

@tidal kiln does arch have maintainer/qa targets for checking source package version/dep declares against the upstream counterparts (from the source tarball)

?

[tidal kiln]

nop

very much not happy about that either

we have to manually check

[fresh cove]

ok

so

ill get this done with your help

and we can both (all) benefit

we cant do this manual thing any longer

same with tests

it takes so long to review upstream docs on how to invoke tests, and find their deps

such a waste of time

if tox.ini { grep command= }

but not always

test_suite, pytest.ini (wait no! pytest.cfg)

no wait, Makefile.

[golden falcon]

unfortunately, tests haven't been standardised

[tidal kiln]

the arch use-case will be essentially fixed if the wheel installer also checks for the dependencies

[fresh cove]

no wait, def PyTestCommand

[golden falcon]

there was some talk about this buried deep in a build issue

[fresh cove]

@golden falcon what are the known barriers to that?

other than doing it, and people to do it

[tidal kiln]

it's just doing it

[fresh cove]

@tidal kiln we need to define a wheel spec for freebsd too, longer term

@tidal kiln no schema/pep/standard/design-decision problems?

[tidal kiln]

it takes a lot of time to come up with a interface and host discussions

[fresh cove]

@tidal kiln by 'do' we mean 'document it in the pep or related pep' and 'implement it in reference tools' ?

and all dependent tasks

if so, no other barriers is good.

its better than "cant be done, too hard problem, heres a list:"

im going to go grab lunch, thank you so much for the insight @tidal kiln @golden falcon

will idle

[tidal kiln]

no worries

let me know if you need anything else

[fresh cove]

roger that, let me know if i can help with anything

including 'getting more downstreams involved in discussions to back you up'

[tidal kiln]

👍

[fresh cove]

Hmm, so importlib_metadata first talks about it being for 'installed packages'

But later, it says "Distribution metadata"

providing methods to get dependency information via requires('distributionname')

Does that referecen to 'distribution' also refer to or support sdists ?

And does it refer to wheels as in a .whl downloaded from pypi?

Or some other notion of 'distribution' (like the class Distribution)

mmm

def requires(self):
"""Generated requirements specified for this Distribution"""
reqs = self._read_dist_info_reqs() or self._read_egg_info_reqs()

self.metadata.get_all('Requires-Dist')

mm.

We couldn’t find any code matching 'sdist' in python/importlib_metadata

good times 🙂

[golden falcon]

You can load metadata from a dist-info folder, it can be on disk or it can be in a zip file (wheel). If you use metadata_path from build, you'd simply pass its output to PathDistribution (or Distribution.at, they are synonymous)

[tidal kiln]

@mossy pulsar btw https://github.com/python/importlib_metadata/issues/329

also, https://github.com/python/importlib_metadata/issues/328

[fresh cove]

Does anyone know of a standard (pyca) method for packages to support MAKE_JOBS

Or is there no support at all?

[lusty quarry]

for this https://mail.python.org/archives/list/[e-mail]/thread/PUWLS2FS2QGJMYZL4MV5Z6L3YRW7FYNV/

is there a way to force the use of a different compiler? the one it's looking for is not free https://www.ibm.com/products/xl-cpp-aix-compiler-power/pricing

[tidal kiln]

that seems like a build system issue

[lavish rapids]

@lusty quarry did you set CC ?

and CXX to g++ ?

[lusty quarry]

yes

I think it's b/c CPython itself was built with that compiler

[tidal kiln]

sounds like they are looking for the compiler in the sysconfig args

it's fine as the default behavior because compiler can have weird compatibility issues, but if the user sets CC/CXX, that should be honored by the build system IMO

[quartz scarab]

hi all! I'm having an issue making a new PyPI release via twine. In particular, I'm getting the error

"This filename has already been used, use a different version. See https://pypi.org/help/#file-name-reuse for more information.".

I realize the error is pretty self-descriptive, but I've been scouting my project page and I really can't find the target filename in there...so I'm having a hard time diagnosing the root cause.

Is there a way to enumerate the filenames uploaded to a PyPI project? Ideally with a timestamp associated with each?

In particular, I'm trying to upload azure_functions_durable-1.0.2-py3-none-any.whl to https://pypi.org/project/azure-functions-durable/

Thanks in advance 🙂

[mortal shore]

filenames are consumed forever once uploaded

even if you've deleted the file

[quartz scarab]

right, I understand that, but I don't think that file was ever uploaded. Is there a record of all uploaded files that I can double check?

I've also never deleted a file from the project

[mortal shore]

the security history and/or journal page on your project should tell you

https://pypi.org/manage/project/twine/journal/ it'll be an URL like that, just with your poject instead of twine

[quartz scarab]

got it, will review it, thank you 🙂

[mortal shore]

I pulled it up in my admin accont and I see 1..2 being deleted in march

by prananth

[quartz scarab]

when you say 1..2 you mean 1.0.1 and 1.0.2 being deleted?
Also, I suspect that would make sense, that was a previous owner, no wonder I was unaware

[mortal shore]

1.0.2

mistyped

[quartz scarab]

all good, thank you so much for diving in, I appreciate it lots.
I guess we'll have to skip a version in the next release then, all good. Thanks 🙂

[mortal shore]

it looks like 1.0.2 and 1.0.1a1 were the only ones ever deleted that I can see from a quick glance

[woeful wharf]

http://packaging-con.org/

[woeful wharf]

(Just found out about this, I’m not involved)

[torpid veldt]

Has there been any thoughts into making a types only .whl format? Eg for pywin32 so twisted could run the checker on Linux CI?

Like a pyi platform tag?

[golden falcon]

don't types-* packages satisfy this need?

[torpid veldt]

I don't think so, they're installed separately

Will need an additional build stage to strip the annotations from the .py source and move into the types directory right?

[tidal kiln]

but those wheels would also be installed separately

I would need to tell the package manager or install tool that I also want to install the typing wheels in some way

[spiral urchin]

What the practical difference between having a types-* package and a pyi platform tag? The only meaningful difference from what I can tell is you can list the typehint wheels with other distributions, which is purely cosmetic, and platform tagging is the wrong abstraction layer for that IMO.

[round rapids]

anyone know of a good way to install [build-system] requirements (a la pip install -r)?
I see https://github.com/pypa/pip/issues/8049 is still open, and it doesn't look like build supports this yet either.

[mossy pulsar]

For what purpose?

[round rapids]

i want to run static checks on setup.py

(in tox, fwiw)

[mossy pulsar]

generate a requirement file in first command and then pip install in second

you can use a python call to read the toml and dump the list into a test file

which in case of tox I'd recommend doing into the envtmpdir folder

or write a tox plugin that does that for you 🙂

in case of plugin you could even implement the caching

to not keep running the install when the requirements don't change

[round rapids]

seems like a sound idea... not seeing any plugins that exist already. not terribly generic tho. know of any examples of the py logic this needs?

[mossy pulsar]

Not from the top of my mind

But should be simple tomli.loads and path.write_text

[round rapids]

ah... love that toml mimicked the json interface

pip install -r <(cat pyproject.toml | python -c 'import sys, toml; print("\n".join(toml.load(sys.stdin)["build-system"]["requires"]))')

grody tho 😅 any idea if this is planned for pip or build? (i'm not entirely sure that issue refers to what this is trying to do)

[tidal kiln]

not planed for build so far

[round rapids]

my first guess was actually python -m build --no-isolation... it seems like that actually means no deps AND no isolation

[tidal kiln]

yes, because we wouldn't be installing deps on the current environment as a side-effect 😅

--no-isolation will still verify the dependencies though, they need to be present, just not automatic install

[round rapids]

the only part I need! :p

[tidal kiln]

🙂

we also have an api btw

[round rapids]

is ther a distinction between build's API and pep517?

[tidal kiln]

yes, we have an api to provision environments

https://pypa-build.readthedocs.io/en/latest/api.html

build provides a more high-level API

[round rapids]

aha! neat 🙂 TIL

[spiral urchin]

This would be easier after https://github.com/pypa/build/issues/181 is implemented

[torpid veldt]

https://twitter.com/webology/status/1424432003957600259?s=19 is this something that pypa could help with?

✨ Jeff Triplett ✨ (@webology)

@graingert That could make a lot of sense. I don't know that I know the maintainers to suggest it, but if I keep using it, I might open a discussion or issue since they have a note about wanting maintained on the repo.

Twitter · Aug 8, 2021, 6:07 PM

eg pulling in pipreqs to pypa

[mossy pulsar]

"PEP 609 -- PyPA Governance | Python.org" https://www.python.org/dev/peps/pep-0609/

The process described there

[torpid veldt]

Thanks

Uhh if pip-tools were moved from jazzband to pypa under https://www.python.org/dev/peps/pep-0609/#determine-who-is-and-isn-t-a-pypa-member
That would make every jazzband member a pypa member?

[mossy pulsar]

We generally only consider active maintainer

Does every jazzband member actively maintains that repo?

If yes then I suppose yes

[torpid veldt]

Yes technically

[mossy pulsar]

I don't think everyone is active maintainer in practice

So technically I don't think would be an issue

[torpid veldt]

Can someone enable registered users only on the libera #pypa channel?

[tidal kiln]

ask in #python-ops

[torpid veldt]

Thanks will do

[torpid veldt]

^

[woven yarrow]

Is #pypa too noisy on Libera?

I am not an op there in any case.

[lavish rapids]

Jaime from Quansight Labs has compiled a glossary of concepts used across the packaging tools with a tilt towards the scientific python ecosystem (conda and pypi both). https://jaimergp.github.io/scientific-packaging-glossary/ Comments are welcome in the github repo https://github.com/jaimergp/scientific-packaging-glossary

[echo ruin]

(it looks like only @mortal shore and Yhg1s can do this: 10:16:25 Julian| flags #pypa 10:16:25 >ChanServ< Entry Nickname/Host Flags 10:16:25 >ChanServ< ---------------------------------------------------------------- 10:16:25 >ChanServ< 1 dstufft +AFRefiorstv (FOUNDER) [modified 11w 6d 13h ago, on May 19 19:22:50 2021 +0000, by dstufft] 10:16:25 >ChanServ< 2 Yhg1s +AFRefiorstv (FOUNDER) [modified 11w 3d 17h ago, on May 22 15:52:23 2021 +0000, by Yhg1s] 10:16:25 >ChanServ< ---------------------------------------------------------------- 10:16:25 >ChanServ< End of #pypa FLAGS listing., but almost certainly that's just an oversight I assume, probably at this point libera supports the cloak-based access we used to have where all of psf/staff or whatever can moderate)

[torpid veldt]

the [project.urls] section seems to be listing domain names, what does that mean? https://www.python.org/dev/peps/pep-0621/#example

repository = "github.com" can't be right surely it should be repository = "https://github.com/pypa/example"

eg the example URL is missing a scheme and a path

[spiral urchin]

Those are just example values. If you find them confusing, feel free to open a PR and tag the authors for review

[torpid veldt]

does it autofill an "https://" ?

[spiral urchin]

That’s up to the installer. If you (as a locker) want https explicitly, put it in the lock file

[golden falcon]

no, the backend shouldn't autofill anything, the package metadata are supposed to be static (unless explicitly marked as dynamic) and should be transposed verbatim

[golden falcon]

this is the repository from PEP 621 project URLs, not PEP 665

[spiral urchin]

Oh I misunderstood, sorry. You’re right the backend should pass on whatever the user provides and not autofill anything for those columns.

Or maybe if you want to be strict, I guess you can error out if the user does not provide a scheme; that’s between you (the backend author) and the backend’s users

[woven plover]

what would be the correct path to get the wheel/install format updated to have a dist-info folder be generally without the version number in it?

[golden falcon]

starting a discussion on https://discuss.python.org/c/packaging/14 I guess

[tidal kiln]

I want to reboot https://discuss.python.org/t/adding-a-default-extra-require-environment/4898 with a concrete proposal, does it make sense to create a new thread?

[woven plover]

i remember once proposing positive/negative extras so something with a + prefixed in extrs_required would install by default but could be opted out with a -) (i wanted this for pytest to enable opting out of junitxml, unittest support and pylib )

[spiral urchin]

See also https://github.com/pypa/packaging/issues/452#issuecomment-907449456

[tidal kiln]

hum, I am a bit too skeptical of your proposal

perhaps there are some motivations I am missing

but I would just fix the packaging api to be able to take a set of extras to evaluate the markers

[spiral urchin]

fix the packaging api to be able to take a set of extras to evaluate the markers
How is this possible without changing the marker syntax?

[tidal kiln]

by adding some special handling to the extra keyword

[spiral urchin]

That would still require a new metadata version because the redefined syntax will not work correctly on existing tools. And if a new metadata version is required, why not redefine the entire thing to make them “read right” as well.

[tidal kiln]

I am not saying to change the metadata, but the packaging api

[spiral urchin]

What API?

[tidal kiln]

any(a_dep.marker.evaluate(environment={"extra": e}) for e in a_req.extras)

to have a simple way to do that

but like I said, I may be missing some motivations

from my understanding that is the main pain right now

[spiral urchin]

Yes and like I said this is changing the semantic meaning of the markers and incompatible to existing tools, so you need to bump the metadata version even if the markers are unchanged syntax-wise

[tidal kiln]

I don't think you get what I am saying

I am just proposing a better api for that, nothing to do with the metadata itself, just an improved way to plug these things together

[spiral urchin]

Say we do that logic above, and projects will start releasing metadata with something like six ; extra == "foo" or extra == "bar". This will not work on the current pip and users will get the wrong behaviour and be confused. So pip needs to emit a message saying “hey this package is using a metadata your pip version does not recognise, you need to either try another version of the package or upgrade pip”. But the way to convey that is to bump Metadata-Version.

[tidal kiln]

why would it not work? it does work with our current approach

using and is what wouldn't make sense, as it would never be enabled, but or yes

anyway, I am not saying this is a bad idea, just that I am skeptical

[spiral urchin]

It would not work (on old pip) as the project maintainers intend to (on new pip)

[tidal kiln]

okay

just make sure you describe that issue in the PEP then

[spiral urchin]

I understand your concern though; redefining == was one of the choices I considered as well. But I really don’t want to be responding to pip bug reports for this, and bumping Metadata-Version is the only way I can think of to avoid this confusion. And once we need to bump the version, we can “fix everything” instead of doing smaller scoped changes (which feel hacky to me)

But if someone can propose a way to avoid the confusion without bumping Metadata-Version I’d be very willing to reconsider the smaller scope change

[tidal kiln]

sure

my proposal for the recommended/default extras would be to add a Recommended-Extra metadata field, which would ask package managers to enable certain extras by default (eg. pip install a would be the same as pip install a[b,c] if b an c are recommended extras)

this would only ask, package managers would not be required to enable them, in fact, I would recommend testing tools like tox and nox to not do that

[spiral urchin]

So it like of sounds like my Requires-Extra but with a different name? (I think I like your name Recommend-Extra better; the extras aren’t actually required)

What do think think about disabling the recommended extras? package[] is the best I can come up with now

[tidal kiln]

yeah, I specifically did not want to force them, and that name seemed to fit better 😅

I think you would negate them, or do that

or pass an option to disable all recommended extras

[spiral urchin]

We need a syntax for another package to use in Requires-Dist so an option is probably not enough. How does negation work?

[tidal kiln]

a[!b]

[spiral urchin]

Hmm

So if a package has extras b c d and b is the default, does a[c] install both b and c or just c? I’m thinking the logic may get complicated

I think I’ll stick with a[] clearing the defaults and a[c] installs only c (because [] already clears the default) for now and see what people think

[tidal kiln]

I mean

sure

that works too

but I wouldn't want packages to be depending on recommend extras anyway

because the package manager might choose not to install them

[blazing lantern]

It probably makes sense to start a new topic and mention it in the old one based on how long the old one is

[bleak forge]

Hello.

Can somebody again give permission to GitHub for https://github.com/pypa/warehouse/pull/9972 to be tested?

[bleak forge]

What is needed to push review of https://github.com/pypa/warehouse/pull/9972 forward?

[spiral urchin]

Time, I presume. Most of PyPA stuff is pretty short-staffed and Warehouse is no exception.

[bleak forge]

Can participating in fundraising activities change the situation? I am talking about https://gitcoin.co/grants/ which starts in a few hours. This - https://gitcoin.co/grants/3451/add-blake2-into-w3c-sri-spec - is the project I am going to work on, and I can add an application for PyPA.

At the very least the participation can measure how serious is crypto community about supporting important infrastructure projects in Python ecosystem.

The Gitcoin backed itself is partially written in Python https://github.com/gitcoinco/web/ so guys will be glad to see PyPI on the platform.

[bleak forge]

I can open a grant request right now with details to be filled later, so that the fund could be used to review hacktoberfest PRs if PyPA decides to take part.

[bleak forge]

I opened a grant https://gitcoin.co/grants/3537/python-package-index to not miss the deadline before the next Gitcoin Round which starts today. But to be accepted, it needs an approval and participation at least of somebody from PyPA team.

I don't think it is possible to fundraise more than $300000 to PyPI through Gitcoin Quadratic Funding model, but at least it is an experiment that can show if it is a viable tool or not.

[blazing lantern]

People can donate to the PSF to help fund projects like PyPI; https://donate.pypi.org/

[bleak forge]

Yes, they can. And how many people donated specifically to PyPI already?

[bleak forge]

Can somebody give some feedback on improved PostgreSQL initialization for development https://github.com/pypa/warehouse/pull/9993 ?

[fleet oxide]

Hi there , I try to run warehouse repo in local but when I run make run I get this error

.a ../libpng/libpng.a  ../gifread/libgifread.a ../pnmio/libpnmio.a ../minitiff/libminitiff.a  -lz -lm
#15 77.82 npm ERR! Makefile:100: recipe for target 'optipng' failed
#15 77.82 npm ERR! make[1]: Leaving directory '/tmp/ead5085c-8807-4bec-9c6b-6febca484861/src/optipng'
#15 77.82 npm ERR! Makefile:14: recipe for target 'install' failed
#15 77.82 npm ERR!
#15 77.82 npm ERR!     at /opt/warehouse/src/node_modules/execa/index.js:231:11
#15 77.82 npm ERR!     at runMicrotasks (<anonymous>)
#15 77.82 npm ERR!     at processTicksAndRejections (internal/process/task_queues.js:97:5)
#15 77.82 npm ERR!     at async /opt/warehouse/src/node_modules/optipng-bin/lib/install.js:18:4
#15 77.93
#15 77.93 npm ERR! A complete log of this run can be found in:
#15 77.93 npm ERR!     /root/.npm/_logs/2021-09-09T12_56_20_902Z-debug.log
------
executor failed running [/bin/sh -c set -x     && npm install -g npm@latest     && npm install -g gulp-cli     && npm ci]: exit code: 1
ERROR: Service 'web' failed to build : Build failed
make[1]: *** [.state/docker-build] Error 1
make: *** [build] Error 2

Has anyone seen this ?

[tidal kiln]

we probably should create a channel for warehouse 🙃

[spiral urchin]

I thought Python packaging is fucked up and npm is doing all the right things and makes things so simple /s

[golden falcon]

apropos of nothing, I had TS hang the other day and tried to exclude the latest release, apparently the only way to do that with npm is to use a range like >=1 <2 || > 2.1, no !=2.0, which I found very weird

[spiral urchin]

FWIW I always feel PEP 440 should support OR expression as well, so things kind of even out

[blazing lantern]

I don't know, but major grants have come in before, e.g. Mozilla funding the completion of Warehouse so it could be launched

[bleak forge]

npm fund is awesome. )

[bleak forge]

Institutional sponsors surely get a lot of benefits to incentivize them to donate (https://www.python.org/sponsors/application/) but for individual supporters there are not much benefits. I can not brag that I've supported PSF like 5+ years ago, because https://www.python.org/psf/donations/ page doesn't keep history.

Gitcoin can help me maintain public profile of projects that are important to me, so the company that want to hire me can access that they'll have better chances in doing so if they already support the same projects that I do.

Quadratic Funding (https://wtfisqf.com/) also allows to use matching fund without being Google, Twilio or Microsoft employers (https://www.python.org/psf/donations/matching-gifts/), increasing the ratio according to amount of people expressed interest.

And the last thing - people can see how much was donated directly to PyPI.

[golden falcon]

So the full amount's raised thru individual donations? This would make more sense if organisation x donated $y to FOSS and then it was divvied up "quadratically" by making token donations, say, up to $10, with the donations themselves being allocated 1-to-1. If I wanna donate to PyPA, or start a crowdfunding campaign for PyPA, why would I want the target to be determined by how much other people have donated and to whom? It might've made more sense if you were choosing which project to pursue, e.g. from the list of PSF fundables, at the time of donation - that is, if the projects were to fall under the same umbrella but not as much in a generic donate-to-FOSS fashion. I suppose it depends on whether Gitcoin can reach critical mass, if campaigns are started regularly (monthly or quarterly), that sort of thing.

[bleak forge]

Yes, the matching fund is distributed according to individual donations only, using the quadratic formula (more people - larger share). The specific matching logic such as 1-to-1 matching can be implemented too outside of Gitcoin - by exporting information about donations to specific projects. Maybe it is even possible to have a specific tag and build up the matching fund for this tag only - let me ask. It is already Gitcoin Round 11 with the first round started in January 2019 and it is being done regularly every few months.

[bleak forge]

Answering the question about dedicated matching fund for Python projects. Right now the matching fund is split into several categories that interest people the most. It might be possible to have a Infra-Python category if a certain amount of people on the platform expresses the interest.

[keen yew]

I have no idea how it works, but Gitcoin is currently the largest source of urllib3 income
GitHub Sponsors will probably surpass it at some point

[bleak forge]

People are already started contributing to https://gitcoin.co/grants/3537/python-package-index even given that the application is not confirmed. It will help people gain more trust if he @pypi team could confirm participation in Gitcoin by tweeting the text

I am verifying my ownership of Python Package Index on Gitcoin Grants at https://gitcoin.co/grants/3537/python-package-index. 🎊🔥⚡⚡⚡🎆

And of course I am waiting for people from PyPI to register on gitcoin.co website so that I could add you to the team.

[blazing lantern]

We can't unilaterally do that. Donations are handled by the PSF and they have their own legal requirements, procedures, etc. for accepting donations. You will have to convince the PSF to accept the gitcoin donations

[bleak forge]

Can you clarify from PSF if Gitcoin Grants == Donations as defined by their laws? From what I know tokens are not considered to be money in U.S. and there are no papers that regulate what and how you can accept. Besides I am not asking for everybody at PyPI or PSF to start using crypto. I am asking just a few folks to support the idea to run it as an experiment. If there are doubts about Ethereum in US, then the team that physically reside in a different country may still benefit from that support.

[blazing lantern]

@bleak forge I'm not a lawyer or accountant, so I don't know how Bitcoin would be viewed from a US non-profit perspective in terms of money received

[woeful wharf]

https://twitter.com/peter_szilagyi/status/1437646118700175360?s=21

Péter Szilágyi (karalabe.eth) (@peter_szilagyi)

Between the 3 Sept and 10 Sept, secure env vars of all public @travisci repositories were injected into PR builds. Signing keys, access creds, API tokens.

Anyone could exfiltrate these and gain lateral movement into 1000s of orgs. #security 1/4

https://t.co/i23jFzAjjH

Retweets

1424

Likes

1794

Twitter · Sep 14, 2021, 5:15 AM

I know that some pypa projects use Travis, so:

• time to move away
• and investigate if this might have been exploited ?

[bleak forge]

I understand that you're not a lawyer or an accountant, and you've already said PyPA can not decide to accept ETH tokens without PSF, so can you ask PSF on behalf of PyPA (and interested parties) for permission to accept ETH from Gitcoin Grants? There are already 9 people expressed support for PyPI through that platform, and they will sure appreciate some heads up.

[blazing lantern]

I don't have the time or knowledge to answer the inevitable questions that will come up about how Gitcoin operates, but you can email [e-mail] and let them know about Gitcoin Grants and see if you can answer their questions

[bleak forge]

I would prefer the thread to be public for reference, but if there is no other way, I will have to do this.

[bleak forge]

What is the PostgreSQL version used in production on https://pypi.org ?

It looks weird that tests are using 10.1 and not even 10.18

If production is using 10.18 I can bump it for tests too.

I wouldn't ask if this info was exposed through https://pypi.org/varz endpoint, for example like NATS does it https://docs.nats.io/nats-server/configuration/monitoring#general-information

I can bump DB for tests in https://github.com/pypa/warehouse/pull/9993 if somebody is willing to review it.

[tidal kiln]

we can move this discussion to #pypi

[woven plover]

do we have any flit alternative that would be willing to integrate setptools_scm - i don't really want to reinvent the wheel there

@golden falcon i just took note you once made a hack about it, i wonder - would it be possible to make a version of that which works as a build dependency?

[golden falcon]

yeah, you could just publish the in-tree backend on PyPI and depend on it iso flit-core

[woven plover]

its not sanely usable for generic usage as its pretty much not allowing configuration atm

[golden falcon]

Well, get_version doesn't support loading the configuration from pyproject.toml. I suppose I could read the Configuration.from_file then pass all the options back as arguments to get_version for it to reconstruct the configuration

Ideally _get_version would've been the function that was exposed and not get_version

[tidal kiln]

I think I should probably integrate it in trampolim

currently I am doing it manually

[woven plover]

@tidal kiln im currently not done with the git full support tho (i think i'll start with just tag archive support) - however likely not before the end of the next week

[tidal kiln]

cool

[woven plover]

@tidal kiln can you add a easy way to bootstrap that configuration form a local file as well (then i would start to migrate a lot of pytest and setuptools_scm itself to trampolim (i dont intend to restart gumby elf)

[tidal kiln]

what do you mean exactly?

a way to bypass the version mechanism?

[woven plover]

currently im under the impression, that if i wanted to use trampolim for setuptools_scm itself, i'd have to be able to bootstrap the integration without entrypoints

but i realized that it can just install older versions

[tidal kiln]

I think you could set backend-path to use the local setuptools_scm

[woven plover]

but then it wouldnt have the entrypoints

aka id have to configure the setup differently

[tidal kiln]

I don't think I need to rely on the setuptools_scm entrypoints in trampolim, do I?

[woven plover]

current core functionality of setuptools_scm does

[tidal kiln]

okay

[woven plover]

but i could change that a bit

[tidal kiln]

anyway, we could always add a way to bypass stuff for bootstraping in trampolim

just need to figure out the details

I want this to be bootstrapable without older versions because otherwise it would be hell for distro packaging

some distro packaging anyway

[woven plover]

true, - in that case it would be easiest to have setuptools_scm implement the entrypoitns different, and only use them for plugins, not for core function

[tidal kiln]

yes, I think the core functionality -- as in API -- should not depend on entrypoints

in trampolim I would only need to call the API to calculate the version, I don't see why that would have to depend on entrypoints

[woven plover]

the entrypoints map scms/support files to their parsers

structurally its more elegant to just consistently use entrypoints

[tidal kiln]

hum

[woven plover]

but bootstrapping wise, thats not accounted for

[tidal kiln]

well

actually

nevermind

[woven plover]

the most evil thing to do would be to publish a setuptools_scm_bootstrap_hack package that just ships the entrypoints and/or to havea bootstrap script that creates a editable install which trigges the metadata creation

[tidal kiln]

setuptools_scm would be a optional dependency of trampolim

so we don't actually have to need it to build trampolim itself

[woven plover]

would it be possible to jsut have plguins for trampolim?

btw, does trampolim have helpers to make local build backends that use it?

[tidal kiln]

not really, but I just published the PEP 621 parser as a standalone package

and was thinking of writing a library to help building build backends

[woven plover]

btw, would you mind if i added a private package to that to update a setuptools distribution object with the pep621 metadata ?

[tidal kiln]

but trampolim actually gives you a lot of control over the package

because you have a hook to modify the package source

like, run arbitrary code and modify stuff

use case is generate code, etc.

[tidal kiln]

I don't follow 😅

[woven plover]

setuptools hasa hook to finalize a distribution in recent versions, so adding a path to use pyproject metadata while migrating away from setuptools would be a help

[tidal kiln]

sure

feel free, but please only rely on public API

https://pypi.org/project/pep621/

[tidal kiln]

real world example: I have a protocol spec in XML and want to automatically generate code with structures to parse the data

https://github.com/openinput-fw/openinput-protocol/blob/main/openinput-device.xml

pretty much the same thing as wayland

[woven plover]

i see, i have a number of similar use-cases

but i tend to commit the generated code to account for generator changes over time (saved me more than once from bugs)

[tidal kiln]

right

I want to integrate a test runner into trampolim in the future

so that you can have an automatic workflow for build > test > publish

[woven plover]

at that oint it would be terrifying to use

[tidal kiln]

oint?

[woven plover]

eh point

[tidal kiln]

ah

😄

[woven plover]

better to integrate well with CI/CD than to invent yet another mini version of it

anyway, calling it a night, thanks for the input ^^

[tidal kiln]

depends on scale, this definitely would not be for everyone

no worries

let me know if there's anything else

I should also go do my things 😅

[bleak forge]

The problem with most Python projects that people need to be very curios about standards to click three pages away and then type pep621 into search to understand what all these projects about.

Just clicked through three pages and decided that it it not what I looking for anyway. )

[servers]

  # Indentation (tabs and/or spaces) is allowed but not required
  [alpha]
  ip = "10.0.0.1"
  dc = "eqdc10"

  [servers.beta]
  ip = "10.0.0.2"
  dc = "eqdc10"

Just found out that TOML indentation is not useful. Like in the example above [alpha] will appear at the top level of parsed file. That's why each nested in project metadata needs to be denormalized as [project.urls] etc.

[spiral urchin]

It’s not syntactically significant, but certainly “useful”

[bleak forge]

Too easy to make mistake.

[spiral urchin]

Only if you come with the assumption indentation is significant.

[bleak forge]

I come from Python.

[golden falcon]

if anybody could make a YAML 2.0 with explicit types and with fewer than 365 different ways to express strings, that would be my dream configuration language

[woven plover]

i want a configuration langue that actually has macros and includes + libraries

[golden falcon]

https://dhall-lang.org/

[woven plover]

that one is interesting, im not sure if sad or nice that it has a compile step

[echo ruin]

googles for "dammit firefox hide the toolbars in full screen"

_encounters a wild @glass sand https://support.mozilla.org/en-US/questions/1265173_

[bleak forge]

If YAML didn't have types at all, leaving type conversion up the app, then maybe PyYAML parsers would not be 200k compressed. That's just too much for a config parser.

[tidal kiln]

@woven plover is there a way to do tell pytest to rewrite the asserts of my helpers?

like pytest.register_assert_rewrite

hum, I think the fancy asserts are broken in the tests themselves, it has nothing to do with the helpers

I don't really understand

https://paste.xinu.at/83BFao/

[woven plover]

@tidal kiln just took a notr, will give it a small dig once the toddler sleeps

[woven plover]

@tidal kiln yikes, they realy should implementa assert repr hook forthat thing,

[tidal kiln]

well, I am the one implementing these tests

the assert wrappers are semi-temporary while we figure out the test api

this is the code btw https://github.com/conda/conda/pull/10886

but I thought since it was a simple set comparison pytest would be able to do the fancy asserts

[woven plover]

i think we dont have set contains helpers for pretty yet (we have equals %& co)

[tidal kiln]

hum

[woven plover]

@tidal kiln yeah, its not yet implemented

[tidal kiln]

I implemented it locally, but still need to fix the tests

some break

[bleak forge]

Are any of PyPA projects participate in https://hacktoberfest.digitalocean.com/ ?

[woven plover]

hacktoberfests are horrendous, so much driveby for maintainers ^^

[bleak forge]

This point definitely needs to be heard.

[golden falcon]

build participated last year I think

the build repo is still tagged with #hacktoberfest actually

[tidal kiln]

yes

I don't think we have much to do though

but I'll be accepting contributions

[woven plover]

@tidal kiln btw, does trampolim do repeatable builds or will it do ?

[tidal kiln]

there's some code that takes that into account

but I haven't verified that, and don't have tests for it yet

but it's 100% on the plan

[woven plover]

@tidal kiln good, i want to move pretty much all of the opensoruce proejcts i deal with to something like it soon

@tidal kiln btw, do you have a hack for editable installs ready?

[tidal kiln]

not yet

[woven plover]

for some extra context, i want what https://github.com/RonnyPfannschmidt/gumby_elf did set out to be, bu preferably not managed by me ^^

[tidal kiln]

I haven't had much time to implement all these things yet