#setuptools

1 messages · Page 2 of 1

versed lily
#

maybe because that's an ancient thing?

keen ibex
#

The project is 14 years not updated, might be broken. Try pip install ., but I am not hopeful

versed lily
keen ibex
#

Fork it and fix issues. Judging by the age, it is probably focused on Python 2, so code might even be incompatible with modern Python versions

median crest
#

yeah, at that point 3.x barely existed and had quite poor adoption. and nobody distributing their code was thinking in terms of the "ecosystem" we have today. they weren't even thinking in terms of "setuptools can release new versions and that could possibly cause a problem"

#

however, I don't see a license here (let alone any PyPI distribution), so....

#

to do things properly you'll have to start by emailing the author

versed lily
#

idk do those developers active in Github

empty sun
#

There's no hurry as I've already been sitting on it for a while. Just wanted to mention that next time there's a pypa/distutils sync to pypa/setuptools, that'll unblock removing a bunch of jank from typeshed's setuptools stubs: https://github.com/python/typeshed/pull/13627

raw dune
#

It seems like it's not producing the android wheel?

robust yarrow
#

I tried the following:

uv build --config-setting=...=/build --config-setting=...=/build

with --build-lib/--build-temp, build-lib/build-temp and build_lib/build_temp

robust monolith
robust yarrow
#

Is it possible to configure the editable_mode behavior in pyproject.toml?

robust yarrow
#

<@&815744082823348274> ^ spam

nimble bridge
#

Ban hammer has been used.

fiery raft
#

I'm trying to ship a .pth file with coverage.py. pytest-cov used to do it, so I'm trying to follow its old-code example, but I think it has some things wrong. The documentation for extending setuptools is a bit confusing to me, and has some gaps. For example, I can't tell what commands in cmdclass will be used when running pip install -e .. Are these described somewhere that I've missed?

fiery raft
#

Doing some reverse engineering, these are the keys looked up in cmdclass when doing pip install -e .:

looking for cmdclass['egg_info']
looking for cmdclass['egg_info']
looking for cmdclass['build']
looking for cmdclass['build_ext']
looking for cmdclass['dist_info']
looking for cmdclass['dist_info']
looking for cmdclass['build']
looking for cmdclass['build_ext']
looking for cmdclass['bdist']
looking for cmdclass['editable_wheel']
looking for cmdclass['editable_wheel']
looking for cmdclass['build']
looking for cmdclass['build_ext']
looking for cmdclass['bdist']
looking for cmdclass['build_scripts']
looking for cmdclass['install_lib']
keen ibex
#

what is the reason for custom .pth files actually?

fiery raft
fiery raft
#

BTW, that page has some odd sentences:

Between these mandatory methods, are listed: initialize_options(), finalize_options() and run().

#

and:

Subcommands SHOULD take advantage of editable_mode=True to adequate its behaviour or perform optimisations.

unkempt birch
#

Messing with those is pretty fragile, can't you just add it as package data?

#

I don't use setuptools on my projects (too slow and don't like the error messages), but I'm pretty sure I added a pth file for a project I was helping and that probably was setuptools, I know it wasn't hard, didn't have to resort to setup.py.

fiery raft
# unkempt birch I don't use setuptools on my projects (too slow and don't like the error message...

I tried using package data, but the .pth file has to go in one of four specific places, not in the coverage/ directory. Using a data_file works as long as the Python version that built the wheel is the same as the version installing it. The version is in the file path (see https://inspector.pypi.io/project/coverage/7.12.1b1/packages/1b/1b/cd5c3391a04a101767961e3e3ccf5a0c987b08d8434d97073dbbd132a049/coverage-7.12.1b1-py3-none-any.whl/), so environments that use py3-none-any (like PyPy or 3.15.0a2) don't work properly.

unkempt birch
#

That's not the right place for the .pth file. Those go at the top level, just like if you had a <package>.py file (single file module).

#

I'll make a PR showing you what I mean.

fiery raft
#

Thanks, i'll take a look soon. The 7.12.1b1 code was putting the .pth in the correct place for binary wheels, but not for the non-binary wheels. I've had a fascinating session with Claude reverse-engineering setuptools. I had been looking at how pytest-cov managed it, and it seems like their code was outdated.

unkempt birch
#

No, it happens to go to the right spot if site-packages is in the most common location. It only goes in the right spot in a specific (though common) case. It must be placed in the correct location in the wheel, in purelib/platlib, not assuming python was built without passing the --site-packages configure flag.

fiery raft
fiery raft
#

i have to decide what to do about -e installs. the pytest-cov code for that doesn't work.

nimble bridge
fiery raft
#

and i got editable installs working also. just a few coverage-within-coverage details to iron out.

nimble bridge
#

Aye nice! I'll look into updating the answer w/ credits then, assuming I can find my SO credentials right now. 😅

fiery raft
#

I'm a little surprised that ../ works there, but it seems to.

median crest
#

wait, is coverage.pth in a folder above the pyproject.toml? o_O

#

(actually, ISTR something about a giant Apache monorepo having that use case....)

#

(for Airflow I guess?)

keen ibex
#

Is there any new info about pkg_resources removal?

empty sun
# keen ibex Is there any new info about `pkg_resources` removal?

Came here to check on this too ^^

Past the due date and no movement on https://github.com/pypa/setuptools/pull/5007

PBR still had reliance on pkg_resources, but that seems to have been solved: https://bugs.launchpad.net/pbr/+bug/2111459/comments/6

GitHub
Remove pkg_resources by jaraco · Pull Request #5007 · pypa/setupt...

Remove pkg_resources
Remove assertion that &#39;import pkg_resources&#39; works; it&#39;s no longer an invariant.
Mark pbr integration test as xfail.
Mark namespace tests as xfail. Supp...

Launchpad
Comment #6 : Bug #2111459 : Bugs : PBR

@jaraco I think this should be resolved now. Running the same tests, I now see:

❯ tox -e py39 -- -p no:cov -p no:xdist -k pbr_integration -v
.pkg-cpython39: _optional_hooks> python /usr/lib/python3.13/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg-cpython39: get_requires_for_build_editable> python /usr/lib/python...
#

Actually there's been almost no movement on distutils/setuptools in a few months other than CI fixes. I believe Jason got a new job, probably related.

I've been patiently waiting on a distutils sync to do some cleanup in typeshed. I'm also so, so close to making setuptools' type annotations complete enough to be first-party (all the changes I still need have a PR open).

bronze berry
#

wonders if top_level.txt still serves any pupose post-eggs

river oak
#

Even if it had any use, PEP 794 should provide a complete replacement (and some more)

bronze berry
#

Yeah

#

I'm looking at some misbehaviour of a Debian Python package where the upstream had:

[tool.setuptools.packages.find]
where = ["."]

Which leads to all sorts of non-reproducible junk in top_level.txt. And debating how much effort I want to spend on fixing this systematically.
Just deleting top_level.txt should probably be fine?

quaint jolt
#

it does

#

importlib.metadata uses it

#

PEP 794 isn't supported yet if I am correct

bronze berry
#

That is at least used in a context where missing is probably better than present and incorrect.

crude bolt
#

i don't think an stdlib module should be reading single-tool non-spec files at all

median crest
#

(I tried to dig into importlib_metadata once and the architecture of it just seemed really strange overall. Like, just organic and not designed with forethought)

#

(so I'm not surprised that it would be doing something like that, though I agree it's not a good idea)

quaint jolt
#

there are much worse sins being committed there 😅

quaint jolt
quaint jolt
quaint jolt
#

and if you are fixing the package manually anyway, you could just fix it instead, but it's your call

#

also, I am struggling to see how that setuptools setting might be causing issues with reproducibility unless you are not building from clean sources

crude bolt
bronze berry
crude bolt
#

i think it's worth though documenting top_level.txt as some form of unsupported/deprecated, if just to avoid users building on top of broken workflows

quaint jolt
#

that'd be my recommendation as it builds the sdist first, which may not include the problematic build artifacts you will have in the second build

#

building a wheel directly from source is a footgun with some backends

#

anyway, this is definitely a setuptools bug, so worth reporting

bronze berry
quaint jolt
#

doesn't seem so

#

the key thing in the command I mentioned is that it builds an sdist, unpacks it, and builds a wheel from that source

#

I would recommend you to do the same

#

setuptools, and some other backends, are not designed to work like that, they are designed to build the wheel from the sdist

#

setuptools is usually the most problematic one, in the others issues mostly come from dirty development trees, etc

#

do this

bronze berry
#

I'll look into that. I want to make some other changes to this code, to run tests away from the code that we'll install.

brittle crypt
humble palm
brittle crypt
#

gotcha

#

why is it that people continue blindly updating through major versions and expect no backwards compat issues?

humble palm
#

and ignoring DeprecationWarnings and UserWarnings and not pinning...

formal zealot
#

My way of fixing this in the scientific ecosystem has been to get more and better docs for meson and meson-python written

brittle crypt
#

I read the issue more thoroughly and the problem seems more nuanced than I first thought

#

the problem seems to be legacy dependencies that still assume setuptools has pkg_resources

#

with build isolation this becomes a problem

crude bolt
brittle crypt
#

this isn't even a problem for projects that release wheels

#

and my criticism, whether deserved or not, was aimed at app developers who have the ability to pin their dependencies

median crest
#

(FD: that's me. Writing for LWN is fun and not as stressful as you'd think, despite that they're even more nitpicky than you'd guess)

acoustic cradle
#

generally speaking open ended version ranges behave better

#

at least until they don't 🙃

#

Given metadata for anything shipped on PyPI is immutable, if people were pessimistic about their upper bounds, you'd quickly end up where a large majority of PyPI couldn't be installed alongside each other

#

for build backends it's less of a problem, given they're (typically) run in isolation

#

If PyPI had metadata patching, pessimistic upper bounds would function better, since you could patch your metadata to bump the upper bound after you've tested the new release-- but that presumes people would actually do that across all of PyPI, which is unlikely! So even with metadata patching, optimistic would tend to be better, but at least you could patch to respond to breakage 🙂

unkempt birch
#

setuptools releases major versions way too often to cap it. And you need newer versions for newer Pythons, newer OSs, etc. Anything with capped setuptools would likely be broken already from other things.

#

I don't understand the insistence that changing setuptools should require a PEP in that issue

#

This only affects Python 3.9+ (would maybe have been nice to have dropped 3.9 first, as that reduces impact on older Pythons, but 3.9+ does have importlib.resources.files and such)

potent basalt
acoustic cradle
# potent basalt Wouldn't this download a file, and then do something weird on installation since...

Well it doesn't exist so it's impossible to know exactly how it would work 😉 but presumably either installers would treat the repo level metadata as authoritative or it would be implemented as separate "patch" (in concept, not the ⁨patch⁩ tool) files that the installer is expected to apply to the metadata before reading it (and maybe PyPI pre-applies those for the metadata that it lifts out of the artifact-- or maybe it doesn't and tools just have to apply thoes patches to the metadata to get the correct behavior)

#

or something like that

#

https://prefix.dev/blog/repodata_patching conda has repodata patching, so it's just patching (afaik) the repository level metadata, and I think it leaves the artifacts alone, but tbh I don't really know if that's true (or if conda artifacts even have metadata inside them like python packages do)

keen ibex
acoustic cradle
#

Oh yea, adding it would be a PEP that would need the installers to support it for sure

keen ibex
#

there is also a matter of local caches of wheels that would have to understand that

unkempt birch
#

Patchable metadata is massive undertaking. It's been discussed a bit, but it would touch a lot and require changes everywhere. And security can be an issue depending on what you wanted to allow patching for. And you'll need something for mass updates on PyPI. Etc.

boreal yarrow
unkempt birch
#

Added one small comment, nothing major though, looks good. 🙂

bronze berry
#

I just ran into https://github.com/pypa/setuptools/issues/2688 again, where our DEFAULT_EXCLUDE list doesn't include any sub-packages, so doesn't do a whole lot of excluding... I'd forgotten how painful setuptools finders can be.

The context for me is building Python modules multiple times from one source tree and getting build directories inside the result wheel.
@quaint jolt: You previously advised building wheels from sdists, rather than directly from the source tree, I guess that's the best solution for me here...

GitHub
Users expect `[options.packages.find] exclude` to be recursive, but...

setuptools version 57.0.1.dev3+ga5131f0b.post.20210528a Python version cpython 3.9 OS Ubuntu 21.04 Additional environment information No response Description [options.packages.find] exclude = arpeg...

acoustic cradle
bronze berry
#

Seeing this with 517/518. But it also happened before 517/518

acoustic cradle
#

sorry I just mean hopefully less people will include pre-generated files in their sdists because IIRC that was commonly done to avoid bootstrapping dependency issues 😅

bronze berry
#

I'm worried that going via sdist is going to break lots of fragile test setups etc. But the only way we'll find out is by implementing it...

#

I'm just shaking my head wondering how DEFAULT_EXCLUDE was ever supposed to work

#

(And I have a sneaky feeling I've been there before)

acoustic cradle
#

my guess is someone made an arbitrary choice 20 years ago and changing it is scary 😛

bronze berry
#

My position too 😛

quaint jolt
median crest
#

👍 that's good practice anyway (at least according to build)

formal zealot
#

Deeply confused about how the Py_LIMITED_API macro ends up being defines in builds managed by setuptools. I see how that’s controlled via the distutils bdist_wheel command’s py-limited-api argument, but then I don’t see how that leads to any macros being defined in extension builds. Anyone have the foggiest idea how that works? Context is I’m working on https://github.com/Quansight-Labs/stable-abi-testing and want to get the setuptools support I’ve hacked together into a more mergeable state.

If PEP 803 were accepted, does it make sense to have a new Extension keyword argument that allows manually toggling an abi3t build? Or should it require a free-threaded interpreter?

acoustic cradle
#

uh

#

distutils/setuptools has a way to inject macro definitions into the build

#

is the method in distutils, not sure if that get moved out if distutils or if that part is still used

#

I don't think setuptools actually sets the macro for you though, I think you're expected to set the macro yourself using those mechanisms (or another)

#

changelog for 25.4.0 says:

Add Extension(py_limited_api=True). When set to a truthy value, that extension gets a filename appropriate for code using Py_LIMITED_API. When used correctly this allows a single compiled extension to work on all future versions of CPython 3. The py_limited_api argument only controls the filename. To be compatible with multiple versions of Python 3, the C extension will also need to set -DPy_LIMITED_API=... and be modified to use only the functions in the limited API.

formal zealot
#

interesting, OK

#

thanks!

acoustic cradle
#

no problem!

formal zealot
quaint jolt
#

@lusty dome I had a look at the distutils adoption

#

setuptools will adopt the distutils module as it is currently in cpython, right? or will the implementation be somewhat replaced?

#

my take considering the Linux distro issue, is that it should be adopted as-is, but deprecated

#

then the Linux distros that need to, can apply their patches

#

some people might also be relying on implementation details of the distutils module, so I think this would cause less breakage

#

I am probably still missing some context though, because overall it seems pretty straight-forward to me

lusty dome
#

Indeed it is more complicated. Setuptools has already adopted distutils, but it’s protected by a flag flip currently disabled by default. The problem with applying the patches is that distros can’t apply their patches reliably to third party packages... and they can’t apply them to the adopted code without affecting other platforms. They need a hook or mechanism to apply their settings anytime Setuptools is installed on their platform. The approach we settled on is they will override sysconfig (root package) and distutils will honor those settings. The devils in the details.

#

Maybe we could meet Friday or Sat and I could give you a tour and answer questions.

quaint jolt
#

yes, that is fine by me

quaint jolt
quaint jolt
#

@lusty dome I am available for the rest of the day, let me know if you want to have a chat

lusty dome
#

How about in 25 minutes. At :45 ?

quaint jolt
#

works for me

lusty dome
#

SETUPTOOLS_USE_DISTUTILS=stdlib as fallback.

quaint jolt
#

@lusty dome can you talk in the next few days?

#

I just have some final details to run by you and some questions about the process

lusty dome
#

It's been quite a week for me, and I'm on-call this weekend, but maybe Saturday. Does that work for you?

quaint jolt
#

yeah, saturday works fine

#

I don't expect it to take much time

lusty dome
#

I'm going to make a coffee, then I'll be on the voice channel.

quaint jolt
#

I am available now

lusty dome
cloud sand
#

it's almost like setuptools is not able to generate metadata for pip to evaluate the python_requires without loading the namespace

river oak
#

The packages = find_namespace: line in setup.cfg looks out of place to me. zipp doesn’t seem to have a namespace package, so setuptools is probably just confused.

cloud sand
#

importlib-metadata depends on zipp, and virtualenv/pytest/tox depends on importlib-metadata, so this bug breaks the entire python 2 CI at my company 😄

#

@river oak do you know how to check why pip picks a given package? for some reason on my local machine it picks 1.2.0 of zipp, but in the CI picks up 3.4.1 😐

river oak
#

setuptools not able to selectively load values without loading everything else is a very known problem though, this is not the first example and won’t be the last. That’s what using an interpreter (Python) for configuration gets tou, every thing is evaluated at import time so there’s no way out 🤷

cloud sand
#

actually in CI this just could be an older pip 🤔

river oak
#

Yeah older pip may be a problem. IIRC Ubuntu ships with pip 9 by default as recently as 20.0 or something

lusty dome
#

It could be pip 8.

#

@cloud sand Have you confirmed that your company is using a PEP 503 compliant index server?

#

FTR, I don't think there's anything wrong with find_namespace:. I use that in all my packages now as a matter of course, unless they need Python 2 compatibility.

cloud sand
#

how do you find that out? 😄 (it's running Artifactory a recent release)

lusty dome
#

It's a red herring - the problem is that some installer is pulling down packages that declare incompatibility with the target environment.

#

In that case, you're unfortunately constrained to a pre-PEP 503 world. If you recall that world, the only way to prevent installation of incompatible packages is to pin your dependencies.

cloud sand
#

So I'm being told it should be 🙂 (was released sometimes last year)

lusty dome
#

For this particular issue, I'd recommend to install zipp<2 and importlib_metadata<3 and importlib_resources<4 (possibly only for ;python_version<"3").

cloud sand
#

seems suddenly now everything passes 😐

#

I guess we'll come back if we can pin it down what was it, did the PEP-503 service crashed in artifactory perhaps 🤷‍♂️

lusty dome
#

The PEP says "A repository MAY include a data-requires-python attribute on a file link.", but a repository really SHOULD implement it if they want happy Python users.

#

But that should answer the question. If you want to test your repository, load the /simple/zipp/ (or similar) and inspect the HTML for anchors with data-requires-python. If it's not there, then pip can't discern either.

quaint jolt
#

@lusty dome I was thinking, it would be really handy to be able to write entrypoint scripts that only use specified install schemes

#

this would prevent pip from shadowing modules that distro installed entrypoints are expecting to be there

#

what are your thoughts?

#

to implement this I was thinking using python -SI in the shebang and then doing something like sysconfig.enable_scheme('arch')

#

would this be a good idea, what do you think?

lusty dome
#

I think I understand what you're saying - if a distro installs a package with a script, you want that script to preclude the possibility that the modules that back that script could be superseded by non-distro module installs. Yes, that seems plausible to me. I'm not aware of any problems with the approach you propose, but I'm also not very familiar with installers. There are probably people better suited than me to consider the proposal. I believe it's mainly distlib and pip that implement those scripts.

quaint jolt
#

what I described requires the addition of something like sysconfig.enable_scheme('arch')

#

we need a method that will add the correct paths to sys.path

#

it's mainly here that I want your opinion

river oak
#

What would enable_scheme() do? Everything in sysconfig is global and has no side effects, and probably needs to continue to be.

quaint jolt
#

it would only add paths to sys.path

#

note this doesn't have to be in sysconfig

#

that's just my initial sugestion

#

but since it's a method that users have to run explicitly, I don't really see the issue

river oak
#

Oh so it adds purelib and platlib from sysconfig.get_paths(scheme='arch') to sys.path? That feels… both weird and unuseful to me.

#

Also there’s https://bugs.python.org/issue43312 if you’re not already aware. Instead of putting things into sys.path (which has side effects), installation tools should just not read sys.path at all, and instead relying on values returned by the “pure” sysconfig functions.

#

Scheme-specific console script sounds weird to me as well. IMO the scheme should decide whether it wants scripts and how, and if it doesn’t want them it can make sysconfig.get_path('scripts', scheme=scheme) return /dev/null or something (or maybe we can use None for that).

quaint jolt
#

well, I presented the usefulness above

#

do you have any alternative suggestions?

#

I am not convinced it is the perfect solution, I think it is an okay solution, so I am all ears for alternatives

#

I am aware of that bpo, I don't think this should affect anything. we would be asking Python to enable a custom scheme, it does not affect the default scheme.

#

IMO the scheme should decide whether it wants scripts and how

#

that is fine, and in my use-case they would the same path

#

note there is nothing really preventing me from doing it already, I just want to streamline things so that we don't end up with another Debian-like issue

#

IMO this is a valid use-case

#

the current mechanisms were not designed exactly for this, but they were also not not designed for this

#

I am just trying to do the best I can to provide users the best UX

nimble bridge
#

Artifactory doesn't have data-python-requires except on the newest versions.

river oak
#

I’m not sure I have better suggestions since I’m not really seeing the benefits here. Why is a runtime function needed? Shouldn’t this be done in the initialisation phase, maybe in site.py? Would it be a good idea to make site.py better documented and structured so distros can better configure the installation for their needs?

quaint jolt
#

sure, I am open to that

#

I had thought about it but couldn't come up with anything particularly good

#

so I just proposed the simplest thing

#

but I am more than happy to discuss alternatives

haughty bramble
nimble bridge
#

🤷🏻‍♂️

quaint jolt
#

@lusty dome hey, do you have some time to go over the implementation details of the vendor config with me?

lusty dome
#

Sure. I'll have some time later today.

quaint jolt
#

okay, great

#

ping me when you are available

lusty dome
#

@quaint jolt I got occupied but free now. Joining voice

#

Looks like I missed you. Will check back later. Feel free to post questions in the meantime

quaint jolt
#

alright

#

I was having dinner

#

@lusty dome joined voice, feel free to hop in when you can

# 
./configure --with-vendor-config=vendor_config.py
# 
/usr/local/lib/python3.10/_vendor_config.py
# 
import _vendor_config
# 
EXTRA_SITE_INSTALL_SCHEMES = {
    'arch': {
        'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}',
        'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
        'purelib': '{base}/lib/python{py_version_short}/arch-packages',
        'platlib': '{platbase}/{platlibdir}/python{py_version_short}/arch-packages',
        'include':
            '{installed_base}/include/python{py_version_short}{abiflags}',
        'platinclude':
            '{installed_platbase}/include/python{py_version_short}{abiflags}',
        'scripts': '{base}/bin',
        'data': '{base}',
    },
}
quaint jolt
#

@lusty dome I have the PR ready but some people requested more feedback

lusty dome
#

acknowledged

quaint jolt
#

if you have time, you can look at the code to see if there is anything you'd like to change

lusty dome
#

I see Éric has asked to discuss the design in a forum. Is that something you're prepared to do?

#

Probably the first thing should be to reference the prior discussions and designs and point out that this PR is an implementation of those proposals/designs.

quaint jolt
#

it's a little bit tricky because this has a lot of baggage and required context, but I think so

#

I opened a thread on discourse

#

my mindset here is essentially "please point issues, or suggest and demonstrate a better alternative"

#

I think there is a possibility of people nitpicking the implementation, I will try to hear the suggestions but won't block progress because of that

quaint jolt
#

@lusty dome the changes probably aren't going in, right?

#

for 3.10 I mean

lusty dome
#

Deadline is Monday I think. Let me look tomorrow and see how close we are. It doesn’t have to be perfect but it needs to be close to an adequate and permanent solution.

quaint jolt
#

thanks

#

I should have fixed all the tests now

#

I also sent an email to Miro regarding the feedback for Fedora

#

I should have been quicker 😕

quaint jolt
#

I have added get_preferred_schemes to the vendor config as some distributors, such as Fedora, want it

#

and this is better than the current approach anyway

restive talon
#

👋

quaint jolt
#

hey 🙂

lusty dome
#

@quaint jolt I'm around today. I'm going to work through the various notifications I have. I'll also be hanging out in the voice channel.

quaint jolt
#

test_resource_opener and test_resource_path

wide stream
#

Would setuptools accept wheel building for POSIX vs. Windows so my linux box does not get all the .exe ? (I have not searched for a issue - being lazy since i see the channel here) 😄

lusty dome
#

It's been requested.

#

Those exes are deprecated. I'd prefer to just eliminate them and rely instead on distlib.

wide stream
#

O, even better.

#

I might take a look at it one day.

lusty dome
#

There's an open PR that's exploring switching to distlib, if you wish to assist there.

wide stream
#

Wow - would be very hard to relate this to deprecating .exe's in setuptools for an outsider

lusty dome
#

I'm unsure if distlib wheels have exes in it.

#

If distlib has exes, then that would only push the issue downstream to distlib.

wide stream
#

loves the project description of distlib

#

haha

quaint jolt
# 
(files(package) / normalize_path(resource)).open('rb')
lusty dome
quaint jolt
# 
self = <importlib_resources.tests.test_open.OpenZipTests testMethod=test_open_binary_FileNotFoundError>

    def test_open_binary_FileNotFoundError(self):
>       self.assertRaises(
            FileNotFoundError, resources.open_binary, self.data, 'does-not-exist'
        )

importlib_resources/tests/test_open.py:50:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
importlib_resources/_common.py:124: in open_binary
    return (files(package) / normalize_path(resource)).open('rb')
/usr/lib/python3.9/zipfile.py:2306: in open
    stream = self.root.open(self.at, zip_mode, pwd=pwd)
/usr/lib/python3.9/zipfile.py:1502: in open
    zinfo = self.getinfo(name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def getinfo(self, name):
        """Return the instance of ZipInfo given 'name'."""
        info = self.NameToInfo.get(name)
        if info is None:
>           raise KeyError(
                'There is no item named %r in the archive' % name)
E           KeyError: "There is no item named 'ziptestdata/does-not-exist' in the archive"

/usr/lib/python3.9/zipfile.py:1429: KeyError
lusty dome
quaint jolt
#

@lusty dome

lusty dome
#

yo

quaint jolt
#

I am in voice

#

please join when you can

river oak
wide stream
#

Is that terribly expensive?

lusty dome
#

It would be an extra layer of abstraction (extra step to encode when the wrappers need to change, and then extra work to decode those back to executables to generate them). I doubt the run-time cost would be a concern. How much does it benefit the use-case to have executables encoded into Python?

river oak
#

Correct, it's not the cost to write that's the concern, but the additional complexity involved in encoding that file in Python source and decoding it when needed.

restive talon
#

@lusty dome im wondering, is there a sensible way to separate setuptools_scm from the core setuptools dependency, im trying to make it usable for other tools that may not want to pull in setuptools (even if the wheel is easy)

lusty dome
#

Absolutely.

#

I would have expected that to be the case naturally.

#

In hgtools, the functionality was usable independent of setuptools.

#

What about setuptools_scm is reliant on setuptools?

#

That's the beauty of entry-points - their inversion of control. They put all the control in the hands of the plugins. All a plugin needs to do is meet the consumer's protocol, and any other behavior is fully under their control.

restive talon
#

@lusty dome the dependency is mainly there ot sort out minimal versions, the real issue is that i have to depend on setuptools to set the required version for the integration

#

@lusty dome so right now i dont know how to make the setuptools dependency optional without making the integration fall on too old versions

lusty dome
#

I'm inclined to suggest not to depend on Setuptools at all, but to explicitly depend on packaging (for sorting out versions). I would expect even old versions relying on setup_requires=setuptools_scm to correctly get dependencies of setuptools_scm, so it should work. What's wrong with that approach?

restive talon
unkempt birch
#

Guessing this is timely for poetry’s new plugin system, but I’d like it for a potential future PEP 517 backend for scikit-build, too. Personally pinning a min version of setuptools In pyproject.toml is fine with me (42+ must be pinned already for good pep 517 support), but not everyone uses that, I guess.

restive talon
#

hmm, actually, i have a idea now, will fix today - this will be a setuptools_scm 7.0

lusty dome
#

@restive talon It sounds like you want to remove the dependency on Setuptools but still enforce that if Setuptools is present that you have a suitable minimal version. For that, I'd recommend packaging and importlib.matadata. Something like

def check_requirement(req="setuptools>=42"):
    """
    Return whether a present package satisfies the specified version.
    """
    target = packaging.requirements.Requirement(req)
    with contextlib.suppress(importlib.metadata.PackageNotFoundError):
        found_str = importlib.metadata.version(target.name)
        found = packaging.version.Version(found_str)
        return found in target.specifier
    return True
restive talon
#

Hmm, yikes, the file finders

#

This will be horrible but work as trigger

lusty dome
#

@restive talon I guess I'm not following. I don't yet fully understand the problem you're trying to solve. If nothing attempts to import setuptools_scm, then you don't have to do the check. Do you want to perform the check only when setuptools_scm is invoked from Setuptools? If that's what you're after, then yes, you would want to only enact the check in the hook(s) the plugin presents to setuptools.

restive talon
#

@lusty dome setuptools before the finalize_dist hook batted against a project that triggers setuptools_scm importation by that hook would then not import setuptools_scm as far as i understood (as the setup keyword would only trigger import on use)

glad quartz
#

Hmm, I found a bug on https://github.com/pypa/setuptools/blob/2234e88b7b820c40f29d7ddadc182b0f130eaa1d/launcher.c#L179 where GenerateConsoleCtrlEvent silently fails and that failure isn't reported at all.

I think it is because it has wrong parameters, the first one should be one of CTRL_C_EVENT or CTRL_BREAK_EVENT https://docs.microsoft.com/en-us/windows/console/generateconsolectrlevent.

This means that if the launcher is killed the underlying python process doesn't receive that signal.

I had found it in (for context) https://github.com/msys2/msys2-runtime/pull/31#issuecomment-854019021.

glad quartz
#

will make a PR to fix them

glad quartz
nimble bridge
#

@lusty dome Since when does setuptools use pip for handling setup_requires?

#

I think that changed early this year, but I can't find setuptools' changelog in the docs.

#

NVM, I'm blind -- it's called History and it was 52.0.0.

lusty dome
#

23 Nov 2019

#

Also, I'm blind too. I read your first two messages, but didn't see the third until just now.

quaint jolt
#

this does not happen in build_meta, right?

#

but it does in the legacy backend

#

right?

#

(just confirming)

lusty dome
#

I don't recall how build_meta handles setup_requires. I seem to think build_meta still honors setup_requires but it should be a no-op if the user is supplying their build requirements in pyproject.toml.

#

Oh, I'm mistaken.

#

It does actively disable setup_requires.

quaint jolt
#

cool, thanks!

restive talon
#

@lusty dome is there something that can be done about better integrating pyproject.toml - bascially the finalize options hook will not have access to a distribution name, if the distribution config files have not bene parsed, its quite a pain at first glance - it seems setuptools_scm will have to parse setup.cfg in addition to pyproject.toml to work, and as things look, potentially the metadata from pyproject.toml as well

lusty dome
#

I agree - setuptools_scm shouldn't be responsible for parsing metadata, except maybe in a situation where there's a standard for providing that metadata (like pyproject.toml), but even then, I'm not sure why Setuptools couldn't provide that metadata parsed to its plugins.

#

Looking into the issue as you describe it, I can confirm that a Distribution calls .finalize_options in init, but parse_config_files is called after the Distribution is initialized. And I agree - there should be a way for plugins to hook onto the Distribution after config is fully loaded. I can see a couple of potential approaches - either add a new hook, after finalize_options and parse_config, that setuptools_scm could use instead. Or, move the config parsing ahead of finalize_options. I'd probably lean toward the latter if feasible, because that would involve less flux and enable other plugins to use the finalize_options with config loaded. What I can't tell with a cursory examination is if moving the config loading earlier would have adverse consequences.

#

Would you be willing to file a bug and ping me on it? And if you or someone is willing to take on the next step of the investigation and see if moving the config parsing earlier is viable (or if not, why not), that would be most appreciated.

patent linden
#

geofft and I have been struggling to build internally to our company a number of python projects (most recently dm-tree, but also stuff like xeus and arrow) whose setuptools build scripts use a variety of mechanisms (mostly raw build_ext custom implementations, but also scikit-build exists) to call out to cmake to drive a build of c++ code, into an extension module directly (often with pybind11), sometimes into a shared library that gets linked into an extension module (cython or raw cpython or something)

our struggle often comes about because we need to pass arguments to the cmake stuff to tell it where to find its dependencies or something, and usually the custom bridging of setuptools to cmake doesn't afford passing arguments through setup.py's arguments into cmake, so we have to patch the sdist, usually for expediency just patching CMakeLists.txt

I wonder if there's any prior work you all have done, or appetite to work together on, some standardization of how setuptools invokes lower level build systems like cmake (or, I anticipate soon, meson)? We have distutils which has a known api for building c files into extension modules and surfaces things like the include path up for control by the invoker of setup.py, could we have a standard mechanism that knows about cmake conventions? Would people be in to this?

#

Put another way, we have setuptools-rust/maturin and these are early enough in the rust ecosystem that it's likely anyone doing python+rust will standardize on this; can/should we try to corral the python+c++ packages that exist in the ecosystem into a standard way for the build systems to cooperate?

quaint jolt
#

I usually just set CFLAGS/CXXFLAGS

#

but that depends on the details

#

I can get most stuff to work using that generally

patent linden
#

For well behaved cmake projects you can often finesse it with a carefully constructed CMAKE_PREFIX_PATH, yeah, but often you need to convince setuptools to pass some -D options to cmake or something like that, and usually when it's a custom build_ext thing, the author of setup.py hasn't bothered to plumb arguments down from setup.py's argv to cmake

haughty bramble
restive talon
#

@jaraco thanks for the details, it seems that i have to do the parsing first in any case for compatibility

I will slot passing more details to setuptools for after i can land a workaround in setuptools_scm

restive talon
lusty dome
#

Some have expressed interest. I don’t think there’s been any active development. I’m away from my computer or I’d search the issue tracker. Feel free to file one or work out a design.

restive talon
#

i see, unfortunately im too stretched to put work into it atm

cloud sand
#

One day I'll do it 😬 if no one gets ahead of me ☺️

restive talon
#

well, im going to merge a few prs for setuptools scm now and then work on it, the toddler dropped aslept shortly after 5

restive talon
#

@lusty dome is there any way to integrate into commands via a entrypoint? i'd like to stop setuptools_scm from doing the write_to always and instead only do it for installs/editable installs

lusty dome
#

Setuptools allows for additional "distutills-commands", but not to hook into specific commands. @restive talon I'll need more details to describe what you're intending to accomplish.

restive talon
lusty dome
#

So you wouldn't want setuptools_scm to be invoked for sdist? That sounds like a pretty big deviation from the current design - where configuration and plugins mainly affect the setup of the build, and the build acts on that configuration. That's not to say that there shouldn't maybe be a way for setuptools to solicit other behaviors at build time, but it's not obvious to me how that would work.

#

What's the motivation for not wanting the file to be generated earlier?

restive talon
restive talon
lusty dome
#

Have you considered not writing the file and just letting setuptools track the version?

#

That's what I do and it avoids the need to write the file.

haughty bramble
tranquil saffron
#

not sure what you mean, setuptools doesn't support PEP 621

haughty bramble
#

Well it certainly will do eventually

restive talon
#

@lusty dome that is the common/suggested way to track the version with setuptools_scm - however due to the relative cost of loading/finding the metadata its sometimes desirable to dump them into a file - thats how write_to was introduced - the problem - it practically needs to put actions into the build/install commands to ensure file creation at the right point in time

restive talon
#

@lusty dome aware of a safe way to invoke parse_configuration inside of a finalize_options call? when i run the helper from setuptools.config i will end up with a recursion and i'd like to avoid creating global context-manager to handle this

lusty dome
#

i'm not aware of a way off the top of my head.

lusty dome
restive talon
#

@lusty dome the error on allow_hosts being specified on setuptools versions that use pip is a little bit of a pain , at least i could resolve it in my env quickly

lusty dome
#

@restive talon I'm not familiar with the error. Sorry to hear it's a pain.

restive talon
#

@lusty dome basically when bisecting setuptools versions for issues, the newer ones need a pip config to disallow network, the older ones need a distutils config, however the newer ones raise a error if its given as pip doesnt support it (it might be a good idea to make it a warning

lusty dome
#

I see. Well, changing it from an error to a warning now won't help with bisecting old versions. And allowing option to be specified but ignored could cause some serious security issues (user said "don't contact any hosts but x, y, z" and then Setuptools issues a warning but then contacts "pypi" or "badpypi"). I'm open to changes that might enable the necessary patterns for testing, but I'd rather focus the little energy I have on deprecating easy_install/setup_requires entirely.

restive talon
#

i see, in that case bisecting being a pain is a acceptable intermediate

haughty bramble
#

@lusty dome why is distutils still maintained in a separate repo to setuptools?

#

Or is it just nobody got around to moving it?

lusty dome
#

There are a few blockers:

  • Tests in distutils aren't exercised in setuptools.
  • Setuptools hasn't adopted distutils by default. Because it's still opt-in and loosely integrated, it's conceivable that it might not become fully integrated.
  • It may prove valuable, especially as Python 3.12 drops distutils, to have an installable package of just distutils (though I'm reserving that until there's a strong use-case).
#

Oh, and

  • pypa/distutils was used to port commits directly from CPython, so has a historical link to the CPython history and is easier to integrate the few changes that landed in CPython since the deprecation/split.
haughty bramble
#

Either way I'm looking forward to the angry comments from people still using lib2to3

restive talon
#

@haughty bramble >150 broken ci pipelines here including deploy checks - due to pinned older coverage and jsmin

haughty bramble
#

Rip

#

Seriously due to missing 2to3?

restive talon
#

yeah

#

its almost funny as the fix is a 1 line change in our ci lib but the guy who merges those typically in seconds is on sick kid leave today

haughty bramble
#

What's the fix?!

#

Probably something in setuptools should be a noop rather than crash

#

@restive talon ^

#

Wait did coveragepy need 2to3?

restive talon
#

@haughty bramble setuptools absolutely has to error, my morning wil lbe spend with digging into the details, i just did late night bandaid about pinning low versions of setuptools

haughty bramble
#

That's weird, it's not true by default right?

#

setuptools should probably support use_2to3=False

#

I think that was to work around an issue where setuptools failed to import lib2to3

restive talon
#

ah, nm, over the nighta new release allowing false was done

haughty bramble
haughty bramble
river oak
river oak
#

The project has been taken over by Airflow maintainers and now have wheels

haughty bramble
#

noice

#

maybe setuptools needs to include a 1 second sleep after raising deprecation warnings, that doubles every month

#

this way your build will go to an hour after one year of ignoring the warnings

river oak
#

IIRC warehouse did something like that a while ago and people just end up sending cryptic “I build is failing for now reason” error reports (and it’s close to impossible for maintainers to realise that brown period behaviour is the cause since users don’t connect the two)

robust yarrow
#

I'm at a loss... how does one include a .pth file that resides next to setup.py?

# 
root/
  pkg/
    ...
  patch.pth
  setup.py
river oak
#

Include as in you want to install it?

restive talon
haughty bramble
haughty bramble
nimble bridge
#

Ahoy. Question for the maintainers: would you be willing to accept PRs that enforce that setup.py / setup.cfg files exist, in the setuptools' build backend?

tranquil saffron
#

has anybody tried bootstrapping setuptools using PEP 517?

cloud sand
#

as in, why?

tranquil saffron
quaint jolt
#

you can bootstrap wheel by running setuptools from source

#

then install it and bootstrap setuptools

tranquil saffron
#

ok, so you've put setuptools on path and you call build_wheel to build a wheel... what happens next?

tranquil saffron
#

I was able to build a wheel by generating egg info. So, the exact process is make all of _distutils_hack, pkg_resources, setuptools and wheel importable, run setup.py egg_info on setuptools, copy the egg-info folder over to your faux site packages, repeat for wheel and call the PEP 517 hook to build their wheels. dist_info won't work, it depends on bdist_wheel which is not discoverable until you run egg_info on wheel. egg_info also won't work unless you comment wheel's setup_requires out, unless pip is available in the environment - setuptools will attempt to install itself with pip.

tranquil saffron
#

Also if you shallow clone the setuptools repo expect your setuptools version to be mangled

quaint jolt
#

needs special handling for setuptools

#

but works

tranquil saffron
#

@quaint jolt any idea why setuptools.egg-info is created and then immediately deleted when you try to run build_sdist (or python setup.py sdist) on setuptools itself without having previously run egg_info?

#

it doesn't get pulled into the sdist either

river oak
#

I seem to recall that's a setuptools implementation detail, the egg-info is created and then used as a "template" to create the dist-info (or whatever metadata in sdist)

tranquil saffron
#

I managed to get it to stick by copying of all its entry points into bootstrap.egg-info

tranquil saffron
#

I think the next time someone asks "how do I include package data with setuptools" we should just point them to flit and be done with it

jagged grotto
wild flicker
#

hi! i have an old package for py2.7 only, i want to use python -m build to make an sdist and a wheel, but the wheel is always marked py3 even though i have

classifiers =
    Programming Language :: Python :: 2
    Programming Language :: Python :: 2.7

and python_requires = <3 in my setup.cfg. what am i missing?

haughty bramble
quaint jolt
#

you need to enable the universal wheel setting if you want a Python 2 and 3 wheel

haughty bramble
wanton salmon
#

What's the status on using pathlib in the setuptools codebase fwiw? I'm continuing my work on adding pep 660 support to setuptools and having to use the os.* and os.path.* utilties constantly is rather annoying as all of my path manipulation experience is with pathlib :p

wanton salmon
#

I took a stab at implementing PEP 660 but I can't seem to find a reliable way to query the importable base of the packages / modules (i.e. the parent directory of {name}.dist-info generated from an in-tree build). For what it's worth this is my first time working in the setuptools codebase so I'm confident there's many things wrong

my stab: https://github.com/ichard26/setuptools/commit/bbe070e735f2e4314da2228bdac1dd2c5c1705e7
blocker: https://github.com/ichard26/setuptools/commit/bbe070e735f2e4314da2228bdac1dd2c5c1705e7#r59675825

GitHub
wip: PEP 660 backend support · ichard26/setuptools@bbe070e

currently the contents for the {name}.pth file is hardcoded for the
psf/black project (because of course :P)

wild flicker
haughty bramble
#

unless you have another py3 wheel

unkempt birch
#

Wow, just noticed the new docs. 🙂

cloud sand
#

I like the hammer, setuptools is used as a hammer that fixes on youryour packaging need, one big hammer to rule them all

wanton salmon
haughty bramble
river oak
haughty bramble
#

so package_dir = tubes=tubes does nothing?

#

it produced an indentical wheel when I removed it

#

The default correspondence is the most obvious one, i.e. package distutils is found in the directory distutils relative to the distribution root.

#

ok awesome

#

and [options.packages.find] exclude defaults to empty?

#

tubes/test/__init__.py is included as expected

river oak
haughty bramble
#

and include_package_data is redundant because there's no package data

#

great thanks a lot!

unkempt birch
#

Just curious, would anyone from the setuptools developers be interested in collaborating on https://iscinumpy.gitlab.io/post/scikit-build-proposal/ ? The late-first-year phase of the proposal would include rebuilding a proper scikit-build - setuptools bridge, ideally playing nicely with setuptools rather than just trying to wrap over it as is done now. A specific letter of collaboration with setuptools might make the proposal a bit stronger.

ISciNumPy
Scikit Build Proposal

I’ve spent the last few years trying to make it easy for anyone to extend
Python with compiled languages. I’ve worked on pybind11, a powerful C++
library that allows users to write advanced Python extensions using just C++11,
used by some of the largest projects, SciPy, PyTorch, Google, LLVM, and tens of
thousands of other libraries, down to ve...

river oak
#

I’ve wondered a while if we could develop an actually good build system (read: not what setuptools is currently doing) that can also be plugged into e.g. Flit and Poetry so backends can build binaries all in the same way.

#

This would require the build system to have some kind of agreed on API for PEP 517 backends to hook onto; probably not necessarily a PEP (because there’s no real need for interop, if the system is actually good we probably don’t want a second one?) but something useful enough for people to rely on, and standardised enough so if we really need a second such system it does not need to do the design and integration work from scratch

unkempt birch
#

Is Flit interested in plugins? I know Poetry is.

river oak
#

Flit is interested in supporting extensions but not sure how best to approach it. So if there’s an established library that does it well I think it’d be happy to use it (likely something simpler than plugin, maybe just import detection, use that if the user lists the extension builder as a build dependency or something like that)

dreamy urchin
# unkempt birch Just curious, would anyone from the setuptools developers be interested in colla...

I have been thinking lately about this (since your first blog post).

I think in the setuptools universe, integration could be achieved as an entry-point (most of the things are).

Maybe a setuptools.externally_compiled_files or something like that... But I wonder which would be the appropriate interfaces.

Would it be enough to imagine that the entry-point value is a function with a signature similar to the following?

def external_compiled_files(language_tag, abi_tag, platform_tag, project_root) -> list[ExternallyCompiledFiles]:
   ...


class ExternallyCompiledFiles(Protocol):
    def source_files(self) -> dict[str, Path]: ...
    def compiled_files(self) -> dict[str, Path]: ...
quaint jolt
#

currently all build system need to reimplement complex logic for native modules

river oak
#

Both Poetry and PDM currently just punt the issue and use setuptools. Not very pretty. https://stackoverflow.com/a/60163996/1376863
(IIRC there’s a documentation page on this but I can’t find it)

Stack Overflow
How to build C extensions via poetry?

To build a python project managed with poetry I need to build C extensions first (an equivalent to python setup.py build). poetry is able to do this according to this github issue. But to me it's not

unkempt birch
#

pdm has a page on this somewhere.

restive talon
#

any tool for that shoudl aslo have things for cython, cff, hpy, mypyc

#

oh, and the rust libs for python interaction/boost & co as well

jagged grotto
#

And add a visualization of how graph is constructed, traversed, enriched, selected for build and processed.

#

The problem is that that all interactive tools for graph processing are mostly in javascript, or require heavyweight installation of binaries.

#

It actually doesn't matter which build system it is. The debugger for build graph is the selling point here.

cloud sand
#

you might want to coordinate with pip maintainers to find a path ahead on this, because feels many people will be affected by this

lusty dome
lusty dome
#

@cloud sand Initial indications are the fix does the trick. Please consider bundling this new Setuptools in a Virtualenv release or yanking the current release until you can.

cloud sand
#

Said before, the bug is not related to virtualenv and can be reproduced without it, so there's no yanking warranted here.

cloud sand
tranquil saffron
#

What's the reason for setuptools using a meta path finder and a custom loader instead of shipping, let's say, setuptools/_internal/distutils and prepending _internal on the sys.path?

cloud sand
#

that would be a question towards @lusty dome

quaint jolt
#

user could be running it from a zip for eg

#

nobody really should be manipulating internal paths directly on code that gets distributed

tranquil saffron
#

the default path finder supports zip paths but I see your point, you get to rely on import semantics this way

lusty dome
#

The main motivation is that because distutils is deprecated in the stdlib and scheduled to be removed in 3.12, Setuptools wishes to offer a replacement, but to replace situations where "import distutils" is used prior to "import setuptools", I deemed it necessary to have an import hook.

#

Now I realize I'm answering a different question.

#

I can't recall if I attempted to simply do path manipulation. I'd definitely be open to a simpler solution.

#

As it's currently implemented, the _distutils_hack intercepts three different imports (import setuptools, import pip, and import distutils) for different reasons/issues.

#

I don't see how that logic could be implemented without an import hook, but once pip removes its reliance on distutils and get-pip removes its "import setuptools" check, it may be possible to simply manipulate the sys.path.

#

(and of course most everybody is running those updated releases)

haughty bramble
#

Why are things doing "import distutils" still now that it's deprecated?

river oak
#

Because the code was written ten years ago and it’s non-trivial to migrate? I literally spent a year of volunteer time to implement sysconfig support for pip and it’s still not feature-parity to distutils. It’s much more complicated than you’d imagine.

quaint jolt
#

there are still blockers for the removal

#

I have opened several bpo

tranquil saffron
#

I don't know if it's intentional but the last 4 releases of setuptools haven't been published on PyPI

dreamy urchin
#

A piece of the workflow has been disabled (cygwin), so now the releases need to be manually uploaded.

We have a PR that fix the failing job, if anyone else would like to review it, we could confidently merge

tranquil saffron
#

I'm sorry, I've never used cygwin and I'm not familiar with setuptools' testing infra so I don't feel confident reviewing the PR

dreamy urchin
#

No problems at all. Thank you very much for pointing the problem out though!

haughty bramble
dreamy urchin
#

Anything else not discussed in the PEP you feel that is still missing?

nimble bridge
river oak
#

At some point I think (personal opinion!) Python should follow the crates.io/docs.rs pattern and standardise the documentation generation interface. But setup.py build_sphinx is of course not the way to do it.

unkempt birch
#

A nice standardized docs (& tests) interface would be very nice. But I think installer is higher priority. 😉

delicate gale
nimble bridge
unkempt birch
#

nox -s docs 😛

robust yarrow
wanton salmon
#

poetry run task docs :p (for the record I don't even use poetry)

cloud sand
#

We must come up with a new standard to unify them all

tranquil saffron
#

I think make docs is where it's at

unkempt birch
#

What is the long term plan for https://github.com/pypa/setuptools/issues/889 ? IPython has just started using this, and https://github.com/twisted/twisted/blob/trunk/setup.cfg uses this too. It's a fantastic way to shorten complex extras with lots of minimum versions, etc. You can self-reference now, but that requires Pip 21.2+, while this works with older pips. I was very exited to see it, as not being able to build 'dev' and 'all' extras is one of the main things keeping my setup.py's around (and not virtually empty), but then found it breaks setup-cfg-fmt. It's also not something that has an obvious parallel when moving to PEP 621.

#

Is the direction here "setup.cfg is going to be deprecated anyway", perhaps?

wanton salmon
#

off topic but you can now define a (for example) "all": ".[docs,tests]" extra now and pip behaves as expected? I did not know that, but that's very nice!

dreamy urchin
river oak
unkempt birch
#

No, you can't use "all": ".[docs,tests] currently, but you can use "all": "pkg[docs,tests] for a package named pkg, and pip behaves as expected (and doesn't try to download the last release or something like it used to)

# 
name = 'pkg'
dependencies = []

[project.optional-dependencies]
tests = ['pytest']
all = ['pkg[tests]']

Works with Pip 20.2+. Ideally, .[tests] or just raw [tests] could be converted to the appropriate metadata by the backend, but at least one PEP would be affected (or a new one needed).

#

Hmmm, does that extra syntax work today? Looking at PEP 508 I see it mentioned (with ONLY when defined by a containing layer warning)

dreamy urchin
#

(specifically in setuptools I have the impression that it would fail RN because the process of handling the extras is convoluted... it first parses it as something unique to .egg-info and then it converts it back for PKG-INFO/METADATA...)

#

By containing layer is defining the extras I mean that the backend would produce the Provides-Extra field in the core metadata file, that would enable the extra to be used as requirement marker

unkempt birch
#

It's also really a mess to use for this purpose, say if you have "tests", "dev", and "all"

#

But does define everything once

#

The in syntax would be better.

dreamy urchin
#

packaging.requirements.Requirement('pytest; extra in "all tests dev docs"') seems to run with no errors

unkempt birch
#

I can try it out on setup.cfg

dreamy urchin
dreamy urchin
unkempt birch
#

Would this make the requirements messier in the metadata? I'm guessing the backend is not pulling these out, but just dropping them into the metadata?

dreamy urchin
#

@unkempt birch I like your comment in the setuptools issue. I think it would be nice to have the .[<extras>] syntax going forward since it allows specifying multiple dependencies at once. At the same time the explicit requirement marker extra can be handy for cases like extra != tests (exclusion).

The . syntax most definitely would require a PEP. The extra marker could potentially work nowadays? (It does not seem explicitly forbidden by the standards...)

unkempt birch
#

Do we need a dot? what about vanilla [tests]?

dreamy urchin
#

Both could work. The . seems to evoke some familiarity (I use pip install -e .[all] all the time in my packages), but any of the two would work fine...

unkempt birch
#

TypeError: 'in <string>' requires string as left operand, not NoneType when I try this in setup.cfg.

#

No, I think that is correct. "extra" is not defined if you are not "in" an extra.

dreamy urchin
#

Oh, that makes sense. It reminds me of a discussion somewhere about someone asking for extra to have a default value.

#

I guess there is no way around that right now then other than relying on pip 😅

unkempt birch
#

Yeah, even when chaining them with or, extra seems to not work.

#

pkg_resources.extern.packaging.requirements.InvalidRequirement: Parse error at "';extra ='": Expected stringEnd

dreamy urchin
# unkempt birch `pkg_resources.extern.packaging.requirements.InvalidRequirement: Parse error at ...

Can this error in particular be a problem with an outdated vendored version of packaging?
I tried the following:

>>> packaging.requirements.Requirement('pytest; extra == "all" or extra == "tests"')
<Requirement('pytest; extra == "all" or extra == "tests"')>
>>> packaging.requirements.Requirement('pytest; extra in "all tests"')
<Requirement('pytest; extra in "all tests"')>

And the parsing seems fine.
The bigger problem may happen when evaluating the markers, because as you pointed out extra can be undefined...

unkempt birch
dreamy urchin
#

I am impressed that the tests passed. I was definitely not expecting that...

#

Are you guys also testing an editable install?

unkempt birch
#

Not there, but I can try that. Also, I had to leave one item in each extra otherwise setup-cfg-format was wiping it out. Though I could try without extras.

#

This seems to work with empty extras too, minus setup-cfg-fmt's bug removing them.

#

(locally)

dreamy urchin
#

It is being a few months that I played with the part of setuptools that process the requirements, but what I remember is the following:

  • having the extras_require part is important so setuptools will generate the corresponding Provides-Extra field in the core metadata file (it can in theory be empty, like the one for https://github.com/pypa/setuptools_scm/blob/ccd9e7c1448044b0c832f641581738e3102ca768/setup.cfg#L72).

  • I remember editable installs depending on a .egg-info/requires.txt file, instead of the PKG-INFO. The way this file is generated and its relationship with the Requires-Dist is complicated, so it might not work for the editable install. Hopefully it is just an impression...

unkempt birch
#

Wow, actually, the metadata looks pretty good.

#

I think requires.txt actually does put the extra in correctly.

#

Or, well, sortof

# 
[metadata]
name = something
version = 1.0

[options]
packages = find:
install_requires =
    pytest>=6;extra == "test"

[options.extras_require]
test =
dreamy urchin
#

[:extras == "tests"]?

unkempt birch
# 
[:extra == "test"]
pytest>=6

[test]
dreamy urchin
#

Hehehe, it accidentally works 😆

unkempt birch
unkempt birch
#

Yeah, that flopped as expected.

tranquil saffron
#

The empty array idiom is a bit jarring, if we wanna do this we should amend PEP 621 to accept a list of strings in addition to a dictionary of lists for optional-dependencies

#

Or add a new property, like optional-dependency-groups

tranquil saffron
unkempt birch
#

I think ideally, if we are messing with PEPs, we should do it properly and allow [tests] or .[tests].

dreamy urchin
#

The separated extra marker does allow some other use cases though (e.g. "negative" extras or default extras)

tranquil saffron
#

I'm not sure if repurposing . is a good idea

#

This just means the current working directory when you do install .[foo] from the command line

#

It's going to mean something different in the spec file

unkempt birch
#

Current:

dependencies = [
   'pytest; extra == "tests" or extra == "all"'
]

[project.optional-dependencies]
tests = []
all = []

With dot:

dependencies = []

[project.optional-dependencies]
tests = ["pytest"]
all = [".[tests]"]

With nothing:

dependencies = []

[project.optional-dependencies]
tests = ["pytest"]
all = ["[tests]"]
#

Pretty sure Anthony doesn't actually read the contents of the comments, just skims them. I was literally suggesting an alternative to interpolation... Anyway, locked, so that's that.

#

Please someone other than Anthony write a pyproject.toml formatter. 😉 (likely, actually, as he hates toml)

robust yarrow
#

I thought a lot about dependency groups for the Hatch rewrite and became confident that they only should be used for features. Deps for testing, docs, etc. should not be there but rather should be part of config for an environment manager

dreamy urchin
#

@robust yarrow does hatch currently supports something like

dependencies = [
   'somedef; extra != "gpu"'
]

[project.optional-dependencies]
gpu = [ ... ]

? (just for curiosity)

tranquil saffron
dreamy urchin
robust yarrow
#

that example would be:

pip install /path/to/project[server,grpc]
dreamy urchin
# robust yarrow <@!819222905522487338> that would translate to a pip command and extras based on...

Not sure I understood, but I think you are talking about using hatch to manage environments, right?

I guess my question would make more sense If I had referred to hatchling instead?

I was curious if the build backend would process that kind of project metadata and produce the corresponding Provides-Extra and Requires-Dist core metadata accordingly (or if hatchling errors when extra markers are given explicitly)

robust yarrow
#

ah, no it would not error during packaging

dreamy urchin
#

Thank you @robust yarrow !

robust yarrow
unkempt birch
#

An idea for "negative/default" extras could be for #build - virtualenv could be the default, but using [venv] could "opt out".

tranquil saffron
#

I wonder if something can be done about venv being outrageously slow. What part is it that takes so long, it makes no sense that unzipping pip and setuptools should take 4 or 5 seconds?

#

Bit of a tangent, I'm sorry

cloud sand
#

Just use virtualenv 😂

tranquil saffron
#

I wonder if the installation could be performed with installer instead of pip or switch off whatever pip options are making pip take so much time

tranquil saffron
cloud sand
#

There can be multiple reasons why venv is slow:

  1. venv by default reaches out to the network to check newer version
  2. installing the packages is done with pip that is a generic purpose installer, starting up pip to do the useful work is 30% of the time, using installer could help
  3. On Windows the files are generated from a zip file, the Windows anti virus treats these operation as hostile, so there's a lot of overhead interupption by the antivirus checking each file (virtualenv works around this by extracting the file to folder, and then copying from there, this way the antivirus considers it safe, so less overhead).
unkempt birch
#

venv is an option for bootstrapping, having minimal dependencies has advantages. However, most users should just use virtualenv, so making it opt out instead of opt in would be great, IMO, as long as it worked.

#

build is designed to have minimal dependencies so it can be bootstrapped into environments; that's why there's a venv option. CC @quaint jolt

quaint jolt
#

I was kinda holding hope the default extras PEP would move forward

robust yarrow
dreamy urchin
#

I have done some effort in https://github.com/abravalheri/validate-pyproject, maybe if you would like to discuss joining forces? (probably it will still be necessary to keep 2 separated things since one aspect of validate-pyproject is to not have dependencies so it can be easily vendored, but we could discuss that)

#

(it is going to be fun for someone looking on PyPI: validate-pyproject and pyproject-validate 🤣 )

cloud sand
#

I've started work on a opinionated comment preserving pyproject.toml formatter, for now only supports the build-system table https://github.com/gaborbernat/pyproject-fmt/blob/main/tests/formatter/test_build_backend.py#L8, but will add project support soon too 🙂 https://pypi.org/project/pyproject-fmt/

PyPI
pyproject-fmt

Format your pyproject.toml file

GitHub
pyproject-fmt/test_build_backend.py at main · gaborbernat/pyproject...

Contribute to gaborbernat/pyproject-fmt development by creating an account on GitHub.

robust yarrow
#

I wish we could standardize on a TOML indentation

#

tomli-w does 4, others write 2, etc.

cloud sand
#

I'm in the 2 group but left it configurable 😂

robust yarrow
#

me too. I opened a PR on tomli-w changing the indent with others in favor but Taneli is not very keen on it, nor making it configurable

cloud sand
#

Do you put one or two spaces before a comment?

robust yarrow
#

2

haughty bramble
cloud sand
#

What would you make configurable?

stable jasper
robust yarrow
#

blasphemy

haughty bramble
#

I like to disable magic commas

wanton salmon
#

fair enough 🤷

cloud sand
#

What's magic comma?

humble palm
haughty bramble
#

@cloud sand in your formatter do you explode collections that end with a trailing comma?

#

Or always collapse them if they fit in the column limit?

cloud sand
#

For now I always expand 🙂 but will add at some point a fit in one line of possible

#

That is if they are no comments, if there's a comment in the array it must always be exploded IMHO

unkempt birch
#

So we have pyproject-fmt, pyproject-validate, and validate-pyproject now? 😄 To me, there are several, distinct things needed: a) validate the content of pyproject.toml to make sure it is valid, b) check for good practices like remembering certain fields, and c) formatting. I think having c separate from a/b would be fine, and I'd also be okay with all being separate tools, I can just add them all into pre-commit. 🙂

#

I'd honestly like black-style magic commas here, if it's something that gets changed a lot, I'd like it one-per-line, but requires often is very short.

cloud sand
#

Will fix 😂

cloud sand
#

It's a bit WIP at the moment

#

I can see also having a compact and a long mode

#

Because I personally prefer the compact syntax instead for nested tables

#

e.g.

[project]
urls."Bug Tracker" = "https://github.com/gaborbernat/pyproject-fmt/issues"
urls.Documentation = "https://github.com/gaborbernat/pyproject-fmt/"
urls."Source Code" = "https://github.com/gaborbernat/pyproject-fmt"

instead of

[project.urls]
"Bug Tracker" = "https://github.com/gaborbernat/pyproject-fmt/issues"
Documentation = "https://github.com/gaborbernat/pyproject-fmt/"
"Source Code" = "https://github.com/gaborbernat/pyproject-fmt"
#

issues welcome though 🙂

unkempt birch
#

I'd leave (nested/unnested) table structure alone, that can be a user choice.

cloud sand
#

Hence saying compact/long form

unkempt birch
#

https://scikit-hep.org/developer/pep621 now has tabs for several backends, including setuptools experimental 🙂

Scikit-HEP
Pure Python packaging

The Scikit-HEP project is a community-driven and community-oriented project with the aim of providing Particle Physics at large with an ecosystem for data analysis in Python. The project started in Autumn 2016 and is under active development.

bronze berry
unkempt birch
# robust yarrow I thought _a lot_ about dependency groups for the Hatch rewrite and became confi...

I do think I understand your point now, by the way, after seeing the extreme (hypermodern-python's dev environment) and the solver time and mess it causes. I don't think it's bad to have a [docs], [test], or similar extra, but not a "dev" or "all" that includes it, since it's only for use by environment setup, like readthedocs/nox/tox, etc. It's quite handy in cases were you can't reuse something like hatch (say [test] for cibuildwheel for testing the wheels)

robust yarrow
unkempt birch
#

Hatch

unkempt birch
#

I'm not getting the inverted extras to work. It says "Ignoring mplhep: markers 'extra != "noplot"' don't match your environment" then installs it anyway.

nimble bridge
lusty dome
#

I am convinced it's a suboptimal UX. What I don't want is a bolt-on fix that strands the legacy implementation. I'd also like to retain the degenerate behavior for use-cases where that's valuable (tests, experiments). All of that leads me to think there should be a separate layer where the validation is done. What I'd really like to see is something where user builds get the validation by default, but I haven't yet envisioned a way to make that more sophisticated use-case the default and not breaking the other use cases. The problem is also complicated by the conflation of name and version. It seems perfectly reasonable for a package to get a default version of 0, except when it happens to users that expected to get a version from elsewhere. At this stage, I have higher priority concerns on my plate (mainly distutils adoption and pkg_resources deprecation), so I'd like to defer this concern until things have settled down a bit.

nimble bridge
#

Hmm... Okie, I think I finally understood what you meant now.

#

I think the bit where we disagree is that you want to preserve the degenerate behaviour in some cases. I don't. I think changing the behaviour to be the tests and experiments is not a big deal -- it's certainly not an unclear failure mode as long as we mention what the user needs to do in the error message. It is very different from the subtle breakages where the error messages will always be esoteric because setuptools doesn't direct control those bits. Like automation uploading to 0.0.0 to PyPI which only fails the second time.

The fix for a broken test or experiment is adding a name= or version= in the right place, in the file that the exception originates from. The fix without the clear error is the same, except you have to trace back to it from underneath all the layers of abstraction on top -- pip wrapped under poetry, in a build system invoked in a CI pipeline. After you noticed a failure in later step, which failed because you already had 0.0.0 published.

The current behaviour silently breaks things, and I think changing subtle mistakes to break loudly is a good thing.

#

None the less, would you be fine with reopening the issue and stating that it's not a no-go but merely low priority right now?

quaint jolt
#

my two cents is that the name field should always be required

#

I can see value in the degenerate behavior for the version, especially because people might be relying on it

unkempt birch
#

Version has a somewhat reasonable possible degenerate value (0.0), but name (UNKNOWN?) does not.

#

Well, an empty string, but that breaks things (as it should...)

haughty bramble
#

inhibitted_halibut or something

robust yarrow
#

that's worse

restive talon
#

how about setuptools_lost_{basename(projectdir)}_{sha512_digest(projectdir)[:7]} and the default for version could be something as horrendous as YYYY.MM.DD+guessed.admirably

unkempt birch
#

You don't want a default name so that an automated release procedure (or manual if you are not careful) doesn't add some weird project to pypi.

#

Or weird version to PyPI.

#

Like I recently uploaded boost-histogram 0.0.0

robust yarrow
#

tbh, no need to change anything b/c pep 621 support will fix it

dreamy urchin
unkempt birch
#

But pep621 requires non-dynamic name?

#

(I’d be tempted to pull it from CMake for scikit-build otherwise)

dreamy urchin
#

That is true, but adhering to PEP 621 is not mandatory

tranquil saffron
#

Well isn't setuptools adding PEP 621 support? Will it follow PEP 621 apart from requiring the name be declared statically?

unkempt birch
#

Downstream tools, like PDM, Hatch, etc. are supposed to be able to depend on name being present in pyproject.toml if [project] is present. I don't see the point of a standard if it's not followed. Wouldn't it be better to modify the standard instead and allow name to be listed in dynamic?

tranquil saffron
#

There was also a suggestion for build to use the name field to detect circular dependencies ahead of time

bronze berry
#

@lusty dome : Got an issue with the _distutils_system_mod mechanism in Debian. Not sure how to make it play well with pip build isolation, so we still don't have correctly functioning editable installs.
When pip does an isolated build, it appears to the setuptools/distutils inside the isolated build that it's building for a virtualenv, so it uses the posix_prefix scheme, instead of the desired posix_local scheme. Can you think of a good way to detect that we're doing an isolated build for a system-wide install?

robust yarrow
#

adhering to PEP 621 is not mandatory

why?

nimble bridge
#

I'm going to not say much more right now (just woke up), but suffice to say: Not adhereing to a standard which one of the tool's current maintainers co-authored and other maintainers participated in the discussion of, that went through rigorous design discussions, basically feels extremely disrespectful of the work that's been put into that effort.

river oak
#

I think what @dreamy urchin intended to say was that you don’t need to use PEP 621 to define metadata. As in, if the backend can’t work within the scheme proposed by PEP 621, just use your own config format (like old-style Flit, setup.cfg, etc.). PEP 621 makes it easier to move the project from one backend to another, but does not force every PEP 517 backend must use it for configuration. (Plus if a backend can’t work within the PEP 621 structure, a project using it likely can’t be converted to use another build system anyway.)

nimble bridge
#

Fair. All I feel strongly about is that if you use the project table for metadata, then you ought to adhere to what PEP 621 says is required/not required.

#

If setuptools has tool.setuptools.whatever section like flit, sure -- it can do whatever it wants there. If it, however, uses the project table -- it ought to do what the standard says. 🤷‍♂️

dreamy urchin
dreamy urchin
# river oak I think what <@!819222905522487338> intended to say was that you don’t need to u...

That is precisely the point. Thank you very much @river oak

This approach is also discussed in the text of the PEP:

(compared to opting-out of this PEP entirely, which is also possible).

So the backend don't need to adopt pyproject.toml for configuration or if they do, they don't need to adopt the project table.

In the case of setuptools, users also can specify the metadata in setup.py if they want to...

dreamy urchin
nimble bridge
#

Awesome. Apologies for the somewhat spirited response earlier -- I hadn't properly woken up. 😅

dreamy urchin
tranquil saffron
#

Ah ok thanks, I misunderstood!

dreamy urchin
bronze berry
lusty dome
#

Ack. I appreciate that. I have seen requests for virtualenv and pip to copy that module to virtualenvs and isolated builds.

unkempt birch
dreamy urchin
# unkempt birch <@!819222905522487338> Do you have a ballpark figure for "soon" (days, weeks, a ...

To be sincere, my wish is to get it down to days... But we never know which other problems may appear.

My priority so far has been to reduce the risk of breaking existing packages, so I tried to test exhaustively and engage with the community for beta-testing...
Hopefully things will go right, but I am also prepared for the chances of having to revert the changes shortly after they land. Setuptools is used everywhere and therefore every single obscure aspect of its implementation is depended upon...

#

There are some integration tests in place for pandas, kiwisolver, brotli, sphinx, botocore, mypy, pytest and pandas... I also have been testing locally with the packages people proposed in the Discourse thread (linesep, minegauler) and the ones you helped me with (pybind, plumbum and the scikit-hep templates)...

I have my fingers crossed that things will go well

dreamy urchin
#

Ok, considering that my last comment in discourse had "thumbs up" from most of the people that were testing the changes before, I believe that not many people will still try the experimental branch. So I am incline to proceed with the merge today 🙂

dreamy urchin
#

(Lucky me... just pushed the merge but GitHub is experiencing issues with webhooks and actions)

nimble bridge
#

🎉🌈🎉🌈

dreamy urchin
#

Thank you all for the help!

wanton salmon
dreamy urchin
#

This PR was complicated so I was going to merge it manually anyway. But I was counting on the CI...

Now it is running! Eventually the new archive distributions will be published...

robust yarrow
#

all our CI is failing on the new version, looking into it now

# 
ignored error: ValueError - invalid pyproject.toml config: `project.license`
...
TypeError: string indices must be integers
#

oh, no support for pep 639

dreamy urchin
#

Sorry Ofek, support for PEP 639 is not implemented yet.

Temporarily you can use tool.setuptools.license-files to pass an array of glob patterns for files that should be included in the archive

dreamy urchin
#

(The default behaves like the usual: LICENSE*, NOTICE*, etc...)

robust yarrow
#

& neither I nor a team mate can repro locally...

dreamy urchin
#

Having a look on the error, it seems that the error comes from the fact that setuptools currently requires license = {text = ...} (exactly what you mentioned about PEP 639)

robust yarrow
#

right but we do:

[build-system]
requires = [
    "hatchling>=0.11.2",
    "setuptools; python_version < '3.0'",
]
build-backend = "hatchling.build"

but tox is for some reason using the setup.py:

GLOB sdist-make: /home/vsts/work/1/s/postgres/setup.py
ERROR: invocation failed (exit code 1), logfile: /home/vsts/work/1/s/postgres/.tox/log/GLOB-0.log
================================== log start ===================================
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py:100: _ExperimentalProjectMetadata: Support for project metadata in `pyproject.toml` is still experimental and may be removed (or change) in future releases.
  warnings.warn(msg, _ExperimentalProjectMetadata)
dreamy urchin
#

O.o

#

Yeah. That is very weird

cloud sand
#

tox 3 does to find out the name of the project

#

But shouldn't use it for actual packaging

robust yarrow
#

looks like isolated_build = true fixes it

cloud sand
#

Yeah

unkempt birch
#

I'd always use isolated_build = true, otherwise tox is not doing what pip would do. (at least if you have a pyproject.toml file)

unkempt birch
#

dprint (Rust) is a pretty nice auto-formatter for TOML. Someone needs to bundle it up in a wheel and write a pre-commit hook for it. 😉 https://dprint.dev/playground/#code/NoIwrglgNgJgtAZwJ4IC4FMC2BdAUAJ3QEdJCEACAXnOF3PvICIF1UwAHVAey6gQD5KAFgCsjADR0GzVh268EAfQQBjTMG6Yo2QUIB0ARj0AmRuQDE5AMJdMmAIYV7MGOhi484aPBD2VAa3QAOxgqJhY2Th4+PS9YRUxWe0ZcXEsAFQALCAoc8ntyFVtEoNRcDWi9CLlopVUcXAB3fAgMRW4w5nwVAHoQHjRFbLQuAHN8e0wegDd0fAQILiC9diQU8vkoFaQMND0IIIhFLk5FoIQ8ZxgT1ApqWgYmOAmJJ8RMrkaoLhV7PlfGHBEKgWipUHAHPhAvMJLhAcDQeCikEAGYQUaMPC4NFQDD4Rr2fCHIKjO40RhzfBcfCvbAWazFAqETBcWbuNLkACyYFxEHYUHQ5CgB3QFCWUCQ5AgKPyqCF6EccqWgvYZGCSvw+XICEm6DgwqCgqKdnVUoo3xJuF2qHY9lQmQQlGAjGtCABmDAqGF-PQAINosxuCAA/language/toml

robust yarrow
#

I guess I can do a patch release capping setuptools, but that's not ideal

dreamy urchin
# robust yarrow I guess I can do a patch release capping `setuptools`, but that's not ideal

Sorry @robust yarrow, PEP 639 is on my radar, but it will take some time. I want first to address all the issues with the thing we just implemented, ideally also have at least a look on PEP 660, and then I can go back to it.

If you want to submit a PR, that would be very welcomed.

Alternatively, I imagine that maybe you could temporarily rename pyproject.toml before calling setuptools? Would that work for you?

robust yarrow
#

I imagine that maybe you could temporarily rename pyproject.toml before calling setuptools?

I capped it but that's a good idea thanks! I'll do that if there's a CVE before PEP 639 support

true warren
restive talon
river oak
#

Yeah we should’ve added those when it’s marked as final

#

PR would be much appreciated

true warren
river oak
#

Good enough to simply mention Flit and PDM, IMO

unkempt birch
#

I'd personally go with Flit, Hatchling, setuptools experimental, and PDM, which I think are the major ones currently. Whey seems to be very quiet, and Trampolim too.

true warren
#

My inclination would be toward mentioning more rather than fewer, as long as it'll be taken as a listing of example implementations, rather than a recommendation or endorsement.

unkempt birch
#

Also, that's a page (meaning it remains up to date), not a post. 😛

true warren
#

Ooooooo, indeed. Thx for the correction. 🙂

nimble bridge
#

PEPs aren't documentation -- I'm inclined to say that it's fine to leave it as is or add, like, one or two examples there.

true warren
#

@nimble bridge That was the other angle I was considering -- if there's maybe value in having that part of the PEP reflect the state of the ecosystem at the time the PEP was finalized... in which case, it would be best to leave it as-is.

dreamy urchin
#

It seems that the Mergifyio bot needs to update some permissions (pypa/setuptools > Settings > GitHub Apps).

Does anyone have the user rights to update it? (Or is it the idea to remove this bot completely from the repo?)

nimble bridge
#

I can do that.

dreamy urchin
#

Hi @nimble bridge, that would be helpful! Thank you very much. I do use it a lot to rebase PRs (much quicker than adding a new remote, pulling the PR and rebasing it manually...)

nimble bridge
#

Done.

#

FWIW, I really like the GitHub CLI.

#

With my git aliases, doing that is as simple as:

g sync
gh pr checkout 100
g rebase main
g pushf

#

(I have alias g=git)

river oak
#

gh pr is the only subcommand I use, the rest of the command is more or less useless

cloud sand
#

GH repo clone is also handy for me

#

Plus, GH repo view Especially with the -w flag

river oak
#

I’m weird and prefer to type the URL into my browser manually instead

dreamy urchin
#

Thank you very much!

I still haven't got my head around gh (and @Mergifyio rebase is quite handy)...

#

(It is very likely that unconsciously I feel prejudice just because their documentation mentions conda as an installation method, despite of the follow up Linux-specific link... Personally I run away from conda unless absolutely necessary... Even mamba feels too slow -- Long live pip!)

humble palm
#

I also only use gh pr checkout 100, should learn a few more but have some of my own commands:

repo is like gh repo view -w, can also do like repo pulls or repo actions to open that tab

upstream is like repo but for (surprise!) the upstream

ci opens the relevant CI webpages iff there are GHA, Travis or AppVeyor config files. can filter like ci gi to only open GHA if there's also Travis

unkempt birch
#

I love gh pr checks too. Also gh pr/repo view -w, (though already mentioned above). I've also used gh release create a tiny bit. The other commands look interesting, but I guess I do center on gh pr most of the time. I just realized that gh co is a shortcut for gh pr checkout, nice. 🙂 I guess I've used gh gist a tiny bit.

patent lotus
#

This needs to bee an invite but its a link to this channel

unknown.png
crude bolt
#

i'm trying to implement my own finder and i'm getting crashes because setuptools expects that the Distributions returned find_distributions(self, context: DistributionFinder.Context = DistributionFinder.Context()) are instances of setuptool's vendored importlib_metadata instead of importlib.metadata

#

(as of python 3.8)

#

is there some way i should get the Distribution class to use from the caller oder something else i can do to properly handle this? my current workaround is try: from setuptools._vendor.importlib_metadata import PathDistribution

dreamy urchin
#

Hi @crude bolt, maybe it would be better to direct this question to importlib-metadata. Am I correct to understand that this error would happen (forgetting a little bit about setuptools) if someone uses your finder together with the latest version of importlib-metadata? (Maybe this is a backwards compatibility issue?)

crude bolt
#

i don't think so, it happens because setuptools expects that the Distribution instances to have attributes the python 3.8 importlib PathDistribution does not have. if i use importlib_metadata (either installing it on my own or importing it from setuptools._vendor) it works fine. i'm also surprised that this isn't a problem with the built-in finders, so i'm not sure whether i need to change my finder or setuptools needs to accept built-in importlib

dreamy urchin
#

Thanks @crude bolt can you file an issue in setuptools then, with a minimal reproducer for your custom finder? I will try to have a look soon.

crude bolt
#

will do

dreamy urchin
#

thank you very much

crude bolt
vernal harbor
#

When I use the universal2 version of Python and write an extension I don't need to specify -arch arm64 -arch x86_64 in extra_compile_args but I do need to specify them in extra_link_args but I wonder if that should be the case.

vernal harbor
#

I guess if you're trying to link against an outside package there's not a way I know of to force it to install the universal2 version cleanly if they distribute platform-specific versions pip will prefer instead during a build

#

but that seems like an esoteric use case

lusty dome
#

@dreamy urchin I'd like to cut a release of Setuptools. Any reason to delay?

dreamy urchin
#

(Sorry for the delayed response)

lusty dome
#

no prob

#

62.5.0 going out now.

dreamy urchin
#

Does anyone knows if the artefacts produced by build_clib are added to the wheel distribution at all?

I went trough the code and the old distutils docs and I have the impression that the binary files produced by build_clib are only used when extension modules produced by build_ext want to link against them...

#

The code for distutils/build_clib seems to place the generated files in a temporary directory...

The code for distutils/install_lib seems to only ensure that build_py and build_ext did run, not build_clib.

quaint jolt
#

I am not certain but that makes sense to me

#

just looking at your research I'd say install_lib not ensuring build_clib is run seems like a bug

#

but try to make sure build_ext itself doesn't do that

dreamy urchin
#

Thank you very much Filipe. It seems that build_ext will check for libraries created by build_clib but not exactly ensure the command runs.

For the purposes that I am investigating right now (PEP 660) , I think I will just assume build_clib does not create any artifact that is distributed, and everything is used by build_ext... At some point I will try to test that theory for editable installs...

quaint jolt
#

that sounds good 👍

restive talon
#

@dreamy urchin @lusty dome is it intended that the finalize dist hook happens before setup.cfg metadata is parsed?

dreamy urchin
# restive talon <@819222905522487338> <@720786343097270374> is it intended that the finalize dis...

I always find this behaviour curious considering that config files do change options...

I did some investigation and find out that this hook was introduced in https://github.com/pypa/setuptools/pull/1877 as part of the finalize_options() method (which was originally stablished by distutils and intended to run at the end of Distribution.__init__).

At that point in time, the parse_config_files method already existed, so we could suppose that it was kind of intentional (but only Jason could clarify that). Maybe the naming could be improved to make it more obvious?

I don't know what would be the impacts in the ecosystem of chaging the momment we trigger the plugins. Do you have a use case in mind?

restive talon
dreamy urchin
#

That sounds like a fair request, we could think postponing this hook or creating a different one for backward compatibility. @lusty dome do you have a suggestion?

lusty dome
#

I haven't had a chance to look into the code. I'd want to investigate whether it's likely that we could postpone the hook (or more likely, push up the config file parsing)... or if there are other dependencies (does config parsing depend on any actions in finalize_options. If parsing happens as part of finalize options, it may be possible to simply increase the priority of that finalizer so that it happens before other unprioritized hooks (such as setuptools_scm). I don't recall any specific dependencies, and I'm open to it changing.

dreamy urchin
#

If parsing happens as part of finalize options,
I think that right now the parsing is separated from finalize_options. If I am not wrong, distutils.core.setup runs parse_config_files as a separated statement, after the distribution object is initialized. On the other hand Distribution.__init__ runs finalize_options at the end of the initialization process.

dreamy urchin
wild flicker
#

hi! i'm trying to transition a setup.py-using cython project to using pyproject.toml, and i noticed that building wheels also includes the pyx/pxd sources files, which doesn't happen with setup.py based wheel building. i can't find a MANIFEST.in-like mechanism for excluding them. am i missing something?

dreamy urchin
#

Can you try include-package-data = false in your pyproject.toml tool.setuptools?

haughty bramble
wild flicker
#

in the wheel and sdist, but in the latter i'd expect them

#

@dreamy urchin putting that into setup.py at least doesn't change anything

#

adding it to the tool.setuptools section of pyproject.toml also changes nothing

dreamy urchin
#

@wild flicker, I tried again with a simple example (that contains a package directory), and include-package-data = false seems to work:

rm -rf /tmp/mypkg
mkdir -p /tmp/mypkg/mypkg
cd /tmp/mypkg
touch mypkg/hello.pyx

cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools", "Cython"]
build-backend = "setuptools.build_meta"

[project]
name = "mypkg"
version = "42"

[tool.setuptools]
include-package-data = false
EOF

cat <<EOF > setup.py
from setuptools import setup, Extension

setup(ext_modules=[Extension("mypkg.hello", ["mypkg/hello.pyx"])])
EOF
pipx run build

unzip -l dist/*.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
    74832  2022-07-04 22:36   mypkg/hello.cpython-38-x86_64-linux-gnu.so
       47  2022-07-04 22:36   mypkg-42.dist-info/METADATA
      103  2022-07-04 22:36   mypkg-42.dist-info/WHEEL
        6  2022-07-04 22:36   mypkg-42.dist-info/top_level.txt
      376  2022-07-04 22:36   mypkg-42.dist-info/RECORD
---------                     -------
    75364                     5 files

Without include-package-data = false it shows .pyx inside the wheel. I suppose you can also use exclude-package-data to be more fine grained.

wild flicker
#

weird. neither including nor excluding nor putting them into the gitignore work for me. must be some weird interaction somewhere

#

disabling setuptools_scm also changes nothing

wild flicker
#

ha! actually it works, after i do a git clean -xdf to get a clean slate. thanks 🙂

cloud sand
acoustic cradle
#

PEP 503 doesn't control filenames

#

just the simple repository URLs

#

PEP.. 427? controls wheel file names

#

and there's a 6xx PEP I think that controls sdist filenames

robust yarrow
cloud sand
#

https://discuss.python.org/t/amending-pep-503-and-pep-427-on-package-normalization-rules/17226 🙂

Discussions on Python.org
Amending PEP-503 and PEP-427 on package normalization rules

Both PEP-427 - Escaping and Unicode and PEP-503 - Normalized Names require the wheel/sdist files distribution name part to be normalised by replacing all non-alphanumeric characters wit the _ character. This means that a package name of a.b-c.d gets transformed to a_b_c_d. I’d like to propose to not make this escape for the . character. The . ...

cloud sand
acoustic cradle
#

Yes

#

we need a single name to refer to it as

cloud sand
acoustic cradle
#

and human beings are bad at determining the difference between . - and _

#

oh sorry, I thought you meant normalizing . at all

#

PEP 503 normalizes to - not _

cloud sand
#

that part 🙂

acoustic cradle
#

that's mostly historical

cloud sand
#

I meant normalizing the distribution name as the PEP implies, no/

acoustic cradle
#

filenames are normalized to _, because - otherwise has special meaning

#

(as does .)

cloud sand
#

What is the special meaning of .?

#

Historically AFAIK was used to separate namespaces in packages, but if we do this normalization that seems no longer possible

acoustic cradle
#

it splits the filename from the extension

#

and it's used in version numbers

cloud sand
#

shouldn't the extension only apply for name.split('-')[-1] of the wheel/sdist name?

acoustic cradle
#

I guess you could have argued to normalize to . instead of _

cloud sand
#

that allows it to use it both for distribution name and version numbers

acoustic cradle
#

the royal you, not you specifically

#

but the choice of character isn't super important other than it's not -

cloud sand
#

I can accept all non-alpha numeric to be _, minus the .

#

to still allow package names to encode namespaces within the distribution name

#

the charachter we normalize to doesn't really matter to me, just want to be able to signal namespaces in the filename 🙂

#

and there's also an element of backwards compatibility and setuptools not really following the current recommendation

#

If I have a rule in my HTTP layer that user A can upload packages starting with https://pypi.org/simple/zope. this normalization breaks that because I can no longer use such rules because https://pypi.org/simple/zope_ could be both a namespace package and a root package named zope-magic 🤔

acoustic cradle
#

PyPI treats zope-sqlahcmeny and zope.sqlalchemy as the same name, and that's something that's unlikely to change, nor should I think it should change.

If people want to be allowed to say zope.sqlalchemy instead of zope_sqlalchemny in their filenames.. it's probably fine? But wheel files IIRC escape . to \_ .. or at least the spec says it does, nto sure if it does in practice

#

PyPI treats them the same because if a human says "install zope sqlalchemy", it's not clear if that means zope.sqla or zope-sqla

cloud sand
#

setuptools does not in practice 🙂 and pypi leaves it untouched

#

so technically yes you can upload both a.b and a-b package names under the same a.b project

acoustic cradle
#

pypi doesn't do much validation of the filename at all I think, which given the relevant PEPs we probably should

cloud sand
#

What I'm arguing here is that allowing . in the filenames (but in the PyPI backend still normalizing to -, e.g. to check if files is already uploaded) should be safe and what users would expect; that way people can use URL pattern permisioning to setup namespace upload rules 🤔

acoustic cradle
#

Do you mean in the PEP 503 URL (e.g. /simple/$THISPART/) or in the filename itself ($THISPART-1.0.tar.gz)

cloud sand
#

filename itself 🙂

#

in the URL is fine

#

that can/should be normalized

acoustic cradle
#

I don't personally see a problem with that, but you'll probably need to get the PEPs ammended if you want that to be PEP valid

cloud sand
#

The way PEP-427 is formulated zope.sqlalchemy-1.6-py2.py3-none-any.whl should be zope_sqlalchemy-1.6-py2.py3-none-any.whl

acoustic cradle
#

the fact nobody is listening to PEP 427 on . is a strong signal that it's fine

#

and might be enough to convince folks to just ammend PEP 427 and 6whatever for leave . alone

#

I'm bad at spelling, pretend I spelled those words correctly

#

You're looking to amend 427 and 625 though, 503 only directly controls the /simple/$THISPART/ URLs (it starts to control sdists by 625 referencing it)

cloud sand
acoustic cradle
#

it's a little confusing because there's multiple things defining an escaped/normalized name, and one of them refers to another one

robust yarrow
#

hmm 🤔

acoustic cradle
#

I'm actually not clear if PEP 427 intended for . to be escaped or not

#

the text says:

Each component of the filename is escaped by replacing runs of non-alphanumeric characters with an underscore _:

#

but

#

the code says

# 
re.sub("[^\w\d.]+", "_", distribution, re.UNICODE)
#

that . in the character set is confusing, unless it was meant to be \.

robust yarrow
#

I'm pretty sure Hatchling does it right with pep427(pep503(name))

acoustic cradle
#

it does it right for PEP 625, since they implemented escaping as "do PEP503, then do PEP427"

cloud sand
#

Yeah but does not do it right for PEP-427 and its regex specification as:

import re

re.sub("[^\w\d.]+", "_", "a.b", re.UNICODE)
'a.b'

robust yarrow
acoustic cradle
#

oh

#

TIL

#

well

cloud sand
#

Though the text seems to contradict the regex specified

acoustic cradle
#

so in PEP 427 the code and the text contradict each other

#

and in practice the code is what everyone is using, and what makes sense

#

since the text would also mean py2.py3 becomes py2-py3

cloud sand
#

seems setuptools decided to follow the regex and not the text, while hatchling followed the text, hence the difference

acoustic cradle
#

er py2_py3

#

and 1.0 would be 1_0

cloud sand
#

though PEP-625 is not yet accepted, so likely would suggest amending that to only do PEP-427 and not PEP-503

acoustic cradle
#

so probably PEP 427 just needs it's text fixed

#

since the text is just wrong

robust yarrow
#

if we change 427 it should be to match 625 imo

nimble bridge
acoustic cradle
#

we actually figured out the text and the code in PEP 427 are at odds with each other

nimble bridge
#

Ah, ok.

acoustic cradle
#

and the code is what (most?) everyone is doing

#

and also what makes the most sense

#

since the text would require escaping version "1.0" as "1_0"

nimble bridge
#

So... do we want wheels that are zope.interface-1.0.0-py3-none-any.whl to be valid? Or push them to be zope_interface-1.0.0-py3-none-any.whl? Or something else?

robust yarrow
#

zope_interface-1.0.0-py3-none-any.whl

imo

acoustic cradle
#

I'm pretty sure that PEP 427 intended for zope.interface-1.0.0-py3-none-any.whl to be valid

#

because it doesn't call out name as a special case

nimble bridge
acoustic cradle
#

it just says that every segment of the filename needs t obe scaped like X

#

and if you apply . -> \_ on anything but names, then the later segments break

robust yarrow
#

but pep 621 says tools should internally store name using pep503

acoustic cradle
#

eh

#

that sounds like something a PEP shouldn't have an opinion on

nimble bridge
#

IIRC, the idea is that METADATA should have the display name, but all operations should use the canonical name.

#

And because - doesn't work in wheel names, changing that to _ is... consistent?

#

My only worry is zope_interface and zope.interface wheels being uploaded, as an attack or whatever.

acoustic cradle
#

pypi can normalize the filename like we already do for project names

#

for uniqueness

nimble bridge
#

And, you have access to the upload tokens for package at that point... so... you can do a bunch more?

acoustic cradle
#

PyPI has Project.name and Project.normalized_name, and unique-ness is checked against normalized_name

#

Name is just used for display

#

I would probably say that requiring any internal uses of - within a filename segment to be escaped to _, but otherwise not caring about what a tool puts in the segments as long as it's valid is probably the right way

#

and just require that tools interpret those sections normalized

robust yarrow
#

I'm strongly against outsourcing resolution of ambiguity to PyPI. the name part should only have one valid representation

nimble bridge
#

Honestly, I'd prefer if everything that's in a distribution or whatever is normalised, with the display name only being in METADATA. I think the argument is that . can safely be in distribution files -- and... yes... at the cost of forcing normalisation on everyone on every ingestion? It's not costly, but it's cognitively annoying.

acoustic cradle
#

I don't think anyone is expecting projects to normalize names inside of the metadata itself

#

I guess I should soften that

#

I think that that it's unlikely we gain consensus on requiring names to 100% always be normalized in every context

#

so there's always going to be places where the unnormalized name exists, ideally as a display name only

robust yarrow
nimble bridge
#

every context -- of course not. I want zope.interface in the UI, not zope-interface.

But we definitely want to only have zope-interface in simple API, because tooling and consistency. The same argument applies for zope_interface in the distribution filenames IMO.

acoustic cradle
#

I don't personally care what the filename name is, as long as - is escaped

#

so I don't really have a horse in this race

robust yarrow
#

so 503 for 621, 625, and now 685. we should use 503 everywhere

acoustic cradle
#

I don't think 621 actually uses 503 FWIW, I know it calls out that they should store it internally that way, but that PEP cannot actually dictate how tools operate internally, and later on in PEP 621 it explciitally calls out that requiring PEP 503 normalization was rejected

#

685 doesn't require storing extra names normalized, just defines how they're compared.

nimble bridge
#

Requiring PEP 503 in the name key was rejected, precisely to allow display names like zope.interface to be presented as such.

acoustic cradle
#

AFAIK nobody is against using PEP 503 for comparison of these things

cloud sand
#

I think PEP-503 is fine for comparing stuff, but the package names for wheel/sdist should keep PEP-427 so the package namespace can be infered from the package name

acoustic cradle
#

and AFAIK nobody is against PEP 503 normalization for the simple API

#

it's just the question of whether PEP 503 normalization should be mandated in places where it's otherwise not required (like the filename)

robust yarrow
acoustic cradle
#

that statement should be removed from the PEP tbh, or moved to a non-normative recommendation

#

it has ntohing to do with the spec in the PEP itself

nimble bridge
robust yarrow
#

that will lead to subtle bugs over time

acoustic cradle
#

well it should be removed there too most likely 😛

#

It's a trade off

#

it's a lot ahrder to get subtle bugs if you enforce that names are normalized, and only ever exist in normalized fashion

#

but like I said, there's no way we actually get consensus on that

nimble bridge
#

I mean, I want PEPs to be left untouched -- making minor updates in the specifications pages.

Now, assuming we move everything over. Which... is a few weekends.

#

FWIW, pip's internals and packaging.metadata have display_name and canonical_name. So, no mixing of the two.

robust yarrow
#

let's back up 🙂 ... what is gained by allowing inconsistency?

acoustic cradle
#

inconcistency is going to be allowed

#

you'll never get consensus on removing it

#

what is gained is that people are attached to their names

#

and get really upset if you say, change foo.bar to foo-bar in the PyPI UI

nimble bridge
#

There's namespacing. Having foo.bar.baz be different from foo.bar_baz in the distribution names makes things work better with Artifactory.

robust yarrow
#

to be clear I get the display name bit. I'm asking why in every other spot to not go with 503

nimble bridge
#

The comment I just made -- tooling like Artifactory, which basically does namespacing permission handling based on names.

robust yarrow
#

I think this is a bad decision, but I'll make Hatchling not follow the regex over the weekend

nimble bridge
#

Yea, I think I agree too FWIW. But setuptools does this today, so it's reasonable to do this for now.

cloud sand
#

We could change to normalize . to _ but that's probably a longer transition period ask and we'd need to communicate that change with stakeholders like Artifactory

#

Would also require enforcing that namespace packages can be only alphanumerical to allow deducing from a_b_c the namespace the user is uploading too.

nimble bridge
#

I mean... we could enforce normalisation rules coupled with a Python version (3.12, or more likely, 3.13) and expect that the platforms will add support in time for that. Then, 5 years after that, everything being built will need to be normalised and we've successfully transitioned to normalised names.

acoustic cradle
#

I don't really like tying things to python version

#

personally

nimble bridge
#

Care to elaborate?

acoustic cradle
#

I don't think it makes sense, what version of Python is "foo-1.0.targz" associated with to know how it should be normalized?

nimble bridge
#

Oh, I meant on the build tooling side.

acoustic cradle
#

is it the python version of whoever happened to create it? they can control python version as easily as they can control build tool version

#

and if we're going to require it, ideally pypi should enforce it

robust yarrow
#

seems weird for non-CPython stuff. like defaulting to utf-8 on Windows change

acoustic cradle
#

I understand the desire to have some project agnostic sigil for migrating some change in

#

I just think it isn't really going to work well in practice

#

and we'll end up with a bunch of things not respecting it

#

or not capable of respecting it

nimble bridge
#

Like, 3.12 enforces that build-backends generate sdists/wheels with that format. Then, starting 3.17, installers start using only the enforced names to consume and reject other variants.

Anything that doesn't work, has been unmaintained for 5+ years and... it's bitrotten?

acoustic cradle
#

you'd be surprised how many projects haven't had a release in 5+ years and still work just fine

#

a number of the projects today were people ging

#

"I haven't released this since 2013, and it's still getting 5 million downloads?!"

nimble bridge
#

Sure, how many of them have a . in their distribution names? :P

acoustic cradle
#

dunno, probably > 0

#

seems trivial to just say that installers should always accept the un-normalized name if they're able to parse it

#

why break things that don't need to be broken

nimble bridge
#

I mean, in this case, I'm pretty sure it's not worthwhile FWIW.

#

Ah, I never posted one of the things I wrote. XD

#

I think I'm firmly in the "don't care" category for this at this point. This inconsitency already exists in the ecosystem and pip's gonna have to deal with it for basically forever.

Whether individual backends support having the . is upto them. 🤷🏻‍♂️

acoustic cradle
#

Yea, that's more or less where I'm at. If people feel strongly and really want to force normalized names in filenames.. I also don't really care

#

I don't think it affects much either way

#

I do think there's value to aligning PEP 427 and PEP 625

nimble bridge
#

And whether we should update the specs to reflect the current reality -- yea, we should?

acoustic cradle
#

since right now they require different things

nimble bridge
#

jinx? :P

#

Ah, no, that's not the word.

acoustic cradle
#

whether the alignment is "the name must be fully normalized" or "the name must have - escaped to _, but otherwise any valid name, normalzied or not is fine", Idon't really care

#

Pratically speaking, the latter is what is used in practice for wheels, what the code snippet in PEP 427 implements (but not the text, but I'm pretty sure the text is a bug in the PEP), and PEP 625 is not accepted or implemented, it's easier to align on the latter

nimble bridge
#

So... PyPI should probably enforce that normalised_name-1.0.0-py-none-any.whl and normalised.name-1.0.0-py-none-any.whl are not considered distinct?

acoustic cradle
#

(but as @nimble bridge mentioned, build backends can emit normalized if they want, we're unlikely to require unnormalized names)

#

yea

#

it should

#

might even make sense to break the filename parts up for distinctness

#

or normalize the entire thing

#

normalized version, etc

nimble bridge
#

We need .whl2!

#

And a .sdist!

acoustic cradle
#

pls no .whl2

#

pls yes .sdist

#

even though I think people didnt' like that suggestion when I made it ages ago

nimble bridge
#

Which reminds me, do we still care about putting the actual contents of the wheel into an inner-archive within the .whl?

acoustic cradle
#

I think it would be a good thing to do

#

we got tons of extra compression that way IIRC

nimble bridge
#

That's what I remember too.

acoustic cradle
#

and if we leave the dist-info, WHEEL can be rev'd to 2.0 which conformant tools should error on

#

I had to step away from packaging stuff mostly during the pandemic

#

my brain got real bad

#

ADHD and all that

#

on meds now though! so hopefully able to sustain participating again, and inner archive wheel is one of the things I want to tackle

#

(but don't let that discourage someone else from doing it if they want to)

tranquil saffron
#

(or if anybody else knows)

dreamy urchin
#

You can have an arbitrary mapping in package_dir... In theory a person can have a frankenstein project layout where each package is stored in a folder with a complete different name...

The empty string works as a special value. With the empty string you are mapping only the parent directory, but not the package folders themselves.

#

The configuration in this repo for package_dir is not wrong... but it is unnecessary (and if I had to guess, breaks setup.py develop?)

tranquil saffron
#

okay, I thought you were always meant to map the parent dir

#

it looks like it does break develop, easy-install.pth points to the project root

dreamy urchin
#

If the maintainer is interested, he can just remove it and use auto discovery (although I think the PEP 660 support should fix editable installs...)

#

Sorry, autodiscovery will not work because the project does specify ext_modules, right?

tranquil saffron
#

Yeah

tranquil saffron
#

I just changed it to '': 'src' - the maintainer was ok with that 😛

dreamy urchin
#

Thank you for spotting that. It is really better this way