#setuptools

1 messages · Page 1 of 1 (latest)

robust yarrow
nimble bridge
#

My understand is yes.

#

Setuptools is trying to get out of the business of "do everything about packaging" to "be a pyproject.toml based backend with a setup.py file".

#

(as I understand it anyway)

#

P Ganssle's post even says that, although he likely used much better words.

robust yarrow
#

just wanted to confirm as I'm switching Django away from setup.py and a triager asked me to post on their mailing list b/c they had a python setup.py bdist_rpm script which I think is unused

robust yarrow
river oak
#

There are also multiple tools to create RPM either from wheel or the source tree, Red Hat also was/is working on a new tool to create RPMs for their Python packages. They can also switch to one of those if RPM is really needed.

humble palm
somber wren
#

hi all - does pkg_resources pull latest information from repositories like pypi or is it only an interface to what is locally currently known about packages?
(i want to check the version of a package and see if it is outdated, so i need to know if i should bother using the pypi api or just use pkg_resources 🙂)

river oak
#

It is only an interface to query locally installed packages (and a disfavoured one—you should use importlib.metadata instead)

quiet latch
#

packaging isn't discouraged, right?

river oak
#

It’s not, but its feature set is discrete to importlib.metadata (pkg_resources actually depends on packaging internally)

quiet latch
#

I know it's discrete to importlib.metadata which is why I ask since some of the stuff pkg_resources exposes is available in packaging

somber wren
#

Cheers!

river oak
nimble bridge
#

Any pointers on who are the best folks to reach out to, regarding a security report in setuptools?

#

@dreamy urchin see ☝🏻

quiet latch
#

setuptools 64.0, exciting!

#

(not out yet)

#

(but incredibly close)

dreamy urchin
#

(CI takes time :P)

#

I am anxiously waiting for the backlash 🤣

quiet latch
#

mr runner, please pick up the job!

keen ibex
quiet latch
#

pep660 editable installs

#

and [insert the rest of the changelog]

#

but mostly editable installs

#

there's a strictness option for the editable installs, I'm not sure if that's new

#

seems to be new, yeah

#

aaand, it's here

unknown.png
dreamy urchin
quiet latch
#

I promote changelog reading :)

dreamy urchin
#

(But I am indeed verbose 😛
Well, better to fail for the excess of information than the lack)

nimble bridge
#

🎉

#

@dreamy urchin Thank you for all the work you have done on this. ❤️

granite flint
nimble bridge
#

It's not a competition. :P

granite flint
#

No, its a CAMpetition

#

...I'll go now

wanton salmon
#

I needed that, I'm currently head deep debugging a bug caused by runtime monkeypatching and I'm losing my mind

#

oh goodness sake of course it's caused by mypyc

#

gahhh, mypyc compiles imports in functions as if they existed in the module scope, polluting globals >.< ... OK I'll go now.

lean quiver
#

Only two pip tests failing with setuptools 64. Could have been worse 😅. I was worried the pip test suite was more dependent on legacy editable installs.

granite flint
tropic widget
#

😔

dreamy urchin
# tropic widget https://github.com/pypa/setuptools/issues/3518

The solution proposed by Bernart in the email thread seems very reasonable (having an additional json file listing the package/module locations). There seems to be some level of consensus at least on the packaging side, but we would need the buy-in from the static analysis tool side.

tranquil saffron
#

At the risk of complicating the installation logic further have you considered using symlinks for lax installs like you do for strict? At least then only Windows users will have to suffer and who cares about them ;)

nimble bridge
#

I do, and so do many other people. For example, most students only have access to a Windows machine, especially at school.

#

Treating the most popular consumer OS as a second class citizen is... something I'd hoped we'd learnt as a community to stop doing for the sake of it.

tranquil saffron
#

We'd be selecting a different installation method based on the operating system's capabilities. I'm not suggesting that the existing method is removed

#

I don't need to be lectured on the importance of Windows for a comment I obviously made in jest

dreamy urchin
#

I have considered trying symlinks when possible, but then I dismissed the idea because I am very weary of the complexity of the code...

(Side note, strict install should work on Windows because it falls back to hard links... but hard links don't work with directories)

#

Another side note: a downside of using a link farm is that we currently don't have a interface to talk to users "Please be careful to not delete the directory X". I tried to use warnings but they are supressed by pip 😜

lean quiver
#

Do we need a pep517 extension to let backend communicate warnings to frontend?

dreamy urchin
#

In the lastest implementation, I added a series of warnings talking about the limitations of each method + this one about not deleting the directory that contains the link farm

tranquil saffron
river oak
#

We might want to yank the latest setuptools, the editable issue is affecting quite a few people doing namespace packages

dreamy urchin
# river oak We might want to yank the latest setuptools, the editable issue is affecting qui...

Hi @river oak, I encourage anyone facing problems to create a reproducer in the setuptools issues.

I agree that pkgutil/pkg_resources-namespaces can be tricky to handle for flat-layout projects. But the implementation offers many tweaks that can be made for those to work (editable-mode=strict, editable-mode=compat, SETUPTOOLS_ENABLE_FEATURES="legacy-editable").
PEP 420 namespaces should work out of the box.

The implementation using import hooks for flat-layouts is meant to be a compromise solution between the 2 distinct views discussed in the context of PEP 660. The minimal guarantee this implementation seeks is to prevent spurious files/packages/modules to end up in sys.path, which I believe many people in the community agree to be a flawed behaviour.

The implementation also allows us to fix historical issues with custom package_dir mapping.

robust yarrow
#

oof Airflow's setup.py looks very complex

dreamy urchin
river oak
unkempt birch
#

I've got a _core.so (generated) & _core/__init__.pyi; setuptools is complaining about _core not being found with packages = find: and that it will be removed from package data soon. I don't think I want to add __init__.py, since it's only there for type info about the compiled file.

#

Should __init__.pyi modules also be found like a package?

dreamy urchin
#

That sounds very reasonable. I might try to have a look on that over the weekend (sorry I got very busy this week).

I am curious: is the _core package top-level? (I understand that you don't have a _core.pyi because otherwise it would not be included as package_data...)

unkempt birch
#

Nope, not top level, it's boost_histogram._core, which has submodules built-in, so there are several _core/*.pyi files - even one _core/axis/*.pyi folder too.

dreamy urchin
#

I see, that makes sense :)

vernal harbor
#

I wish I could shake the hand of whoever did the editable install machinery in 65.x -- this is amazing. My org has a super large monorepo namespace package that was impossible to make editable the old way, and this is about to make our development loop MUCH more enjoyable

#

I guess I just wanted to come here and say thank you x 1000 😅

tiny mountain
#

A setup.cfg question.

I'm trying to find a way to define dependencies between libraries living in a monorepo as installable source directories. I finally found a way to do it with https://peps.python.org/pep-0440/#direct-references e.g.:

./lib/
    foo/
    moo/ # depends on foo can have the following in install_requires

f"my_package @ file://localhost/{path_to_foo}#egg=foo

  • egg is probably not needed
  • and path_to_foo is an installable source directory

It appears however that this wants an absolute path, and if I'm using setup.cfg there's no way to construct such path using cwd or os.environ.

❓❓❓
Do you know if this must be an absolutely path?
Do you know if there are ways to achieve dynamic path using declarative config?

CC: @nimble bridge

tiny mountain
#

After reading through the first 3 pages in Google and some of setuptools code I conclude that this isn't possible. String interpolation is done via configparser.ConfigParser and there's no way I can see for env variables to be injected. Sadly I may have to go back to using setup.py 🙂

river oak
#

#egg=whatever is definitely not needed. This is an alternative syntax before the name @ url form was standardised, and is ignored in the form you mentioned.
The path must be absolute. The standard package metadata does not allow relative paths or environment variable expansion in package specification.
Various workflow tools support referring to a relative path with various tool-specific syntaxes, but you can’t do this in setup.cfg

tiny mountain
#

Thank you for confirming this!

robust yarrow
robust yarrow
#

is PEP 685 implemented?

robust yarrow
quiet latch
#

Since [tool.setuptools] table is still in beta, should I prefer using setup.py for configuration such as license_files, options.packages, and options.packages.find to avoid releasing an sdist that could in future not be buildable?

And second thing, I noticed thatinclude_package_data now defaults to True, is that specific to projects that migrated to using PEP 621 pyproject.toml or all projects? And can I rely on this staying the new default if I use pyproject.toml?

#

hmm I do also wonder, does setuptools support having all 3 files at the same time

#

lol

humble palm
humble palm
quiet latch
#

actually nevermind

#

wouldn't work for me either way

#

I distribute a license for a vendored dependency using a name project-name.LICENSE (which is actually why I added license_files field instead of relying on the default as I did before)

quiet latch
#

in case of setup.cfg, setup.py takes precedence if both set the same field

#

in case of pyproject.toml it's simply not possible to set in both since you have to declare it as dynamic field

#

So if mixing all 3 together worked, I would expect to need to use dynamic for any field set in setup.cfg or setup.py and for any set in both setup.cfg and setup.py it would just use the same precedence as without pyproject.toml

#

but yeah, I don't know if it does and whether it's supposed to 😄

quiet latch
#

Hmm, I'm now not even sure that using license_files in setup.py would save me as once it becomes part of [project] table, me using it in setup.py means that it would at some point probably start erroring out due to not being specified in dynamic field.

potent shoal
#

i'm working with some colleagues on reliably rebuilding wheels for a bunch of PyPI packages, and we just realized that setuptools' --py-limited-api only controls the filename of the generated wheel, and doesn't actually enforce that the underlying build is abi3 compatible with whichever Python version is passed into the flag. is that behavior documented anywhere?

unkempt birch
#

Is there a way to pass custom setup keyword arguments via setup.cfg or pyproject.toml? Plugins can add custom keyword arguments, but don't see a way to pass them except via setup(), so just curious.

patent lotus
#

It seems like a very tried-and-true design pattern for packages + package-data somehow became deprecated:

setup.py
my_package/
  __init__.py
  __main__.py
  ...
  data/
    subdir1/
      data1.yaml
      data2.yaml
    subdir2/
      data3.yaml
      data4.yaml

There's a new warning about my_package.data.subdir1 is deprecated as "data" and should be listed in "packages".

What? Why should something that's not a package be listed in packages? What's the advice about "package namespace search" to discover these directories and list them as packages? (possible relevant discussion : https://github.com/pypa/setuptools/discussions/3323)

#

@dreamy urchin ^

dreamy urchin
#

@patent lotus please refer to the discussion in https://github.com/pypa/setuptools/issues/3340

The pattern is not deprecated, but in the light of Python's behavior since PEP 420, the configuration needs to be adapted to eliminate ambiguity

patent lotus
#

copy, thanks for the link 🙂

tiny mountain
#

Still thinking about ways to host multiple libraries under the same namespace in the same monorepo.
Just wanted to bounce ideas, and get feedback from the gururs of setuptools 🙂
I want to use package_dir={'': '$MONOREPO_ROOT'} to host all libraries under the same folder structure:

$MONOREPO_ROOT:
company/
    lib/ 
        foo/
            setup.py # package/import names are company.lib.foo
        moo/
            setup.py # package/import names are company.lib.moo

Each setup.py would use package_dir to point at the very top of the hierarchy and would only include its own subfolders. Do you see issues with this? Have you seen this done before? I'm not sure how I would express dependencies between libraries using this schema.

rigid herald
quaint jolt
#

if I understand correctly, script-files installs a script file to .data/scripts`, right?

#

if that's the case, why is that deprecated?

granite flint
# quaint jolt if that's the case, why is that deprecated?

Isn't that because the preferred approach for a long time has been to use a console_scripts entrypoint instead, with the business logic inside the package (like with data_files vs. package_data? Or is there something I'm missing here?

quaint jolt
#

yes, but if you want to include other kind of scripts, console entrypoints are not great

granite flint
quaint jolt
#

yeah

granite flint
quaint jolt
#

we support it in meson-python

copper dock
#

I tried putting a binary executable in that field(or scripts= in setup() function) in my project but it doesn't work

quaint jolt
#

gotcha

nimble bridge
#

@lusty dome Any thoughts on whether pkg_resources/setuptools could migrate to platformdirs instead of using + vendoring appdirs?

nimble bridge
lusty dome
#

Yes. I've only become aware of platformdirs recently. It would still need to be vendored.

#

I'm hoping to dive into setuptools issues tomorrow.

lusty dome
#

I responded in that issue. I hope my tone is "matter of fact" and not hostile. I expressed some preferences/perspective, but am legitimately okay with whatever direction the pip maintainers wish to take.

nimble bridge
#

Thanks for your responses! ❤️

#

I don't have time right now to write fully polished replies right now. 😅

#

But I've written some scrappy replies that I've not tried to polish the language of, so please don't take those as strong opinions or whatnot.

celest dove
#

I was looking at this PR I was at-mentioned on, which I think will fix the imp removal blocker for Python 3.12: https://github.com/pypa/setuptools/pull/3685

As mentioned in my comment, I see a bunch of failures from tox, but it's also unclear to me how I can create a local whl from this branch to drop in and try on my CPython branch. The setuptools dev guide doesn't explain that process. Does only @lusty dome know how that works?

GitHub
`pkg_resources`: Flip import conditionals around `pkgutil.ImpImport...

Summary of changes

In Python 3.3+ importlib.machinery.FileFinder is always present, therefore we no longer need to keep the conditional
In Python 3.12+ pkgutil.ImpImporter is removed, so we need a...

lusty dome
#

I'd half expect standard build tools to work. Does build not do that? Actually, looks like pip wheel git+https://github.com/abravalheri/setuptools@issue-3631 does the trick.

lusty dome
#

And of course, if you don't actually need a wheel, just pip install the same target.

celest dove
crude plaza
#

i was switching over an old extension module at work to use build instead of pip wheel, but it broke rather mysteriously

#

after a while, i was able to track down the problem to a bug in the setup.py script that got exposed when __file__ was a relative path instead of an absolute one

#

yes, this was a bug in the setup.py script and it was using the legacy setuptools build ... but maybe setuptools should be setting __file__ to an abspath, given that most __file__ are abspath?

crude plaza
cloud sand
nimble bridge
#

So... I'm looking at the packaging failure and wondering: can we have hashcmp not attempt to parse the version, and use the raw string?

#

That would avoid the pkg_resources's resolve from trying to parse invalid versions in the environment while scanning, IIUC.

river oak
#

Assuming the string is normalised, probably

robust yarrow
crude plaza
river oak
#

History: The setuptools implementation predates Core Metadata, and the latter was intended as a formalised iteration of setuptools’s metadata “format” (it was just a key: value storage without any formal specification). But since the standard looks so much like the original, the implementation in pkg_resources was never updated

#

Oh actually I’m wrong, the old format did have a backing specification and it was based on RFC 822. (The part where implementation was reused and never changed is correct, I think)

acoustic cradle
#

lol

#

folding in RFC 822

#

it does not surprise me that caused a bug somewhere, though I am surprised somethign emitted that METADATA

#

when I was looking before I didn't see anything that actually folded long lines I thought

#

though maybe my memory is just bad

river oak
#

I remember folding was a problem back when Description was emitted as a field by some tools (setuptools?) instead of the message body

#

But very few people really care reading Description so the disturbance was minimal

acoustic cradle
#

it's problematic because you can't actually unfold Description

#

except one of the PEPs implemented a different folding mechanism than RFC822

crude plaza
acoustic cradle
#

Ah

#

I think your METADATA is valid FWIW

#

I was just surprised because I thought I knew all of the oddball behaviors of the various main tooling, and assumed it was one of those

keen ibex
#

while it is an interesting bug, when you have a version string that long, I think the problem is not in the tool 🤣

acoustic cradle
#

Technically the bug exists regardless of the length of the version string, because RFC822 doesn't require folding in any case, so you can fold for a one character version number and that's valid

crude plaza
stable jasper
true warren
keen ibex
#

In Poland we sometimes joke that "I love you" in German can sound like "shoot to kill" order

true warren
#

Oh, huh...it's not just English, then?

keen ibex
#

yeah probably (I think we should go to #off-topic if we want to continue this topic)

nimble bridge
crude plaza
#

it looks when doing an editable install for some package, setuptools will remove random other things from easy-install.pth if they happen to be on PYTHONPATH

#

is this a bug?

restive talon
#

whats the correct way in setuptools to have a importable backend helper, it seems to me like i need to import whats in build_meta + add own attributes to make things work (all i really want to do is add a version attribute in setuptools_scm )

quaint jolt
#

@dreamy urchin is there any way to tell setuptools that an extension depends on the specific interpreter version?

#

basically I want to pin the interpreter version, eg. if CPython 3.11.2 is being used to build, set CPython 3.11.2 as a requirement

#

I am trying to make Cython correctly set the interpreter dependency when it uses the CPython private API

dreamy urchin
#

Hi Filipe, not sure if I understood the question right, but here it goes... The part of setuptools that builds C-extensions is still something I don’t understand completely, but as far as I know, setuptools requires python_requires (or requires-python for pyproject.toml) to be specified at distribution level, and there is no method for "feedbacking" it from ext_modules.

Maybe you could do something using the setuptools.finalize_distribution_options hook to set python_requires? (https://setuptools.pypa.io/en/latest/userguide/extension.html#customizing-distribution-options)

dreamy urchin
#

(Or if you just want to use for single package, tlyou could try calculating python_requires in your setup.py).

dreamy urchin
restive talon
#

but its ugly

dreamy urchin
#

Ok, yes, if you want to completely obliterate setup.py that makes sense.

#

Yeah, sorry, I don't think I have a better answer than the approach you are doing for the time being.

In the long run, I really want to make it possible to allow customizations to be loaded directly from the pyproject.toml spec as "in-repo" plugins.

restive talon
#

similar to hatch custom plugins?

dreamy urchin
#

I admit I don’t know a lot about how hatch works internally :’)
What I had in mind is just to re-use the same kind of entry-points that setuptools already has defined (e.g. setuptools.finalize_distribution_option) but exposing it in a way that it can be used directly from the project itself without requiring an installation to expose entry-points.

restive talon
restive talon
#

@dreamy urchin can setuptools be made to only consider entrypoints from build requirements when using pyproject.toml
In particular when using extras

Setuptools_scm file finders needs a way to opt out /in

quaint jolt
#

but it turns out the Cython situation is more complicated...

dreamy urchin
dreamy urchin
restive talon
#

aka

[build]
requires = ["setuptools-scm[find-files]"]

would be required to make it consider the plugin entry-point

smc-files = "the:name[find-files]"
dreamy urchin
#

that sounds very challenging, specially because there is no built-in support in importlib-metadata and pip for handling/filtering the extras part in the entry-point specification grammar (as far as I know).

But just because I don't have a clue on how to start supporting that, it does not mean it is impossible...

restive talon
robust yarrow
#

for context, at work we are trying to remove runtime dependence on setuptools and this is the very last transitive dependency that uses it...

cloud sand
#

@dreamy urchin @lusty dome I think the two maintainers that could chip in

restive talon
dreamy urchin
# robust yarrow for context, at work we are trying to remove runtime dependence on setuptools an...

My personal opinion is that importlib.metadata is a better and faster way of dealing with entry-points. But I think we are only going to officially deprecate pkg_resources usage directly a setuptools installation once we find out a way of untangling the codebases... (Maybe Jason has other ideas about). I am not sure if the plan after that is to ressurect pypa/pkg_resources and distribute it independently from setuptools.

restive talon
robust yarrow
#

@dreamy urchin would you be comfortable adding a comment there, as a maintainer, saying that? even though I'm involved in packaging and know our plan, that person does not really know that and it would appear to an outsider that since I am not a setuptools maintainer I lack context

dreamy urchin
#

Hi @robust yarrow, let's see if Jason has other plans. Otherwise I am happy to share my toughts directly on the linked issue.

lusty dome
#

One way to eliminate the complexity and conditionality is to use the backports unconditionally. Another way is to move the conditionality into a single private module so it only happens one (see Setuptools._importlib). Users are encouraged to move off pkg_reapurces asap and not wait for stdlib versions to sunset.

robust yarrow
dreamy urchin
#

@robust yarrow Jason is preparing a PR with the oficial deprecation.

robust yarrow
#

oh awesome!

haughty bramble
dreamy urchin
#

We managed to remove/delay pkg_resources imports in the critical path recently. Now it is only used for legacy builds. (Although since it is a separated codebase pypa/wheel might still import it in bdist_wheel).

#

The complete removal is going to be more viable after pip 23.1 adopts --use-pep517 by default, but we still need to discuss that in detail.

#

(And check which circumstances that might affect existing projects. E.g. how long do we have to support if users call pip with --no-use-pep517?)

haughty bramble
#

The use in bdist_wheel is minimal and could be removed nicely

quiet latch
dreamy urchin
#

(Or develop and other commands)

robust yarrow
#

at work we are experiencing an issue where merely importing the vendored distutils increases RSS by 10 MB whereas the one from the standard library does not. does anyone happen to know why?

acoustic cradle
#

A complete guess would be that the distutils hack is causing problems

#

it looks like that does some shenagins where it deletes distutils from sys.modules then inserts the vendored copy

#

I wonder if something is causing that to happen multiple times

haughty bramble
#

Not long left on 3.7 support

Ends in 3 months and 2 weeks
(27 Jun 2023)

lean quiver
#

We keep setup.py develop in pip for now.

#

We keep setup.py bdist_wheel too. Or is it identical to what the default build backend does?

dreamy urchin
haughty bramble
nimble bridge
dreamy urchin
#

Exactly that is the difference between the legacy and the non legacy backends that I was referring to

#

(They do differ in only this aspect)

nimble bridge
lean quiver
#

Thanks @dreamy urchin @nimble bridge . So this means it should be safe for pip to use the default build backend instead of calling setup.py egg_info + setup.py bdist_wheel.

nimble bridge
#

I mean, as safe as Hyrum's law would allow.

restive talon
#

@dreamy urchin is there a way to add a build time plugin to setuptools (so that writing the version file of setuptools_scm can be deffered from config time to build time

dreamy urchin
#

If the idea is just to add a version.py or VERSION.txt file somewhere, I believe it should be possible to add a custom build step.

Right now, you can use finalize_distribution_options to modify the cmdclass and build.sub_commands to do it. In the future, I want to introduce a dedicated setuptools.build_steps entry-point for that.

I made an example last year for custom build steps using this methodology: https://github.com/abravalheri/experiment-setuptools-plugin (but it might not be super complete, e.g it does not support editable installs). Jason also has some toy examples.

restive talon
haughty bramble
river oak
#

I think pkg_resources style namespace packages are natively supported on Python 3?

haughty bramble
#

No you can't mix and match them

#

You need to upgrade all packages to use implicit namespace packages simultaneously

river oak
#

I guess it’s not too difficult to pull that code out yourself, declare_namespace_package is essentially implementing a custom module loader

#

(And zip-safety is not a thing now either)

haughty bramble
#

I'm saying the docs need an update, but I'm not too sure what that is

river oak
#

Right

haughty bramble
river oak
haughty bramble
#

Where's the repo for the packaging guide?

river oak
#

IIRC it’s somewhere under pypa

haughty bramble
gleaming mirage
#

Could the setuptools namespace package implementation be changed so that it becomes compatible with pkgutil-style packages? This would make it easier to transition away.

haughty bramble
haughty bramble
gleaming mirage
#

Maybe you could finagle it by walking up the stack? 🤷‍♂️

haughty bramble
restive talon
#

@haughty bramble this will need a compatibility package that moves namespaces to Metadaten +pth files first, then turns optional after all calls are removed

haughty bramble
restive talon
# haughty bramble ?

to elaborate, there is need for a meta package that declares old style namespace in a no files used manner while pkg_resources is installed/depended uppon, but have it as optional dependeny, then once nothing has it, it wont be installed and wouldnt be used anymore

dreamy urchin
restive talon
#

(at least when trying to be multi location compatible)

dreamy urchin
restive talon
#

Hmm, based on the pep, it's actually OK to drop the files

haughty bramble
#

so it looks like there is a pathway upgrade to pkgutil then once that's all done upgrade to pep420

haughty bramble
restive talon
haughty bramble
dreamy urchin
haughty bramble
#

sorry discord suggested I make a thread and I thought it would pull the 4 messages in the chain inot a thread

haughty bramble
#

are they xfail?

dreamy urchin
#

Mixing editable with non editable seem to fail

haughty bramble
#

maybe mark the broken editable mixing as xfail and document it?

dreamy urchin
#

(I am testing the default editable mode in setuptools, which I believe will use MetaPathFinder for these examples)

haughty bramble
#

is that different to the PEP 660 mode?

dreamy urchin
#

(newer pip may not even support editable installs without it)

haughty bramble
#

23.0 still does the old setup.py develop if there's no pyproject.toml

dreamy urchin
haughty bramble
#

what are the limitations then we can document it on the packaging tutorial rather than advising against it

dreamy urchin
haughty bramble
dreamy urchin
haughty bramble
#

2 months left on pytho 3.7

dreamy urchin
#

I will try to commit this change and make a PR later this evening

dreamy urchin
haughty bramble
#

probably worth a matrix with 3.8-3.11

#

how long does it take to run?

dreamy urchin
#

a bit, let me time it

#

The main problem is to process the resulting data later into a nicer table 😛

#

Oh, wait, they might already have a tool for that

#

I will wait until this round finishes and try again

dreamy urchin
haughty bramble
#

are you adding a test for all three in one environment?

dreamy urchin
#

I want to add installing legacy pkg_resources first and then pep420 (inverting the order).

haughty bramble
# 
Python 3.11.2 (main, Feb  8 2023, 14:49:25) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import itertools
>>> list(itertools.permutations(("native", "pkg_resources", "pkgutil")))
[('native', 'pkg_resources', 'pkgutil'), ('native', 'pkgutil', 'pkg_resources'), ('pkg_resources', 'native', 'pkgutil'), ('pkg_resources', 'pkgutil', 'native'), ('pkgutil', 'native', 'pkg_resources'), ('pkgutil', 'pkg_resources', 'native')]
>>>
dreamy urchin
#

I will focus my work here on experiments that help moving away from pkg_resources (preferably into pep420). If anyone wants later to add tests for combinations of the 3 of them, that is fine, but I think for now the more pressing matter can be simplified down to 2x2 combinations.

haughty bramble
#

when I install zope.interface I don't get a zope/__init__.py

#

it's just a native namespace package on a pip install of the wheel

dreamy urchin
haughty bramble
#

there's no zope/__init__.py in the whl

#

but I do get import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('zope',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('zope', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('zope', [os.path.dirname(p)])));m = m or sys.modules.setdefault('zope', types.ModuleType('zope'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p) in zope.interface-5.5.2-py3.11-nspkg.pth

haughty bramble
#

Should the namespace_packages kwarg in setuptools.setup be deprecated?

#

Oh it already is, I just missed the warning for it: SetuptoolsDeprecationWarning: The namespace_packages parameter is deprecated, consider using implicit namespaces instead (PEP 420).

dreamy urchin
quaint jolt
haughty bramble
#

What reads namespace_packages.txt in .dist-info?

dreamy urchin
#

pkg_resources will read this file and "pre-declare" any registered namespace

haughty bramble
gleaming mirage
#

Is setuptools going to deprecate bdist_egg any time soon?

humble palm
gleaming mirage
#

True, but it's not clear what "eggs are deprecated" actually means. The bdist_egg command itself doesn't seem to be deprecated, and there's no warning when you run it.

restive talon
dreamy urchin
#

Given that using .egg files is deprecated, there is no reason for producing such files to not be considered deprecated...

tranquil saffron
quaint jolt
#

I tried to but wasn't able to, even though I should have triage permissions

#

@dreamy urchin are you able?

dreamy urchin
#

I suppose you need permissions in both repositories to transfer (so I also cannot do that)

high talon
#

i'm trying to make it easier to install platform & accelerator-specific torch libraries for a source-only application via an invocation like pip install -e .. i see that dependency_links has been deprecated (or removed?) and as a workaround, my goal is to simply append requirements in the form of

"torch @ https://download.pytorch.org/whl/cu117/torch-1.13.0%2Bcu117-cp310-cp310-win_amd64.whl"

is there a pre-existing, possibly private, function that can generate those urls, provided an index url, package name and a version, inferring /enumerating the index, inferring the platform and inferring the version of python?

#

i do not want to reinvent the wheel here, so i'm hoping someone familiar with setuptools internals can suggest how to approach this

#

i might also be wrong, and find_links may just work when passed as a karg to setup? this would simplify things

dreamy urchin
#

Sorry @high talon if there is something like that, I am not familiar with it.

high talon
#

do you have any suggested approach for what i am trying to do here?

dreamy urchin
#

i might also be wrong, and find_links may just work when passed as a karg to setup?
In non-deprecated usages, setuptools is not used to install anything, so the find_links option has no meaning for setuptools

high talon
#

Since its version 19.0, released on 2019-01-22, pip ignores the setuptools options dependency_links is what stackoverflow says, which is why ididn't try, but is that actually true?

#

sorry i meant dependency_links, which i can see from the source is concatenated into find links

dreamy urchin
#

TL;DR all the support in setuptools for things like find_links or dependency_links is deprecated, if not already removed.

high talon
dreamy urchin
#

I think the best workaround is to create a repository proxy

high talon
#

i suppose if the user is already going to do

python setup.py install

i can invoke pip with the --find-links command directly

river oak
#

Perhaps you need to explain what you want a bit better, maybe with an example of your expected output.

dreamy urchin
high talon
high talon
high talon
# river oak Perhaps you need to explain what you want a bit better, maybe with an example of...

i am doing two things

  1. trying to author a single, easy to invoke script that installs a repo - a torch-based application - into the current virtual environment editably. the script will infer platform features like CUDA versus ROCm support and install the correct version of torch for the user's platform.
  2. trying to not revisit the long discussions about how or why find links was deprecated, etc. etc., since none of this is going to be published to pypa.
#

it would be great if i could author a setup.py that is compatible with pip install [-e] [.|https://...]

river oak
#

Sounds like a proxy package repository is indeed your best choice (or maybe no proxy is needed at all? Pytorch already has a pretty compliant release page)

high talon
#

okay, but the proxy would still need to be passed to dependency_links, which is removed, so i'm back to my original issue

#

i am trying to avoid asking the user to do anything arcane...

dreamy urchin
river oak
#

Sounds like a script like this would work pretty well?

subprocess.run([sys.executable, "-m", "pip", "install", "--find-links", pytorch_url, "-", *requirements])

And your users can simply call python install.py to run it

dreamy urchin
high talon
high talon
dreamy urchin
river oak
high talon
high talon
river oak
high talon
#

i felt like i was reinventing setuptools so i knew i was doing something wrong

high talon
#

everything becomes much easier - CI/CD, developer expectations, etc.

#

there are maybe 100-1,000 applications torch-accelerated right now, maybe 10 new good ones being authored every week, so there is a very high value in figuring out how to use torch correctly with the wider python packaging ecosystem. for what it's worth, poetry has explicitly determined to not solve this problem, so i really appreciate your assistance here

river oak
#

I think one of the fundamental point setuptools tries to make from removing dependency_links and others is exactly not every piece of distributed code should be an installable library, and your code likely should not be an installable library if it needs these features

high talon
#

well if pip will transitively invoke subprocess.run([sys.executable, "-m", "pip", "install", "--find-links", pytorch_url, "-", *requirements]) because it's in a setup.py, i think my problem is solved

#

and this will become the authoritative way to solve this looming issue for thousands of developers

dreamy urchin
river oak
#

The “problem” is pip does not allow you to do that unless you do some very clever tricks to fool it 🙂

high talon
high talon
#

like i can monkeypatch this eventually

#

i can also compute the URLs, i can reinvent the package-spec-to-url resolution process

river oak
#

One thing I’d caution is your users may not appreciate your clever tricks. With that said Python is just a programming language and obviously you are entitled to be able to do anything you want with a Turing-complete tool.

high talon
high talon
#

like my journey has taken me to

    canonical_pkg_name = canonicalize_name(package_name)
    tag = Tag(python_version, tag_str, platform_name)
    wheel_file_name = f"{canonical_pkg_name}-{version}-{tag}.whl"
...
    url = f"{repository_index_url}/{quote(wheel_file_name)}"

which like i said is reinventing things already in setuptools

dreamy urchin
#

The https://download.pytorch.org/whl/cu117/ part is definetely setuptools has no knowledge about...
Looking at the torch-1.13.0%2Bcu117-cp310-cp310-win_amd64.whl part we can see the trivial {dist_name}-{version} , a +{cpu} local version that is project specific (so setuptools also don't know about it), and then more -{python tag}-{abi tag}-{platform tag} which is something you can probably use packaging.tags for.

The template is {dist_name}-{impl_tag}-{abi_tag}-{plat_tag} and you can see how pypa/wheel uses it in https://github.com/pypa/wheel/blob/c87e6ed82b58b41b258a3e8c852af8bc1817bb00/src/wheel/bdist_wheel.py#L381

high talon
#

gotchyu

#

ideally there would be a method get_wheel_name(platform=None, python_version=None, ...) that would use the currently running machine's platform and python version by default

#

but i suppose that does not exist

dreamy urchin
#

If I am not wrong next(packaging.tags.sys_tags()) will give you a pretty specific tag, but I am not sure how pypa/packaging works... Definetely the correct tag is somewhere inside the iterable produced by packaging.tags.sys_tags().

high talon
dreamy urchin
#

if you implement the hipotetical select_correct_tag, you can do:

tag = select_correct_tag(packaging.tags.sys_tags())
impl_tag, abi_tag, plat_tag = tag.interpreter, tag.abi, tag.platform
high talon
#

if that's not possible, just pip install URL would also be chef's kiss

#

right now it sounds like the clever trick is to install the package temporarily*, then somehow investigate the side effects to create a setup.py

dreamy urchin
high talon
#

gotchyu

dreamy urchin
#

Not sure how PyPI will respond to that... But you may have to use some tricks to avoid the PKG-INFO to be populated with the dependencies for the machine you use to build your sdist

#

Of course, I agree with @river oak , that is pretty much something that is abusing the packaging infrastructure...

high talon
#

yes, i think this will never interact with pypi

high talon
#

🥺 it is very hard to ask pip programmatically to give me a list of packages that would be installed as though i had --extra-index-url passed

#

pip does not want to run recursively under dry-run. it would be great if i could just know where its lockfile / whatever is placed so i can move it temporarily and get the text result, then parse it.

nimble bridge
#

Are you working on torch itself, or looking to use torch in an application that you're building?

high talon
#

that was quite the journey

#

my feedback is that if you are going to support "@ packageurl" you might as well support extra-index-urls / find-links in setup.py again

high talon
# river oak One thing I’d caution is your users may not appreciate your clever tricks. With ... 
"""
The name of the package.
"""
package_name = "yourpackage"

"""
The current version.
"""
version = '0.0.1'

"""
The package index to the torch built with AMD ROCm.
"""
amd_torch_index = "https://download.pytorch.org/whl/rocm5.4.2"

"""
The package index to torch built with CUDA.
Observe the CUDA version is in this URL.
"""
nvidia_torch_index = "https://download.pytorch.org/whl/cu118"

"""
The package index to torch built against CPU features.
This includes macOS MPS support.
"""
cpu_torch_index_nightlies = "https://download.pytorch.org/whl/nightly/cpu"

"""
The xformers dependency and version string.
This should be updated whenever another pre-release of xformers is supported. The current build was retrieved from
https://pypi.org/project/xformers/0.0.17rc482/#history.
"""
xformers_dep = "xformers==0.0.17rc482"

def dependencies() -> [str]:
    _dependencies = open(os.path.join(os.path.dirname(__file__), "requirements.txt")).readlines()
    _alternative_indices = [amd_torch_index, nvidia_torch_index, cpu_torch_index_nightlies]
    session = PipSession()

    index_urls = ['https://pypi.org/simple']
    # prefer nvidia over AMD because AM5/iGPU systems will have a valid ROCm device
    if _is_nvidia():
        index_urls += [nvidia_torch_index]
        _dependencies += [xformers_dep]
    elif _is_amd():
        index_urls += [amd_torch_index]
    else:
        index_urls += [cpu_torch_index_nightlies]

    if len(index_urls) == 1:
        return _dependencies

    try:
        # pip 23
        finder = PackageFinder.create(LinkCollector(session, SearchScope([], index_urls, no_index=False)),
                                      SelectionPreferences(allow_yanked=False, prefer_binary=False,
                                                           allow_all_prereleases=True))
    except:
        try:
            # pip 22
            finder = PackageFinder.create(LinkCollector(session, SearchScope([], index_urls)),
                                          SelectionPreferences(allow_yanked=False, prefer_binary=False,
                                                               allow_all_prereleases=True)
                                          , use_deprecated_html5lib=False)
        except:
            raise Exception("upgrade pip with\npip install -U pip")
    for i, package in enumerate(_dependencies[:]):
        requirement = InstallRequirement(Requirement(package), comes_from=f"{package_name}=={version}")
        candidate = finder.find_best_candidate(requirement.name, requirement.specifier)
        if any([url in candidate.best_candidate.link.url for url in _alternative_indices]):
            _dependencies[i] = f"{requirement.name} @ {candidate.best_candidate.link.url}"
    return _dependencies

setup(
    # "comfyui"
    name=package_name,
    description="",
    author="",
    version=version,
    python_requires=">=3.9,<3.11",
    packages=find_packages(),
    install_requires=dependencies(),
)

here is a working snippet

high talon
#

@nimble bridge i guess i have more feedback. based on the other proposals i'm seeing about packaging, and my experience with poetry and doing something sophisticated with pip:

  • you are reinventing setup.py in configuration files.
  • python is a serialization format.
    i think you know this and the maintainers agree. it's tough. you get these security asks about boms or whatever that are impossible. then it's impossible to persuade devs at bigcorp who "just" want things to "work". many audiences, no one perfect solution.
    looking forward to making useful contributions in the future. i will post my snippet for the torch end users!
nimble bridge
#

FWIW, I see that you're using pip's internals -- that's not supported and can (will) break arbitrarily at any time.

high talon
wanton salmon
#

Nice work with modernizing setuptools! I've been a flit user for ages now, but moments ago I needed a backend that was a bit more powerful. I didn't really feel like learning how to configure hatchling so I decided to give setuptools a(nother) whirl. The docs are pretty good (minus the unavoidable pre-PEP517 cruft) and getting my package to work was dead simple:

[build-system]
requires = ["setuptools >= 61.0.0"]
build-backend = "setuptools.build_meta"

[project]
name = "dobackup"
authors = [
    {name = "Richard Si", email = "sichard26@gmail.com"}
]
classifiers = [
    "License :: OSI Approved :: MIT License",
    "Development Status :: 4 - Beta",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3 :: Only",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Typing :: Typed",
    "Private :: Do Not Upload",
]
license = {file = "LICENSE.txt"}
dependencies = ["click >= 8.0.0", "rich >= 10.14.0"]
requires-python = ">=3.8"
dynamic = ["version"]

[tool.setuptools]
py-modules = ["dobackup", "gcrepos"]
dynamic = { version = { attr = "dobackup.__version__" } }

[project.scripts]
dobackup = "dobackup:main"

[project.entry-points]
"lily.tools" = { dobackup = "dobackup:main", gcrepos = "gcrepos:main" }
#

I'm really glad to see pyproject.toml be treated as a first-class citizen by setuptools now.

split sluice
wanton salmon
#

setuptools also has the advantage of being such an old player, I'd be able to package a setup.py based project without issue even to this day

haughty bramble
#

I originally misdiagnosed this as a pip issue and there's some messages in #pip #pip message

haughty bramble
nimble bridge
#

Oh, that's the Python import system being wonky.

#

And the fact that cwd is always on sys.path before everything else.

dreamy urchin
#

Yeah, when I was investigating this, I reached the conclusion that Python import system will treat packages provided by MetaPathFinders and PathEntryFinders differently than packages in sys.path... No clear why or how.

You can try pip install -e . -C editable-mode=compat (pip 23.1.1) to try the previous behaviour

stable jasper
haughty bramble
dreamy urchin
#

You mean the fact that "." makes setuptools use the same code path as compat mode is a bug?

#

Probably that is an edge case that was not considered and could be fixed... it would ruin your workaround though 😛

haughty bramble
#

it seems include-package-data=True stops working when I do this though

robust yarrow
#

@haughty bramble are you sure versioneer is required in your case?

haughty bramble
#

unless there's a way to support __git_revision__ in hatch-vcs?

restive talon
#

@dreamy urchin any recommendation for deferring setting the version attribute and writing a file containering version metadata until after keywords and options happened

dreamy urchin
#

I don't think right now there is a entrypoint for that. If I am not mistaken setuptools.finalize_distribution_options runs before parse_config_files.

restive talon
dreamy urchin
#

The entry-points for setuptools.finalize_distribution_options will run after the keyword arguments of setup.py are processed (but before setup.cfg and pyproject.toml are processed).

#

That is how it works nowadays

restive talon
#

@dreamy urchin where should i put the data of a keyword so that the finalize hook can use them correctly?

dreamy urchin
#

Does something like getattr(dist, keyword, None) work for you?

restive talon
#

🤦‍♂️

dreamy urchin
#

I believe right now setuptools might be doing setattr(dist, keyword, value) in setuptools.Distribution.__init__ for all entry-points defined in distutils.setup_keywords

#

Sorry, I am not sure if I understood your questions...

restive talon
#

@dreamy urchin for some reason i was under the misconcpetion that its not automatically added since a while

dreamy urchin
#

By looking at the code, it looks like it is implemented like that. Please let me know if you find problems

restive talon
#

hmm i messed up there, ^^ it will all work out

granite flint
#

Hey @dreamy urchin , is there a plan for when the Setuptools-specific config will be officially declared out of beta, and are there plans to change it before then? It seems that its been giving a fair number of people pause in adopting pyproject.toml-based metadata as a whole, so I was curious what the plan was to move forward on that (so I can help inform interested users accordingly). Thanks!

dreamy urchin
#

Given that I did not receive many proposals in how the configuration on pyproject.toml should be different from setup.cfg, it should be fine to keep them similar. So I think we can move it out of beta soon (I will probably wait a little bit as I want to see if the latest release affected many people).
However, my plan is to still keep some reservations on the docs:

image.png
#

So it should be fine for people to use [tool.setuptools.packages.find] or [tool.setuptools.package-data], but I don't think people should be using [tool.setuptools.eager-resources]...

#

I also don't think that [tool.setuptools.dynamic] dependencies = {file = requirements.txt} should get out of beta soon. There are a bunch of corners to polish (specially with optional-dependencies)...

granite flint
granite flint
# dreamy urchin So it should be fine for people to use `[tool.setuptools.packages.find]` or `[to...

FWIW, I don't think I've ever seen eager_resources used in the wild (in setup.py, setup.cfg or pyproject.toml, and in fact, a grep.app search: https://grep.app/search?q=eager_resources&regexp=true&case=true&filter[path.pattern][0]=setup.py reveals only 5 total usages in setup.py, 0 in setup.cfg and 0 in pyproject.toml on all of GitHub.

By contrast, I have seen zip_safe widely used, at least in the former two; I'm assuming it just gets cargo-culted and I'm not sure people were aware it only applied to eggs—even I wasn't, until recently when I read that table in the docs, despite having read of lot of Setuptools, distutils and related packaging docs. Indeed, there are over 11 600 total usages across GitHub in setup.py files alone per grep.app: https://grep.app/search?q=zip_safe&regexp=true&case=true&filter[path.pattern][0]=setup.py&filter[lang][0]=Python , 1400 in setup.cfg and 145 in pyproject.toml.

Interestingly, this is a decrease of an order of magnitude each time, and nearly 100:10:1; by comparison, there are 45 800 setup.py (with at least name ?=) to 8800 setup.cfg (with at least [metadata]) to 1900 pyproject.toml (with at least tool.setuptools), i.e. only a 25:5:1 ratio, so the relative usage decreases by nearly half with each, from 25% to 15% to 7.5% of each using zip_safe, respectively.

acoustic cradle
#

Well

#

"it only applied to eggs" is a slightly confusing

#

because it doesnt' mean only .egg files distributed on PyPI

#

because setuptools called the things that it installed onto the filesystem "eggs" also, which is why you had things like .egg-info (which predated .dist-info), .egg-link, etc.

#

So if you ever did something to cause setuptools to install your thing, you would get an egg

#

Common ways this would happen in the past:

  • Doing setup.py install
  • Using easy_install (no matter what the "source" artifact type was).
  • Using pip install --egg before it was removed.
  • Using setup,py develop or pip install -e (though this particular usage of eggs didn't affect zip_safe, because dev installs never zipped IIRC).

The first one is historically the big one

gleaming mirage
#

Does pyproject.toml even need to support zip-safe? IIUC, eggs are basically irrelevant now, so projects that are modern enough to use pyproject.toml shouldn't need this setting.

dreamy urchin
#

Thank you Donald. I changed the PR to also mention setup.py install in the table.
If anyone also would like to suggest changes, please feel free to send a PR review 🙂

dreamy urchin
restive talon
#

is there any suggested way to render the current version into dependency names as well (context is wanting to port the toga project group to pyproject.toml, those currently invoke setuptools_scm directly as they have a few dozen packages that have strict version number interlinking and they render the version into both the actual version and a number of dependencies

granite flint
# restive talon is there any suggested way to render the current version into dependency names a...

There's nothing standards-based that would allow this, and someone else might have a better solution, but at least using Setuptools/setuptools_scm, the ways I can think of (which you probably also have) are:

  • List dependencies as dynamic under tool.setuptools and either
    • Add a setting or custom hook to setuptools_scm itself to enable this
    • Or, write a separate custom Setuptools plugin (that runs after setuptools_scm) that injects the project version into the specified dep versions
    • Or, write the custom logic in a setup.py and define dependencies there
  • Or, do it as a pre-processing step to generate a pyproject.toml with the injected version, via one of several means
dreamy urchin
#

Yeap I agree with CAM. They can migrate as much as possible to pyproject.toml and keep the programatic bits in setup.py (using dynamic accordingly).

split sluice
#

Hi, apologies for using this server for support, but I'm not sure if I'm arguing with pip or setuptools.

I tried copying just my code and the pyproject.toml into my Dockerfile -

COPY requirements.txt .
RUN python -m pip install --requirement requirements.txt

COPY pyproject.toml src .
RUN python -m pip install .

And it sat on Getting requirements to build wheel: started until it timed out -

#11 [5/6] RUN python -m pip install .
#11 0.545 Processing /
#11 0.549   Installing build dependencies: started
#11 2.863   Installing build dependencies: finished with status 'done'
#11 2.865   Getting requirements to build wheel: started
Error: The operation was canceled.

But I changed it to copy everything and it succeeded -

COPY . .

With verbose logging it looks like it's stuck on -

#9 4.418   Getting requirements to build wheel: started
#9 4.418   Running command Getting requirements to build wheel
#9 4.554   /tmp/pip-build-env-hz7ahlkr/overlay/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py:66: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
#9 4.555     config = read_configuration(filepath, True, ignore_option_errors, dist)

But I don't understand exactly what it's reading.
I've listed every file in my project manually, including the .git folder, and it still gets stuck. AFAIU there shouldn't be any difference between listing all the files and just COPY ..
I... I'm sure I'm just missing something, but I can't see anything else that I would need. And I don't know why listing all files manually wouldn't have gotten it what it needed.
Can anyone see what I'm missing here?

pastebin of the pyproject.toml: https://paste-new.pythondiscord.com/ZA4A

haughty bramble
restive talon
#

anyone aware of a setuptools plugin that adds files to a editable wheel or a wheel, but doesnt let them show in the source tree (unless inplace build is forced)

restive talon
#

is there any suggested mechanism for making part of the project under build importable with setuptools, hacking together a own build backend that imports setuptools feels most gruesome, but is whats necessary with PEP621 atm

random harbor
#

i'm using a custom setuptools extension to add to install dependencies. This extension cds into git submodules and gets their names and versions. It works fine when building sdist (srd distro), but bdist/wheel is built using the sdist in a temp directory, so my setuptools extension cannot cd into the git submodules.

#

How do I circumvent this issue?

gleaming mirage
#

Your custom extension could probably save the necessary information inside the sdist while building it.

random harbor
#

As metadata? How?

#

I thought of writing to a file and adding that as metadata when building sdist. But the problem is that the same process is repeated when building wheel as well, so that's not a solution.

gleaming mirage
#

The idea is that you detect if the file already exists, and if so, you just use the information in it instead of doing Git stuff.

random harbor
#

Oh, right. How do I add to metadata file when building sdist?

gleaming mirage
#

I don't mean the metadata file. I just mean a regular file, specific to your extension.

random harbor
#

Will that be picked up by sdist tar.gz?

#

Lemme try.

gleaming mirage
#

I don't really know how to do it specifically in setuptools, but there must be a way to include a generated file.

random harbor
#

I'll try. Thanks.

gleaming mirage
#

I don't think you should include this information in the project metadata, since it's for the project's own use, not external tools.

random harbor
#

Okay. Makes sense.

random harbor
#

Even the cwd changes to the temp directory when building wheel. Otherwise I could use the path.

random harbor
#

I managed to circumvent this issue by using the env variable $PWD. The problem is, this env variable is absent in Windows. Any way I can set env variable once on python -m build so that I can get the env var when building wheel in the temp directory?

gleaming mirage
#

How would that work for someone building a wheel from the sdist (without the original Git repository)?

random harbor
#

It won't work. But that's not an issue for me right now.

gleaming mirage
#

Then why are you even making an sdist? Just build the wheel directly.

random harbor
#

How?

#

I just know python -m build.

gleaming mirage
#

build has an option for it, it's probably -w or something.

random harbor
#

python -m build --wheel (-w probably works, too).

#

Thanks.

bronze berry
#

We've noticed a few packages in Debian are picking up the debian directory as a package (and nobody notices) if they use find_namespaces Here's an example with setuptools itself: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1041091#10. This directory is something that usually doesn't exist in upstream's source directory, but always exists in a Debian source package. I can't think of any way to reliably stop this from happening, without patching setuptools. (And setuptools has no built-in blocklist, so it feels wrong). Any thoughts?

dreamy urchin
#

I suppose we can consider changes to https://github.com/pypa/setuptools/blob/v68.0.0/setuptools/discovery.py#L183. However I would not like to get into a complex situation where we have a default blocklist but people want to re-use parts of the blocklist while allowing other parts[^2]... That can became a configuration hell quite easily and explode the number of configuration "knobs and levers".

If devs are relying on the auto-discovery, setuptools has a blocklist[^1], but if they want to customise find:/find_namespace:/find_packages()/find_namespace_packages() I think they should start from a blank slate (to avoid the explosion of knobs and levers I mentioned before).

Nonetheless, I was thinking about find_namespaces and find this morning (for a different reason) and I was wondering if it would make it easier to if we have a "convention over configuration" approach...
How about if we skip directories that have a custom marker (e.g. directories that contain a .skippkg file)? Would that make like easier?

For example in the setuptools repo itself we have to keep an eye on https://github.com/pypa/setuptools/blob/v68.0.0/setup.cfg#L32 everytime we rename a directory or create a new one. And sometimes we forget that. A file marker would make maintenance easier for that use case. Not sure if it helps in the case of a debian folder, unless there is a central template that devs are incentivised to copy from...

[^1]: I think we can have a PR to add debian to https://github.com/pypa/setuptools/blob/v68.0.0/setuptools/discovery.py#L213 (there is no debian package on PyPI right?)
[^2]: This is not far-fetched, people already asked me for that.

bronze berry
#

Ah, I there's the list I didn't know existed 🙂

#

There is python-debian on PyPI, but we can always explicitly declare the package name

bronze berry
dreamy urchin
#

Thank you very much @bronze berry, just to clarify (so we don't have unmet expectations): the list works when people omit packages = , py_modules = and ext_modules = (i.e "auto-discovery" kicks in). If packages are using find:/find_namespace: in setup.cfg or find_packages()/find_namespace_packages() in setup.py, the probablem of including directories by mistake can still happen...

bronze berry
#

Yeah, it's at least a step in the right direction

#

find_packages should be safe, because there's no __init__.py

dreamy urchin
#

Thank you very much @bronze berry , I plan to cut a release early next week (probably in a couple of days). I can include the PR in that.

vagrant patrol
#

how do you include a custom directory in setuptools? i have an include directory for headers with an extension but it doesnt get copied to the build

dreamy urchin
thorn urchin
#

I'm seeing an issue where my setup script generates Python files while building native extensions, but when building with build with the setuptools backend these files don't end up in the final package. Is that by design?

dreamy urchin
#

Hi @thorn urchin, the answer is probably no, but it is hard to guess without a minimal reproducer.

I would recommend you checking:

  1. Are all the necessary files present in the sdist? (see https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html
  2. Are all the build dependencies listed in the [build-system] requires list (inside pyproject.toml)?

If these questions are true and you still see the problem, please consider opening a new issue with a minimal reproducer in the Github repository.

haughty bramble
#

But then you know that your sdists are broken and you probably shouldn't ship them

boreal yarrow
#

just wondering, are there any plans to merge distutils into setuptools, or will they stay separate git repos forever?

keen ibex
#

I think the grand plan is to migrate all necessary code from distutils to setuptools

split sluice
keen ibex
#

the way it is now, it's more "vendored" than "integrated"

restive talon
#

i`m observing the finalize options hook running before the version keyword when running setup.py - is there anything i can do about the order?

simple quiver
#

Hi guys. How I can introduce hook for setuptools.finalize_distribution_options in pyproject.toml for current project? So I don't want to introduce additional package like setuptools_scm, but define some script near pyproject.toml and call it for finalize_distribution_options inplace. Is it possible to do this?

restive talon
simple quiver
restive talon
#

nope, entryponts come from installed package data, the package under build is not installed

#

(thats why setuptools_scm has to list version schemes and parse functions explicitly in its self-build)

simple quiver
#

Then, can I put folder inside build-system.requires? like "my-build-dep @ ./buildscripts"?

#

The problem I want to solve - read version and probably other project information from environment variables... Maybe there is already plugin exists.

#

Anyway, thanks for help, that clarifies things a little bit.

restive talon
#

@simple quiver why do you want to override that way ?

#

(that setup sounds flaky without context)

simple quiver
#

As I mentioned - version. From environment variables. Maybe to add some custom CI logic(ultimately - just env variables).

restive talon
#

but why env variables?

simple quiver
#

Commonly used in CI in other places of my pipeline, like CMake, etc. So it will be great to set single environment variable and forget about everything.

restive talon
#

like whats the end goal there and why do you think the env vars are a good idea

#

so your projects would no longer build safely outside your ci ?

simple quiver
#

probably the best idea, since gitlab-ci globally set some variable with calculated version for whole pipeline and all jobs. So wherever package is referenced or installed or uploaded from particular pipeline, it will have good version.

simple quiver
restive talon
#

btw, for version, you could abuse setuptools_scm + the setuptools_scm_pretend_version env var to get there quickly

simple quiver
#

Have no idea what is the setuptools_scm_pretend_version , but probably will be easy to find in sources. Thanks for pointer.

#

But I want to achieve another thing - lets assume that I has package_a as arfifact of such pipeline, and package_b, that must depend on exact version of package_a from this particular pipeline. Can I specify project.dependencies = ["package_a == {project.version}"] ? Or this definitely will require new plugin?

restive talon
#

this needs dynamic metadata, take beeware as inspiration

simple quiver
restive talon
#

yeah, it has a monorepo and does version linked releasing

simple quiver
#

Is there any other ways to intercept or modify setuptools data locally without introducing build separate plugin package like setuptools-scm?

dreamy urchin
dreamy urchin
# simple quiver Is there any other ways to intercept or modify setuptools data locally without i...

For local modifications the simplest way is setup.py.
It is a common misconception that the setup.py file itself is deprecated. The only part deprecated is treating it as a CLI tool.
setup.py is a perfectly fine configuration file (like other Python files are for other tools, like conftest.py, noxfile.py etc...).

You can combine pyproject.toml/setup.cfg + setup.py (provided you configure pyproject.toml's [project] dynamic accordingly if you are using pyproject.toml). Ideally you can keep all your static configuration in a declarative file such as pyproject.toml/setup.cfg and only use setup.py for the parts where custom programming is necessary.

haughty bramble
split sluice
#

FTR - is there a deprecation plan in place for the setup.py?
Or will it live on forever?

haughty bramble
split sluice
gleaming mirage
#

Why should setup.py be deprecated? It's useful.

keen ibex
#

Until we get a declarative, PEPefied API for language extensions compilation, setup.py will be in use

dreamy urchin
# split sluice FTR - _is_ there a deprecation plan in place for the `setup.py`? Or will it live...

Not that I am aware.

My personal opinion is that there is always going to be a valid use case where people need to derive some form of configuration programmatically. Not always static metadata is a good solution for every usecase.

Since setup.py is already there, people are already using it, I don't see much point in removing it completely.

There is value, though, in moving the static bits to a declarative format like pyproject.toml (E.g. other tools can read it, it does not require arbitrary code execution for bundling the distribution archive etc...)

#

Something that could make sense in the long run (my personal opinion again), would be introducing a "paranoid mode" (via an env var for example) in which end users concerned with arbitrary code execution could disable setuptools reading setup.py and also allowlist/denylist some plugins...

But that is just brainstorming... (And if we consider how wheels are published/installed may make less sense than simply having pip instroducing a flag to block installation from sdist)

dreamy urchin
sullen prawn
#

It really is precarious; the kicker is that when we look at something like aggregate behaviors of malicious programs in Python, the overwhelmingly vast majority of them either rely on runtime installation of malicious payloads (a la os.system('pip install <malware>') or perform their staging/execution within the setup.py themselves. Heuristics wise, it is exceedingly difficult to detect negative behaviors in setup.py versus some of the solutions that exist out there. So much so that we've more or less conceded that we'll always have a large number of positives from setup.py files and accepted that that's just the state of the ecosystem.

As it stands now, I'd say a solid 70-80% of our false positives are sourced from setup.py behaviors that require package specific exceptions be made in order for them to no longer be detected as such.

gleaming mirage
#

It would be great to have a documentation page explaining why declarative configuration should be preferred to setup.py. That would make it easier to convince project maintainers to switch.

sullen prawn
#

https://blog.phylum.io/sensitive-data-exfiltration-campaign-targets-npm-and-pypi/
On cue, here's a demonstration of why, from a security perspective, setup.py is very concerning to us.
Arbitrary exfiltration of ssh and kubeconfig credentials, unilateral attack across ecosystems, etc.
We didn't catch this, fortunately Phylum did. Blanket detecting 'curl' commands in setup.py leads to an enormous amount of false positives; and trying to enumerate out all variants of curl commands (without just detecting the use of the word 'curl' directly) makes this incredibly difficult to programatically track. You can probably picture a world where '.kube' and '.ssh' aren't spelled out as clearly; or where obfuscated payloads or even simple string concatenation might be used to avoid this entirely.

I don't know, there's plenty of examples out there of how setup.py is being levied heavily to target developers and maintainers in order to further create supply chain issues.

indigo tide
#

While I agree that setup.py is visibly being used to do this, removing setup.py doesn't fix anything here. installing a malicious wheel is going to be just as devastating, and isn't difficult for attackers to go this route, they just haven't had to. It's also significantly harder to detect and analyze statically. There are good reasons to ship a binary in a wheel (it's kindof the purpose of those) and someone could easily have a non-malicious sdist with a malicious set of wheels that pip will prefer by default.

river oak
#

It is impossible to apply declarative configuration for all cases, and the possibility of a setup script will live on in that sense. Whether setup.py will serve the use case forever is another question but I don’t see the point for setuptools to deprecate it in favour of something else. If you want to create a new build script API you can just do it in a new build backend instead, why mess with setuptools at all 🤷‍♂️

dreamy urchin
#

Also, setup.py is not the only way of executing arbitrary code during the build process. Any backend that supports local "programatic" customisation will have the same problems (I believe poetry and hatch also support local customisation). Finally build dependencies (as in pyproject.toml's [project] requires = [...] array) also constitutes a form of injecting arbitrary code execution during build...

The best way of preventing those in the end-user's machine(as @indigo tide pointed out) is by using pre-built wheels.

The environment for building the wheels can be made more safe by isolation techniques (e.g. containers, vms, etc) + allowlisting/denylisting build dependencies. Introducing a "paranoid" mode in setuptools that blocks local code execution could be a way of improving that for people interested in repackaging distributions (as it is the case of OSs). Still it would be required for the repackagers to analyse case by case, because some distributions simply require heavy customisation to be able to be transformed into a wheel file, which is made possoble by setup.py.

indigo tide
#

except the wheel itself can contain a malicious binary that's placed in such a way that it's preferred over something that's near guaranteed to be imported, like typing_extensions, or even just malicious on top of the expected functionality of the package, I was not suggesting that pre-built wheels solve this at all, quite the opposite, that one of the very next places it could go if you push back on setup.py is it can just stop happening at install time and happen at import instead.

#

the real answer is don't install packages without vetting them first and (unfortunately) there are bad actors out there, so we need people policing for malice

dreamy urchin
#

Yeah that is true.

Pre-built wheels just solve the problem of "arbitrary code execution at installation time", but in the Python ecosystem you can have arbitrary code execution at import time.

river oak
#

Any central third party package distributor is arbitrary code execution as a service basically

indigo tide
#

@sullen prawn speaking of this line of thought, I'd like to bring something to anyone playing defense's attention in case it isn't already on people's radar's prior to a post I intend to make some time next week. It's not a vuln or disclosure process for that would be pretty clear, it's a means and methods thing, but it may be worth balancing the "this should be publicly discussed" with "make sure people playing defense are already considering this", lmk if you have a preferred place to discuss it if you're open to it, DMs here fine if that's fine too.

sullen prawn
# indigo tide <@227921192772698114> speaking of this line of thought, I'd like to bring someth...

You can fire me a DM at your convenience!

Good discussion in the above; I concur that vetting your dependencies is highly relevant, but in the same vein, the exponential nature of SBOMs to inflate themselves in any real code base makes ensuring that you’re using secure code incredibly difficult. I get the point— building your own deps from the sdist has an inherent level of risk; but there is a real world scenario where that action could occur in a manner entirely occluded to you the package user; in other words, you could be installing a wheel for “selenium”, but one level down, you’re building and installing something from the sdist instead. And really, you’re probably not going to have a way to know this is even happening, unless you’re climbing that entire dependency tree and cross-checking that you, indeed, have the appropriately matching specifications for a given wheel in a package.

Edit: Also just for clarity's sake, I'm not really championing for any specific fix here. We struggle to write effective heuristic detections for setup.py, but I truly don't have an excellent solution in many respects on what a 'remedied' version of this would look like. I don't want it to seem I'm screaming "setup.py is ruining security!" I just happen to get an (I believe) fairly unique perspective given... the work I do, perhaps, and enjoy engaging in dialogue about some of those issues.

indigo tide
stable jasper
# haughty bramble

I’m in this picture, but unfairly someone cropped out the part where I say “write a hatch plugin instead” 🐦

restive talon
worn laurel
#

hello hello! I have a really odd issue and I hope it's okay I ask here, because I honestly don't really know where to ask this:

I have a Wheel I'm building with setuptools and setuptools-rust (currently still the old-school python setup.py bdist_wheel approach).
For some reasons in GitHub Action's envrionment for MacOS Python 3.11 (and only 3.11) the wheels built with the x86_64 Python environment provided there are incorrectly marked as being universal2. They are not universal2, they are only built for x86. The other Python versions correctly mark this as an x86 only wheel.
Does anybody have an idea what could be going wrong here?

Example run: https://github.com/SkyTemple/skytemple-rust/actions/runs/6355353799/job/17263321360

I realize this is most likely not the right place to ask... if anybody could guide me to the right place that'd be great.

#

Twine or PyPI also don't reject them. I thought they scanned the wheels to make sure they are valid?

#

right now because of this i have a wheel on PyPI that is marked universal2 but is actually x86 only because of this

#

ImportError: dlopen(/opt/homebrew/lib/python3.11/site-packages/skytemple_rust.cpython-311-darwin.so, 0x0002): tried: '/opt/homebrew/lib/python3.11/site-packages/skytemple_rust.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

#

file shows: skytemple_rust.cpython-311-darwin.so: Mach-O 64-bit x86_64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|NO_REEXPORTED_DYLIBS|HAS_TLV_DESCRIPTORS>

#

So this is either an issue with GitHub Actions, setuptools, wheel, or Python 3.11 itself I guess.

keen ibex
worn laurel
#

where do you think this issue most likely is and do you know where I can best report it?

keen ibex
#

My guess would be setuptools-rust or setuptools itself (don't know how those 2 are working together). You might also check out Maturin, kinda "all-in-one" rust-python solution

worn laurel
#

It's not setuptools-rust. The other package I linked does not use setuptools-rust.

#

(it uses setuptools-dso)

#

So I think this is a more general issue. I can kinda not imagine this being a general Python 3.11 MacOS bug so I guess I'll try to start raising an issue at GitHub Actions, unless somebody has a better idea. I really doubt this will get any traction there.

#

Sadly I don't have a Mac to replicate this issue outside of the GitHub Actions runners, so I have no idea.

#

what package is actually responsible for generating the wheel identifier that should be used?

keen ibex
#

in setuptools? I have no idea. I don't use setuptools at all

worn laurel
dreamy urchin
# restive talon right now setup.py is the recommended mechansm to put dynamic code into setuptoo...

For the wheel itself (bdist_wheel) the example in https://github.com/pypa/setuptools/discussions/3762#discussioncomment-4658230 shows how to add an arbitrary file to the archive.
However for the editable_wheel I don't think that is directly supported. For the "lenient" mode or the "compat" mode, the files are taken from the project directory, so you could in theory add a file there... for the "strict" mode - which uses a symlink/hardlink farm, it is possible to add a custom file to the file tree in the auxiliary directory (https://peps.python.org/pep-0660/#what-to-put-in-the-wheel) by implementing the interfaces in https://setuptools.pypa.io/en/latest/userguide/extension.html#supporting-sdists-and-editable-installs-in-build-sub-commands, but I have the impression that a minority of users is using that mode.

dreamy urchin
# dreamy urchin For the wheel itself (`bdist_wheel`) the example in https://github.com/pypa/setu...

I understand that in theory we could have an arbitrary file added to the editable wheel in addition to any .pth file or custom MetaPathFinder, but then we are bound to get into the "namespace-hell" problem with editable installs isn't it[^1]? So far we haven't got back from the core devs any instructions on how to implement "editable" namespaces (https://github.com/python/cpython/issues/92054).

Setuptools current editable wheel implementation does handle namespaces reasonably well (it is probably the most complete implementation available), but it is not without hassle...

[^1]: Or worse, end up in a situation where we want a namespace-like behaviour, but the folders actually contain __init__.py...

restive talon
#

@dreamy urchin i'd also like to see a general hook to place files in the build output (i want to have the next version of apipkg generate lazy loading .py file for api modules or alias modules defined in .pyi files)

dreamy urchin
# restive talon <@819222905522487338> i'd also like to see a general hook to place files in the ...

Does the following help?

class CustomSubCommand(Command):
    def initialize_options(self) -> None:
        self.build_lib = None
        self.editable_mode = False

    def _get_build_lib(self):
        if self.editable_mode:
            return self.distribution.src_root

        with suppress(Exception):
            return self.get_finalized_command("bdist_wheel").bdist_dir

    def finalize_options(self) -> None:
        self.build_lib = self._get_build_lib()

    def run(self) -> None:
        if self.build_lib:
            outdir = Path(self.build_lib)
            outdir.mkdir(parents=True, exist_ok=True)
            (outdir  / "file.txt").write_text("hello world", encoding="utf-8")

    def get_outputs(self):
        return [Path(self.build_lib, "file.txt")]
dreamy urchin
restive talon
woeful bronze
#

Hello everyone, I'm building a package with a pyproject.toml using setuptools and I tried to hook into the build process according to PEP 517.
What I'm trying to do is to collect the version of a submodule (which is not a python project) based on it's git tag and hash and store it in an additional metadata file in dist-info. Therefore, I added a prepare_metadata_for_build_wheel/editable function which uses setuptools original one and add a file there. However, when I display the metadata directory the file is created but it is not written to the resulting dist-info directory. This is a minimal example of my code:

import os

from setuptools import build_meta as _orig
from setuptools.build_meta import *

def _write_version_to_metadata(directory: str):
    # Read version string, insert a fixed one here
    # version = get_vcs_version()
    version = 'v2023.10-dev50-ghash'

    with open(
        os.path.join(directory, "version.txt"),
        "w+",
        encoding="utf-8",
    ) as file:
        file.write(version)

def prepare_metadata_for_build_editable(metadata_directory, config_settings=None):
    dist_dir = _orig.prepare_metadata_for_build_wheel(
        metadata_directory, config_settings
    )
    _write_version_to_metadata(os.path.join(metadata_directory, dist_dir))
    return dist_dir

Is there something wrong with what I'm doing? Or am I doing something which I'm not intended to do with adding an additional file to dist-info? (Basically, I'm trying to do this https://setuptools.pypa.io/en/latest/userguide/extension.html#adding-new-egg-info-files but for the new egg-less build process)

dreamy urchin
# woeful bronze Hello everyone, I'm building a package with a pyproject.toml using setuptools an...

For many reasons, setuptools does not "re-use" the metadata directory, but instead tries to regenerate a consistent copy of it (also related to the fact that the code that generates the final wheel is in pypa/wheel not in pypa/setuptools).

Setuptools will try to ensure that all the files itself writes to the .dist-info in the prepare_metadata* hooks are also present in the final wheel... However, if you are overwriting the method, now this responsibility transfers over to your custom backend. Unfortunately this is absolutely non-trivial...

If you would like to contribute with the improvement in setuptools focusing in "re-using" the .dist-info directory (instead of "re-generating" it), that would be a very nice.

While we don't have that implemented, another solution would be working directly in a setuptools plugin. I suppose you could try o use https://setuptools.pypa.io/en/latest/userguide/extension.html#defining-additional-metadata. I think pypa/wheel probably will simply copy the file over to .dist-info (but that should be tested, I don't remember for sure).

If you don't need this file to be necessarily in the .dist-info directory and if you have the flexibility of write this file as a regular data file inside the package directory, you can also use a sub-command or customise an existing command https://setuptools.pypa.io/en/latest/userguide/extension.html#extending-or-customizing-setuptools. There is an example here generating custom files: https://github.com/pypa/setuptools/discussions/3762

woeful bronze
# dreamy urchin For many reasons, setuptools does not "re-use" the metadata directory, but inste...

Thank you for the answer.

Actually, your last option was what I went with. I just let the build hook create a file in the repository before I call to setuptools prepare_metadata* functions. This works fine and is good enough for my use case for now (even though I feel it would be cleaner in the .dist-info directory).

I actually tried this https://setuptools.pypa.io/en/latest/userguide/extension.html#defining-additional-metadata first and I ended up with writing build hooks because I couldn't get it to work with my pyproject.toml based setup. It felt like that this guide is outdated for the current process but maybe I just did something wrong.

I could give an improvement for re-using the .dist-info a shot. I'm not particular familiar with the setuptools code base (had a look there though when I tried to figure out why my file got lost) so it would be nice if you could point me to relevant code locations. Also I will not be able to allocate too much time rn, so it probably would take some time until I'd be ready to contribute something.

dreamy urchin
dreamy urchin
#

There was a pile of PRs pending and I was kind of hopping to cut a release on the beginning of November (but I got busy...), so I just pushed a tag for 69.0.0.

This version mainly removes deprecated stuff (mostly related to pyproject.toml).
In theory the changes should be relatively safe for old packages that don't use pyproject.toml.

#

I haven't touched the PRs that I considered were still in the "review" stage (e.g. fixing type check errors and all the spin-offs)

granite flint
#

Hey, just wanted to say that you're a real hero, @dreamy urchin . Not only do you contribute a tremendous amount of work for the ecosystem, but you are consistently kind, thoughtful, humble and helpful to others, which is a relatively rare combination given how much you have to deal with. Thanks for everything you do! 💛

dreamy urchin
boreal yarrow
keen ibex
#

is there a plan to integrate distutils code into setuptools properly instead of vendoring it?

boreal yarrow
#

I wonder what would be needed for this? Move over the test suite and CI? Anything else?

keen ibex
boreal yarrow
#

Hm, not sure what you mean.

restive talon
boreal yarrow
#

I'm wondering if there is much room for improvement if distutils API and the monkeypatching need to stay as is. But either way, centralizing the development seems like a step in the right direction to me. (not that my voice has much weight in this..)

restive talon
keen ibex
split sluice
#

So -- what exactly is distutils used for?

keen ibex
#

setuptools was (kinda still is) a wrapper for the original code of distutils from stdlib

restive talon
split sluice
#

It's been removed from CPython, and now it only exists as a vendored dependency in setuptools, right?

Why can't we just update setuptools to not need distutils and then archive distutils?

#

Oh

#

Uhh

#

Move the setuptools patches back to distutils and archive setuptools?

keen ibex
#

check out how setuptools code works. you will encounter distutils-related code like 2-3 functions deep

keen ibex
restive talon
#

if it was simple/easy it would have been done in a jiffy

#

its jarringly hard to sanely reintegrate those 2 things

keen ibex
#

honestly, I once tried to understand the setuptools code to see how hard it would be to take the extensions part and replicate it or isolate into a separate package, turned out close to impossible without hauling around a lot of legacy. "simplest" (only from code standpoint, not taking into consideration time, compatibility or users) way would be to just rewrite it entirely

split sluice
#

So uhh... What if we archived both?
I mean, I feel bad saying we should just abandon all the time and love and energy that's gone into these tools
But the PyPA now owns.... three? PEP 517 backends?
What if we picked just one to focus our efforts on?

split sluice
keen ibex
#

setuptools is the default and backwards compatible backend for great many projects

gleaming mirage
#

Flit doesn't support dynamic configuration. Or native modules.

keen ibex
#

and practically the only one that supports extension modules (like code in C/C++ etc)

split sluice
#

And to be clear, I am speaking from almost complete ignorance
Just trying to understand the options on the table

split sluice
keen ibex
#

from my point of view, until we have a PEP-standard way to declare and compile extension modules, setuptools are there to stay

keen ibex
#

#extensionlib is a hope for the future, but I think it stalled for now. not sure what the plan is exactly

jaunty dagger
#

hi, quick question: for a python extension module I currently use setuptools as a build-backend where the extension compilation requires certain compiled components present on the system – suitesparse and sundials. the cibuuldwheel docs here mention about a build_py distutils/setuptools command class (https://cibuildwheel.readthedocs.io/en/stable/faq/#actions-you-need-to-perform-before-building) that can be used as a pre-installation script that I would like to use in my setup.py for both editable installations and building wheels (we currently provide a separate python script which we can directly integrate here then). I can include the suitesparse and sundials tarballs directly in the sdist to save users' bandwidth and allow their compilation and therefore building extensions on a platform I don't publish wheels for – but it does dramatically increase the size of my sdist from ~6 MB to around 70-80 MB. the extension in specific is a C++ solver that wraps sundials and is optional based on specific environment variables we check for in setup.py, and another thing I feel is that including the source code for suitesparse opens us up to potential software licensing issues (it is GPL-licensed while ours is BSD) – this can be bypassed by downloading the tarballs at runtime. my goal is to simplify the installation: why have two scripts when I can embed them into one and pip install can attempt to compile things up but fall back to the non-accelerated alternative if it doesn't work. in a nutshell, I want to seek opinions on embedding pre-installation scripts in sdists, the purpose of sdists viz. pre-installation, and the purpose of build_py especially in cases like this where one needs to compile something beforehand. I am an intermediate packager at best so I would appreciate thoughts from those who have expertise around this area :) thanks!

boreal yarrow
wind wolf
boreal yarrow
#

hm, would it be possible to completely replace distutils via _distutils_system_mod.py at import time?

#

Does anyone do this by any chance?

haughty bramble
haughty bramble
#

Where are setup(data_files=[...]) supposed to end up?

#

Oh I found them it's directly in the root of the virtual environment

haughty bramble
#

Why are data_files deprecated? The note says it's unsupported in wheels but I think that's outdated because I just tried it with a wheel and it worked

dusky willow
#

You should use package_data

#

Or at least, that is the advice I got a long time ago

haughty bramble
dusky willow
#

Why do you want to do that?

haughty bramble
haughty bramble
#

@dusky willow ^

dusky willow
#

Does jupyter not use standard entry points based plugins? I'm not familiar with the ecosystem on that

haughty bramble
#

I think it's legit to want to use the .data/data whl format feature

#

And this is the only way it's exposed

dusky willow
#

I don't recall, do data files get namespaced when installed?

haughty bramble
acoustic cradle
#

the problem with data_files historically was that there was not a standard place to put them

#

so it was a big ??? where they would end up

haughty bramble
dreamy urchin
# haughty bramble Is the standard hashed out now? Could it be undeprecated?

By the way I find that files inside "package directories" work fine most of the time (at least for the same use cases that data_files is actually able to handle 100% of the times, i.e. the use cases where there are no requirements about specific paths).

The major valid concern that I saw once in the Python discourse (packaging category - https://discuss.python.org/t/should-there-be-a-new-standard-for-installing-arbitrary-data-files/7853/74) justifying data_files was the performance hit when you use entrypoints for plugin discovery.

However, you still have the ability to co-locate arbitrary files inside a single folder, if you create a namespace package.

For example, imagine you have the myapp package on PyPI that has a plugin system that allows 3rd parties to add HTML, CSS and JS files... The myapp docs can say that plugins need to add files into the myapp_assets namespace package. So when the plugins are installed, pip will place them inside the same directory (please correct me if I am wrong here).

This way you have the same benefit of data_files in terms of plugin-discovery performance, while also being able to use the extra niceties of importlib.resources.

gleaming mirage
#

data_files could still be useful for non-Python directories.

river oak
#

The ability to put files outside site-packages is considered a misfeature since it breaks the (virtual) environment abstraction. There’s still use case of course but some design would be needed to fit it into the environment layer. Or arguably software (Jupyter…) should simply not rely on system paths to begin with.

gleaming mirage
#

How does it break the abstraction?

past compass
dreamy urchin
# gleaming mirage `data_files` could still be useful for non-Python directories.

If you are willing to accept that such non-Python directories will be placed relative to a parent directory chosen arbitrarily by the package installer depending on the way it was invoked and/or system configurations: yeah, definetely useful for that. But that is the catch.

The only use case I know that fits this category was discovery of asset files from plugins that I discussed above (for extreme performance optimisation that you rarely see in packages). But even this use case can be solved in an alternative way, via namespace packages.

If you know any other use case, please let us know. I think this is a very nice thing to have in mind. This is the last discussion we had on the topic, as far as I remember: https://discuss.python.org/t/should-there-be-a-new-standard-for-installing-arbitrary-data-files/7853.

Kudos for Miro that compiled the main things that people ask for https://discuss.python.org/t/should-there-be-a-new-standard-for-installing-arbitrary-data-files/7853/63. As summarised in the thread, manpages and desktop files will not work out of the box if the end user employs virtual environments.

gleaming mirage
#

Thanks, that's helpful.

#

I guess it's fine for this feature to stay deprecated, but the docs still shouldn't claim that it doesn't work in wheels, since that's incorrect.

dreamy urchin
restive talon
#

@dreamy urchin I'm wondering, is there a easy to instruct in process way to test setuptools plugins that are supposed to generate/use artifacts and/or affect file lists for sdist

I still can't fathom a safe way to make and integration-test correct and sdists aware file finders for setuptools-scm

robust yarrow
dusky willow
#

I have a workflow where we build the wheel then run wheel tags to re-tag the wheel

#

Oh but that wouldn't allow cross compiling to a different arch bitness

robust yarrow
#

oh true

#

yeah the issue is that a contributor switched a project of mine to pyproject.toml which broke the build for Windows

hot hamlet
#

How can I use setuptools to clone a git repo that isn't a Python package? I'd like to clone a repository of C code so I can generate bindings from its headers

restive talon
hot hamlet
dreamy urchin
# robust yarrow is there a way to pass this option via `--config-settings`? https://setuptools.p...

The "backward compatibility" of --config-settings is more or less like the following in setuptools (roughly equivalent):

get_requires_for* ~~> f"python setup.py {config_settings['--global-option']} egg_info" ~~> workaround to extract dependencies
prepare_metadata_for* ~> f"python setup.py {config_settings['--global-option']} dist_info --output-dir {metdata_dir} --keep-egg-info"
build_sdist ~> f"python setup.py {config_settings['--global-option']} sdist --formats gztar --dist-dir {output_dir} {config_settings['--build-option']}"
build_wheel ~> f"python setup.py {config_settings['--global-option']} bdist_wheel --dist-dir {output_dir} {config_settings['--build-option']}"
build_editable ~> f"python setup.py {config_settings['--global-option']} editable_wheel --dist-info-dir {metadata_dir / '*.dist-info'} --dist-dir {output_dir} --editable-mode {config_settings['editabe-mode']}"

(subject to human errors from my side :P)

However, please note that we cannot consider this feature stable.

The only known "stable" mechanism so far for passing configuration parameters for individual commands is via setup.cfg.
If no other tool is relying on the DIST_EXTRA_CONFIG environment variable, you can also use it to avoid modifying an existing setup.cfg and instead add an extra one.

dreamy urchin
dreamy urchin
robust yarrow
dreamy urchin
#

Probably something like the following:

python -m build -C "--build-option=--plat-name win_amd64"

(or a variation? The stability of this approach is not guaranteed)

A more stable solution is to add a command configuration section to setup.cfg:

[bdist_wheel]
plat_name = win_amd64
proven sapphire
#

Unable to install a python package because no matching distribution found for setuptools>=40.8.0

lofty tiger
#

Hey all, I have a fairly restrictive environment that I am trying to make an in-house package nice and easy to install to. The setuptools version is waaay below v60 so my existing PEP517/518 pyproject.toml-managed project proceeds to fail spectacularly when attempting to install from source (wheels are not an option 😔). Apparently doing the install inside a virtual environment with a non-ancient setuptools is "too complicated", so my next best option is to bundle a "pip freeze"'d requirements file.
Don't suppose anyone could weigh in on this before I commit myself to bundling a requirements.txt with every release?

keen ibex
#

That sounds.... weird. What is "too complicated" in using newer setuptools and virtual environment?

lofty tiger
#

The end user is not technical, and iirc the environment is some old release of an enterprise linux running py36

#

I have chosen not to wage the war of "please fix your environment" and have consigned myself to releasing an install script.

haughty bramble
lofty tiger
#

possibly? How well does it handle package resources (importlib.resources.files)?

keen ibex
#

It should

lofty tiger
#

wow yeah that works perfectly

dreamy urchin
#

Just to double check: I suppose telling your user to run pipx run build is not a possibility, right?

haughty bramble
#

I'm not sure why wheels are not an option though

river oak
lofty tiger
# haughty bramble I'm not sure why wheels are not an option though

It’s less that it’s not an option and more that it’s need to be delivered as source code, and they are used to just running pip install -r requirements.txt. Anything that doesn’t resemble that without reasonable justification results in a long discussion on why it’s necessary.

#

Building the wheel on their side was the closest to what I was originally envisioning

gleaming mirage
#

Put . in your requirements.txt. 😉

lofty tiger
#

All I need is the ability to perform a single command build/install in an environment running py36 that doesn’t depend on anything not in stdlib by default

#

Or py37

lofty tiger
#

They’re fine running pip install but on setuptools<60 that results in an empty UNKNOWN-0.0.0 being built instead of the actual project

gleaming mirage
#

Isn't it the version of pip that matters here (rather than setuptools)?

lofty tiger
#

Afaik no. It’s setuptools that’s shitting the bed, not pip

gleaming mirage
#

But if you have a PEP518 project, then pip is supposed to download whatever's in your build requirements (including newer setuptools).

lofty tiger
#

It’s supposed to. But from what I can tell, pip installs newer setuptools but then doesn’t use it, or fails. Googling leads me to believe that because a lot of Linux distros rely on a pinned version of python and pip, it won’t let you upgrade the versions for backwards compatibility reasons. Good example of this is upgrading pip on Ubuntu 18.04

gleaming mirage
#

Really? What does the install log look like?

#

😛

dreamy urchin
#

If your Python version does not support setuptools>62.5.0, then you cannot build projects that use pyproject.toml. That is the sad truth.

vagrant patrol
#

is there a way to expose library and header files from ext_modules to others that install your project? so like if you did pip install somelib and you had your own extension module you could use

#include <somelib.h> // links against library properly
river oak
#

Specify it as a data file to go somewhere in the environment, and tell the dependant package to add that somewhere to the include path. There’s no standard around this (the entire extension dependency area is very under-specified) but I believe the convention is to put the headers under $dataroot/include

#

And if the downstream uses setuptools (as an example) it adds the path with Extension.include_dirs

stable jasper
gleaming mirage
#

Does setuptools provide a way to put stuff in that directory, though?

stable jasper
#

Yes, I don’t use setuptools so I don’t know the option, but I know that it exists.

jupyter packages rely on the data dir, and they used setuptools before they switched to hatchling.

river oak
#

IIRC there’s a headers key for this

gleaming mirage
#

I can believe that they use the data dir, but headers?

gleaming mirage
river oak
#

It’s been a while since I did it. Use data_files or something if it’s not there. I think they all go into the same place in the end anyway.

stable jasper
#

data_files sounds right. #installer supports the headers key, so if you e.g. build a DEB/RPM/Pacman/… package using #installer, the headers will go in the right place.

#

So if you use it, someone creating a native package from your PyPI package will simply have to run

python -m installer --destdir=... dist/*.whl

and everything will be put in the right place.
This makes it easier for people to package your library for a Linux distribution and everyone wins.

@quaint jolt started a conversation about creating a more well specified standard for this, but so far, this is the best we have.

vagrant patrol
stable jasper
river oak
#

include is the more popular convention in general in C et al

gleaming mirage
#

That's not the same thing. It's a directory named include that's installed under the data key, not something that's installed under a key called include (which doesn't exist).

#

The key intended specifically for headers is called headers.

river oak
#

Oh OK I thought that was a repo’s source tree

#

sysconfig uses include though so I can see why there’s confusion

vagrant patrol
#

the issue for me hasn’t been the headers, it’s been making the actual shared object available to others

river oak
#

Isn’t platlib for this?

vagrant patrol
#

im really not familiar with python packaging, i just develop libraries. what does platlib do?

restive talon
#

Is there any way to better manage the order of the finalize options hook and the setup keyword hook

Setuptools-scm is in deep trouble as depending on version and build backend the order of these hooks maybe reverse

dreamy urchin
#

The only thing it is implemented right now, is that it sorts the finalize hooks according to an optional .order attribute (lower numbers go first, 0 is the default)

#

I don't think the same is implemente for the keyword hooks.

#

The keyword hooks always go first than the finalize hooks

restive talon
dreamy urchin
#

distutils.setup_keywords hooks run with order=0, it is not currently possible to modify that.
setuptools.finalize_distribution_options hooks run with default order=0, but you can modify that by seting an order attribute in the function (the higher, the later it runs, you can have negative numbers too)

#

hooks with the same order attributed (or default 0) run according to the order they are found in the disk by importlib-metadata

#

(so on further analysis, my previous phrase "The keyword hooks always go first than the finalize hooks" is incorrect)

brittle crypt
#

The bdist_wheel command has now been merged from wheelto setuptools and will be in the next release

#

this should make it much easier to work on the long-standing warts of the implementation, as there is no longer any need to coordinate and synchronize releases between the two projects

#

once there is a new release of setuptools with this included, I will deprecate the bdist_wheel command in wheel and ultimately remove it

gleaming mirage
#

Awesome!

brittle crypt
#

I am currently working on migrating the wheel file handling code from wheel to packaging, overhauling and polishing the API in the process so that it'd be more palatable to other build back-ends than just setuptools

unkempt birch
#

@brittle crypt I’ll try to make a branch of scikit-build-core based on the packaging PR and provide feedback soon

stuck dew
#

I am working with a package system Nixpacks

In it, the python provider is doing some unclear things:
https://github.com/railwayapp/nixpacks/blob/9fae921e719f540965347b662571311273353a1c/src/providers/python.rs#L192-L194 ({create_env} && {activate_env} && pip install --upgrade build setuptools && pip install .)

The main thing I am unsure about is pip install --upgrade build setuptools. To get you as much context as I can without you looking at the code; this step is ran if it is pip installing through a pyproject and not a requirements file. It seems to be an attempt to handle https://github.com/railwayapp/nixpacks/blob/main/examples/python-setuptools/pyproject.toml. However, I am of the understanding and opinion that this explicit install of build and setuptools is largely not required here. It doesn't seem to be needed in any way. Is it not better to just let pip figure out how to install all required dependencies on its own? Does PEP 517 make this whole line unnecessary? It really feels like it is not required here.

It isn't like there is some "build" step going on. It basically will download the dependencies and then start the python program through basically python main.py.

I have asked the maintainers and ,,, they don't know. I am sure someone knows why it was done this way. But none of the main people seem to know. The whole system is a bit of a mess in some places.

[...] but it was my impression that pip install . both built and installed the project
[...] I thought that build, at least, was how pip dealt with it
I know that setuptools isn't needed
^ from the guy who wrote this part. Not sure though if that really changes anything I said. Because it shouldn't be required to explicitly instal these dependencies? I would really appreciate it if someone could help make this behavior more clear to us

The main motivation behind all of this, is that I am looking to clean up the python provider and make some additions / improvements. So I was wondering if anyone could think of a reason why I shouldn't remove the pip install --upgrade build setuptools? Or better yet, if you have a suggestion for a better, more generalized way to handle this

quaint jolt
#

if you have any issue tracking this in the nixpacks upstream, feel free to ping me there

boreal yarrow
restive talon
keen ibex
#

All the best and a lot of patience to setuptools maintainers. The storm will pass

prime mesa
keen ibex
prime mesa
#

Why not? Mostly as a temporary solution to unblock other people's workflows

prime mesa
#

Ok, I've found the culprit was cssbeautifier which indeed used those API. I thought that project were being affected more randomly. Can't tell if that's the case right now though

#

I've had a report for pybind11, which doesn't look like it should fail, which is what raised that suspicion

restive talon
#

Half a decade of deprecation is still not enough:)

split sluice
keen ibex
restive talon
#

So deprecation is not enough, infrastructure to make them notable needs to be implemented everywhere

keen ibex
#

yeah, I was thinking something in line of package installers bubbling up the deprecations on installation or something

restive talon
#

They needed to be visible in container builds,ci and more

#

It's so easy to just miss them in builds until the build goes up in fire

keen ibex
#

yeah

restive talon
#

So given current tools, I suspect the only general vehicle will be something like open telemetry

Or a similar behemoth

unkempt birch
#

People also tend to forget there are three levels of Deprecation warnings, with the highest level not including "Deprecation" in the name (FutureWarning) - it's the generally only one that shows up for "users". It doesn't help that pip eats output by default or that setuptools is often really verbose.

lucid oyster
#

Hey Folks,
I wanted to setup setuptools locally for development, is there are guide for this? (Not very familiar with python ecosystem, wanted to hack around)

humble palm
lucid oyster
#

Thanks @humble palm I was following that but I've been getting errors while running tox commands.

  1. Set up a venv
  2. pip install -e ".[testing,docs]"
  3. After this most of the commands error out for me

Is this the correct way?

humble palm
#

if you run tox, you don't need to set up a venv or pip install, it does that stuff for you

#

what tox command did you run? what error did you see?

lucid oyster
# humble palm what tox command did you run? what error did you see?

So I'm trying to get 68.1.2 working. After using tox, initially it complained about not finding https://github.com/jaraco/build@bugfix/630-importlib-metadata, which is linked to https://github.com/pypa/build/issues/630. But that gives a 404 now. After removing that clone line, running tox resulted in a lot of test failures:

==================================== 285 failed, 1276 passed, 22 skipped, 11 xfailed, 2 xpassed, 1149 warnings, 63 errors in 155.65s (0:02:35) =====================================
py: exit 1 (156.29 seconds) /home/vyom/patching_pkgs/setuptools/noble/setuptools-68.1.2> pytest pid=71712
.pkg: _exit> python /usr/lib/python3/dist-packages/pyproject_api/_backend.py True setuptools.build_meta
  py: FAIL code 1 (175.56=setup[19.27]+cmd[156.29] seconds)
  evaluation failed :( (175.66 seconds)
humble palm
#

what command are you running? just plain tox?

lucid oyster
#

Yup, just plain tox

humble palm
#

are you on a fork?

lucid oyster
#

I'm on the Ubuntu source package for setuptools, patching RCE CVE for setuptools

humble palm
#

hmm, ok then it seems to be about a year old

lucid oyster
#

Is there any way to recover that branch (jaraco might have it locally), but except that there's no way, right? (unless the commit can be found)

humble palm
#

I doubt it can be recovered

lucid oyster
#

I don't want to give up that fast on this. Can you help me to understand what I need to possibly do to fix the problem in https://github.com/pypa/build/issues/630. I can checkout from a commit near that date on the main branch and can maybe add the patches that the issue talks about. (screams in dependency hell)

humble palm
#

good to hear you're giving up! you might be better off getting some help from a setuptools maintainer at this point 🙂

lucid oyster
#

Hmm, I'll do some research before directly reaching out to the maintainers. Thanks for the help so far Hugo!

dreamy urchin
#

Old commits of setuptools may include workarounds that are bound to the situation of the ecosystem in the time they were written...

If you see a pinning in the tox.ini to a specific branch/commit, you may try to bump that dependency to the version that came immediately after the issue was solved. In general, just removing the pinning altogheter might also work.

lucid oyster
#

Removing the pinning didn't work

dreamy urchin
#

If you tried to regenerate the tox environment without the pinning and it does not work, then you may have to find out the version where the fix was released...

lucid oyster
#

So, I've been trying different commits in https://github.com/pypa/build/pull/631 to see if anyone works because they are from the branch I'm referring above to, but I'm getting an attribute error:

  File "/home/vyom/patching_pkgs/setuptools/noble/setuptools-68.1.2/.tox/py/lib/python3.12/site-packages/jaraco/packaging/metadata.py", line 8, in <module>
    source_dir: util.StrPath,
                ^^^^^^^^^^^^
AttributeError: module 'build.util' has no attribute 'StrPath'
keen ibex
#

what version of build do you have?

#

looking at releases, I would say you want 1.0 for that

lucid oyster
#

1.0 also gives the same error

# 
XFAIL setuptools/tests/config/test_apply_pyprojecttoml.py::test_utf8_maintainer_in_metadata[international-email] - CPython's `email.headerregistry.Address` only supports RFC 5322, as of Nov 10, 2022 and latest Python 3.11.0
XFAIL setuptools/tests/test_build_py.py::test_excluded_subpackages - reason: #3260
XFAIL setuptools/tests/test_dist.py::test_read_metadata[Metadata Version 1.2: Project-Url-attrs5] - Issue #1578: project_urls not read
XFAIL setuptools/tests/test_dist.py::test_read_metadata[Metadata Version 2.1: Provides Extra-attrs9] - provides_extras not read
XFAIL setuptools/tests/test_egg_info.py::TestEggInfo::test_requires[extras_require_with_marker_in_setup_cfg]
XFAIL setuptools/tests/test_virtualenv.py::test_pip_upgrade_from_source[pip<20.1] - pip 23.1.2 required for Python 3.12 and later
XFAIL setuptools/tests/test_virtualenv.py::test_pip_upgrade_from_source[pip<21] - pip 23.1.2 required for Python 3.12 and later
XFAIL setuptools/tests/test_virtualenv.py::test_pip_upgrade_from_source[pip<22] - pip 23.1.2 required for Python 3.12 and later
XFAIL setuptools/tests/test_virtualenv.py::test_pip_upgrade_from_source[pip<23] - pip 23.1.2 required for Python 3.12 and later
XFAIL setuptools/tests/test_virtualenv.py::test_pip_upgrade_from_source[https://github.com/pypa/pip/archive/main.zip] - #2975
XFAIL setuptools/tests/test_integration.py::test_python_novaclient

Using without any version (ig it gets latest compatible). I think some failures are expected here 🤔

nimble bridge
#

Heyo! Are y'all aware that setuptools 72.2.0 failed in the release pipeline and isn't on PyPI?

#

Flagging this coz I saw a changelog entry and no release on pypi.org for it.

potent basalt
nimble bridge
#

Indeed!

keen ibex
#

is there some docs for the API of extension modules builders classes? If not, maybe I could get some help. I would like to use setuptools for building extension module, but without passing the build through the setup() method. What is the API there?

keen ibex
#

basically I would want to do:

ext = Extension(...)
# some config methods run on ext
ext.build() # or whatever the method is called
dreamy urchin
#

Extension objects are basically data objects that hold data. The action happens in build_ext.run

split sluice
#

Wait, it tries to defer to Cython?
Wait, Cython bundles distutils?

#

This whole ecosystem hurts my head braindamage

#

AAAAA

river oak
#

The magic of monkeypatching

dreamy urchin
#

I don't think that except ImportError in Cython happens in normal circumstances as Setuptools provides the distutils package on Python 3.12. So there should be no circular import.

#

Probably that _build_ext_module = sys.modules.get('setuptools.command.build_ext') would be None because the import machinery has not finished executing setuptools.command.build_ext yet.

nimble bridge
#

https://urbanists.social/@fancysandwiches/113120369825013877

Since I'm sure the issue tracker is usually filled with people complaining.

urbanists.social
Fancy Sandwiches
Fancy Sandwiches (@fancysandwiches@urbanists.social)

I maintain a Python package at work, but I don't have to touch it particularly often. Our package build stopped working in our CI environment the other day and I found out it's because setuptools is basically deprecating every CLI command.

The solution was dirt simple, and in the end I think my package is better because of these changes. Next s...

keen ibex
#

so, setuptools is removing all commands?

keen ibex
split sluice
quiet latch
#

oh nice, there's a document in packaging docs about it now

#

previously I only recalled Paul Ganssle's blog post about setup.py commands

humble palm
keen ibex
#

is requires.txt just a legacy thing or is it still used by setuptools?

#

also, what is the status of eggs and egg-info?

humble palm
keen ibex
#

ok, but I still see .egg-info spawning here and there when setuptools is used...

raw dune
river oak
#

Eggs are mostly dead but egg-info still pops up here and there in legacy installations, especially if the environment does not have wheel installed

keen ibex
raw dune
keen ibex
dreamy urchin
#

By default setuptools does not parse requirements.txt . However, the users may opt-in to such files (we do recommend using requirements.in though (https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#dynamic-metadata).

.egg-info folders are something normally added to the sdist to store metadata files -- and or to the source tree as metdata caching (the name is a bit outdated, but the reason for existing is legit)

stuck robin
#

Is there a way to configure setuptools such that it creates a multi-package distribution / wheel?

I'm trying to remove bespoke build-time shenanigans in a project which has two root directories foo/ and bar/. At runtime, bar/ is "mounted" into foo.plugins (which depending on the age of the setup, convenience, or other constraints, can be done by moving all the contents into foo/plugins/, extending foo.plugins.__path__, or symlinking some_dir/foo/plugins to bar and putting the latter on the PYTHONPATH PEP420 style).

I was looking at setuptools' packaging directives wondering if I could maybe just tell setuptools how to set it up, but I'm not sure it's an option? As far as I can tell package_dir would not be able to do the remapping: foo/plugins does have content so it seems to get confused if I tell it that

[tool.setuptools.package-dir]
"foo" = "foo"
"foo.plugins" = "bar"

("error: package directory 'plugins/a_plugin' does not exist"), and I can't find a way to tell it to create multiple packages from just one pyproject.toml / setup.py

dreamy urchin
#

@stuck robin in principle that seems fine ("foo" = "foo" is probably unnecessary), but to analyze it in the proper way it would be nice if you can open an issue with a minimal reproducible example and full trace logs.

Not however that it will make the editable install more cumbersome. It requires a pathentryfinder/metapathfinder and it is a well know limitation of static analysis tools that don't support such dynamic aspects that are native to the Python language.

#

Any particular reason why it is not possible to use a directory structure in the source code that mimics the same directory structure that is needed to be installed in the site-packages?

stuck robin
# dreamy urchin Any particular reason why it is not possible to use a directory structure in the...

That the project is 10+ years old, has a number of outstanding working copies, that most of the work happens in the plugins rather than the core, and thus that this would have a very widespread impact (every development working copy would be full of conflicts).

Making packaging simpler and more standard would be great, but it's a very secondary concern overall, so the question on our side is more if standard tooling can support our odd layout (and we could remove custom build stuff).

stuck robin
#

Although thinking more about it it might be the wrong direction entirely, creating separate pyproject.toml (one for the main project and one for each plugin "pack") could work a lot better if pip can correctly set them up in the site-packages on install (and if the rest of the packaging pipeline supports doing that)

dreamy urchin
#

if pip can correctly set them up in the site-packages on install
Pip is able to "reassemble" namespace packages. But that also implies a change in the directory structure, so not necessarily it changes much for you.

Please feel free to open an issue in the github repo if you are still facing trouble. Discussing with a reproducer will make things easier to investigate

stuck robin
# dreamy urchin > if pip can correctly set them up in the site-packages on install Pip is able ...

Pip is able to "reassemble" namespace packages.
Indeed, that does seem to work quite nicely.
But that also implies a change in the directory structure, so not necessarily it changes much for you.
The software itself doesn't care too much about directory structure (the current bespoke packaging basically does that but ad-hoc at build time, and core actually handle PEP420-style plugin directories), the concerns are entirely on the meat-bag side 🙂

dreamy urchin
#

But that is he point if you can change the directory structure of the source code to match that intended to go to site-packages, ir does not change much if you do a single package or offload the plugins to their own separated packages.

Without changing the source code directory structure everything will be complicated.

stuck robin
wanton salmon
#

Not sure if you're aware of this already, but the "this is the latest development version" warning is not friendly to dark mode users. Despite that, I like it. It's a good reminder.

image.png
#

In particular, the flask icon is invisible as it's white and so is the background.

split sluice
#

That's created by Read the Docs
I don't think you can change it

#

I don't remember seeing the RtD badges on the setuptools docs firT
Did y'all migrate or is this just for PRs right now like CPython?

wanton salmon
#

Oh yeah, this seems to be a RTD thing. I get the same thing on pip. Whoops.

stable jasper
nimble bridge
#

Yea, I have a personal TODO to look into Furo + RTD add ons but things have been slow on my end.

restive talon
#

would there be any way to make more details of the distribution part of the file finder hook - atm setuptools_scm cant even safely opt out/in

glad laurel
plucky atlas
#

Q: when doing codegen with setuptools as build system it seems the usual way to do it is to write out generated files during the early build phase and get them picked up during wheel build.

Is there a way to do the latter directly in-memory without having to go through FS updates, so the working copy does not need to be touched in case e.g. it's mounted readonly?

dreamy urchin
#

I don't think so. If I am not mistaken setuptools relies on puting files on the disk before zipping/targz-ing them.

#

Not sure if you're aware of this already

quaint jolt
split sluice
#

Is there any way to keep a .egg-info folder from being created when building your package?

image.png
unkempt birch
#

(side note) You don't need a setup.py anymore for a (basic) extension module from what I understand.

split sluice
unkempt birch
#

Should be in 74.1.0+

#

For realistic projects, there are always a lot of custom things here that depend on the compiler/platform/etc, but the basics are available in pyproject.toml.

split sluice
#

waaaaaa

dreamy urchin
#

you can try to set these kind of parameters in setup.cfg, but there is always at least one call to the metadata_directory passed by the frontend

vernal lagoon
#

Hi, anyone who is maintainer of setuptools and online ?

past compass
vernal lagoon
#

any comments on my idea proposed at : https://discuss.python.org/t/draft-proposal-artifact-dependency-graph-adg-generation-for-python-wheels/74286

Discussions on Python.org
Draft Proposal: Artifact Dependency Graph (ADG) Generation for Pyth...

Hi everyone, I’d like to share a draft proposal that could eventually become a Python Enhancement Proposal (PEP) if there’s enough community interest and support. The idea is to integrate automatic Artifact Dependency Graph (ADG) generation into Python’s packaging workflow. This post contains the full draft specification, and I welcome any feed...

obtuse parrot
wanton salmon
#

If we need to say, sorry but setup.py install is essentially dead and any bugs will remain, that's fine!

robust yarrow
#

this project is used at work for a particular CLI feature https://gitlab.com/libvirt/libvirt-python
when locking UV tries to build it because there are no wheels and the source distribution that is on PyPI is still using core metadata version 2.1

is there a way to force a higher version of metadata so that the source distribution has the *.metadata file on PyPI thus easing the resolution of transitive dependencies

normal hill
#

With the new ext-modules support in pyproject.toml, how do we specified conditional compiler flags? For example we need to pass -std=c++11 on OS X.

normal hill
#

Ahah,

[tool.cibuildwheel.macos]
environment = { CXXFLAGS="-std=c++11" }
lusty dome
#

@potent shoal I've noticed you've wanted to transfer issues from packaging-problems to setuptools. Would you be interested in membership to setuptools-team so you can make such transfers?

potent shoal
restive talon
#

Id like to discuss expanding the file finder api to pass over distribution/ configuration and/or just allowing to opt out of them from setuptools configuration as they keep making trouble for setuptools_scm

vernal harbor
#

Now that setuptools can do ext-modules I'd like to gently nudge for the standardization of something out of ChimeraX's bundle builder: library-modules and include-modules . I included these so a ChimeraX extension could depend on, say, numpy, and when I make the list of compiler arguments I call one of either

numpy.get_lib() or numpy.get_include() from their __init__.py, and add the appropriate -I or -L flag.

It was so useful that I made it the standard for all ChimeraX packages that make dynamic libraries or ship headers.

(Being able to make dylibs and executables would be sick too)

mental flame
#

@dreamy urchin @lusty dome would you be open to backporting https://github.com/pypa/setuptools/pull/4766 into a 75.3.1 release? there are some users who still need to build/release Python 3.8 wheels that can't use newer setuptools versions with this fix in it. I'd be happy to do the work of prepping the PR.

GitHub
Fix "Wheel naming is not following PEP 491 convention" by di · Pull...

Summary of changes
Fixes #3777.
Marking this as draft as there is one outstanding failing test that I have been unable to resolve (setuptools/tests/test_easy_install.py::TestSetupRequires::test_set...

lusty dome
#

That seems reasonable. I thought it would violate semver, but now I see we classified the file renaming as a bugfix. In retrospect, that probably should have been a minor bump as it was a change in behavior and not merely correction of previously-intended behavior. But that's convenient for us today, for purposes of the backport.
If you wouldn't mind creating the PR, that'd be great. It should be fine for it to target the main branch. You'll probably want to hand-add the changelog entry instead of using towncrier, and you'll need to have the version of the changelog for the backport release and also merged with main. I can't recall whether the backport version should appear chronologically or numerically, so you'll want to scan the changelog for another example (or just pick one; I think I'd lean toward numerical).

dreamy urchin
dreamy urchin
normal hill
#

Mostly works, worked around it for the 1 platform that needs it by setting it up inside of cibuildwheel instead and documenting the need for the flag on older versions of OS X

mental flame
dreamy urchin
#

I see! thank you for pointing it out. Yeah the postprocessing should be straightforward. I haven't had the time to approach it yet. If anyone in the community would like to submit a PR that would be great.

normal hill
#

I can submit a PR this weekend, currently working on a CPython threading bug introduced in 3.13 (#130522) 😦 . Has there been any discussion somewhere on how conditional logic might look? Maybe [tool.setuptools] for the base and [tool.setuptools.<platform tag>] to override it?

#

This is essentially what cibuildwheels does for its pyproject.toml configuration, simple yet enables a fair bit

wanton salmon
#

We're trying to phase out .egg distributions, but that bug results in a .egg being produced.

#

I'm not too worried about it. It seems like downstream will be able to work around it.

dreamy urchin
mental flame
#

Backport

restive talon
#

it seems theres multipe versions of setuptools that produce wheel filenames from the build backend that lack the actual file extension - i haven't been able to find a related issue was this missed?
i#ll update details once i make my script better (im creating a all most all setuptools version regression test for setuptools_scm after the last fallout and clear differences in hook order between legacy install that keep happening on enterprise distros and modern tools=)

restive talon
raw dune
#

Regarding

Setuptools no longer accepts options containing uppercase or dash characters in setup.cfg. Please ensure to write the options in setup.cfg using the lower_snake_case convention (e.g. Name => name, install-requires => install_requires). This is a follow-up on deprecations introduced in v54.1.0 (see #1608) and v54.1.1 (see #2592).

#

I would urge you to revert this as I think it is going to break a ton of the ecosystem

robust monolith
#

At this point I would strongly recommend that setuptools yank 78.0.0

restive talon
#

Only if there's a clear timeline for cleaning up before bringing the error back

The depreciation always gets hidden by the ecosystem of frontends

raw dune
#

@restive talon I'm sorry but that doesn't seem like the right trade-off

#

What's the cost to continuing to allow these fields in perpetuity?

#

I'm all for standardizing ways to...

  • Lock or constrain build dependencies
  • Surface warnings from build backends
#

but the timeline for those things is much longer

restive talon
#

im suggesting a tradeoff thats in favour of the project maintainers - setuptools is a high cost low reward thing - and as far as im aware entirely volunteer work - imho it gets the nebraska launch codes

robust monolith
#

This is a very "just my opinion", I already experience enough pain helping maintain pip, I wouldn't want to deal with setuptools

But reading that setup.cfg isn't really owned by setuptools, it's predates setuptools by at least a decade, from other Python packaging tools, is it reasonable for setuptools to enforce new standards on it?

If parsing setup.cfg is really dragging down setuptools would it make sense to release a "setuptools2" that could, say, completely drop support for setup.cfg. And leave setuptools in bare minimum maintenance mode for a few years before stopping support entirely?

My 2cents as a bystander, understand if it's not considering all the issues you face maintaining setuptools.

raw dune
#

What maintenance costs were resolved by removing support for this syntax?

#

The amount of time a disruptive change like this costs package maintainers across the ecosystem is significant too, a high burden should be placed on intentional breaking changes.

raw dune
#

(I'm all for protecting maintainers, but it seems like to some extent the project should just be in a minimal maintenance mode given much of the ecosystem depends on it and it's non-trivial to avoid getting a new version with breaking changes)

quiet latch
#

new packaging specs still come up from time to time and new features get introduced so I'm not sure it can really be in a minimal maintenance mode

#

Other build backends certainly gain popularity but setuptools is not dead

raw dune
#

I don't think setuptools needs to be "dead"

#

And I think it's great to support new packaging specifications and add new features

#

I think these things should be additive though, it's different than removing old functionality.

quiet latch
#

that's understandable, I was mostly referring to the remark on that the project should be in minimal maintenance mode

raw dune
#

I don't think it would be controversial to move new features into another package, but understand that's a big change.

#

(As in, controversial to consumers)

quiet latch
#

setuptools-new, followed by setuptools-newer, and setuptools-newer-than-new-and-newer :)

raw dune
#

I don't think that's fair at all

#

There is no other "legacy" default build backend

#

And, hopefully, in the next year we can normalize the idea of build constraints and locking.

quiet latch
#

are the dependencies that failed to build due to this that old?

#

like, were they actually pre-pep517

raw dune
#

requests does not appear to use PEP 517

#

I don't see a build system declaration

quiet latch
#

welp, that's sad

raw dune
#

I'd love to see that change yeah, but.. it's the reality

robust monolith
quiet latch
#

you can do an upperbound on the setuptools version

#

but in practice you probably won't

#

because it's not very, well, practical

robust monolith
#

But people don't, and there's millions of package versions out there, and old metadata can't be updated on PyPI

quiet latch
#

can you use PIP_CONSTRAINT?

#

for build backend dependency definitions

robust monolith
#

It can, but I'm not sure that having millions of users immediately have to learn build backend constraint systems is better than having build backends adhere to not breaking backwards compatibility in the same backend name , but maybe I'm wrong, as I say, I'm going to kick off this discussion on the Python board

robust monolith
restive talon
#

@raw dune as long as nobody is willing to finance the cost of keep endless backward compat - its somewhere between unfair and evil to expect it of volunteer projects - and so far i dont see anyone willing to shell out what would mak it possible so imho its entirely ok to break things that dont change over time to ensure volunteers dont have to deal with excess mess

robust monolith
#

Then I think there needs to be some innovation in the tooling space, because the current set up, as I outlined in the discuss thread, is that Python packaging ecosystem has been set up in a way that backwards incompatible changes can be too disruptive to bear, and there's not a clear answer to users in terms of "just install the older version"

keen ibex
#

tbf setuptools has a versioning scheme that is pretty hard to use with breaking changes (major bumping all the time not for breaking changes only)

robust monolith
#

I guess, like zanie mentioned in the thread, should users, even pure Python libraries, be encouraged to write ["setuptools>=78.0.2,<79"], if so we should update the packaging documentation, which currently reccomends ["setuptools >= 61.0"]: https://packaging.python.org/en/latest/guides/writing-pyproject-toml/

The objections to this in the past has been that old sdists might not work with new versions of Python, but maybe for setuptools based packages that's okay?

keen ibex
quiet latch
robust monolith
quiet latch
keen ibex
robust monolith
#

Understanding isn't the issue, if one package build depends on setuptools>10 and is broken by setuptools v40+, and another package build depends on setuptools>50, there is no way a user can build them together with current tools

#

So here's a thought, if a backend requirement has a lower bound, what if frontend tools, by default, selected the minimum version of the backend where a wheel was available for that backend?

#

It doesn't solve the problem at large, but it would generally limit breaking changes to when users upgraded their Python and found the setuptools selected no longer works with old packages

#

Not sure if that's better or worse

keen ibex
robust monolith
#

Each package is built in an isolated environment, but the user has no way to constrain the build requirements on a per install basis:

Scenario 1:

  • foo requires build setuptools>10, by default selects newest setuptools and build fails because it is broken by setuptools 40+
  • bar requires build setuptools>50, by default selects newest setuptools and works

Scenario 2:

  • User sets "PIP_CONSTRAINT" with file setuptools<40
  • foo requires build setuptools>10, with constraint setuptools<40 then setuptools 39 is selected and succeeds
  • bar requires build setuptools>50, with constraint setuptools<40 there is a logical incompatibility and build fails
keen ibex
#

that's just a tooling not supporting it, you can always install them in separate steps changing the constraint in between (not the best way, I agree, but still doable)

robust monolith
#

Only if you know the version of foo and bar you want, you might not, you might have complicated install requirements that need to backtrack, and your building the sdists to extract their requirement metadata

#

But yes, that's probably the only solution, under the current tooling and current ecosystem, if build backends introduce large backwards incompatible changes with the ecosystem the expectation is then put on users who deal with sdists to pre-build all the wheels they might need, with their own per-build build constraints as separate commands, and then point their front end tool to that directory of wheels

restive talon
#

sometimes i'm of the impression that it needs 2-5 people to work full time on specs and base tools for a while
a key issue that brings everything to a hold is the pace of only volunteer work - its practically impossible to keep up with times and evolve the ecosystem with the engineering time available for it

and the problem space has easily between 5-10 years of work for a team without even ballooning

bronze berry
#

The latest wheel has un-bundled packaging, presumably because setuptools now has native bdist_wheel support, and we don't need to bootstrap wheel into virtualenvs any more.
This means that we can't bundle wheel 0.46.0 inside virtualenvs or Python's venv anymore, without also bundling packaging.
That lead me to try to clean this up for Debian, I've filed:

GitHub
Stop including 'wheel', setuptools 70.1 has native bdist_wheel supp...

setuptools has native bdist_wheel support these days, so I don't think it's necessary to bundle the wheel library any more. Or am I missing something?
Possibly we could include a no...

GitHub
Remove check for wheel in --no-use-pep517 by stefanor · Pull Reque...

setuptools includes native bdist_wheel support since 70.1.0, we don't need the wheel library.

GitHub
Bump the setuptools in test.wheeldata by stefanor · Pull Request #...

setuptools > 70.1 includes bdist_wheel, so we no longer need to bundle wheel.

#

That last one needs an ack from a setuptools maintainer

bronze berry
#

Actually, let me move the rest of this discussion to #general

median crest
#

In third-party discussion, is it properly stylized "Setuptools", or "SetupTools", or what else?

haughty bramble
humble palm
haughty bramble
#

Someone should pick one and fix them all to that

river oak
#

President is leading the Trend capitalizing random common nouns these days so I don’t see anything need fixing

gray timber
#

The German language capitalises all nouns. Let's do that instead?

robust yarrow
#

since this issue is apparently a very specific feature request directly from setuptools, I'm curious what maintainers think of my alternative idea described here https://github.com/pypa/pyproject-hooks/issues/157#issuecomment-2810109615

Perhaps we can meet in the middle then and say that installers should expose user-configured verbosity to build backends. This way installers don't have to do anything special and the builders can use a standardized enum to influence their output. Like the person before mentioned, builds can be very complex and often call other processes such as when compiling extension modules which can take a long time and users often confuse lack of output with something being wrong/nothing happening.

I think this also solves the warning manifestation because builders can simply output them at the end of the build, or right before exiting with an error.

#

it would require some sort of proposal/PEP of course

bronze berry
#

The wheelesque #!python shebang from https://github.com/pypa/distutils/pull/332 breaks any direct execution of scripts from the build directory before installation. That breaks a lot of Debian packages that run things during build. They'll be fixed when they move to a PEP-517 build, but there's friction involved in getting there...
So, I think I'm going to have to revert that bit in Debian's setuptools for Debian 13 ("trixie").

GitHub
In build_scripts, unconditionally set shebang to #!python. by jarac...

In the past, distutils then Setuptools was a (/the) primary installer for packages. It therefore took on the responsibility of rewriting Python shebangs in scripts.
Later, Setuptools became primari...

quaint jolt
#

are you running the scripts manually?

#

why don't you just call python <script.py>?

#

or fix whatever code is not doing that if it's not debian

bronze berry
#

People run scripts manually from the build directory, yes

bronze berry
bronze berry
bronze berry
#

Build-system code is always very fragile to changes like this, so distros probably get impacted far more than the rest of the community

quaint jolt
#

then I think it's a reasonable request to setuptools to revert the changes

bronze berry
quaint jolt
#

I commented in the issue and the distutils PR, I think this is fixable

bronze berry
#

I hadn't even realized that setup.py install was broken. I assumed it was just the build directory behavior that was different 😛

keen ibex
#

wasn't that deprecated for some years now?

bronze berry
#

Doesn't mean it isn't widely used

#

It was the only way to drive setuptools for decades

median crest
#

not sure I understand... the apparent goal is to build a wheel and then test installation via Setuptools? But won't the end user be installing it with something else anyway?

bronze berry
median crest
#

I guess my question is more just what that pipeline looks like, at a high level

boreal yarrow
#

installed scripts also seem to have a broken shebang now. /me reverts to older version

bronze berry
# median crest I guess my question is more just what that pipeline looks like, at a high level

There are some layers of indirection involved, but you can get an idea of an example here: https://salsa.debian.org/python-team/packages/python-argcomplete/-/blob/master/debian/rules?ref_type=heads#L9-11
The high level is what I described above: Importing libraries out of the build/lib* directory is very common (when there are C extensions built there), and sometimes running scripts out of the build/scripts-* directories.

GitLab
debian/rules · master · Debian Python Team / packages / python-ar...

bash tab completion for argparse

quaint jolt
boreal yarrow
#

But I see it when doing setup.py install too (yeah, I know, deprecated), in the installed files.

compact ingot
#

Does anyone know what's going on with
https://github.com/pypa/setuptools/pull/4901

Based on the discussion, there was talk about figuring out a proper SPDX license expression before removing the classifier.

But it looks like a different commit removed the classifier, and now setuptools has no declared license.

That's a big problem for any pipeline that checks the licenses of all dependencies.

GitHub
Remove deprecated classifier by abravalheri · Pull Request #4901 ...

Summary of changes
A possible additional step is to add a SPDX license expression like license = "MIT" to the [project] table in PyPI.
This PR does not implement this additional s...

compact ingot
#

Is the intent for setuptools to no longer be MIT license? Because since the 19th, it has shown up with no declared license in the metadata on PyPI.

435753574-897ea7dc-3164-4ae5-a991-7c58ec01133f.png
#

It used to look like this:

435753607-54db680f-d34d-44aa-a077-7640ee22d9b3.png
#

This is a really big deal for people who are trying to make sure they're complying with license rules. Because there's no declared license, automated tools see this as being under exclusive copyright:
https://choosealicense.com/no-permission/

Choose a License
No License

You’re under no obligation to choose a license and it’s your right not to include one with your code or project. But please note that opting out of open source licenses doesn’t mean you’re opting out of copyright law.

gleaming mirage
restive talon
#

anyone ware of examples of extending setuptools for alternate pass in of files to the manifest builder and/or introducing generated files to an sdist

lusty dome
#

I'm not aware. My understanding is the files finder protocol is pretty simple - one hook to present all of the files in the sdist. Perhaps there needs to be another hook. In https://github.com/pypa/setuptools/issues/4990, I'm exploring ways to make that hook more flexible. I'm open to proposals for improvements or alternates to file finders. In my estimation, setuptools-scm basically owns that hook, so modulo any compatibility/deprecation concerns, let's make it the hook you want to have.

GitHub
Issues · pypa/setuptools

Official project repository for the Setuptools build system - Issues · pypa/setuptools

restive talon
#

Let's bikeshed about the signature and its backwards compatibility then implement it

fiery raft
#

Is there a way to fix some clang arguments inside build_extension()? I have a problem with how pyenv builds on Mac (https://github.com/pyenv/pyenv/issues/2890), and it results in warnings that I am really tired of seeing. I've poked at all kinds of sysconfig variables, etc, and nothing seems to affect how clang is eventually invoked.

#

I even tried using redirect_stderr() around the build_extension() call, but it doesn't intercept the output, because of subprocess or whatever.

lusty dome
#

@fiery raft - short answer, probably. And definitely if there's something that needs fixing in distutils/setuptools. My suggestion - see if you can isolate the issue to happen with just setuptools, and then file a bug with distutils (or setuptools, but it'll likely get transferred to distutils). Hopefully, it's something I can also recreate on my mac. You also may be able to reproduce the issue and add some breakpoints for troubleshooting by using pip install -e /path/to/setuptools and running pyproject-build --no-isolation to invoke the build.

#

I'm guessing you're not still around PyCon, but if you are, I'm in 317 for a couple of hours today.

fiery raft
#

Thanks, I am not at PyCon. The whole rabbit hole of setuptools and distutils is new to me, so I'm not sure I have the energy to keep debugging through it. The problem starts with pyenv which puts the duplicates in LDFLAGS to begin with, so any fix elsewhere is kind of a hack.

gleaming wedge
#

Hi all, what would be the current recommended way of opting in to abi3 wheels dynamically when building with setuptools? Currently I've found the following ways of building correctly tagged abi3 wheels:
• Setting py_limited_api in [bdist_wheel] in setup.cfg files,
• Setting py-limited-api in [tool.distutils.bdist_wheel] in pyproject.toml files,
• Subclassing bdist_wheel in setup.py (e.g. https://github.com/joerick/python-abi3-package-sample/blob/main/setup.py),
• Using DIST_EXTRA_CONFIG similarly as described in https://github.com/pypa/setuptools/issues/4129 but to set py_limited_api in [bdist_wheel]
The first two don't allow the tag to really be set dynamically without editing the configuration files. The third is not ideal for projects that are trying to move away from using setup.py files, so it looks like the last one might be the best way for these projects?

gleaming mirage
#

@lusty dome I saw that in https://github.com/pypa/setuptools/issues/3085#issuecomment-2899334790 you said that it wouldn't be worth trying to reclaim the pkg_resources name on PyPI. Why do you think that? Seems like the setuptools project would have a strong case for claiming that name.

GitHub
Setuptools should not rely on pkg_resources · Issue #3085 · pypa/...

This issue basically tracks the root goal of obviating pkg_resources, in particular that Setuptools depends on it. Over the past few years, myself and others have worked hard to create importlib.re...

keen ibex
#

Imight be wrong, I think it is because pkg_resources is deemed deprecated and its functionalities were replaced by other packages (importlib.metadata, importlib.resources and packaging)

gleaming mirage
#

I know, but it would still be useful to have a fallback.

lusty dome
#

I’m sure I could get that name… it’s more about the long term maintenance burden versus the value provided. IIRC, there’s at least one case that might use it. If there’s more than one or that case is compelling, I’ll consider it… and only if there’s not a suitable workaround (pin Setuptools).

median crest
#

I noticed something a bit concerning: with Setuptools versions before 77 (with the experimental PEP 639 support), it would apparently happily put License-File: entries in core metadata. Which packaging complains about, because the metadata version isn't 2.4
That implies there are a bunch of subtly broken wheels up on PyPI (including for example the latest version of imageio and probably several before that) and that makes it harder for non-Pip installers in Python to parse metadata

#

... ah, apparently I can just disable the validation there. But still...

restive talon
#

i'd appreciate a hint for better testing on https://github.com/pypa/setuptools/pull/5056/files - id like to avoid introducing a new hook, in particular since i need to keep the old ones in place as well for setuptools-scm

GitHub
Add optional Distribution parameter to walk_revctrl file finder by ...

i made a first draft using claude - the tests need some simplification for the unit part and im looking for suggestions on a integration test thats not excessive

File finder entry points for SCM-b...

versed lily
#

how can I fix this ? ```Unknown distribution option: 'console'
warnings.warn(msg)
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help

error: no commands supplied

keen ibex
#

Where did you start? How did you get to that point?

versed lily
#

I typed python setup.py at that time

#

this is the current log

keen ibex
versed lily
#

the package installed successfully