#build

1 messages Β· Page 1 of 1 (latest)

warm lance
#

Is build supposed to enforce pep 621? Or is it the responsibility of the build backend?

warm mulch
#

It is the backend’s responsibility

warm lance
#

Seems a bit trusting

warm mulch
#

That description applies to a lot more than just PEP 621 πŸ˜›

tawdry nimbus
#

Backends are required to validate the table, e.g. via a JSON schema, which is what I believe e.g. Setuptools does. Maybe it would be better if a canonical one was provided in the spec, as there was an initial attempt for with the original PEP 518.

mighty yew
#

the JSON schema for it would be very simple, so I think someone could simply open a PR against the PEP to add that

arctic briar
#

Not the PEP -- that does a good-enough job.

#

And pyproject-validate or something like that, by @worn token (if I remember correctly) does exactly that.

#

Close enough. :P

#

(also, yay Furo!)

#

(and booo non-hidden toctree on landing page)

karmic steppe
worn token
arctic briar
#

That's what the left sidebar is for! 😝

worn token
#

Huha, I might be weird. Maybe someday I will reconsider it (although I really like the fully expanded ToC in the index page)

#

(Thank you again for the amazing theme, @pradyunsg)

arctic briar
#

^>^

worn token
mighty yew
#

wondering because it seemingly solves the same issue

#

just no CLI I mean

tidal bramble
#

Any thoughts on using the new pip zipapp as an alternative to ensurepip for isolated envs?

arctic briar
#

You can use pip with --python as well, now.

#

And, that's probably a better solution than the zipapp for build; since users would get the same copy in the build environment as the environment from build.

#

That said, why? :P

tidal bramble
#

Well, --python's not been released yet so it'll be some time before it finds its way into distros and Python/ensurepip, and then we can't assume that pip's available in build's environment, or if it is that it isn't ancient, or that ensurepip works at all

#

Caching the zipapp would also save time from boostraping isolating envs

arctic briar
#

Well, the zipapp isn't supported and might change names.

tidal bramble
#

I thought the plan was to release the zipapp for 22.3?

arctic briar
#

... which is also when you'll get the --python flag. :)

#

And, unlike the --python flag, zipapp is going to be treated as experimental.

tidal bramble
#

right, but to use the --python flag you need to have pip 22.3. We can't assume that pip is installed or that it's in a usable state, so we've got to provision it some other way - with ensurepip, or by downloading and unpacking the wheel, or by using the zipapp, or by taking virtualenv on as a dependency.

#

With ensurepip we'll invariably want to upgrade pip too

severe tundra
bronze aurora
#

Approved but seems I don't have right to merge because the PR modifies the CI πŸ€·β€β™‚οΈ

#

@mighty yew perhaps you could make Henry also a maintainer on PyPI πŸ€·β€β™‚οΈ or we should enable to automatically publish on release creation

#

Just to reduce the bus factor a bit

mighty yew
#

I agree, I just need to document the release process

violet pebble
#

I'd recommend adding an extra person to PyPI, documenting the release checklist (RELEASING.md is often used for that), and automating releases πŸ™‚

mighty yew
#

I may be being too hard-headed but I don't want to use github actions to publish artifacts as it is not a secure environment by design

mighty yew
#

released

#

I am flying home in a bit, I will try to document the release process in the weekend

severe tundra
#

Thanks! I'm not sure what "secure environment by design" means. GHA does a tremendous amount of work to be secure, and PyPI is currently working on sigstore support so GHA would be able to authenticate directly with Actions. There can be holes, of course, but it is very much designed to be secure. You can lock down pretty much every element of releases, require them to be signed, etc. (Of course, remember I'm on cibuildwheel which really often is used for releases from CI πŸ™‚ ) Happy to follow whatever procedure you want, I don't really care, but just want to clarify that GHA is designed to be secure and to be used for releases, deployments, etc - it's a selling point of the product.

undone coral
#

I'm also curious what 'not a secure environment by design' means. For reference, @severe tundra is talking about the first part of this work: https://github.com/pypi/warehouse/issues/12465 (which, for the record, has nothing to do with Sigstore πŸ™‚ )

GitHub
[META] OIDC federation roadmap for Warehouse Β· Issue #12465 Β· pypi/...

This is meta-issue, tracking different strands of development/design for Warehouse's use of OIDC. Goals There are two high-level goals for Warehouse's OIDC work: Enable "OI...

onyx coral
#

secure isn't a binary property, but I'd not feel bad about having GitHub Action release stuff via the OIDC work

undone coral
tidal bramble
mighty yew
#

it's on my TODO list for this weekend

#

might also get some time at work go do it

#

need to check that

fallow prairie
#

Is there any way to invoke build (or setuptools directly, if need be) to report the MANIFEST built from the current MANIFEST.in, without going through the whole build process?It's annoying having to wait for the virtualenv creation every time. I can run with --sdist --no-isolation, but if there's a "more proper" way to do it I'd rather use that. Don't have to mess around with clearing dist/ afterwards.

Thanks!

karmic steppe
#

MANIFEST isn't part of any spec so build wouldn't really be able to help here

karmic steppe
#

I think MANIFEST (without .in) isn't even generated as a physical file in setuptools, is it? There's a setuptools-specific .egg-info/SOURCES.txt in sdist (though I imagine it isn't necessarily equivalent to everything that's matched by MANIFEST.in) and other build backends may not even generate any file listing all files in the sdist at all. In the case of wheel, there's .dist-info/RECORD though.

Anyhow, that doesn't really matter, getting this kind of information is backend-specific, same as the actual MANIFEST.in file so you might have better luck by reasking in #setuptools directly :)

fallow prairie
#

That's what I thought overall, backend-specific sort of thing, and as such it wouldn't make sense for build to have that pass-through. Will inquire there, thanks!

severe tundra
#

I think we are ready for a Flit-based release. I think I can do it early next week if it falls on me. Thoughts? Okay?

mighty yew
arctic briar
#

4 people other than you.

tidal bramble
#

perhaps the next release should be v1

violet pebble
#

7 million downloads per month agree πŸ™‚

#

or hop into calver instead, like quite a few other PyPA projects

arctic briar
#

This is a library with an API, so depends on whether y'all wanna get into the debate on breaking vs not breaking.

mighty yew
#

I need to look carefully at the API to see if there's anything we need to fix there

karmic steppe
#

It seems like that page could be reordered

tidal bramble
#

PR welcome

tidal bramble
#

Thanks @crude plaza :)

thin crypt
#

Drive-by thought so I don't end up forgetting it: has there been anyone who's looked at embedding build reproducibility information (ie SOURCE_DATE_EPOCH) into sdists so that downstream wheels being built from those sdists can use the same setting as the likely accompanying wheel on PyPI?

warm mulch
#

People had asked for it, not sure if anyone actually worked on it

mighty yew
#

no, but that's a good idea, it's tricky though

sleek berry
#

Actually getting reproducible is probably more than just that, though. Probably requires pinned versions of tooling etc.

thin crypt
#

Are builds across different versions of python-build guaranteed to be reproducible?

#

(given same inputs+environment of course)

onyx coral
#

I don't think python-build has much control over it

#

that's mostly the build backend

arctic briar
thin crypt
#

That's pretty cool! Thanks πŸ™‚

arctic briar
#

IIUC, He talked about it this week in a Stockholm Python meetup, so there's at least one more person who cares. :P

dry mural
# thin crypt Drive-by thought so I don't end up forgetting it: has there been anyone who's lo...

not exactly your case, but someone contributed a PR to maturin that allows SOURCE_DATE_EPOCH=0 to get 1980-01-01 0:00 UTC https://github.com/PyO3/maturin/pull/1334

GitHub
Support SOURCE_DATE_EPOCH environment variable in wheel building by...

This adds support for the SOURCE_DATE_EPOCH environment variable like wheel does when building wheels. If this environment variable is set, the mtime of the files in the wheel are set to this value...

lean tundra
#

These strategies are pretty common in Bazel (and Nix I assume)

#

same with redefining __DATE__, __TIME__ and other non-reproducible built-ins when building C code

severe tundra
#

Latest version of clang (16 IIRC, whatever just came out) actually sets __DATE__ and __TIME__ when SOURCE_DATE_EPOCH is set.

mighty yew
#

didn't older versions also use to do that?

#

I know gcc does

severe tundra
hushed flicker
#

What's the release cadence of build?

severe tundra
#

I still need to get set up as a releaser. It's still on my todo list! 😊

hushed flicker
mighty yew
#

yeah, we can make a release I think

severe tundra
#

0.11? I can try to get set up. I just need to make sure I put the correct public part of the key into https://github.com/pypa/build/issues/561 - I'll try to do this later today or at least by early next week. (Don't wait on a release for me, though, if you want to do one sooner)

GitHub
Onboard maintainers to the release process Β· Issue #561 Β· pypa/build

Our release process requires that maintainers have their PGP key listed in https://pypa-build.readthedocs.io/en/stable/installation.html, and preferably have their key signed by one of the other ma...

mint briar
#

Is there anyway to tell build to install all the build dependencies before starting the build? In my case I am trying to build wheels from extracted source tarballs.

#

oh, I think I already solved this.

#

I need more coffee before touching these.

#

@thin crypt I am trying to see what all things it will take to create a system which can build reproducible wheels for possible linux distributions/versions combinations and then provide in a special index.

thin crypt
mighty yew
#

I am very interested in this too!

#

@mint briar I am sure you already know about this right?

#

I may have an extra sticker that one of the people behind the project gave me at FOSDEM

#

will give it to you at PyCon if I can find it

arctic briar
#

:)

mighty yew
#

ah πŸ˜…

arctic briar
thin crypt
#

reproducible-builds

jaunty matrix
lean tundra
jaunty matrix
#

I don't think so, the closest I could find to current situation is the following text from PEP 625:

Code that processes sdist files MAY determine the distribution name and version by simply parsing the filename, and is not required to verify that information by generating or reading the metadata from the sdist contents.

so the behavior is allowed technically but I would say is undesirable

#

I added the ability to prevent normalized names due to a real-world use case that I'm struggling to remember

#

oh I remember! it was Bloomberg asking because they use Artifactory which somehow uses dots for namespacing

lean tundra
#

I guess that just describes the observed standard, but doesn't make any mandates. And the docs for builf_sdist specify what the directory name should be, but seems to stop short of mandating an archive name.

#

Seems to me that the archive/directory relationship is heavily implied in these docs but not actually specified either way. Would be nice if something explicitly said that they don't need to match if that is indeed the case

onyx coral
#

"what should things be named" is in a bad state right now

jaunty matrix
arctic briar
jaunty matrix
arctic briar
#

I don't think there's a Bloomberg position on this topic at the moment. It's Bernat's position.

jaunty matrix
#

does the Python infra no longer require dots to indicate namespaces?

#

iirc it's an Artifactory thing

jaunty matrix
arctic briar
#

I think the problem he's concerned about is something that Bloomberg's internal infrastructure teams needed to solve, and design an answer for.

onyx coral
#

I still think the current state of things is broken fwiw πŸ˜›

arctic briar
#

And, I think 625 needs to start getting enforced on upload by PyPI.

onyx coral
#

I'm -1 on PyPI changing it's enforcement of filenames until the current state of things gets resolved

arctic briar
#

What's left?

jaunty matrix
arctic briar
#

Is it that PEP 625 isn't respected by all backends? What do you say is left/broken?

onyx coral
#

The fact that the spec was changed in a backwards incompatible way outside of the PEP process, without a plan to deal with that backwards incompatibility

jaunty matrix
#

if ^ issue was resolved I wouldn't have had to release that change today

arctic briar
#

I'm not aware of what you're referring to here TBH.

onyx coral
#

this thread has the background

arctic briar
#

Wait, I thought we have agreement that we're normalising all distribution names blanket?

#

Do people not want that?

onyx coral
#

The original spec did not do that

#

that was added without discussion when PEP 427 ported over to packaging.p.o

#

nobody can actually do that currently in all cases, because PyPI will reject the upload (I think mostly around .)

#

some tools normalize and can release to PyPI

#

er cannot*

#

some tools do not normalize, and can release to PyPI

arctic briar
#

Huh, wow.

onyx coral
#

I'm not personally willing to break the projects that currently work, to implement an undiscussed backwards compatible change that was done without a plan to how to deal with the fallout

#

I don't really care what happens, but either:

  • The backwards incompatible change should be explicitly decided on, and we should figure out how we plan to cope with it
  • The backwards incompatible change should be backed out.
arctic briar
#

Can we make PyPI normalise names, and allow those uploads as well?

jaunty matrix
#

the issue I linked has the most context about that

onyx coral
#

I'm -1 on making any changes to how PyPI handles filenames until the community has an agreed on path forward, because anything we change is likely going to just make the situation more complicated

arctic briar
#

Ohkie! I guess I have a lot of reading to do then!

onyx coral
#

and the situation is already very messy

#

with various bits and bobs doing different things

#

IIRC Warehouse interprets things one way, setuptools another way, twine another way, flit another way, and most of the other backends another way

arctic briar
#

OK, so before I dive into reading and research, my current understanding is that the spec doesn't match all implementations, the spec does not match what was discussed and that there's no concensus on what the spec and intended behaviour even is.

Is that correct?

onyx coral
#

The current spec at packaging.python.org more or less clearly states that filenames are normalized (it uses the english word should rather than RFC MUST, but I don't think anyone interprets it as RFC SHOULD).

#

Pretty much everything made after that, does normalization as you expected

#

everything made before that... does things a little differently πŸ˜‰

arctic briar
#

And, we don't have agreement to make that change in things made before that?

#

(please don't feel obliged to elaborate on how I'm wrong, if I'm wrong -- that's a "Pradyun, you need to read the links we sent you instead of asking more questions here", and just say that)

onyx coral
#

I don't know if have agreement or not TBH, the maintainers of those older tools never really weighed in

#

other than myself IIRC

arctic briar
#

LOL OK

onyx coral
#

which I don't care what the final rule is, just that it's clear and we know how we're getting from point a to point b

#

IIRC setuptools doesn't do any normalization to filenames, it just escapes "unsafe" characters

#

I think twine does the same, but with slightly different escaping rules

#

Warehouse implements a "check if filename matches what we think it should be", but that check is wrong under both the old and the new way afaik

arctic briar
#

Hehe, OK. Thanks for patiently answering my questions. That's all useful context to have going into reading these discussions later.

onyx coral
#

IIRC the history too

karmic steppe
onyx coral
#

the original wheel spec was inconsistent

hollow basin
#

so, I guess "python packaging is a mess" is still valid after all 🀣

onyx coral
#

the text of the spec mandated some really strict rules that were actually broken

#

and wouldnt' work in reality

#

but the example code implemented something closer to setuptools "escape unsafe characters" rule

#

so warehouse/setuptools/wheel/etc all implemented the "escape unsafe characters" rule

arctic briar
#

FWIW, my understanding is that this is basically the packaging.tags problem but with package names.

#

There's too many implementations flying around, for doing the same set of things.

onyx coral
#

then when the spec was ported over to packaging.python.org, some things were "cleaned up", with little to no discussion, and one of those things were requiring strict normalization of the filenames, rather than the escape unsafe characters rule.

#

then later projects were created and followed the spec on packaging.python.org, but of course since there was no PEP/discussion none of the older tools noticed the change

#

so they just kept on doing what they were doing

#

and the world just kind of existed like that, because tools like pip etc were permissive in what they accepted so both rules worked

#

but Warehouse was not intending to be permissive, but the check was wrong in edge cases, and when trying to decide how to fix it, we could either fix to to support the old rule or the new rule, and in either case I think we would break one side or the other... or we could become permissive? but it's unclear if that is ok (e.g. should installers have to be permissive? or should they be allowed to assume strict-ness)

#

then other random tools like Artifactory wanted to use . in the name to mean certain things

#

Probably the right answer is to make a PEP that defines project names, like what PEP 440 did for version numbers, defines normalization, equivalence, etc, and sets how to represent those names in filenames and such.

arctic briar
#

OK, I wasn't sure if it needs to be a PEP but looking through the text, it might be the best angle for this.

warm lance
#

How do you set the build-tag for a wheel with build? Or do you just rename the .whl after building?

manic cave
#

You can't just rename, the build tag must also be listed in the WHEEL file.

#

I think the answer is going to be backend-specific, as build itself doesn't create wheel files.

violet pebble
warm lance
#

What reads the WHEEL file? I thought everything went off the filename

warm mulch
#

pip reads WHEEL. The wheel metadata format version is important, and the tag list is easier to interpret than the file name (which can compress tags)

severe tundra
#

You need to use wheel 0.40.0's new wheel tags command to change the tags (and rename the wheel) after the fact.

#

That will work with any backend. Otherwise, yes, it's backend specific.

#

wheel tags --build=1 --remove Pillow-9.4.0-pp39-pypy39_pp73-macosx_10_10_x86_64.whl would have renamed the wheel and correctly set the metadata too. (without --remove it would make a copy)

strange sage
#

(I'm not sure this belongs here of course but...)

#

Anyone know where files like tmpd9422c45cacert.pem are coming from and why they end up in the --outdir of a python -m build?

#

Obviously that's a certificate chain, but I don't know how/why they are appearing, I'm literally only running python -m build

#

(I mean I'm running other stuff, but I don't think anything else that should be outputting into there)

hollow basin
#

Are you using any custom package sources?

strange sage
#

What's a package source?

hollow basin
#

Like PyPI

strange sage
#

Ah a custom index -- nope

tidal bramble
#

it's pip's bundled copy of certifi extracting its certificate chain using importlib.resources which uses tempfile under the hood. But I don't know why it'd appear in outdir, unless your outdir is also your tmpdir...

#

right, you are using nox's create_tmp which sets TMPDIR

#

this isn't really what create_tmp is for, if you want a a folder you can throw away at the end of the run, just use tempfile. The create_tmp folder is persisted!

strange sage
#

Thanks for the info!

hushed flicker
severe tundra
hushed flicker
#

thanks for the update πŸ‘

severe tundra
#

I'd like to try for a 0.11 release soon. I'm off W-F, so either tomorrow or this weekend / early next week.

tidal bramble
#

do we wanna do a pre-release to test the new pipelines etc?

severe tundra
#

The only new part of the pipeline is that the SDist/wheel is available for download as an artifact. It still is a manual release process.

#

1.0.0 seems to be out πŸ™‚

thin crypt
#

Congrats on 1.0.0 folks!!! πŸŽ‰ So appreciative of all the hard work that goes into maintaining this tool πŸ‘πŸ‘πŸ‘

arctic briar
warm lance
#

V1 very exciting

hollow basin
#

what about docs? website still points to 0.10.0

hollow basin
severe tundra
#

Only @mighty yew has access to rtd. I’d be happy to but don’t have access to it.

mighty yew
#

sorry for the delay, I'll add you later when I am at the computer

ashen salmon
#

Hi, I am encountering a strange issue when running python -m build and the repo has symlinks:

tarfile.LinkOutsideDestinationError: 'spglib-2.1.0rc3/test/package/FetchContent/c/main.c' would link to '/main.c', which is outside the destination
#

It seems to be related to tarfile.py (which is on cython side?):

        if member.islnk() or member.issym():
            if os.path.isabs(member.linkname):
                raise AbsoluteLinkError(member)
            target_path = os.path.realpath(os.path.join(dest_path, member.linkname))
            if os.path.commonpath([target_path, dest_path]) != dest_path:
                raise LinkOutsideDestinationError(member, target_path)

dest_path there should point to target_path = os.path.realpath(os.path.join(dest_path, name)) instead of dest_path

violet pebble
#

looks like that hasn't been released in 3.10 yet, but it has been in 3.11.

can you try adding something like this to https://github.com/spglib/spglib/pull/321 ? otherwise it's defaulting to 3.10:

        uses: actions/setup-python@v4
        with:
          python-version: 3.x
ashen salmon
#

Oh, does it take the minor from the OS installed?

tired knoll
violet pebble
#

yep, 3.x is now 3.11. it'll be bumped to 3.12 some after 3.12 is out, but they generally don't do it immediately. so it's like "a recent stable"

ashen salmon
#

Oh, so if I specify literally as 3.x that would take the latest? I was also asking about if there is no python-version, would it pull in the latest minor or jost the current version

#

Also, 3.11.5 does not seem to be used on setup-python: https://github.com/actions/setup-python/issues/725, and #108209 does not seem to be released for python 3.11 either https://github.com/python/cpython/compare/v3.11.5...3.11

GitHub
Issues Β· actions/setup-python

Set up your GitHub Actions workflow with a specific version of Python - Issues Β· actions/setup-python

GitHub
Comparing v3.11.5...3.11 Β· python/cpython

The Python programming language. Contribute to python/cpython development by creating an account on GitHub.

violet pebble
#

Oh, so if I specify literally as 3.x that would take the latest?

not necessarily: when 3.12 comes out, 3.x will still point to 3.11 for a while

tired knoll
ashen salmon
violet pebble
#

not with actions/setup-python. they do have things like 3.12-dev, but they always point to tagged pre-release versions

tired knoll
ashen salmon
violet pebble
#

you can however use deadsnakes for the curring edge

violet pebble
tired knoll
ashen salmon
#

I was hoping for a 3.x-dev syntax so there wouldn't need to update the action, but I guess I should be deprecating old python versions anyway

ashen salmon
tired knoll
ashen salmon
#

That would work with 3.x-dev, not just 3.12-dev?

tired knoll
#

It allows the action to pick pre-release versions

tired knoll
ashen salmon
#

But would it also move 3.x from 3.11 to 3.12?

tired knoll
ashen salmon
#

Tagental, but pypa does not use tj-actions/changed-files right? It seems more actively maintained, but I'm encountering some bugs with it. Wondering if you guys had some experience with it

tired knoll
#

All it does it make pre-release versions available for picking

#

There's not a pre-release called 3.12-dev

violet pebble
ashen salmon
tired knoll
violet pebble
tired knoll
violet pebble
tired knoll
#

Huh
What version does that give you?

#

Is it the same functionality both ways?

violet pebble
#

yes, both give Python 3.12.0rc1, the latest 3.12 pre-release

tired knoll
violet pebble
tired knoll
#

Ohh
I get what they're saying now

severe tundra
#

The recent verions of setup-python also read pyproject.toml too. Though IIRC you have to manually set it as the version-file.

#

Next build release (1.0.1, out soon) will avoid enabling data_filter on buggy Python versions (3.8.17, 3.9.17, 3.10.12, and 3.11.4)

severe tundra
#

PyPI is claiming there was already a build-1.0.1-py3-none-any.whl. Is there a way to see previously used names?

#

I don't see it in the security history

#

I'm guessing there was an old "build" project that was deleted, and it had a 1.0.1. Surprised it would be python3 only, though. I'd like to see a list of the "taken" filenames.

plush moss
#

I'm seeing an issue where my setup script generates Python files while building native extensions, but when building with build these files don't end up in the final package. Is that by design?

hollow basin
#

sounds like an issue with your build backend

#

build is just calling build backend api in isolated environment

plush moss
#

thanks, so I should check with setuptools πŸ™‚

severe tundra
#

I would second the mention in #setuptools, I bet you are missing files in your MANIFEST.in, and build (by default) makes your wheel from your sdist, so it's probably missing in the sdist.

severe tundra
#

I'm thinking we could probably release 1.1 soon, with the external pip speedup , no "wheel" injection, and importlib_metadata fix. That would be the final 3.7 supporting release.

tidal bramble
#

looks we'd omitted the "v" from the git tag in the past

severe tundra
#

Oops, didn't notice that.

severe tundra
#

Another perf idea: what about using --no-compile (pip backend)? This would only compile the parts needed to build, rather than everything. It’s really only for interactive usage. uv does this by default.

tidal bramble
#

unrelated to the above, I'd still really like us to print out the built dists in a machine readable format. I'm annoyed that I have to glob the dist folder every time or construct my own manifest that I can refer to later

tidal bramble
#

are we ok to add --env-impl for uv or would we prefer to do something different?

hollow basin
#

Why? Why would that matter?

tidal bramble
#

why would it matter if there was an env impl option, if it's called --env-impl, if there's support for uv at all?

hollow basin
#

The option

#

Do it like with virtualenv, if installed, use, else fallback to venv

#

Uv > virtualenv > venv

tidal bramble
#

I don't feel comfortable doing that personally, changing the default installer has more serious consequences than switching to a functionally identical venv creator. We'd also probably be forced to limit looking for uv under sys.prefix or provide an opt out because unlike virtualenv which needs to be co-located with build, uv can be anywhere on $PATH

hollow basin
#

well, I meant for envs only, not for installation

tidal bramble
#

oh, we're not using uv venv if that's what you mean

#

sorry for the confusion

rain prism
#

So --env-impl controls the implementation used to install things into the build env? not creation of the env?

hollow basin
tidal bramble
#

yes, the only acceptable value is venv+uv

severe tundra
#

Why not virtualenv-pip and venv-uv?

#

Also, I think virtualenv has been more reliable for us; a common β€œsolution” when people come to us with β€œcan’t create venv” problems is to tell them to install virtualenv. Why is the default being changed to not do that in some cases? Fast is good but working is better.

#

(If I read the comments correctly)

tidal bramble
#

the default is closer to venv-or-virtualenv+pip which is a bit of a mouthful and unless we wanna let people choose between venv and virtualenv, I don't see any reason to expose it as an option

tidal bramble
#

but I'm happy to revert this change and we can do it separately

severe tundra
#

Ah, okay.

#

Given the installer affects the env, but can be mixed and matched, and we aren’t using this to allow a user to pick between venv and virtualenv anyway, why not just β€”installer=uv / β€”installer=pip?

tidal bramble
#

yeah, I guess we don't need to draw attention to the venv implementation

#

or we might decide to use something other than venv with uv in the future

severe tundra
#

We could always add an extra option like --venv=venv --installer=uv in the future, but we can automatically chose --venv for now and only expose it if needed in the future.

tidal bramble
#

renamed to installer with the default being pip

rain prism
#

thanks! that makes sense to me

#

Might be kind of silly to --venv=uv --installer=uv someday but πŸ€·β€β™€οΈ

severe tundra
#

Venv could be automatic based on the installer if not specified. πŸ˜‰

tidal bramble
#

My wish would be to only ever have to use venv, someday

severe tundra
#

build 1.1: 4.2-3.2-ish seconds. build main: 3.2-2.9 seconds (venv, pip), 1.7-1.4 seconds (venv, uv).

#

I wonder if we shoud indicate somewhere if we are using pip from build's environment.

#

Testing this on building scikit-build-core, which uses a few more hatchling plugins, I get 16.6 seconds (1.0.3), 9.2 seconds (1.1.1), 7.8s (main), and 4 seconds (main with uv)

tidal bramble
#

~1s d between 1.1 and main with pip is surprising

severe tundra
#

It varies a bit, I’m doing very loose comparisons.

severe tundra
severe tundra
#

PyPI has turned off uploads. If they come back online, the artifacts can be downloaded from GHA and uploaded to PyPI. I'll check again midday tomorrow.

severe tundra
#

build 1.2.1 is out.

rain prism
#

Realizing this is a more appropriate channel, since I reference build πŸ™‚ #packaging message

severe tundra
severe tundra
#

#pyproject-hooks 1.2 is the thing that broke us. That was not easy to track down...

hollow basin
#

can build build wheel from an sdist? does it require special flags to do so or is it smart enough to figure that out with only --wheel?

violet pebble
#

it's the default: Building wheel from sdist

$ python -m build
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - hatch-vcs
  - hatchling
* Getting build dependencies for sdist...
* Building sdist...
* Building wheel from sdist
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - hatch-vcs
  - hatchling
* Getting build dependencies for wheel...
* Building wheel...
Successfully built norwegianblue-0.20.1.dev5.tar.gz and norwegianblue-0.20.1.dev5-py3-none-any.whl
hollow basin
#

yeah, but what I wanted it to pass the sdist I already have and build a wheel

#

pip wheel --no-deps ./<tar archive>

mighty yew
severe tundra
#

I believe we’ve generally planned to allow it one step, but it hasn’t happened yet, since it’s so easy to do manually. (I could be wrong, though) Pip and cibuildwheel allow building directly from SDist. I think we would still have to manually untar in cibuildwheel, though, since we need to read the pyproject.toml, even if build allowed it.

hollow basin
open kayak
#

Hm, if the Python build system would work using a filesystem abstraction instead of being hardcoded to use the actual filesystem, we could use the tar file as file system.

hasty swift
#

It wouldn't be very performant though, tar files don't support random access like zip files

grand gust
raw belfry
#

I can see the following:

d----- 3/4/2025 3:52 PM walacor_python_sdk-0.0.2.dist-info

# 
+---d_python-sdk
Βͺ   Βͺ   .coverage
Βͺ   Βͺ   .gitignore
Βͺ   Βͺ   .pre-commit-config.yaml
Βͺ   Βͺ   CHANGELOG.md
Βͺ   Βͺ   LICENSE
Βͺ   Βͺ   pyproject.toml
Βͺ   Βͺ   README.md
Βͺ   Βͺ   requirements-dev.txt
Βͺ   Βͺ   tox.ini
Βͺ   Βͺ   
Βͺ   +---.github
Βͺ   Βͺ   +---workflows
Βͺ   Βͺ           ci.yaml
Βͺ   Βͺ           pr-checks.yaml
Βͺ   Βͺ           release-main.yaml
Βͺ   Βͺ           release-test.yaml
Βͺ   Βͺ           release.yaml
Βͺ   Βͺ           
Βͺ   +---docs
Βͺ   Βͺ       github.ref
Βͺ   Βͺ       versionhistory.rst
Βͺ   Βͺ       
Βͺ   +---scripts
Βͺ   +---tests
Βͺ   Βͺ   Βͺ   test_basic.py
Βͺ   Βͺ   Βͺ   __init__.py
Βͺ   Βͺ   Βͺ       
Βͺ   +---venv
ΒͺΒͺ   +---walacor_python_sdk
Βͺ       Βͺ   main.py
Βͺ       Βͺ   _version.py
Βͺ       Βͺ   __init__.py
Βͺ       Βͺ
severe tundra
#

The wheel is empty. Since you are using setuptools it's probably not finding the package.

# 
unzip -l dist/*.whl
Archive:  dist/walacor_python_sdk-0.0.2.dev2-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
    11357  03-04-2025 15:00   walacor_python_sdk-0.0.2.dev2.dist-info/LICENSE
     1252  03-04-2025 15:00   walacor_python_sdk-0.0.2.dev2.dist-info/METADATA
       91  03-04-2025 15:00   walacor_python_sdk-0.0.2.dev2.dist-info/WHEEL
        1  03-04-2025 15:00   walacor_python_sdk-0.0.2.dev2.dist-info/top_level.txt
      466  03-04-2025 15:00   walacor_python_sdk-0.0.2.dev2.dist-info/RECORD
---------                     -------
    13167                     5 files
#

You have "where" pointing to inside the package

#

It should point to where the package is.

#

(Or you can use hatchling, which tends to work out-of-the-box)

tired knoll
#

uv* ferrisWink

severe tundra
#

It works if I delete those lines

# 
diff --git a/pyproject.toml b/pyproject.toml
index 365463b..54fd407 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -54,9 +54,6 @@ Documentation = "https://apidoc.walacor.com"
 Homepage = "https://www.walacor.com"
 [project.scripts]

-[tool.setuptools.packages]
-find = { where = ["walacor_sdk"] }
-
 [tool.setuptools_scm]
 version_scheme = "python-simplified-semver"
 local_scheme = "no-local-version"
raw belfry
#

ohh let me try it πŸ™‚

tired knoll
#

Oh there's a repo?

raw belfry
#

I am deploying it in the following way, do i need to change something here

jobs:
  build:
    name: Build the source tarball and the wheel
    runs-on: ubuntu-latest
    environment: release
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.x"

      - name: Install dependencies
        run: pip install build

      - name: Create packages
        run: python -m build

      - name: Verify build output
        run: ls -l dist/ || exit 1

      - name: Archive packages
        uses: actions/upload-artifact@v4
        with: 
          name: python-dist
          path: dist

  publish:
    name: Publish build artifacts
    needs: build
    runs-on: ubuntu-latest
    environment: release
    permissions:
      id-token: write
    steps:
      - name: Retrieve packages
        uses: actions/download-artifact@v4
        with:
          name: python-dist
          path: dist
      
      - name: Verify downloaded packages
        run: ls -l dist/ || exit 1

      - name: Publish to PyPI (OIDC)
        if: "! contains(github.ref_name, 'rc')"
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          repository-url: "https://upload.pypi.org/legacy/"

      - name: Publish to Test PyPI (OIDC)
        if: contains(github.ref_name, 'rc')
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          repository-url: "https://test.pypi.org/legacy/"
raw belfry
severe tundra
#

No, though you can just do pipx run build instead of installing then running build. Doesn't really matter though.

raw belfry
#

I am very new to packaging and writing CI/CD pipelines and not sure if i am even doing things correctly!

tired knoll
severe tundra
#

I like that action too. πŸ™‚

raw belfry
#

I really appreciate it guys,
let me remove those two lines and try again!

#

just off topic,
did you ever tried some workflows so many times that at the end of the day you had like 40+ commits or something ?! xD

tired knoll
#

That's what branches are for
Squash when merging

severe tundra
#

Yeah, I keep force pushing to a branch, or squash and merge when you are done. I've gone over 100 tries to get something working in CI in the past. πŸ˜‰

raw belfry
severe tundra
#

It was declared incorrectly (if you used src/walacor_sdk/__init__.py, then it would be where = "src", for example). But no, using pyproject.toml config it looks in standard places (like the main directory and the src directory) automatically.

raw belfry
severe tundra
#

Did you try to upload the same version number?

raw belfry
#

nop

severe tundra
#

There should have been more info there on why it failed.

raw belfry
#

I tried git push origin 0.0.4rc1 and git push origin 0.0.5rc1

image.png
# 
INFO     Response from https://test.pypi.org/legacy/:                           
         400 Bad Request                                                        
INFO     <html>                                                                 
          <head>                                                                
           <title>400 This filename has already been used, use a different      
         version. See https://test.pypi.org/help/#file-name-reuse for more      
         information.</title>                                                   
          </head>                                                               
          <body>                                                                
           <h1>400 This filename has already been used, use a different version.
         See https://test.pypi.org/help/#file-name-reuse for more               
         information.</h1>                                                      
           The server could not comply with the request since it is either      
         malformed or otherwise incorrect.<br/><br/>                            
         This filename has already been used, use a different version. See      
         https://test.pypi.org/help/#file-name-reuse for more information.      
                                                                                
                                                                                
          </body>                                                               
         </html>                                                                
ERROR    HTTPError: 400 Bad Request from https://test.pypi.org/legacy/          
         Bad Request
#

interesting since i have not bumped those versions yet!

raw belfry
#

can't really find any issue πŸ˜“

severe tundra
#

Is it uploading multiple times somehow? Don't see why it would. Once you've uploaded a filename you can't ever upload it again, that's what that message usually means.

raw belfry
# severe tundra Is it uploading multiple times somehow? Don't see why it would. Once you've uplo...

It is a nightmare, I have deleted the repo from test PyPI and created a new one, when i created my first tag everything ran so smooth, and I was very happy, and afterwards I tried to add a new tag and it failed, but it failed because somehow the sdist and wheel file kept the old version instead of retrieving it from the tag version, but it is very confusing because now i don't understand where is the problem!!!

tag: 9999.0.1rc1
https://github.com/Keshish/d_python-sdk/actions/runs/13675935030

tag: 9999.0.2rc1
https://github.com/Keshish/d_python-sdk/actions/runs/13676026945

violet pebble
#

for both of those, it's creating the same filenames, without the rc number: dist/d_walacor_python_sdk-9999.0.1.tar.gz

#

well, both with 0.1 as well

raw belfry
#

yes i see that but why? why it is working on the first step and when i create a new tag it fails to create a new dist with the newer tags

violet pebble
#

could be because both the tags are on the same commit?

#

if there's more than one, setuptools-scm will have to pick one of them, so looks like it's choosing the first in both cases? try making a new commit, then tagging that

raw belfry
severe tundra
violet pebble
#

I think local_scheme = "no-local-version" is needed? otherwise you get a version with a plus in it, which PyPI rejects

severe tundra
#

No, only if you push a non-tagged version. And if you do, you'll get name collisions.

warped citrus
warm lance
severe tundra
#

Quoting PEP 660:

An β€œeditable” wheel uses the wheel format not for distribution but as ephemeral communication between the build system and the front end. This avoids having the build backend install anything directly. This wheel must not be exposed to end users, nor cached, nor distributed.

lament onyx
#

Editable shouldn't be part of the cli

But its great its part of the api

severe tundra
#

I've got two lock-step PRs to add, no-wheel and drop Python 3.8, I'd like to followup by adding 3.14 and touching up coverage.

#

I'm a bit stuck for 3.14 until tox gets fixed, though.

mighty yew
#

didn't we have codecov configured as report only?

severe tundra
#

I think it’s informational for the patch, but not the total, but it’s also not required, it does make a red x though.

mighty yew
#

ah, right

#

ahhh, by lock-step you meant dependent PRs, not blocked PRs

#

right?

severe tundra
#

Yes. Basically they need to go in relatively close to each other, because Python 3.8 will be slightly less ideal when the one with passing ci goes in, then that will make the 3.8 removal PR pass CI.

#

It basically will start downloading wheel on 3.8 even if you don’t need it

mighty yew
#

I don't have much experience with it, but it sounds like the Github PR merge queue feature might be helpful for these situations

#

I think we have it disabled, though

severe tundra
#

I think so, haven’t used it either

slender herald
#

Merge queue is more for "PR A and PR B pass CI independently and don't conflict, but fail CI when combined" (classic example: one PR adds a new use of an internal API that the other PR renames, so the combination contains a call to an internal API name that doesn't exist anymore)

severe tundra
#

But if the second PR only passes if the first one is in, wouldn’t that work too?

karmic steppe
#

You can't queue it unless it already passes the required checks

karmic steppe
#

maybe there's some admin overwrite that allows to add it to queue with failing checks but I'm not too sure

#

Note that the queue only waits for certain amount of time (or number of PRs) before all PRs are merged, it's not quite meant for this scenario where you want to merge multiple things at once or not at all. If the second PR fails, that won't stop the first one from being merged (perhaps there's some setting to change but afaik none of the available ones do that)

karmic steppe
tired knoll
#

The "bypass rules" checkbox will jump the queue

karmic steppe
#

consistent with the link I posted, thanks for confirmation

#

I currently only use the queue at work so couldn't really check

severe tundra
#

I'm going to merge those two PRs if no one has obections (reviews welcome!)

severe tundra
tidal bramble
#

you can ping me for reviews if it feels like it's taking too long

severe tundra
#

Thanks! I merged https://github.com/pypa/build/pull/899 trying to get CI green again, feel free to check that, also the other coverage one (896) is ready. I'm hoping we can get to the ModuleNotFoundError one; if that goes in we can do a patch release. (I'm kind of after that 3.14 trove classifier πŸ˜‰ )

#

Longer term, I'm planning on proposing a --config-json or similar flag (from gpep517); if that goes it that would make a minor version. But if we get a bugfix in, that can be a patch versions sooner. Ahh, forgot about dropping 3.8. Guess next version needs to be a minor release. I'll probably try to write something up about that soonish, then.

blazing bridge
#

(build doesn't include any non-Python code, does it? CI here is just a matter of running tests and making sure of version compatibility?)

severe tundra
#

Yes

#

Coverage testing is what we’ve been working on trying to improve

tidal bramble
#

I'm personally not too fond of the requirements.txt/constraints.txt format and I'd prefer if there was some other way we could constrain the build deps

green tusk
#

When building a single project, a pylock.toml would make sense, as the user likely wants a reproducible build env.

tidal bramble
#
  • is there an easy way to generate a pylock.toml from your build-system.requires?
  • can pylock.toml actually be used as input for constraints? We don't want to be installing any old package in the build env
rain prism
#

I don't think anyone supports locking build dependencies yet, since build dependencies are multiple sub-resolutions

#

We're looking at it in uv but need to solve those hard problems still

#

I also don't know of anyone accepting a pylock.toml as a constraints file yet, I don't think pip even supports it as an input file of any sort?

severe tundra
#

Packaging hasn’t made a release with support yet

left lintel
#

pylock explicitly does not support build dependencies

tidal bramble
green tusk
green tusk
tidal bramble
#

that's an interesting thought

severe tundra
#

New release 1.4.0 out with --quiet and --metadata.

arctic briar
rain prism
#

An example hard problem is... uv performs one resolution right now to generate the uv.lock file

#

if we have to build things along the way, we'll perform another entirely independent resolution for those build dependencies

#

now let's say you want those resolutions to be in the uv.lock, how do you unify these sub-resolutions?

#

what do we do when we encounter packages without static metadata and we need to build to determine their dependencies to finish the main resolution?

severe tundra
#

But they would be smaller and therefore simple/fast I'd think (same could be true for independent "task" environments)

rain prism
#

I don't think the intent is to have a unique lock for every sub-resolution, I think we'd try to have coherent versions with forking on conflicts as we do elsewhere

#

You could argue that's a bad idea, but I'm just sharing the complexity of the design space β€” we haven't actually decided what to do

severe tundra
#

I think you'd want a unique lock; one package should be able to pin scikit-build-core or setuptools and still build with something that requests a different pin / limit of it.

#

In theory packages could always pin the build requirements. It's not currently that common, but not "incorrect"

rain prism
severe tundra
#

What would you fork on, though? It's not a different Python version, for example

#

(I am also thinking of tasks, where you might have incompatible or unrelated task environments, like in nox/tox, etc)

rain prism
#

We already fork on arbitrary things, e.g., extras

severe tundra
#

I didn't know that could fork

rain prism
severe tundra
#

I feel like it would still be cleaner if isolated build environments had isolated locks

rain prism
#

I think I'd prefer that I was using a consistent version of a build dependency if possible

#

instead of introducing more versions to my dependency tree

severe tundra
#

It feels odd that adding a dependency could change the build of another dependency that is supposed to be building in isolation.

rain prism
#

but we've considered multiple modes

severe tundra
#

I think a "single shared build environment" mode would make sense to have, could save time and make it easier to keep versions consistent, but maybe opt-in due to possibly not being possible depending on packages

arctic briar
rain prism
#

@dry mural has probably thought about this more than me too πŸ™‚

arctic briar
#

Catching up now -- yeah, enforcing consistency across build environments in a resolve basically adds an additional dimension of selection invalidators, while also exploding the search space.

I didn't think y'all would want to tackle that in the first cut of this feature.

rain prism
#

I'm not sure we would!

blazing bridge
#

Doesn't look like a lot of new functionality, but doc improvements are definitely appreciated.

#

(well, "code changes", I should say, given that it's a patch release)

left lintel
left lintel
violet pebble
#

also broke Pillow's CI, which runs check-manifest which uses build

severe tundra
#

Hopefully 1.4.2 solves the issues. check-sdist is (my) altertative ot check-manifest that supports more backends. πŸ™‚

violet pebble
#

yes, it's working, thanks for the quick fix!

severe tundra
#

Sorry it wasn't yesterday, still working out the new release system πŸ™‚

violet pebble
#

np! you'll be ready for the next one πŸ™‚

severe tundra
#

conda-forge is now using rattler. Forgot about that (Mar 11, 2025 is when I first tried to move it over πŸ˜‰ )

#

Do we want to enable release immutability? And full length SHA pinning for actions? (In settings). I also will set the default workflow permissions to read only.

#

I think we can also disable the "allow actions to create and approve PRs" selection, I don't think we use that here.

tidal bramble
#

I feel like 1.4.1 should've been 1.5.0

severe tundra
#

Yes, it had a feature (build constraints), I think I would have called it 1.5.0.

#

(and of course, in hindsight the fix that needed a fix would have looked better in a .0 release πŸ˜‰ )

tidal bramble
#

can we mark LLM contributions going forward? there's been a lot of activity recently which I assume is AI led

severe tundra
#

I’ve noted any PRs where I used AI (tool and model). I didn’t mark the tool for the lazy typing one since it wasn’t AI, though it was just an oversight (https://github.com/henryiii/flake8-lazy was used).

severe tundra
#

I think for me it's just the one test annotation PR, actually.

tidal bramble
#

It was the dep chain printing PR I saw this morning and a few other bigger ones that I recall - the docs refactor is one

severe tundra
#

I addressed your feedback on 1020 but it was already merged, made a new PR.

tidal bramble
#

Thank you

tidal bramble
#

It got merged before I could get to it again but I trust your judgment

severe tundra
severe tundra
#

I think we are good for a patch release?

tidal bramble
#

just a package name without any context

severe tundra
#

I think this could be fixed with just reverting the lines 147-149 (and test)?

tidal bramble
#

the printing changes would've to be reverted as well, otherwise the bullet points will be printed on the same line

#

that only leaves renaming "origin" to "kind" and adding "step" as a value

severe tundra
#

I think that's fine

#

Did you want to do that, or I could try in a bit.

severe tundra
tidal bramble
#

you were too quick for me πŸ˜„

severe tundra
#

We okay for a release? I can try to trigger it (I think I activate the "pre-release" job and set the type to "patch")

tidal bramble
#

sure

severe tundra
#

Docs still broken. It's running docstringfmt in readthedocs and failing. 🀦

severe tundra
#

Okay with a release?

tidal bramble
arctic briar
#

Hello there!

bronze aurora
#

Nice to see you here πŸ™

mighty yew
#

😁

umbral gale
#

Hello, hello, any reason to still use python -m pep517.build . or is build already a 100% replacement?

bronze aurora
#

Should be 100%, unless you run into a bug 😁

umbral gale
#

Speaking of bugs, I find no more tox4 bugs 😦 .. anyway, wrong chat πŸ˜„

umbral gale
#

Btw - I created a github action using build called will it build? πŸ˜… which builds the package and runs twine check on it. Well, somebody had to do it πŸ™‚ https://github.com/jugmac00/will-it-build

GitHub
jugmac00/will-it-build

Build and verify a Python package. Contribute to jugmac00/will-it-build development by creating an account on GitHub.

bronze aurora
winter spindle
mighty yew
#

it uses an in-tree backend IIRC

#

it means it uses the code in there to build

winter spindle
#

ok, that should create an archive with setup.py and poetry folder right?

mighty yew
#

idk

#

depends of the implementation

winter spindle
#

well it should imo do it, or else how pip install it?

mighty yew
#

it would build a wheel and install the wheel

winter spindle
#

the wheel doesn't contain the folder, poetry

mighty yew
#

something wrong is going on there

#

that is not a valid wheel

#

well, maybe valid, not proper

winter spindle
#

that's what I am complaining

#

both source archive and wheel doesn't have poetry folder

#

and that looks weird to me

#

because the one from pip wheel poetry_core has that folder

mighty yew
#

what exactly are you doing?

#

I simply cloned poetry-core and ran python -m build

#

I have a proper wheel

winter spindle
mighty yew
#

I just did the same with the tarball and it works

winter spindle
#

weird

mighty yew
#

what path is the wheel on?

#

it should be at dist/poetry_core-1.0.2-py2.py3-none-any.whl

winter spindle
#

yeah it is

#

wait, I noticed I downladed it inside of a git repo

#

is that a problem

#

ok that is the problem, when I try it elsewhere I get a proper wheel

mighty yew
#

okay great

#

but that is weird

#

being inside a git repo should have no impact

bronze aurora
#

Depends on your backend

mighty yew
#

yes

#

but I don't see a reason why poetry would behave differently here

bronze aurora
#

Perhaps the repo already has a pyproject.toml and poetry picks the git repo as root for some operations?

winter spindle
mighty yew
#

but poetry sholdn't be picking any pyproject.toml

bronze aurora
#

poetry uses pyproject.toml as config so πŸ™„

mighty yew
#

outside the directory we provide

winter spindle
#

essentially I forgot to change directory and that is the problem

#

so there should be no pyproject.toml in root directory

#

but it is still weird to me

#

my directory structure was C:\MINGW-packages\mingw-w64-python-poetry-core\src\poetry-core-1.0.2 if that matters, essentially was packing poetry-core for msys2.

winter spindle
#

looks like a problem with poetry...

mighty yew
#

I opened this thread

#

https://discuss.python.org/t/add-a-hook-to-get-usage-information-for-the-config-settings-parameter-of-pep-517-build-hooks/7351

Discussions on Python.org
Add a hook to get usage information for the `config_settings` param...

Hi all. As some of you may know, PEP 517 allows us to pass a config_settings parameter to build hooks. Pre PEP 517, we were able to do python setup.py --help to see which extra configuration options the backend supports. Right now, the same is not possible because PEP 517 frontends do not have any information. How do you feel about possibly add...

#

you may be interested

#

😁

bronze aurora
#

I personally have nothing against it, but consider it a niche need. I tend to specify all my package configuration in config files, so personally don't see when I'd want to use this πŸ‘

mighty yew
#

yes, it more niche but setuptools for eg does have options you can pass

#

and there are other backends, for eg. ones that build native code, that allow you to configure various aspects of the build

#

like changing compile flags and whatnot

bronze aurora
#

I know it has, but none of those are dynamic enough that I'd want to change as a one off

mighty yew
#

yeah yeah

#

I get that

#

but I don't think there are really any drawbacks with this proposal, as it's an optional hook, so

bronze aurora
#

At least the way I use Python packaging, but as I'd say have nothing against adding such behavior

#

Though for this to get approval I feel you likely need at least two of flit/setuptoools/poetry maintainers for it

mighty yew
#

config_setting is niche, but evidently there is a need for it

#

yeah

bronze aurora
#

Config settings IMHO as API feature makes more sense to me

#

Because tools might want to move some build backend folders out of their default locations

#

But cli invocations I find generally are good with defaults

tidal bramble
#

flit and poetry don't support config settings at all, I think

bronze aurora
#

Generally if no backend wants to implement it, it's deemed not important enough to make it a pep

tidal bramble
#

setuptools has --global-option basically as a shim for pip but it's not working the way it's meant to

#

the values are appended instead of being prepended

#

i.e. it works like --build-option

bronze aurora
#

Generally the PyPA has lately taken the approach of first implement a poc then standardize

#

So unless you get commitment from major backends that they'll implement the spec you're fighting an uphill battle

boreal quest
#

Hello all, I am just discovering build. I am trying to move my project build to poetry I have been told that using build for creating my artifacts in github actions would be the way yo go (rather than reinstalling poetry). Before testing it in docker for building my wheels on manylinux I am trying it on my mac and at the moment I get this error:
python -m build --sdist --wheel /Users/alleon_g/Documents/0x-TechWork/scikit-decide/tmp/galleon/scikit-decide/.build/lib/python3.8/site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first. warnings.warn(

#

Any idea what I am doing wrong ?

bronze aurora
#

Ah this happens via poetry? You get that error because setup.py contains imports from distutils that's no longer supported

#

If you're getting this via poetry you should raise a poetry issue to address if

boreal quest
tidal bramble
#

You have a module called "build" in the root of your project which is what is causing this warning to be emitted. python -m build will not work at all for you in this case, try pyproject-build (or pyproject-build.exe if you're on Windows)

#

or rename your build script ;)

#

Oh, and you will also have to declare setuptools as a build dependency if you're importing it in your build script

boreal quest
tidal bramble
#

@mighty yew πŸ‘‹ - is it possible to cut a new release for the Unicode fix and the pipx entry point?

mighty yew
#

will do tomorrow

#

please ping me if I forget πŸ˜›

mighty yew
#

did a release now

severe tundra
#

What timeline are you thinking for the next release, @mighty yew ? It would be nice to get a few of those fixes out, as I think there might be a bug or two left to hunt down, but it's harder to be sure because #274's fixed bug a little too easy to trigger. (The other fixes are all nice to have too)

#

I'm also wondering if we can do a little work on the tracebacks in the next version (after a release) - we might be able to reduce the unneeded lines just a little, which would help users opening issues on packaging-problems who don't even know what part to copy-paste.

mighty yew
#

I was thinking soon

#

I think the tracebacks went awry at some point

#

the used to be nice

#

well, maybe not

#

may be selective memory πŸ˜›

tidal bramble
#

I don't remember them looking different to be honest. It'd be nice to get some negative space between the backend traceback and pep517's at least or just hide the pep517 one

mighty yew
#

yes

mighty yew
#

looks much better now

#

for eg

# 
Traceback (most recent call last):
  File "/home/anubis/.virtualenvs/build/lib/python3.9/site-packages/pep517/_in_process.py", line 280, in <module>
    main()
  File "/home/anubis/.virtualenvs/build/lib/python3.9/site-packages/pep517/_in_process.py", line 263, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/home/anubis/.virtualenvs/build/lib/python3.9/site-packages/pep517/_in_process.py", line 213, in get_requires_for_build_sdist
    backend = _build_backend()
  File "/home/anubis/.virtualenvs/build/lib/python3.9/site-packages/pep517/_in_process.py", line 86, in _build_backend
    obj = import_module(mod_path)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 855, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/anubis/git/trampolim/trampolim/__init__.py", line 5, in <module>
    import trampolim._build
  File "/home/anubis/git/trampolim/trampolim/_build.py", line 23, in <module>
    import trampolim._tasks
  File "/home/anubis/git/trampolim/trampolim/_tasks.py", line 10, in <module>
    class Session():
  File "/home/anubis/git/trampolim/trampolim/_tasks.py", line 11, in Session
    def __init__(self, project: trampolim._build.Project) -> None:
AttributeError: partially initialized module 'trampolim' has no attribute '_build' (most likely due to a circular import)

ERROR Backend subproccess exited when trying to invoke get_requires_for_build_sdist
mighty yew
#

if anyone wants to test the PR in different scenarios it would be great!

lament onyx
#

btw, whats the state of non setuptools editable installs?

bronze aurora
#

There's a pep being in review

warm mulch
bronze aurora
#

that one πŸ˜„

mighty yew
#

@bronze aurora is there any way to update test/constraints.txt automatically?

bronze aurora
#

don't think so, how would you know what are the new lowest limits? πŸ˜„

mighty yew
#

there could be a tool that would resolve the dependency for the lowest version

#

maybe the NYU funding will help with that

#

since they will be working on resolvelib

#

and the pip dependency resolver

#

I really hope they extract the dependency resolver to resolvelib

lament onyx
#

wasnt it vendored already?

mighty yew
#

resolvelib is vendored in pip, yes

#

not sure if that's what you're asking

#

resolvbelib cannot resolved python dependnecies

#

pip itself has the logic for that

bronze aurora
#

perhaps @arctic briar may know πŸ™‚

mighty yew
#

it would be good to put it in resolvelib so that we could re-use it for things like this

arctic briar
#

Yea, there's basically 2 things here: resolvelib is the core "just the algorithm" part and lives separately. pip has all the Python-specific logic baked into it which we'd want to generalise and pull out into a separate package.

#

That's not really something that is covered explicitly in any existing funding IIUC, but both of the existing funding seem pretty broad-enough that if someone wanted to include it, they could!

#

I'm not the decision maker in either of those so, well, IDK how they're planning on allocate the funding and direct it. I do trust that the folks involved would be thinking carefully about how to do that properly.

mighty yew
#

yeah

#

this is something I would like to do at some point

#

as I need it for some projects

winter spindle
#

I just found that build gives raw ansi outputs on Windows cmd. Is that expected?

H:\>python -m build -w -n
←[91mERROR←[0m Backend 'mesonpep517.buildapi' is not available.
mighty yew
#

if it's a tty, yes

#

I guess some terminals don't support that

#

sighs

#

are you on windows 32?

#

well, it seems windows has some settings to enable/disable this by default

#

we'd have to use a windows api to force enable

#

which is out of scope for this project

#

so I am just gonna disable colors on windows

#

shouldn't really be a big deal anyway

arctic briar
#

Can you not depend on colorama, and call colorama.init()?

#

I prefer if we do not treat Windows as a second class citizen.

mighty yew
#

it's literally just two words that get colorized

#

ERROR blah blah blah

#

WARNING blah blah blah

#

this package is critical for bootstraping python environments, so I do not want to add unnecessary dependencies

#

but I can make colorama a extra and enable that functionality when it is installed

arctic briar
#

πŸ€·β€β™‚οΈ

warm mulch
#

If it’s that unimportant, remove the colours everywhere

mighty yew
#

since a lot of people are keen on this, I made colorama a hard dep on windows but not actually a runtime dep

#

as I mentioned in the PR, if this causes issues for anyone, we will reconsider the approach

severe tundra
#

I think that will work fine. Colorama happens to be one package that you don't even need to interact with (just an init), so it doesn't complicate the code significantly, and can be skipped.

#

I also love colors πŸ™‚

mighty yew
#

please ping me if you are working on it and have any issues

mighty yew
lament onyx
#

@mighty yew are there any plans to add a simple api for build (at first glance the invocation conveniences are hidden in main) i want to go from directoy to exact wheel file so i can drop wheels into containers on k8

mighty yew
#

it should be fairly simple already

lament onyx
#

at first glance quite a bi pf plumbing is missing (i also just took note that it wont return the created filenames)

mighty yew
# 
builder = ProjectBuilder('path/to/project')
with build.env.IsolatedEnvBuilder() as env:
    builder.python_executable = env.executable
    builder.scripts_dir = env.scripts_dir
    env.install(builder.build_system_requires | builder.get_requires_for_build('wheel'))
    builder.build('wheel', 'path/to/output/directory')
#

that should be all you need if you want the isolation

lament onyx
#

hmm, actually, that looks reasonably doable, will try details

mighty yew
#

please do and let me know if you run into any issues

lament onyx
#

will do

mighty yew
#

ahh, sorry

#

the env.install really needs to be split into two

#

I forgot about that

#

but yeah

# 
builder = ProjectBuilder('path/to/project')
with build.env.IsolatedEnvBuilder() as env:
    builder.python_executable = env.executable
    builder.scripts_dir = env.scripts_dir
    env.install(builder.build_system_requires)
    env.install(builder.get_requires_for_build('wheel'))
    builder.build('wheel', 'path/to/output/directory')
#

because the get_requires hook needs the build system requirements

#

to simplify this, perhaps we could make IsolatedEnvBuilder take a ProjectBuilder argument, to automate the necessary setup

#

would result in

# 
builder = ProjectBuilder('path/to/project')
with build.env.IsolatedEnvBuilder(builder) as env:
    env.install(builder.get_requires_for_build('wheel'))
    builder.build('wheel', 'path/to/output/directory')
#

@bronze aurora @severe tundra what do you think?

lament onyx
#

should be a contextmanager that takes the env builder class optionally and runs necessary steps,

#

but yeah, a full plumbing api would be nice

mighty yew
#

how so?

#

ah

#

the other way round

lament onyx
# 
builder = ProjectBuilder('path/to/project')
with build.env.prepare_builds(env_class=IsolatedEnvBuilder, builder=builder,  builds={"wheel"}) as env:
    mywheel = builder.build('wheel', output_directory)
#

and something that brings it all together aka ```py
results = build.run_for_project(project_dir="path/to/project", output_dir="path/to/distdir", distributions=["sdist","wheel"])

mighty yew
#

I am not sure if I am convinced by these approaches as the code footprint is fairly similar

#

and we shouldn't be reusing the environment to build different distributions

#

as they may have different dependencies

#

to build multiple distributions, you would do

# 
builder = ProjectBuilder('path/to/project')
for distribution in ('sdist', 'wheel'):
    with build.env.IsolatedEnvBuilder(builder) as env:
        env.install(builder.get_requires_for_build(distribution))
        builder.build(distribution, 'path/to/output/directory')
#

which is still fairly reasonable

lament onyx
#

oh, i see, makes sense

#

single distribution type plumbing might be nice then

#

aka ```py
builder = ProjectBuilder(project_path)
wheel_file = build.wheel_of(builder, output_dir=output_dir)

#

(the idea here is to prevent cargo culting a common pattern for doing something that has multiple steps and concepts while the user only cares for the inputs and outputs)

bronze aurora
#

I'm personally neutral on this, happy to accept new plumbing APIs with tests though, perhaps we should not make it part of core though, but instead have a plumbing module that defines these, and just expose it at root πŸ™‚

lament onyx
#

totally fine to have this called plumbing

#

build.plumbing.wheel_for(...)

lament onyx
#

for now im going to use ```py
def build_plugin_wheel(plugin, output_dir="dist"):
mod = sys.modules[plugin.class.module]
project = Path(mod.file).parent.parent
if project.name == "site-packages":
error_print(f"{plugin} is not installed editable, {project}")
import build.env
with build.env.IsolatedEnvBuilder() as env:
builder = build.ProjectBuilder(str(project), python_executable=env.executable,scripts_dir=env.scripts_dir )
env.install(builder.build_dependencies)
env.install(builder.get_dependencies("wheel"))
return builder.build("wheel", output_dir)

lament onyx
mighty yew
#

but I only want it as an optional build dependency

severe tundra
#

From the Git sources or from SDist? It is not needed from SDist, the version file is baked into SDists. It's only required for building from Git sources. (Obviously, it's also not needed for wheels)

#

(Having a hard-coded "fall-back" is possible too, I think, and then setuptools_scm is only used to identify "inbetween" versions, which is still sounds useful - I'd like to do at least this with cibuildwheel, actually, so the logs would contain the exact version used if someone uses a non-tagged version) But I haven't tried that "fallback" personally.

mighty yew
#

from everywhere

#

we generally do not build distro packages, arch at least, from sdists

tidal bramble
mighty yew
#

because I may want to build from a sdist

#

and a dependency on setuptools_scm makes it harder to bootstrap a python environment

tidal bramble
#

I understand that, I'm saying that an sdist produced with setuptools_scm still needs setuptools_scm to build a wheel. The version exists in some format in the sdist but setuptools won't know how to read it.

severe tundra
tidal bramble
#

Ah yes, pretty clever, if you opt into generating a version.py that might work.

lament onyx
#

@mighty yew i wish distro would spend some time to integrate with tools like setuptools_scm better, it's practically already implied by a large set of tools including pytest

#

@mighty yew as for making independent sdists, that sometimes is very error prone, but non setuptools build backends could do better

lament onyx
#

@severe tundra currently setuptools_scm is also required in sdists (as we dont have a modified sdist command that actually edits the metadata (its error prone, the tools for ini files and toml are missing and the setup is actually cheap and getting cheaper)

mighty yew
#

@lament onyx the only thing we need for better intergration really is having it work out of the box from git archives

#

the thing about wanting it only as an optional dependency is only for build

lament onyx
#

@mighty yew working from git archives is planned for about this week, git 2.32.0 finally added a %(describe) format and so first class git archive support is coming

lament onyx
#

hmm, now if i just had access to git 2.32 for the ddetails

lament onyx
#

@mighty yew the next release of setuptools_scm will support git archives when a .git_archival file is added there also will be a example content with the full metadata

jaunty matrix
#

@lament onyx will the next version work independent of setuptools?

lament onyx
jaunty matrix
#

ah cool

severe tundra
#

Maybe setuptools_scm could depend on the "renamed" package and just add the setuptools parts? pypa_scm or something like that?

#

setuptools_scm needs a channel πŸ˜‰

lament onyx
#

Seems like it

mighty yew
#

@bronze aurora hey, do you remember what was the source of the ModuleNotFoundError: No module named 'encodings' error on pypy windows?

bronze aurora
#

I think some bootstrapping issue within pypy

#

you see that error generally if you have a pypy executable on its own that does not have the expected layout alongside it

mighty yew
#

hum

bronze aurora
#

encodings is one of the first stdlib module that it tries to import

mighty yew
#

it was sporadic, right?

bronze aurora
#

so if for some reason the stdlib is missing alongside the exec you'll see an error like that

mighty yew
#

hum

#

the reruns always tun into it

bronze aurora
#

I think was fairly stable πŸ™‚ but ran into that with two different pypy minors

mighty yew
#

ah!

bronze aurora
#

do reruns give you the same pypy? πŸ˜„

mighty yew
#

so the issue is with the github actions provided pypy

#

I'm gonna make that check optional again

#

but we should figure it out

bronze aurora
#

sometimes some github actions run on a earlier ring, so you might pick up a newer pypy or older πŸ™‚

tidal bramble
#

What’s the situation with the ununderscored functions in main. are they public, should they be moved into a separate module if so?

mighty yew
#

they are

#

I'd like to keep them in main for as they are simple helper functions

#

but we might move them if we go with a plumbing module

#

they are just meant to make rolling out your own cli easier

#

without any other api design implications

#

the scope is just that

tidal bramble
#

The PyPy3 tests take a very long time to run, any thoughts on testing 'via wheel' and min versions only?

severe tundra
#

Dang, this just missed the 0.5.0 release

We could do a 0.5.1, perhaps? It's a pretty simple fix. Don't want to volunteer someone else's time, though. πŸ™‚ (PS: GitHub now has approved releases, with lots of security controls, like requiring multiple signoffs, etc)

bronze aurora
#

Agreed

tidal bramble
#

Was a comment deleted, I don't see what you're quoting?

mighty yew
tidal bramble
#

Right :P

void condor
mighty yew
#

@severe tundra are you gonna do a PR for this?

#

just so that we do not duplicate efforts πŸ˜…

severe tundra
#

Happy to if you want me to, up to you.wound be till tonight if I do it

#

Also need to know if you want to copy out the files on failure or not if I do it πŸ™‚

mighty yew
#

I ended up doing the PR as it was pretty simple

#

let's not keep the files, but I'll do a PR improving the error message

#

it's easier to get a concise UX there

severe tundra
#

See my question on #cibuildwheel - is there any snazzy marketing lingo we should add in the docs for why someone should chose build over pip? πŸ˜‰

severe tundra
mighty yew
#

the pypy shipped by github actions is broken

#

I asked this to BernΓ‘t in one of the PRs

lament onyx
#

hmm, time to read release notes, AttributeError: 'ProjectBuilder' object has no attribute 'build_dependencies'

#

ah, found it, broken on 0.4

mighty yew
#

yeah, we renamed that

#

once we hit 1.0.0, the API will be stable

#

if breaking changes are necessary, they will be properly deprecated

lament onyx
#

hmm, i wonder if there is a low cost deprecation library that adds alias warnings & co for next to free

mighty yew
#

that seems pretty straight forward

lament onyx
#

@mighty yew the devil tends to be in the details ^^ i think i'll create a lib for it in a bit (i have some patterns inside of pytest and a few other projects that i want to expose using a small stable api (using versions to pick deprecation error targets, and integrating it with surrounding tooling, so that under test you get a different type of error/test instead of the testsuite breaking once a feature hits the breaks on users setup

mighty yew
# 
def deprecated(msg):
    def decorator(func):
        def wrapper(*args, **kwargs):
            warnings.warning(DeprecationWarning, msg)
        return wrapper
    return decorator
#

isn't that like 90% of what you need?

#

anyway, having a lib for more custom use-cases would be great

severe tundra
lament onyx
#

@mighty yew well, thats not even 10% unfortunately (i want deprecations to turn into errors at designated "breaking" releases, but also allow testsuites to still excerpt deprecated code until its removed)

arctic briar
lament onyx
arctic briar
#

Well, color me interested in that too. :)

lament onyx
#

I'll reach out to you once I have something to nerd over

The idea is to create a small stable core, make it vendoring aware and support global install interaction, and then pushing it to pytest/pypa

tidal bramble
#

stdout = b"ModuleNotFoundError: No module named 'encodings'\ndebug: OperationError:\ndebug: operror-type: ModuleNotFoundError\ndebug: operror-value: No module named 'encodings'\n"

bronze aurora
#

Overriding the python path has no impact on the stdlib path

#

So it must be something else, likely a venv bug πŸ€”

tidal bramble
#

have we tried using 3.7?

tidal bramble
severe tundra
#

Why does python -m build generate a src/*.egg-info directory? Someone reported that it does. From PEP 376 it seems that dir should only be related to installing?

bronze aurora
#

It's because setuptools generates it

#

It works by creating files in place

#

If you build a wheel also generates a wheel directory

#

Not much we can do about it, it's the backends problem partially

severe tundra
#

Guessed it was setuptools, just was wondering why it generated something related to installing in the source.

tidal bramble
#

setuptools runs egg_info during get_requires_for_build_x to find missing build dependencies. The egg_info command creates the egg-info folder, as you would expect

severe tundra
#

Do they need to be in the SDists? PKG-INFO is required but are the .egg-info files? build has an egg-info folder in the sdist, for example, but it still seems to install if you manually remove it.

#

And some files are duplicated in it, including PKG-INFO (Edit: that's the only one duplicated, actually)

bronze aurora
#

Pkg info is egg info with minor transformations

#

It's generated from egg info

#

Egg info is always first step in setuptools

severe tundra
#

I'm not surprised that it's generated to make PKG-INFO, just that it's included in the SDists (and left in the directory). Does something use it?

bronze aurora
#

Probably

warm mulch
#

I don’t think anything standard-ish uses it (except setuptools to generate dist-info and PKG-INFO), but setuptools just leaves it there because, well, why not

#

Of course there are always some in-house tools that rely on those nonstandard behaviour but that’s not anyone else’s problem

bronze aurora
#

It's setuptools problem if they remove it and people complain 😬

#

Hyrums law

lament onyx
#

setuptools_scm uses it in the sdists ^^

warm lance
tidal bramble
#

none of the build devs can do that

warm lance
#

oh woops

#

wait really?

tidal bramble
#

yeah, there's no overlap

warm lance
#

q.q

tidal bramble
#

tomli < 1.1.0 won't be able to read a binary file btw, you'll probably have to decode it and pass it onto loads if you wanna maintain compatibility

warm lance
#

oh hang on this is running on py2.7 so I can't use --strict-markers

#

and --strict-config xD

warm lance
#

@arctic briar ^ fyi

#

There's no pep517 channel?

mighty yew
#

now there is

mighty yew
#

@lament onyx is there any way to tell pytest-xdist that one of the tests cannot be run in parallel to others?

#

btw, let me know if you have a better way to mock that a module is not available, preferably one that does not need to modify the global environment

tidal bramble
#

just use a FileLock inside the test function I guess

mighty yew
#

I mean, I can use a multiprocessing lock instead

tidal bramble
#

you can't, xdist doesn't use multiprocessing

bronze aurora
#

You don't need lock

#

Xdist parralelizes at process level not thread

#

So you're fine

tidal bramble
#

yeah, it does look like all the failures are in the same worker

mighty yew
#

btw, what do you think of a release after the toml v1 PR?

bronze aurora
#

{VW}

#

works for me πŸ™‚

#

for future reference I'm ok with doing a release after every commit πŸ™‚

mighty yew
#

ack

warm lance
#

@tidal bramble I also recommend pytest-randomly to find undesirable test interactions

#

btw you don't need a finally in yield fixtures

tidal bramble
tidal bramble
#

is there a way to approve CI for a PR as a whole?

#

don't wanna have to do it again when a change is pushed

warm lance
warm lance
#

it's pretty great

mighty yew
#

I have change the CI approval setting to "Require approval for first-time contributors who are new to GitHub"

#

which is the best we can do

#

seems like github really is struggling with their free CI offering

warm lance
void condor
warm lance
#

Amazing

warm lance
void condor
warm lance
#

Hmm I suppose just having a bot go and approve any PRs I can would work too

tidal bramble
#

while I'm working on adapting the isolated env class for pip I've been wondering if we should not return the responsibility for managing pip to the user and copy pip from build's env like pip does. It would simplify things massively on our end and would allow us to drop the optional dependency on virtualenv. It would also allow pip to use build out of the box.

bronze aurora
#

What do you mean? You can't copy pip from build env as it might not be compatible.

#

Pip test suite always target the same python so that's special case

#

@mighty yew tried that in early days, against my early recommendation not too, and was a constant source of problems πŸ€·β€β™‚οΈ

#

Also you have no control over what pip version you're seeedin with that, it even could be a host patched version that's not venv compatible

#

So in short that's not a viable path IMHO

#

Works for a ci where it's a highly controlled environment but not in general

tidal bramble
#

we'd keep the pip version check that we have now, we just won't be installing pip for the user. If pip is devendored then it might not work but won't pip have run into that issue with isolated builds on Debian and RHEL?

#

how does it cope?

#

@warm mulch

bronze aurora
#

That sounds super unhelpful for the user, they often can't control the host pip and just working Vs fix your environment for you there's a strong UX difference, so from that proposal gets a strong -1 πŸ€·β€β™‚οΈ

#

Not to mention with the virtualenv you can do cross python builds

warm mulch
#

I think it depends on what you mean by "copy". If you can make sure the pip you're copying from is of the right version I guess it's fine, but I also don't see how it's useful to anyone

tidal bramble
#

The question that's on my mind is how to best depend on pip in a venv - version munging and multiple subprocess calls to upgrade pip and uninstall setuptools, only to more often than not reinstall it afterwards, probably isn't it. I think it's closer to, require the user to have pip and then either use pip to install pip in the build env, which might also cure our ensurepip woes, or do something like pip does which is create a zip file out of the pip package and shell out to that. If either of them proves to be sufficiently frictionless we could drop virtualenv but not before then.

bronze aurora
#

Why not use virtualenv with a custom seeder to provision the seed packages that you want exactly

#

I mean that's the reason I made the seeding process entirely replacaby

#

*replacable

#

Especially in places like pip CI where taking on other dependencies is not an issue

#

And you can ask virtualenv to use venv module to create virtual environments if you're worried about virtualenv creation logic differing

tidal bramble
#

I'll merge the isolated env PR later today if there are no objections

mighty yew
#

I'll do a review later

#

I have been away this week

tidal bramble
#

alright πŸ™‚

tidal bramble
#

@mighty yew, let me know if there's something I can do to make reviewing it easier

mighty yew
#

I don't think there's anything, I have just been busy with work

#

and not feeling super well

#

but I have time today, and will have a look

tidal bramble
#

fair enough, hope you feel better soon

mighty yew
#

sorry! I did not end up being able to do it as the afternoon flew by and I had a dinner today

#

just got home

#

I will really try to take a look tomorrow

tidal bramble
#

no worries, just opened another one for you to review πŸ˜›

tidal bramble
# 
/Users/dimitris/code/python-packaging/build/src/build/util.py:docstring of build.util.project_wheel_metadata:13:py:class reference target not found: importlib.metadata.Distribution
#

as far as inscrutable errors go this is a top contender

#

I don't even know what the number means - you might think it's the line number in the source file but it's not

warm lance
#

Column number?

tidal bramble
#

🀷

warm lance
#

Docstring offset?

#

ah so sphinx-autodoc-typehints generates a docstring for sphinx to consume

#

So maybe the line number in the generated file?

tidal bramble
#

That would make sense

tidal bramble
#

Now if you can figure out why it can’t find importlib.metadata 😜

warm lance
#

Oh at least pipx run tox -e docs

tidal bramble
warm lance
#

Looool

#

Was about to tell you that you don't need to install your package to get the version out

#

You can just use build.util.project_wheel_metadata .....

mighty yew
#

pretty early but

warm lance
#

Ah that looks sick

#

And it's lazy right?

tidal bramble
#

@mighty yew I've tried to address all of your feedback from last night

#

definitely not looking forward to rebasing ;P

sleek berry
#

@arctic briar Any thoughts on how we should handle Tag in .dist-info/WHEEL files in Debian packages when there are >1 Python 3 versions supported? (e.g. when there are C extensions installed for both Python 3.9 and 3.10, as is the case at the moment).

warm mulch
#

I believe the Tag field is multi-value so you can just put in all the tags, one on each line

sleek berry
#

Ah, neat. I couldn't find the definition of it

craggy scarab
#

Hi there, I was wondering: we are in the process of using python -m build everywhere, and trying to get rid of setup.py, but there is one thing which I can't manage to find: how can we extract the version string and package name in a generic way without setup.py? I would expect to replace a call to python setup.py --version with python -m build --version (and same or equivalent for the --name option), but cannot find anything... An option would be to first build the wheel, the read it and extract everything, but knowing this is something we would generally need in tools prior to building anything, we have a problem... It would be so nice if there was a generic way to get that information, independently on which build system is used! Thanks you!

tidal bramble
craggy scarab
#

@tidal bramble Thank you for the pointer! Indeed calling pep517.meta.load() could be a good enough thing if our tooling remains in python (but I share the same hope that we will get a CLI somehow to access this).

#

Another related question if you don't mind: right now one of the big problems of pep517 (well, I personally see it as a problem πŸ˜… ) is that if you have no internet connection, you cannot build, or pip install -U . your package, because the build dependencies are installed on the fly, in isolation, and without caching. You can fix this problem by running without build isolation, but then to do so you need to run it once first to install the tools, which you cannot obviously if you don't have internet (happens a lot while coding on the go, or on offline sites).

#

So the question is: can we achieve the same thing with pep517.meta.load() ? AKA run it without isolation?

craggy scarab
#

Just wondering how internal this is πŸ˜„

mighty yew
#

it's public

#

you can use it

#

but please pin the version

#

we are probably gonna tweak the API

craggy scarab
#

Thank you! It's pretty early stage on our side and we may not be ready to fully jump yet, but as soon as there is a common mainstream build tool to do everything, we would prefer to rather use it for EVERYTHING.

mighty yew
#

well, the build capability is there already

#

I am not sure about your requirements though

#

but I don't think we'll see much progress here

#

unless there is something that we forgot, in which case, please let us know

#

the CLI is stable btw, you can rely on it

#

the API might get some tweaking until 1.0.0

craggy scarab
#

No for now it is clear; our requirement is simply to have a good and clean way to build, query version and package name, and this with or without internet access. The isolation prevents offline use but can be disabled, and now with an API to query the metadata, we should be good for now (and can also work without isolation as per above). Now if we need to have a wrapper for getting the metadata from another CI script (jenkinsfile, bash script, whatever), we can write something on our side to do so; but if there is an easy way to retrieve the metadata via build.util, it would be nice to also have it available via python -m build!

#

Right now if I want to get the version from a bash script, I need something like: ```
$ python -c 'import build.util; import json; print(json.dumps(dict(build.util.project_wheel_metadata(".", isolated=False))))' | jq .Version -r
0.1.dev4+g3904d9c.d20211122

#

if the metadata could be extracted directly (as a JSON for example), with something like python -m build --metadata [--no-isolation] it would simplify things πŸ˜‹

#

(here I used JSON as an intermediate mostly because I would expect the build command to expose the metadata as JSON, obviously it can be simplified if we JUST want the version; for the sake of discussion!)

tidal bramble
#

are there any other metadata that you need apart from the version?

craggy scarab
#

we use extensively the Name and Version; a bit less the other fields except for crawling our packages (the same way we would do on the information returned by devpi or nexus): Summary, Requires-Python, Requires-Dist, Platform.

#

If there was a CLI to be done, I would rather have it output all the metadata, so everybody's happy (I think? at least more than implementing --version, then someone wants --name, then --requires-python, etc...

warm lance
#

Different metadata takes different compute resource to find out

#

If it happens to be in static PEP621 then it's only one file

#

But some stuff might need a full build step

tidal bramble
#

you can self-depend with a recent version of pip

# 
[project]
name = "foo"

[project.optional-dependencies]
bar = [
  "baz",
]
qux = [
  "foo[bar]",
]
warm lance
#

It's weird that self depending is different from literal inclusion though right?

tidal bramble
#

What’s literal inclusion?

violet pebble
# craggy scarab <@!262972466811502592> Thank you for the pointer! Indeed calling `pep517.meta.lo...

I'll need to replace some setup.py --foo calls at some point, but pep517.meta.load is very slow, taking about 10 seconds:

$ time python3 setup.py --version
9.0.0.dev0
python3 setup.py --version  0.44s user 0.19s system 88% cpu 0.719 total

$ cat 1.py
import pep517.meta
dist = pep517.meta.load(".")
print(dist.version)
$ time python3 1.py
9.0.0.dev0
python3 1.py  8.23s user 1.74s system 96% cpu 10.321 total
#

that's probably fine for replacing a setup.py --long-description in a build script, but some setup.py --name in CLI things should be faster

tidal bramble
#

Build without isolation should be considerably faster but it lacks a CLI

warm lance
#

Rather than using a dep reference

wispy stream
warm lance
wispy stream
severe tundra
#

Just a minor irritation today, just noticed that --no-build-isolation in build still checks the exact requirements are available in the current environment. There is a way to disable it (--skip-dependency-check), but it seems like most of the time, both are implied (and that's how pip works). For example, if you use oldest-supported-numpy, that will never be present in the environment. And, if you didn't use it, but instead pinned the oldest supported numpy's by hand, those would probably be older that what's in the environment, and the reason for building out of isolation is to use a newer numpy anyway. I personally would have slightly preferred --no-build-isolation worked exactly like pip, and --dependency-check was an opt-in, rather than an opt out. There are cases were it might be useful (checking to see if an unpinned dependency is present except for special ones like oldest-supported-numpy, but often it is incorrect, especially for pins)

#

Build ignores PIP_CONSTRAINTS_FILE when doing an isolated build, so there's no way to "fix" a build if someone put an uncapped/unpinned setuptools and then used 2to3, or something like that, right? Just polishing off my anti-version capping post.

#

(From build, that is, this works with pip)

tidal bramble
#

I guess the original motivation was for downstream packagers to verify that their build env contains all of the project's build deps. It makes sense for it to be on by default in that context

severe tundra
#

Only if you exactly match the pins, which packagers often will not. If you ship numpy 1.20, for example, you want to build against that, not an older one. So for pins, it's nearly never true, only for unpinned dependencies.

#

And not always, like for oldest-supported-numpy, for example.

warm lance
severe tundra
#

That was an old example (a sphinx plugin that hasn't had a release in something like 8 years, but it actually did have one to fix this a few weeks ago)! I don't remember which one. Technically, I think it was a plugin dependency.

#

My question was more about how do we add one if we really need it - the fact setuptools is talking about upper caps really bothers me, adding them is a disaster.

warm lance
#

Probably best to rename setuptools each major version

#

So you can depend on setuptools56 and setuptools57 in one environment

severe tundra
#

But then you'd need to keep up the old versions. That's kind of the problem with a cap, the old one will not work with the next <insert anything here>

#

In isolated environments, that should be safe, but if you can't inject a pin into an isolated environment, that's a problem.

warm lance
#

Same with numpy

#

Maybe there could be another package with the header files in

severe tundra
#

And my version capping post is now public. πŸ™‚ https://iscinumpy.dev/post/bound-version-constraints/

ISciNumPy.dev
Should You Use Upper Bound Version Constraints?

Bound version constraints (upper caps) are starting to show up in the Python
ecosystem. This is causing real world problems with
libraries following this recommendation, and is likely to continue to get
worse; this practice does not scale to large
numbers of libraries or large numbers of users. In this discussion I would like
to explain why alwa...

tidal bramble
severe tundra
#

NumPy must always be pinned at build time if it's a build time requirement.

warm lance
#

So you can have numpy-h-1-21 and numpy-h-1-20 installed together

severe tundra
#

You can't use a wheel compiled against a newer version of NumPy on an older version of NumPy.

warm lance
#

Isn't the build requirements only the .h files?

#

You don't actually need the full numpy

severe tundra
#

But you can do it the other way around. So NumPy is always pinned to the lowest supported version, and there's an official metapackage oldest-supported-numpy to do just that.

warm lance
#

Right but that could depend on specialist numpy-h-1-21... Packages

worn token
severe tundra
#

Since the oldest supported version depends on a ton of factors. Like OS, Python version, Python implementation, etc.

#

My post mostly talks about runtime requirements, but many of the arguments apply to build-time. If you pin/cap setuptools, then you might not support the next Python release, the next architecture, etc.

#

But then eventually you'll end up with a setuptools that doesn't work because it dropped something, and then you can't pin it inside the isolated environment with a flag (at least in build).

worn token
#

@severe tundra have you also tried the PIP_CONSTRAINTS_FILE with pip install --use-pep517? I remember trying it when I was proposing some integration tests for setuptools, but it also seem to be ignored (or I just did something plain wrong)...

tidal bramble