#build

1 messages · Page 2 of 1

severe tundra
#

No, I just remembered build filtering out PIP_ variables.

#

Yes, it's related. Setuptools really wants to try to improve, but it's a mess - no warnings are shown normally, capping is a bad idea, not capping can be unfixable, etc. 🙂

tidal bramble
#

I don't think build filters anything of pip's out

severe tundra
#

I'll try PIP_CONSTRAINTS_FILE on pip soon then.

tidal bramble
severe tundra
#

I may be remembering a discussion incorrectly.

#

Yes, the solution seemed reasonable.

#

And things are much better than they used to be. Pip no longer dumps a wall of text when you add a single -v. 😉

#

I'll have to play with constraints files then. Could be it's pip and build, not just build like I was thinking. Hurts that I'm not currently running into this, but just needing it for completeness in my article and possible future use.

mighty yew
#

I don't think I really understand the motivation behind the behavior you describe

#

you should not be pining dependencies if you want to build with other versions

#

build will only warn you if the build dependencies, as specified, are not met

#

if you have a upper bound on numpy for eg. and want to test a newer version, you should setup the environment yourself and run with -ns

#

or, of course, write a specific tool to do that, if it's an use-case that warrants it

sleek berry
#

Any thoughts on what Debian should call its PEP-517 build interface? pep517? build-api?

onyx coral
#

pep517 is the name of a project on pypi i think, would that cause any problems?

sleek berry
#

Don't think so. The question for us is whether this general approach will continue to be called pep517 in the future (assuming it gets updated by future PEPs)

warm mulch
#

build-api is probably more in line with how others are calling it.

Flit: flit_core.buildapi
Poetry: poetry.core.masonry.api
Setuptools: setuptools.build_meta

sleek berry
#

I was forced to make a quick decision, so stuck with pep517 for now.

arctic briar
#

I'd generally encourage not using PEP names for identifying things -- it's obscure and unclear to anyone who isn't familiar with them. Plus, it's at least PEP 517+518+600 that you'd be implementing.

desert bear
bronze aurora
#

IMHO a good name for it would be pyproject 🙂 because mirrors the file format that introduces all these new standards, and we had a discussion in the past to name the new standards pyproject style projects 🙂

sleek berry
#

Editable installs don't make sense in our context. Aha, that's a good argument for pyproject. Let me see if renaming is reasonable

tidal bramble
#

building is just one aspect of the pyproject file though

#

my original suggestion for build was "pyproject-build" and that's what the console script is called

bronze aurora
#

For now, but I imagine that might change in the future 😊

sleek berry
severe tundra
severe tundra
#

Actually, it happened with both venv and virtualenv, so that wasn't it. And it's now working again, so maybe a PyPI issue...

warm mulch
#

Or maybe a Debian thing? IIRC the base Python image is based on Debian, which contains some very old distribution formats that importlib.metadata does not recognise (like zipped eggs).

severe tundra
#

It was happing on GHA too, for about 1-3 hours, then it just cleared up.

tidal bramble
tidal bramble
#

I think I'll close https://github.com/pypa/build/pull/361 and open individual PRs for all the tiny improvements I made, doesn't look like there's any interest in isolated env reuse and the PR has sat dormant for far too long.

GitHub
env: refactor `IsolatedEnv` by layday · Pull Request #361 · pypa/bu...

The IsolatedEnv is made responsible for env creation to allow pip
to implement custom env creation logic.
IsolatedEnvBuilder takes an IsolatedEnv class as an argument
so that bringing your own Isol...

severe tundra
#

If @mighty yew could comment on any major worries or implications, then I could try to do a review for details. I mostly didn't want to jump in on something I wasn't an expert on. And if smaller bits could be introduced first independently, that always makes review easier.

mighty yew
#

I still have this on my TODO

#

but I pulled back on oss work for the past few weeks

#

feel free to open smaller PRs, but ideally I would like to merge this sometime

tidal bramble
#

That PR has been the biggest waste of my time in open source

severe tundra
#

@bronze aurora approved it, I think we were all just waiting for @mighty yew to approve.

mighty yew
#

yes, sorry for not being able to get to it

#

I don't think it's wasted work though

#

I will try to pick it up

jaunty matrix
#

are we sure pip doesn't want to use build?

severe tundra
#

Please do, I'd be happy to reopen and merge once you review, personally. 🙂

warm mulch
jaunty matrix
#

@warm mulch are they enumerated somewhere?

warm mulch
# jaunty matrix <@!548380029667508224> are they enumerated somewhere?

https://github.com/pypa/pip/issues/6264

Plus these test failures indicating venv does not work as a drop-in (may overlap with issues mentioned in the other issue) https://github.com/pypa/pip/pull/10720

GitHub
Virtualenv with --system-site-packages breaks pip's build isolation...

It seems that if you create a virtualenv with --system-site-packages, the system packages (but not the user packages) will be on the PYTHONPATH in the PEP 517 isolated build environment that pip cr...

GitHub
Try switching to venv for isolation by pradyunsg · Pull Request #10...

I wanna see how difficult this would actually be, by seeing how the CI reacts to this.
Closes #6264, hopefully.
Closes #7953, hopefully.

tidal bramble
#

The PR was created specifically to allow pip to use build, it was born out of the issue that you linked.

warm mulch
#

I know

tidal bramble
#

Would there have remained any blockers?

#

I’m not picking it up again but if FFY00 does he might like to know.

warm mulch
#

Someone needs to do the integration work? 😛

#

I don’t think there are technical blockers, but I also may have missed something, pip has way too much compatibility baggage to keep track of

mighty yew
#

maybe I can get $dayjob to give me some hours for this if the pip maintainers think it is valuable and a good path forward

#

but I have other stuff to do too, but I would be much more productive there than whatever I am doing right now, so I don't think it would take that much of my hour budget

#

but it would from my weekend + mental health if I do it then, so I have to find a time where it makes sense for me

warm mulch
#

Don’t push yourself too hard on this; someone still needs to write code in pip to integrate build. We are all in a similar situation and can’t promise things will happen when.

mighty yew
#

it seems like a good candidate for the packaging sprints maybe

#

I have time allocated for that, as I am attending pycon

worldly spoke
#

should see if I can squeeze a trip to PyCon - It's only 1 state away for me for a change

mighty yew
#

it's an ocean away for me 😅

#

but I was able to sort it out with my employer

#

so, yay

bronze aurora
#

A bit more than that, Salt Lake city being mostly west coast 😂

worldly spoke
#

Can I drop installing the wheel dep now?

warm mulch
#

Yes you can drop it, build populates a separate venv to build the package anyway so the globally installed wheel is no longer useful

#

Probably want to also add a pyproject.toml to specify minimal configuration

[build-system]
requires = ["setuptools"]
dim needle
#

Will do (I also noted this in my review lol) that, thanks!

worldly spoke
#

Cheers

violet pebble
worldly spoke
#

Nice, can't hurt. And I guess it should only improve.

dim needle
worldly spoke
#

I got told things that the build docs don't mention at all today. We should address that I feel.

warm mulch
mighty yew
#

the PEP text is not very clear, so in practice some people will expect a default value 😕

#

sorry, no, requires is mandatory

#

build-backend is not

#

I do not think this was good but 🤷

warm mulch
#

IIRC that was explicitly designed, least friction or something (I was not involved so it’s second hand info)

dim needle
#

I'll add to bandersnatch's configuration anyway since there's no harm in putting it there

mighty yew
#

also note that the default backend is the legacy one

dim needle
#

yup yup I alluded to that above but good to confirm 😄

mighty yew
#

so always better to specify

mighty yew
#

I did not consider that

#

but still, I feel like we would be better off in the long run without this behavior

#

now we are stuck with it

#

and it is a footgun IMO

dim needle
#

I guess technically putting wheel in build-system.requires is redundant since setuptools will append wheel anyway to the build deps

mighty yew
#

yes

#

I suspect some older versions might not do that, but I am not sure

dim needle
#

Fair, I'll keep it then unless someone yells 🙂 I use flit for my personal projects anyway so I don't really know what's best-practice with setuptools :/

mighty yew
#

it is a good idea to give a minimum version, so you can just verify that the one you pick does that

warm mulch
#

Otherwise the two backends are identical though so defaulting to the legacy backend does not really matter, especially if you don’t use setup.py at all

dim needle
dim needle
warm mulch
#

Yeah

mighty yew
#

a good version would be 42

dim needle
#

I realize I have an open PR for this which I've ignored for the past few months, uhh tbh I haven't been motivated to work on it since I get the vibe it has to be rewritten to be merged (since I chose the "stick a .pth in site-packages" approach which was met with division)

#

I mean also it also needs changes for the legacy backend but that's secondary to design issues 🙂

warm mulch
#

Some meaningful-ish setuptools versions if you want a lower bound:

  • 36.6.0: PEP 517 support added
  • 40.9.0: setup.cfg-only support added (i.e. use this if you don’t include a setup.py in the project)
  • 42.0.0: Ditched easy_install
  • 43.0.0: Automatically include pyproject.toml in sdist MANIFEST
  • 45.0.0: Dropped Python 2
dim needle
#

I already made the change on the PR but thank you anyway! I went with 42.0.0 by suggestion of FFY00

warm mulch
#

If you go with 42 remember to add include pyproject.toml in MANIFEST.in to avoid surprises

dim needle
#

yeaa I just realized that and was in process testing out whether the generated sdist is installable (edit: I'm dumb and tested this in the most wrong way possible, anyway I'll fix it) since we definitely did not add pyproject.toml to MANIFEST.in

worldly spoke
#

What can we do to make pyproject.toml included by default? Feels like it should 😕

warm mulch
#

Is it already since setuptools 43.0.0?

#

If not it should be reported as a bug

arctic briar
severe tundra
#

Anything before setuptools 42 doesn't properly support PEP 517 - I've been bit by that before. You also have to include wheel, not sure if that was mentioned.

#

There's a huge PEP 621 PR (with a broken-into-six-stages version) in setuptools. It also really needs the smarter default package discovery PR, IMO. I've done a little testing in scikit-hep/cookie, and it's working.

dim needle
#

oh did someone make an alternative to my (terrible) PEP 660 PR? if so that's great

#

hmm I don't see one, I see it's for PEP 621 instead

#

It does come with the advantage being written by someone who 100% knows way more than me though so yeah for whatever that is worth.

thin nimbus
dim needle
#

a file extension that if placed in site-packages will extend sys.path (used to implement editable installs for setuptools atm) with its contents IIRC

thin nimbus
#

ah, neat!

tidal bramble
#

per Dustin's email, I suggest disabling all "tox" tests in CI

jaunty matrix
#

why?

mighty yew
#

can't we just ask github to up the limit for the org?

jaunty matrix
#

(I have no knowledge of the email)

mighty yew
worn token
#

The email was motivated by setuptools recently adding some very over the top integration tests 😅.

But @mighty yew idea is actually very good. Even after the new integration tests for setuptools being prevented to run for every commit, it is difficult to provide the same developer experience + quality of packages without doing the tests against all platform combinations... So asking for increasing the limit is a nice thing.

Is there any way in GitHub to run a subset of the tests just when the "merge" button is activated and aborting the merge if it fails?

mighty yew
#

I think it could be possible with automerge

#

I don't know if it creates an event

tidal bramble
#

I still don’t think we need quite so many tests or runners, the test suite takes an hour to complete

#

And I don’t see what purpose the tox tests serve

worn token
#

(It is good to have these tests scoped to the PR instead of running them after the merge, because the author would still be involved in the process, and protects the main branch from failures that would require reverting things)

jaunty matrix
arctic briar
violet pebble
# mighty yew https://mail.python.org/archives/list/pypa-committers@python.org/thread/PCBCQMJF...

Only run CI workflows on PRs and pushes to the default branch
Without this, workflows will be triggered for pushes to branches that do not have corresponding PRs, and pushes to branches that do have corresponding PRs will run each workflow twice.

An example to add to your workflow:

on:
  push:
    branches:
      - main
  pull_request:

I find this branch restriction unfriendly to contribution: it adds an extra impediment in that I can't easily test feature branches my own fork

I wrote more on it here: https://github.com/python/typeshed/issues/7066#issuecomment-1025188218

#

as it happens I saw this yesterday, would this solve it? by allowing forks to build pushes and only doing a single build for upstream

    # We want to run on external PRs, but not on our own internal PRs as they'll be run
    # by the push to the branch.
    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository

https://github.com/has2k1/plotnine/blob/master/.github/workflows/testing.yml#L10-L12

severe tundra
#

Inspired by @jaunty matrix 's work in https://github.com/ofek/hatch/pull/136, I'm going to see if I can get build's CI working properly. If so, I think we should probably do a release and then convert to Flit.

worn token
severe tundra
#

@jaunty matrix is the expert at the moment. 🙂 Maybe once I get this working. Spending pretty much the whole morning getting my windows box updated - pretty sure it's going to go 10->11 as soon as I start running again. I tried to run the build test suite then MSVC's update restarted my computer (without asking)...

#

winget is nice, especially since it's built in now, but Windows apps don't have nice update experience, they pop things up all the time so you can't work while it's running

worn token
#

Oh yes, trying to running tox/nox on windows is also fun! (lots of pop ups for each subprocess or network permissions)

jaunty matrix
#

I'm kind of forced to use Windows because of assistive technology offerings, but I think it's awesome nowadays 🙂

severe tundra
#

python -IM ensure-up --upgrade --default-pip is failing with no module named encodings - this is from venv, I think.

severe tundra
#

Windows

severe tundra
#

Ahh, finally found that at least some of the tests pass if I avoid tox. Possibly pypy on windows is buckling under the nested virtual environments?

mighty yew
#

humm

severe tundra
#

Ahhh, known issue, I guess the skips are not being applied properly

# 
PYPY3_WIN_VENV_BAD = platform.python_implementation() == 'PyPy' and os.name == 'nt'
PYPY3_WIN_M = 'https://foss.heptapod.net/pypy/pypy/-/issues/3323 and https://foss.heptapod.net/pypy/pypy/-/issues/3321'
#

Found ways to improve the error messages though 🙂

#

Is there a good way to detect running in tox?

#

I should only need to apply this skip if running in tox (just in a single regular venv is fine, interestingly, since I'm always running in a venv - just not the one tox sets up)

severe tundra
#

I should be able to just try to make a venv in the conftest, and if I can't, then that means build can't. :) I'll finish this up tomorrow, and then split up my changes into several small PRs; some cleanup, nicer error messages, and 11 skips for PyPy Windows. I've tried patching in venv.EnvBuilder as pytest.skip, but that overskips, so it's still only happening in some places (so my idea above won't work properly)

#

Next PyPy release is in 10 days, wonder if we might be able to get a fix in...

severe tundra
#

Should be fixed in PyPy for the next release.

jaunty matrix
#

fyi for platform conditionals there is special recognition in Mypy for sys.platform, not os.name

severe tundra
tidal bramble
jaunty matrix
#

correct

tidal bramble
#

Is that a recent development?

jaunty matrix
#

forever afaik

severe tundra
#

If you only supported one, I'd definitely go with sys.platform, more detailed. Though it seems like they could support both. I'm fine with a standard, though, and sys.platform is a better standard because you can tell macOS from Linux. 🙂

#

For pyodide, os.name is posix (? guess that makes sense) and sys.platform is emscripten (much better).

tidal bramble
#

is a behaviour change

#

I'm not sure if it actually matters, I assume cygwin doesn't report nt for os.name anyway

severe tundra
#

cygwin returns posix.

#

Not sure about msys2.

#

I think it is a change on msys2, where it does report nt but returns msys

severe tundra
tidal bramble
#

I haven’t had a chance to look at the new output, I’ll do that tonight

severe tundra
severe tundra
#

For reference, Pyodide's build system currently is rather like a all-at-once conda; it has to manually specify packages in a "run" field (soon to be split into build/run or just build), then topologically sort and builds everything. They should be able to use pure Python wheels, but built packages have to be compiled. They've just reworked the system from a "record and replay" system (yuck) to a "intercept and modify" (much better), so it's a more reasonable cross compile system.

#

By switching to build, besides supporting non-setuptools, they are also moving to something more "normal" with less manually specified dependencies, etc. They still have to have some control over build requirements - if "numpy" is requested, they have to provide the version they have built locally, not whatever is pinned, etc.

#

I think most of the private internals were just to mimic the output, which it fine to reimplement, IMO.

tidal bramble
#

It would simplify a few things but the main problem they are having appears to be that there’s no way to pass a different environment builder to the CLI and I guess they don’t wanna monkey patch it?

severe tundra
#

Increasing the bus factor on PyPI would be nice, too 😉

#

I think we should start thinking about a 0.8.0 - maybe right after the next Python 3.11 alpha is out? I'd like to try the move to Flit just after 0.8.0.

warm mulch
mighty yew
#

depends entirety on the backend

#

if you are building a sdist anyway, it is basically just the extra unpacking time

#

if you are not, you also add the time it takes to build the sdist

#

but I don't think there are any problematic backends TBH

#

building a sdist is mostly just packing anyway

tidal bramble
#

The only thing that’d cause a noticeable slowdown is not reusing the build env. I don’t actually recall if we do that in build

mighty yew
#

we don't

#

the dependencies might be different

#

a possible optimization would be reusing it and adjusting the environment if needed

dry mural
#

would it be possible to add an options for backends that ensure correct sdists to say that the build should go directly from tree to wheel? There are some cases where you need to call external tools for building sdists where it makes a difference

jaunty matrix
#

for large projects it would be a massive slowdown

#

we package an app/monorepo as a wheel and builds take ~30-40s

warped citrus
#

hey everyone, python -m build --outdir .buildwork pollutes my src folder with an *.egg-info directory

#

can I ask it not to do that?

tidal bramble
#

that's setuptools' doing (i.e. the backend's), I don't think you can opt out of it

warped citrus
#

ah, thanks!

bronze aurora
#

Switch to hatchling and you no longer get that

arctic briar
#

Or flit. Or poetry.

#

Basically anything except setuptools. XD

mortal steppe
#

I am trying to push a package to pypi with github actions. every time I run the action, build tries to build version 0.0.1 of my project. Whenever I try it locally, build builds 0.0.2 I already have 0.0.1 on pypi and so my builds keep failing.

# 
import pathlib

HERE = pathlib.Path(__file__).parent

README = (HERE / "README.md").read_text()

VERSION = "0.0.2"

setup(
    name="logs4thelazy",
    packages=["l4l"],
    version=VERSION,
    license="MIT",
    description="A simple and easy to use library for logging in python",
    long_description=README,
    long_description_content_type="text/markdown",
    author="Yosi Frost",
    author_email="yosi_frost@icloud.com",
    url="https://github.com/FrostyTheSouthernSnowman/logging4thelazy",
    download_url="https://github.com/FrostyTheSouthernSnowman/logging4thelazy/archive/v_0.0.2.tar.gz",  # I explain this later on
    keywords=["logging", "python3", "simple"],
    install_requires=[],
    classifiers=[
        "Development Status :: 4 - Beta",  # Chose either "3 - Alpha", "4 - Beta" or "5 - Production/Stable" as the current state of your package
        "Intended Audience :: Developers",  # Define that your audience are developers
        "Topic :: Software Development :: Libraries",
        "License :: OSI Approved :: MIT License",
        "Programming Language :: Python :: 3",
        "Programming Language :: Python :: 3.4",
        "Programming Language :: Python :: 3.5",
        "Programming Language :: Python :: 3.6",
        "Programming Language :: Python :: 3.7",
        "Programming Language :: Python :: 3.8",
        "Programming Language :: Python :: 3.9",
        "Programming Language :: Python :: 3.10",
    ],
)
#

python-publish.yml

# 
# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries

# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.

name: Upload Python Package

on:
  release:
    types: [published]

permissions:
  contents: read

jobs:
  deploy:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3
    - name: Set up Python
      uses: actions/setup-python@v3
      with:
        python-version: '3.x'
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install build
    - name: Build package
      run: python -m build
    - name: Publish package
      uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
      with:
        user: __token__
        password: ${{ secrets.PYPI_API_TOKEN }}
violet pebble
#

do you need to git push --tags as well? and maybe only do the deploy on tags

mortal steppe
#

the action was triggered when I created a new github release. Does github not do that automatically?

severe tundra
violet pebble
mortal steppe
#

I created the tag and the release on github

severe tundra
#

The problem was that the version was not changed in the files before tagging/releasing. You can use various tools for pulling the version from git (setuptools_scm, several other backends have support too). But if you manually update, you have to do that before you make the release in GitHub (which creates a lightweight git tag if you type in a new tag name).

mortal steppe
#

Ok

#

Thank you

tidal bramble
#

I wonder if we want to start bundling tests in the sdist

#

all the distros I looked at grab build from GH I presume for this reason, that there are no tests in the sdist

#

even those that generally prefer PyPI as a source (e.g. nixpkgs)

severe tundra
#

I almost always bundle tests in SDist (and when iMinuit dropped tests from the SDist, and then distro packagers had a fit and they were readded). Tests shouldn't be in the wheel, but they are very useful in the SDist.

#

(I also had a fit, since I needed then for conda-forge, but the lead developer of iMinuit didn't care about what I thought, it took debian or something like that to change his mind)

strange sage
#

Why shouldn't they be in the wheel? Or is this just the old "do you want to be able to run tests in production" disagreement

tidal bramble
#

generally tests don't reside inside the package - we don't want people shipping a top-level "tests" folder

#

that gets installed

strange sage
#

Ah, ok.

mighty yew
#

sometimes shipping tests might make sense, but it depends a lot on the use case

mighty yew
#

we also have signed tags, which some distros like arch use

onyx coral
#

sdists or upstream will vary wildly on the distro, and sometimes the package itself

lament onyx
#

imho its the gold standard to include the testsuite in sdists
that being said, stuff like sqlites additional extensive e2e testsuite should indeed ship in separate bundles ;P