hatch

[tropic void]

@umbral bridge did i miss any way to put config overides for hatch/hatch-vcs into a github action (test-pypi uploads with dev versions are a bad match)

ah wait, the setuptools_scm one is still respected

[umbral bridge]

@turbid bane https://github.com/ofek/hatch-mypyc/releases/tag/v0.15.0

[umbral bridge]

I'm open to feedback here https://github.com/pypa/hatch/issues/701#issuecomment-1374595003

[tropic void]

@umbral bridge i beleive im missing how to spell what extras to install in a environment (i want to have the test/dev extras of a package in the test env

[umbral bridge]

https://hatch.pypa.io/latest/config/environment/overview/#features

[tropic void]

🤦‍♂️

@umbral bridge i hope https://github.com/pypa/hatch/pull/710 will spare me that miss next time ^^

@umbral bridge any opinion on providing a set of reusable workflows that encapsulate the flow of build artifacts, test them in a matrix that matches the hatch envs, and finally after all pass uploading the artifacts to pypi/test-pypi

[umbral bridge]

I've never implemented something like that but that sounds quite achievable with GitHub Actions syntax

[tropic void]

https://github.com/asottile/workflows/ has an example (albeit necessaryly icky in some bits)

the idea is coming up as im removing more and more tox usage, tox 4 is as icky as tox3 but has breaking changes so i'm just dropping it every time something goes wrong

[tropic void]

@umbral bridge is there any way in hatch to replicate tox envlist running different sets of environments like tests (matrix) + linting ?

[umbral bridge]

not yet no, unless you have a script that is a series of hatch run invocations

[tropic void]

i created https://github.com/pypa/hatch/issues/711 for now

[tropic void]

@umbral bridge perhaps it makes sense to investigate the idea/concept of a "pipeline" which links environments and artifacts (like build wheels for different platforms, then test against them, have linting and doc building in parallel

also computed matrix values wold be nice (like all supported pypy/python versions)

[umbral bridge]

that would be useful but I don't think it would be a Hatch feature as that would be better suited for CI

[tropic void]

a key problem with most ci is that one cant safely test pipelines tho ^^

but yeah, its possibly also overreach

[umbral bridge]

also as far as the issue you opened I do plan on addressing that use case but just so you know tox and Hatch environments are conceptually very different

tox thinks of environments as "executable" whereas Hatch thinks of environments as a workspace where you can execute arbitrary commands

[tropic void]

im will aware, envlist as in tox makes no sense when facing matrixes and scripts as part of the feature set ^^

[tropic void]

@umbral bridge aware of a nice mapping between github python names and hatch python names (pypy3.7 vs pypy-3.7)

[umbral bridge]

no but you can create an environment collector plug-in https://hatch.pypa.io/latest/plugins/environment-collector/reference/

which you could first easily test with a custom environment collector https://hatch.pypa.io/latest/plugins/environment-collector/custom/

[tropic void]

that woud be a pain to replicate/integrate - is there any other way to pick the python for the matrix

well, for now, lets start with a python -c command and its output and cry

[radiant cloak]

@umbral bridge What's needed to make https://github.com/pypa/hatch/issues/507 possible?

[umbral bridge]

basically adding a dependency on the PEP 517 library and having a conditional that if the build backend is not Hatchling then use that

[low gyro]

PDM is backend-agnostic

but I don't get a reason to do so

[umbral bridge]

yeah I kind of don't see the purpose

alternatively, what I could do would be to introduce the concept of hidden managed environments and just have an environment that provides build that can be invoked

I think that would be preferable so I don't add another dependency that would be rarely used

actually that is a super interesting idea because that solution would overlap with something else people have been asking about which is top level commands similar to other package managers like Cargo that have for example fmt

[low gyro]

You don't have that for plugins?

[umbral bridge]

in that case, the hidden environment could have opinionated versions of for example Black

[umbral bridge]

what do you mean?

[low gyro]

I mean that hatch plugin would provide new command for hatch

[umbral bridge]

oh I see, no actually there is no current plug-in type for adding new CLI commands

I could add that but the implementation would make me a bit sad lol right now the main command loads configuration and creates an application object that gets passed around with the plug-in manager and other stuff for sub commands. if I were to allow the CLI to be extended then the plug-in manager would have to execute first outside of Click main command

essentially that means I'll have to use/modify global state, which I'm not a fan of but there is no other way to do that

[low gyro]

I see

[umbral bridge]

you each have given me a new task for my never-ending todo list 😅

[low gyro]

I know what you mean

I have a similar list for stuff in Poetry 😄

[umbral bridge]

I don't have time right now but I'd like to transfer everything from this text file to a project in the repository so people can see the roadmap/help

[low gyro]

that calls for a migration script! never do something for 10 mins if you can spend 10h trying to automate it 😄

[radiant cloak]

To do what?

Be backend agnostic?

[low gyro]

yeah

[radiant cloak]

I can think of one: Using hatch's CLI with meson-python today.

But, I'll wait to hear what you're typing. :)

[low gyro]

I mean, backend-agnostic solution is build. Projects like Hatch or Poetry give more "specialized" options

[low gyro]

what would be the use case here? I don't know Hatch inside-out, but I assume it is somewhat chained to the backend

[umbral bridge]

I think what @radiant cloak is saying is that if Hatch is to ever become a/the main tool of a workflow then you should never have to reach for another CLI

[low gyro]

there is still a lot to be done before (if ever) Python gets it's CLI tool

[radiant cloak]

Well, I'm not saying that. Status quo is actually enforcing a point that I'm making in my blog post for https://discuss.python.org/t/python-packaging-strategy-discussion-part-1/22420/

[radiant cloak]

... like?

[low gyro]

like, there is currently no one tool I would point to as a good candidate

[radiant cloak]

nods.

[low gyro]

I'd say that taking language extensions from setuptools, plugins and envs from hatch and dependency solving from poetry and you get some kind of base line

and still having pip as low-level tool

[umbral bridge]

I think the only missing piece for Hatch is a lock file, but that is a massive missing piece that I am not well-positioned to help Brett on and kind of blocks the creation of associated commands that would exist if there was a lock file

[low gyro]

oh, and of course, something for workspaces

[umbral bridge]

(the other missing piece is extension modules for Hatchling but I don't want to talk about that anymore...)

[low gyro]

I wonder how hard would it be to extract extension modules logic from setuptools into separate package, which could be used by every tool equally. and used by tools like mypyc to be more universal.

[umbral bridge]

that was actually my original idea to also expose utilities for building but Henry (who knows much more than I do about that) said that that is not the way but rather every thing that can compile would have its own plug-in

[low gyro]

and of course, logic for extension modules should be easy to express via pyproject.toml, without a need for .py script

we had a PR for this in Poetry

[umbral bridge]

I don't want to hold myself to this timeline but I'm thinking if in 6 to 8 months there's no movement on extension module stuff then I can just continue the path of what I did for Mypyc and just make a plug-in for each thing like Cython which would have nice static configuration but would under the hood use setuptools

that would be such a bummer though

[low gyro]

yeah

also I don't really like the idea of making a facade for setuptools everywhere

[umbral bridge]

neither do I but I guess it's a trade-off between that and not allowing users to do what they wish with another build backend

[low gyro]

kinda

[turbid bane]

There were ideas for mypyc to gain its own compilation driver (for direct x86-64 ASM generation), but I highly doubt that will ever happen or at least not in the remotely foreseeable future.

mypyc is still in the "let's not crash constantly or run into unsupported python features" stage of its development 😅

[low gyro]

from what I have heard, mypyc lacks hands to work on it with enough knowledge on C etc

[turbid bane]

Yeah. All of the people most experienced and skilled to work on mypyc have regular day jobs and can't dedicate too much time to mypyc. I'm slowly working towards becoming a mypyc core dev (got the triage bit a few days ago!), but my relative lack of C experience is still a major problem.

[golden merlin]

Hatch doesn't replace pipx at the minute either right? That's another "major" tooling piece that some unified tool presumably needs a really good version of.

[umbral bridge]

yes that is true but I would never ever reimplement its logic, instead I would depend on it

[golden merlin]

Yeah sure, however the functionality ends up getting exposed yeah, even if it's just wrapping it pretty directly.

[low gyro]

I wouldn't say that there are 3 separate tools. pip, pipx and unnamed-unified-project-tool

all have their use

[golden merlin]

Today? Or in the future?

[low gyro]

future

I don't see a point in having all 3-in-1

[golden merlin]

Oh, in the future I'd hope the point of unnamed-unified-project-tool is precisely to have 3-in-1

[low gyro]

the UI/UX would be a mess

[umbral bridge]

I think we have to differentiate between "tools" and user facing tools

for user facing tools I think pip would never be invoked manually ever in the scenario in which we have a unified tool

I think the future state is similar to Rust where there is a unified packaging tool and another tool just meant for managing Python

[low gyro]

pipx for global installs
"unified" for projects (both libs and apps)
pip for those who are special snowflakes or need low-level control

[umbral bridge]

actually perhaps that tool meant for managing Python would subsume pipx

[low gyro]

like pipx built into pyenv?

[radiant cloak]

FWIW, one of my arguments is that pip is special within the Python packaging ecosystem as the only user-facing CLI tool that ships with Python; and we should aim for it to be the unified project workflow tool.

It's a long way to get there, and there's a bunch of pieces to figure out -- but that's likely the most tractable way to get to a point of having a unified tool.

[umbral bridge]

that is interesting, that runs contrary to what folks were saying in that thread. I think Paul specifically said that the tool would not be pip

[low gyro]

well, pip is shipped with Python, so that doesn't mean that future unified tool couldn't be shipped as well

pip is nowhere near being the project workflow tool

[radiant cloak]

I know. I'm not Paul, and that is a strategy discussion. :)

[turbid bane]

It's more likely the $new-unified-packaging-tool becomes the chrome of the desktop world, you use the preinstalled edge to install chrome.

[low gyro]

so something like today state of matters

[turbid bane]

uhh, although there's no one unified tool right now that most people can agree on, no?

[low gyro]

yeah, but if we take PDM/Hatch/Poetry as the $TOOL, that's the same

[umbral bridge]

also if that were to happen that would be a major backwards incompatible change assuming we wanted to go for reasonably named commands. otherwise we have to add weird commands or under a top level command group that managed virtual environments

my personal advice would be to let pip continue to excel at dependency management and everything that that entails and let others work on UX in other tools 😃

[turbid bane]

So are you suggesting we just take a pre-existing tool and market it as the tool and put into the core distribution today-ish? I suppose that's less far off than I thought since hatchling did get into the PUG, but I can't help but remember the pipenv mess.

[low gyro]

kinda. I don't think any currently available tool is good enough, that would be another tool all together

[radiant cloak]

I'm gonna cop out of this until I post my thingie on the the discuss.python.org thread.

[turbid bane]

Agreed.

[umbral bridge]

I agree currently, but after Hatch supports lock files and the associated commands, would that be good enough in your opinion?

[low gyro]

Don't know. Depends on the implementation. Also, does Hatch support dependency resolution? Or do you pack it as part of lock file system?

[umbral bridge]

Hatch uses pip

[low gyro]

so no

[radiant cloak]

IMO no, because the environment management isn't sufficiently flexible/powerful for many workflows.

And, at least today, pip doesn't have a multi-platform lockfile story either.

[low gyro]

correct me if I am wrong, but dependency resolution in pip doesn't take into consideration already installed packages, right? only resolves dependencies passed within one call?

[umbral bridge]

can you please explain that? Hatch supports everything tox/nox do including plug-ins and extra granular modifiers

that is the part of Hatch that I've thought about most so it's surprising that there are cases in which it falls down

[radiant cloak]

@nox.session
def lint(session: nox.Session) -> None:
    session.install("pre-commit")

    if session.posargs:
        args = session.posargs + ["--all-files"]
    else:
        args = ["--all-files", "--show-diff-on-failure"]

    session.run("pre-commit", "run", *args)

I couldn't figure out how to do the equivalent of this with hatch, for example. It might just be my ignorance tho?

[umbral bridge]

https://hatch.pypa.io/latest/config/environment/advanced/#context-formatting

e.g. https://github.com/pypa/hatch/blob/f43a338a7aa664ae085e57600eedf2f6e8b62ed6/hatch.toml#L55

[low gyro]

@umbral bridge btw why does Hatch have additional hatch.toml file? why not just pyproject.toml?

[umbral bridge]

as documented you can do either https://hatch.pypa.io/latest/intro/#configuration

[low gyro]

yeah, I know, just wondering why you added the 2nd file option

[radiant cloak]

Less typing, IIUC?

[umbral bridge]

at work some tox files were a few hundred lines...

[radiant cloak]

Well, that works too.

😄

[low gyro]

ah, kinda makes sense

I wasn't thinking about envs as tox replacement, just a command runner

like npm scripts

[umbral bridge]

yeah we just finished switching one of our monorepos to Hatch from tox

[turbid bane]

I really liked the flexibility of nox, but it has been on my to-do list to try hatch envs

mostly because I'd like to fix black's contributing workflow story

[low gyro]

for me, GH Actions replaced need for tox mostly

[turbid bane]

Yeah I get that. I like nox/tox mostly because it presents an unified and predictable way to run common development workflows. It also hides away dependency management which is a nice plus.

[umbral bridge]

that is generally true for me as well specifically for the Python version part

for other matrix factors like product versions or certain options I still put in Hatch, the CI just provides the Python version difference on each runner

[low gyro]

well, you could put other options in CI matrix

[turbid bane]

It's quite nice to be able to run a CI-like job locally though. When I used to work a side project a while back, I'd use nox to test multiple Python versions at once locally. Coverage and other things are also easier.

[low gyro]

true

[umbral bridge]

if you have the money for that many concurrent runners 😉

[low gyro]

however I'd say 99% of time, working on minimum supported version works just fine...

[low gyro]

true, didn't think about it

[turbid bane]

That is fair. I've stuck with the same Python 3.8.5 installation that's over two years old.

I only create environments with a different version if I have to test something that's version specific

[umbral bridge]

3.8.1 for me 😂

[low gyro]

still on 3.7

[turbid bane]

I have a Python 3.9.5 install ready to go, I just haven't made the switch yet haha

[radiant cloak]

... merely maintaining foundational pieces for the ecosystem, that's all. security isn't that important, right?

[low gyro]

I mean, I have 3.10 as default, but I use 3.7 for projects

[radiant cloak]

what's a few publicly known CVEs between friends?

[turbid bane]

TBH when I first set up this laptop, I was still in the "don't touch system python at all under any circumstances" camp. All of my Python installations have been built by hand. 😅

[crystal idol]

With QtPy, I store all the matrix info (platform, Python version, conda/pip use and Qt/Binding versions for each binding) in the declarative GHA workflow file https://github.com/spyder-ide/qtpy/blob/master/.github/workflows/ci.yml, and then have GHA handle the first three and have a bash script shim that handles testing each version of each Qt binding on the runner via Conda environments https://github.com/spyder-ide/qtpy/blob/master/.github/workflows/test.sh . The idea is to keep all the version information in one single-sourced declarative matrix (though there are a few bits that crept into the script that I need to factor back out). Not the most proud of it, but it works. Do any of the major environment management tools support conda environments?

[low gyro]

Poetry works with Conda from what I know

(however I never really understood a purpose of conda/anaconda)

[umbral bridge]

Hatch has a plug-in for Conda environments https://github.com/OldGrumpyViking/hatch-conda

[crystal idol]

Eventually I'll probably help the Spyder projects migrate to Hatch as the build system for PyPI packages, since almost all the stuff we actually maintain ourselves is pure Python (though we of course depend heavily on binary packages). But right now there's enough work just to migrate off other legacy packaging stuff project by project...dynamic setup.pys, PEP 518, PEP 621, pkg_resources -> packaging, importlib resources/metadata, other deprecated import constructs, other packaging metadata, etc, plus adding/updating standardized CI infra, pre-commit, etc. What I really want to do is develop a configurable, comprehensive cookiecutter template with all of it and use Cruft (or similar) to manage it, which would make future upgrades (e.g. to Hatch) much simpler, but we're a ways away from that now.

[low gyro]

well, it's nice that we can finally weed out the setup.py files everywhere they are not used for extensions

since GH now supports pep 621 graphs

[crystal idol]

For one, managing binary dependencies (including Python itself) in a cross platform, compatible fashion, which are a ubiquitous issue in scientific Python thus it's extensive use in anything related to that. It's sorta like Nix, but specifically for Python. The recent mega-thread goes into some detail https://discuss.python.org/t/python-packaging-strategy-discussion-part-1/22420/

[crystal idol]

It's a long, slow process of course, given the layers of cruft that have built up over time to paper over previous deficiencies, and the inherent unwilling to fix "what isn't broken"

[crystal idol]

Yup, just saw that, but Poetry still doesn't (unlike every other backend now)...is there a new ETA on that? Last I heard there was at least a plan to do it for the next major version.

[low gyro]

that's my personal pain, there is no ETA, but I plan on working on getting it out there when I have some time of $LIFE and $WORK

I have like 3 pages of TODO list concerning Poetry 🤣

[orchid birch]

I'd really love a comprehensive comparison of modern packaging tools like hatch vs pdm+tox vs poetry+tox vs ...? I'm actually working on something like that at $work since we're also on the modernizing kick, but 1) we have different requirements than in open source; 2) not sure I can share the outcome of that research. Maybe such a thing already exists?

[tropic void]

well, define different requirements, its unclear if you speak in magic secret sauce language ^^ its not like you have to disclose a blueprint of a defense weapons project ^^

[orchid birch]

That’s true. I’ll see about distilling our internal discussions into something that can be contributed to in public.

[cursive lion]

Adding some sort of summary/comparison to https://packaging.python.org has been suggested a couple of times in the big discussion at https://discuss.python.org/t/python-packaging-strategy-discussion-part-1, and I agree it would really help people decide what tools to use

[crystal idol]

On the other hand, since it necessarily involves a degree of subjectivity, opinion, and judgement of directly competing tools, and most PyPI members have some involvement in at least one of the tools, I worry it could invite controversy and disagreements. I was going to respond to Barry with a brief one-line summary of the main advantages/use cases of each major backend, but even as a individual message on a semi-private Discord from a technical spec writer with no direct involvement in any of the projects, I didn't really feel it was my place to do so.

[orchid birch]

If we do it, we can try to be as objective as possible, and not judge a project on its choices or roadmap. It's still possible people will debate this or that feature, and why wasn't their favorite package manager included in the list, etc.

As an example, we could say, "CLI command to manage, update, add, and remove dependencies". Some projects have that, others don't. That's a statement of fact, though you could say Project X has that feature on their roadmap.

[shut mantle]

reminds me of this table: https://twitter.com/nletcher/status/1496051932414504961

[orchid birch]

Darn. It looks like my packages, which previously built, tested, and uploaded w/pdm+tox are now failing to built locally with no code changes. Something in all the toolchain updates is causing the build to fail.

I'm mentioning it here just because rather than try to figure out what's going wrong, it might just be time to do the planned conversion to hatch on these projects.

[umbral bridge]

lmk if I can help, as you know I'm always online 🙂

[orchid birch]

Thanks, will do! How about hatch migrate-from --pdm 🤣

[low gyro]

both projects use pep 621 so it shouldn't be that hard 😉

and you could always write "quick'n'dirty" translation script

[orchid birch]

Migrating, even manually, shouldn't be too hard, although I also have to integrate GitLab pipelines for CI. Probably the biggest mental mapping is between pdm+tox -> hatch environments.

[umbral bridge]

oh it's missing a WHEEL file, that is bad indeed lol

to keep it simple you can retain tox for now and just change the build system to Hatchling

[orchid birch]

Good point. I can do the migration in steps.

Maybe it's worth trying to pin pdm versions to see if an upgrade to that tool (or its backend) broke it. Then again, maybe Frost will Just Fix It

[low gyro]

PDM is backend agnostic, so you could just switch the backend declaration and wait for fixes to PDM

[tropic void]

@umbral bridge i came to the conclusion that hatch-sumo will not be sensibly doable as potential tool for bootstrapping hatch itself,

Im wondering if there Could be a feasible solution that wraps the build env in such a manner that Metadata can be avaliabe using a multi stage process and git checkouts

Aka start with checkouts + pythonpath to get to editable wheels with wrong Metadata but enough to get the next iteration to have correct Metadata)

[umbral bridge]

that sounds complex, I would need more of a write-up to understand

@tropic void fyi https://github.com/platformdirs/platformdirs/issues/128#issuecomment-1384159191

[low gyro]

Poetry awaits that new package too 😄

[tropic void]

@umbral bridge had them on pluggy a well

im currentl ver stretche, the focus atw ork currently leaves no room for setuptools_scm and i gto 2 kids these days ^^ - that being said around apri i'll get more active

@umbral bridge as for the bootstrapping, the idea is to have a setup where a python + importable folder of all the bootstrap needed packages can be used to generate the editable wheels of all of them as a starting point (with potentialy incorrect metadata for stuff that needs entrypointss)

after the partially broken wheels are maked, one can make a python env with those editable but broken installed
this env can then be used to build full artifacts

for extra fun, after building all the full artiacts, a zipapp that can build any of the apps should be bundled for downstreams

@umbral bridge is tht enough detail?

[umbral bridge]

yes I think I understand and you probably would be able to do that currently with a Hatchling builder plug-in

[tropic void]

@umbral bridge what do you mean with that in detail ?

[low gyro]

also "distro packagers strike again"... I will never understand their motivation...

[tropic void]

my understanding of the mappignsi s a bit limited, but i am of the impression thathere is a lotof potential footguns when tee metadata is missing

@umbral bridge i took note of a little pain detail with editable installs, i cant ask them for file lists (for execnet i want to add something that enables sending in memeory pure python distributions to a blank no site python process and have stuff like entrypoints work

[umbral bridge]

what I mean is make a builder plug-in that does whatever you want it to do

then build with Hatch

[tropic void]

@umbral bridge in the bootstrap situation hatch, hatching as well as all plugin packages would not yet be available as importable/editable distribution

[umbral bridge]

• https://hatch.pypa.io/latest/plugins/builder/reference/
• -t https://hatch.pypa.io/latest/cli/reference/#hatch-build

right on the target machine I don't really have insight about how to set up as you wish, I'm only saying for building the stuff that you send to the target machine you can do arbitrary stuff with a builder plug-in

[tropic void]

@umbral bridge hmm, i should have made threads, bascialy in the "editable install has no files" situation, its simply hard to pick what to send, i will solve later

as for the bootstrap situation, im thinking maybe it makes sense to make tool that can consume pyproject.py well enough to provide live-updated "editable" wheels even of itself by adding its soruce dirs to the import path, and also registring a dynamic distribution finder that provides "realtime metadata" as in entrypoints from the direct pyproject.toml file

then bootstrapping the complete hatch ecosystem could have a toolset that given a folder of git checkouts/source tarballs is able to produce wheel artifacts for all of them as well as a zipapp of a hatch with all the build dependencies include so that it could produce them

(as for the editables alternative that loads distribution metadata from pyproject.toml for editable installs, woudl evil-eye be a good name

[orchid birch]

That's also not a bad idea, but it looks like this might not be a PDM problem. I'll look at tox when I get a chance.

[umbral bridge]

can you try a quick switch to Hatchling? that would give you a clear answer as to where the problem lies

[orchid birch]

Will do, when I get back to my laptop

[umbral bridge]

@tropic void fyi I just saw https://github.com/jorisroovers/gitlint/issues/378#issuecomment-1385042770

is there an option that would allow tags that are able to be pushed on every commit?

[tropic void]

yes, gimme a sec to get the example (its hatch baseda s well ;P

[umbral bridge]

nice! feel free to comment directly there if you want

[tropic void]

https://github.com/pytest-dev/iniconfig/blob/main/.github/workflows/deploy.yml#L43

[orchid birch]

I didn't actually do that, but I did install tox<4 into a venv and tried to run tox -e py311 from there. Worked as expected. Therefore I think it's a regression or change in tox 4. I closed the issue on the pdm project and reopened it on the tox project: https://github.com/tox-dev/tox/issues/2880

[umbral bridge]

my next recommendation then would be to do the inverse of my previous: switch to Hatch environments as a minimal remedy

[orchid birch]

... or just bite the bullet, rewrite to use hatch and ditch tox entirely. TBH, I'm really coming around to the one tool idea.

[umbral bridge]

to give an example we went from this https://github.com/DataDog/integrations-core/blob/eddd2d91a0fb57acff4fae419a6faee3f92ce59a/mongo/tox.ini to this https://github.com/DataDog/integrations-core/blob/master/mongo/hatch.toml

which, subjectively, is much nicer to read and extend 🙂

[orchid birch]

Agreed. Other dimensions include being able to reason about what will actually happen, and teaching this to a wide cohort of Python users of varying experiences.

One thing that does bother me a bit, after having done a full-hatchification of an example project, is that pyproject.toml does tend to get really big and a bit unreadable. Using hatch.toml could help but then there's tension between one-file-to-rule-them-all. Probably some IDE support would also help a lot, like being able to collapse sections.

[umbral bridge]

yeah the idea of the separate file for configuration is almost entirely due to environment definitions being very verbose

at work I saw some tox.ini that were a few hundred lines and putting that in pyproject.toml I view as a nonstarter

[umbral bridge]

that final comment makes much more sense to me since metadata generation is entirely up to the build backend, so… try Hatchling after all? 😂

[orchid birch]

Indeed. https://github.com/tox-dev/tox/issues/2880#issuecomment-1385850651

[orchid birch]

Apparently, poetry's build backend is broken too (as reported on that ticket), but I didn't try it. I ran into an unrelated problem switching to hatchling (some other backend-related config in pyproject.toml probably needs changing), but build's backend uses flit and making just this change to my pyproject.toml makes it all work:

[build-system]
requires = ["flit_core >=3.2,<4"]
build-backend = "flit_core.buildapi"

@umbral bridge I'll go back and try hatchling, but it does lead me to a question: does the hatch documentation clearly specify which pyproject.toml settings control hatch front-end and which control hatchling build backend?

[umbral bridge]

yes the first three pages on the configuration tab talk about metadata/dependencies and building, after that is environments and other configuration stuff

[umbral bridge]

can you link the file?

[orchid birch]

I haven't tried poetry and haven't pushed the flit change, but for reference, here's the report that it's broken w/poetry: https://github.com/tox-dev/tox/issues/2880#issuecomment-1386067389

And the issue (also closed) linked to by that comment: https://github.com/tox-dev/tox/issues/2870

[mint bramble]

@umbral bridge https://github.com/tox-dev/tox/issues/2880#issuecomment-1386178601?

[low gyro]

Interesting

[orchid birch]

👋

[umbral bridge]

Hatchling would still work in that case, if I understand the issue correctly

[mint bramble]

Yes, All is good with hatch

[orchid birch]

W/hatchling I also have to add [tool.hatch.version] section (it still fails, differently now, still investigating)

Oh, probably need to add hatch equivalents of [tool.pdm.build] and [tool.pdm.dev-dependencies]`

[low gyro]

Does Hatch even support dev dependencies?

[orchid birch]

Well, basically, the moral equivalent of includes/excludes/source-includes and dependencies.

My point being that it's not trivial to switch out the build-backend

[low gyro]

Yeah

[orchid birch]

IWBNI it were! #standards

[umbral bridge]

whereas tools like PDM and Poetry have the concept of dependency groups like dev, Hatch thinks in terms of environments which can define their own dependencies

it's more scalable

and doesn't pigeonhole you into just dev or some small subset of reserved names. you can make arbitrary environments

[umbral bridge]

oh it looks like that is a table that PDM looks for to define extras

yeah Hatchling supports that because that is defined explicitly in PEP 621 as optional dependencies

@orchid birch

[tool.hatch.version]
path = "src/flufl/lock/__init__.py"

[tool.hatch.build.targets.wheel]
packages = [
    "src/flufl",
]

[tool.hatch.build.targets.sdist]
include = [
    "/docs",
    "/test",
    "/tox.ini",
    "/conftest.py",
]

[orchid birch]

Thanks @umbral bridge - that's where I was headed before I got distracted by other $work. Will test shortly.

[low gyro]

I don't like the idea of providing dev deps as extras in a built package

[umbral bridge]

yes neither do I

[low gyro]

Any -> Hatch/Hatchling migrator when

||/jk||

[umbral bridge]

yeah never lol the setuptools migrator is quite robust though https://hatch.pypa.io/latest/intro/#existing-project

it even inserts the TOML manually to avoid messing up any styling, like comments

(b/c tomlkit is imperfect)

[low gyro]

but still is (probably) the only package that supports full TOML syntax with styling, not data only like tomli

[subtle bloom]

that'd be surprising if true

[low gyro]

(we are talking Python packages only here) if there is some alternative, it sits quietly and doesn't advertise itself.

[subtle bloom]

is this not full? https://github.com/samuelcolvin/rtoml

havent done much digging tbh

[low gyro]

nope, it is just tomli + tomli_w in rust with Python interface

[shut mantle]

Yeah, most libraries don't have a way to preserve style, comments, etc.

[low gyro]

I mean, it's no wonder, most of the time it's not needed

[shut mantle]

yeah, it makes sense for that to be an entirely different library

same as there's ruamel.yaml for YAML

[subtle bloom]

yea im trying to understand what full means (here to learn)

also obligatory possibly #off-topic

[cyan turret]

I have to admit that is true, but if you don't use too many dynamic features(copy, inplace update) of python collections, tomlkit works quite well.

[orchid birch]

@mint bramble tox 4.3.5 fixes my build from main w/no changes. Thanks for the quick turnaround!

[mint bramble]

👍

Thanks to masenf really 😅

[rancid mesa]

is there a more elegant way to update stuff in envs than hatch env prune && hatch run ...?

[umbral bridge]

update what exactly?

if you re-create without changing version pins or ranges you will probably get the same result because pip caches

and if you do modify those then Hatch will automatically sync

[turbid bane]

@umbral bridge I just wanted to let you know that it appears mypycified Black seems to build correctly now. I just tried it tonight.

[umbral bridge]

that makes me happy, thanks for letting me know!

[turbid bane]

thanks for all of your hard work! ❤️

[umbral bridge]

it's been hard recently keeping track of stuff for Hatch but I'll have more time to devote like usual soon

I have a rather large project at work with kind of a tight deadline that involves learning a new framework

[turbid bane]

I'm still busy (actually more busy as of lately) unfortunately. Trying my best to balance my free time between Black and mypyc, but it's hard

I feel you, it's rough. It's not fun catching up on the notifications when you come back...

[umbral bridge]

but, good news is what I am learning I'm actually going to start incorporating in Hatch

I'm learning Textual, which is a rather new TUI framework

[turbid bane]

Rich is nice, never tried Textual though.

How's it?

[umbral bridge]

it has some kinks which are quickly resolved if you join their Discord lol but otherwise it is absolutely fantastic

[turbid bane]

sounds awesome

[umbral bridge]

the first place (maybe only?) will be the interactive new project functionality which people have been asking to be improved

basically a split pane with a form on the left and the constantly updated rendering of pyproject.toml on the right

[turbid bane]

oooh I like that

[golden merlin]

@umbral bridge have you used urwid

[umbral bridge]

no

[tropic void]

@golden merlin i have before, and i am very happy textual exists these days

[rancid mesa]

default env deps. i usually write down just "pylint", "mypy" for dev purposes, etc. you're saying i should do "mypy==0.991" etc. instead?

(i'm used to the poetry workflow where you'd let your dev deps dangle freely and just update periodically)

[tropic void]

@umbral bridge is there any writeup on workspaces

[umbral bridge]

no not yet, no time for Hatch until next week

[tropic void]

@umbral bridge i see, im really interested in taking part, if feasible please hit me up when things go down

[cyan turret]

Sorry, but @radiant cloak 's article led me here. Adding interoperability contributes to the possibility of the adoption of a workflow tool.

Supporting PEP 517/PEP 621 are also doing the same

[umbral bridge]

@tropic void I tagged you in https://github.com/pyOpenSci/python-package-guide/pull/23#discussion_r1091109537 to get your opinion about that, if you have time

[tropic void]

I just took a look, that thread is a oppiniated can of worms with terrifying turnaround

[umbral bridge]

lol that is correct though I'm just asking particularly about the use of the ignore file by default

[tropic void]

Im inclined to just reply that personally I default to hatch + hatch-vcs these days as it does the right thing reasonably safe,

[umbral bridge]

that person seems to not appreciate the fact that Git cannot be relied upon to be present in all build environments

[tropic void]

Setuptools_scm basically does a git archive call because the details between git versions are so messy

@umbral bridge i added a reply that hopefully sets stuff straight

@umbral bridge the person in question actually added support for describe Metadata in git archive (bringing full git archival support to setuptools_scm

So I'm under the impression they are in a very specific bias bubble

Which greatly seems to differ from ours

[umbral bridge]

I see, that makes more sense

[tropic void]

@umbral bridge got any suggestion for adding hatch-vcs tests to setuptools_scms ci pipeline?

[umbral bridge]

probably something like this https://hatch.pypa.io/latest/how-to/plugins/testing-builds/ where instead of a plugin you pin local setuptools_scm

[radiant cloak]

@umbral bridge: Would you be open to splitting hatching docs into a dedicated section within hatch's documentation?

I'm considering doing this for flit; and it might make sense to do with hatch too.

[umbral bridge]

yes

[radiant cloak]

Neato! I don't wanna be a cookie licker; but would a PR be welcome?

[umbral bridge]

sure! what were you thinking?

[radiant cloak]

Not sure -- probably a separate hatchling.pypa.io?

[umbral bridge]

Maybe first can we just do a dedicated section in the current documentation?

[radiant cloak]

Or, a hatchling tab in the regular doc?

Yea, basically one or the other.

[umbral bridge]

sounds good. just FYI there is a list of redirects in MkDocs configuration that you would have to update

[radiant cloak]

Well, to set expectations: I'm 100% OK with you doing this before I come around to it; and I don't know when I'll come around to this.

Wow, we're almost a month into 2023.

[tropic void]

i think i missed the memo on making hatch auto-reinstall in a env on pyproject changes between runs

[umbral bridge]

Hatch doesn't reinstall, only syncs dependencies

[tropic void]

@umbral bridge so one would have to purge on metadata or packag detail fixes?

[umbral bridge]

yes, for example if you add a new entry point

[tropic void]

@umbral bridge it makes sense, it would be nice if there was a way to enforce jsut a editable install as part of the sync (i had some packaging mistakes, with tox i could jsut iterate until its right, with hatch i had to hatchet around a little - im prett sure it comes as a user surprise that editable install is not doene as part of the sync

[umbral bridge]

does tox update metadata? or do you mean it automatically does a full reinstall if metadata changes?

[tropic void]

tox reinstals on every run, however i doesnt sync on every run

the idea is, that when developing, metadata changes easier than all deps ,additionally the default is non editable, so reinstall is needed for code update

[tropic void]

should hatch normalize names? i just hit a isuse where aplying pyproject-fmt to a project ended up with a denormalized package name (underscores instead of dashes in metadata) and a naively written script that epected normalized names fel down

related to https://github.com/tox-dev/pyproject-fmt/issues/55

[tropic void]

Upstream decided its a bug 😊

[umbral bridge]

names should be normalized, are they not?

[tropic void]

@umbral bridge i have a project where pyproject_fmt did denormalize the input - and the denormalization passed trough hatchling into the wheel right into package metadata after install

[umbral bridge]

actually I think I don't on purpose

[tropic void]

maybe warn tho

[umbral bridge]

(where allowed by the spec)

this is strange though, I have a very strong desire to normalize… I wonder why I don't

oh! I remember

I can't find the links right now but people did not want normalization because they wanted control of the rendering on PyPI

Hatchling definitely did normalize at one point before people considered that a bug

[elfin crater]

PyPI is careful to preserve the casing and such of the project name, because some people are very particular about how their project name is displayed

[umbral bridge]

btw I'll be recording a Talk Python podcast episode on March 1, let me know if any of you would like me to talk about something in particular

about Hatch, obviously 🙂

[tropic void]

oh, awesome

well, i'll use to issue to have a coworker upstream some stuff

[cursive lion]

fixed and released in https://github.com/tox-dev/pyproject-fmt/releases/tag/0.8.0

[dull rapids]

Names are important! I get unreasonably peeved when ASCII transformations are performed on things that aren’t ASCII.

In Swizerland, there’s no “ẞ”/“ß”, they write “SS”/“ss” instead for words that Germans would write with “ß”. That’s totally fine, but performing the substitution is just wrong for names. There’s both people with the surname “Weiß” and people called “Weiss”. You can’t just make one into the other, it’s different surnames!

I think what I’m trying to say is that PyPI should probably just have enforced lower-kebap-case-ascii, and told people to use whatever unicode they want in the README header. That way instead of having normalization, we’d have full flexibility for names and also unambiguous package identifiers.

[tropic void]

@umbral bridge can you make hatch use twine for publish - i have credentials configured that hatch ignores while twine works

[umbral bridge]

what kind of credentials?

[tropic void]

i have a pypirc with uload cappable tokens, twine uses them ,hatch publish asks me for credentials

(ran into this while uploading the wheel for vcs-versioning)

usually i dont notice because on most projects i just use a github pipeline and build + twine on ci

[umbral bridge]

then that would be a feature request to consider that file for configuration

I would prefer to not add more dependencies especially when there's already a working implementation without them

[tropic void]

@umbral bridge im more lie - why not just use twine to implement publish to begin with, im not a fan of dozens of parallel implementations if standard tools are already around

[umbral bridge]

less dependencies and custom control of the output to have it flow well with the rest of the CLI

if Hatch just subprocess everything the output wouldn't be consistent and be a nice experience

it would be like a giant Bash script in spirit lol

[tropic void]

well, as far as i understand, twine has python apis and rich usage - so i do wonder if a better integration could be done

in particular since twine seems to implement atomic multi artifact upload these days and imho thats not something one should reimplement if dependency is feasible

[umbral bridge]

sorry, what does that mean exactly?

[tropic void]

@umbral bridge atomic artifact upload means that you compose a release on pypi in a way that all files land at once rather than appearing one file at a time by making a "transaction"

hmm, wait, let me re.check, i think i missread something

[umbral bridge]

is that a HTTP thing it's doing or is that related to that new PEP I was reading about last year?

[tropic void]

the pep is still draft,, my mistake

[umbral bridge]

also one other thing which isn't too important I guess but if I were to use the APIs provided by Twine I would have to switch away from HTTPX because I don't want to ship both that and requests

[tropic void]

hmm, good point, maybe a good opportunity for someone to make the switch for twine first, these days im a fan of annihilating requests usage :

@umbral bridge other topic, im under the impression that a lot of the pluggy hooks in use by hatch are for obtaining lists of classes, i wonder if a direct entrypoint for each of those would make sense instead (while removing the need for pluggy)

[umbral bridge]

so, I considered that and was actually about to remove it but it came down to configuration being less pretty for users

[tropic void]

would that be different if there was a hook/plugin to auto-create those entrypoints

hmm - does the pep actually allow to declare dynamic only for a specific entrypoint subset

[umbral bridge]

because say you have a project Foo and it provides multiple plug-ins, when configuring a user couldn't just reference [tool.hatch.build.hooks.foo] or [tool.hatch.metadata.hooks.foo], they would have to do [tool.hatch.build.hooks.foo-build-hook] or [tool.hatch.metadata.hooks.foo-metadata-hook]

which is suboptimal and I take UX very seriously

[tropic void]

so its about namespacing

if the entrypoint name for different hook types is different, you can just use the same names for different hooks still

you`d have something like

[project.entry-points."hatch.hooks.metadata"]
foo = ...
[project.entry-points."hatch.hooks.build"]
foo = ...

as example - for hatch-vcs you`d have

[project.entry-points."hatch.hook.metadata"]
vcs =  "hatch_vcs.metadata_hook:VCSMetadataHook"
[project.entry-points."hatch.hook.version_source"]
vcs =  "hatch_vcs.version_source:VCSVersionSource"
[project.entry-points."hatch.hook.build"]
vcs =  "hatch_vcs.build_hook:VCSBuildHook"

[umbral bridge]

yeah I like that much better!

I do wonder if I should do that though considering there are already lots of plug-ins and I don't know what that migration path would look like off the top of my head

[tropic void]

well there would be a need for a plugin that generates those entrypoints automatically and/or change the code

imagine a hatch plugin that ensures entrypoints listed match expectations

once the tools are there, people can migrate over a deprecation period (while leaving pluggy support in,

then at some point, pluggy support turns opt in and eventually gone

but the basic key tool would be a "migrator" script that uses cst to find the plugin names and replaces hooks with entrypoints

[umbral bridge]

yeah this seems doable, thanks for the tip!

when you get a chance do you mind opening an issue so I can track this?

[tropic void]

@umbral bridge please take a look at https://github.com/pypa/hatch/issues/752

[umbral bridge]

Thank you

I'm a big fan of having less dependencies too

you'd be surprised how many people are nitpicky about the number of dependencies, even if it's just 3 or 4 lol

[tropic void]

im a fan of more sensible dependencies

there is certain details i really like to see librarized for synergy

like creating cacheable buildenvironmens, creating cirtualevs from locs and/or requirement sets

tox ,hatch, poetry, pre-commit - too many tools share and reimplement the workflows

imagine if all of those where able to share a build env with for example hatch + hatch-vcs for dozens of projects

[elfin crater]

You can use whatever characters you want in your readme

[dull rapids]

hmm good point maybe a good opportunity

[tropic void]

@umbral bridge would you be interested in working toward a common pattern for vendoring friendly packages and tools in hatch to support it

I have certain needs for pytest /setuptools_scm that are yet to be fully defined, and I'd like to get setuptools/pip covered as well to ensure that a python best practise is established

[umbral bridge]

yes, preferably based on a build hook plugin

[tropic void]

@umbral bridge aware of prior art wrt setting up environments remotely - say on a ssh host/ a container on podman/docker/k8s/oc (both either as a container that will stay avaliable or a something that quickly starts)

i have a idea wrt making execnet work nicely remotely by having it use hatch tools to make remote environments, then connecting to them via the avaliable methods

[umbral bridge]

not remotely per se but there is the environment plugin I wrote for containers https://hatch.pypa.io/latest/plugins/environment/reference/#known-third-party

[tropic void]

@umbral bridge thats a nice starting point, the thing i want to enable is creating environments locally or remotely that then can be nicely used in combination with execnet

[umbral bridge]

yeah should be totally possible

[tropic void]

@umbral bridge is there any way to run hatch so that it would start a process in a environment whose unbuffered stdio is to be connected via socket/pipe

[umbral bridge]

creation happens once and then you're able to execute commands within environments

but it sounds like creation needs to be interactive?

[tropic void]

@umbral bridge

whops

@umbral bridge thing is - i want to run a execnet subprocess that requires a io stream to bootstrap, classically this is done by having a subprocess and taking over stdin/out to use for the binary protocol execnet uses

[umbral bridge]

actually since the interface is so granular you can just pretend that the environment is always created using that method and on the run method do your actual stuff

[tropic void]

@umbral bridge i think i figured what i want, but in part that makes a problem with how hatch manages environments vs platforms

for example system vs virtualenv on top of docker/ssh to a vm/k8s - its not clear to me if there is a sane way to add that to hatch in a acceptable way

[umbral bridge]

what about the interface is insufficient?

[tropic void]

@umbral bridge gimme as sec to verify that im not accidentally doing a false generalization

@umbral bridge ok, currently the differences between local, possibly remote, docker and others are big enough that layering something like virtualenv vs system on top of them would sabotage sanity

[winged cargo]

how might i add a .pth file to my project with hatch?

[tropic void]

There is a hatch plugin for that, unfortunately I forgot the name

[umbral bridge]

https://github.com/ofek/hatch-autorun

[languid relic]

Listening to your episode on talk python to me. Nice interview! 👏

[umbral bridge]

fyi https://github.com/pypa/hatch/pull/808, the licensing should be fine

[proper tide]

Why not just get upstream to release a wheel?

https://github.com/asweigart/pyperclip/pull/131 hmm no movement here

[umbral bridge]

yup

• https://github.com/asweigart/pyperclip/issues/213
• https://github.com/asweigart/pyperclip/issues/238
• https://github.com/asweigart/pyperclip/pull/229

and I tried messaging directly

and I tried searching for alternatives but they all have dependencies

[proper tide]

Hmm maybe a fork and a 541

[low gyro]

How does Hatch use this?

[umbral bridge]

for now just as an option for the command to find the path to the config file, but I find the ability to copy/paste broadly useful

[cinder gazelle]

So it's not possible to use hatchling with python -m build and a dynamic version from a package in a src/ directory, is that accurate?

For additional context, it seems like python -m build generates the sdist fine but in generating the wheel from that, hatchling can't find the path to parse the version information from. I'm guessing this is intentional

[cursive lion]

I do that, also using hatch-vcs. For example with https://github.com/hugovk/norwegianblue/blob/696693b767e17978086edc0741416031d71383ce/pyproject.toml#L66-L70

[cinder gazelle]

I'm not using hatch-vcs. I'm specifically storing it inside src/project/_version.pyand relying on the regex to find it. I can put together a reproducer happily if it's intended to work, but hatch's docs only talk about using hatch build and the packaging guide which shows hatch as the first option doesn't show a dynamic version. So I'm trying to glean what I can from the documentaiton

[umbral bridge]

can you show your project config?

[umbral bridge]

without the config to see my guess is that the version file is not included in the source distribution

[cinder gazelle]

https://gist.github.com/sigmavirus24/9e976b6f38646097a69b13158497e580

The file is. I opened the sdist generated by python -m build

Added the output from python -m build

The difference is the sdist doesn't have src/ as the prefix. So it seems the way the regex works will only work with non-src projects or only if using hatch

Anyway I have to go elsewhere. I'm not on discord often, but the offer stands to put together a repo to reproduce this

[umbral bridge]

remove [tool.hatch.build]

the sources option is getting inherited by the source distribution configuration and therefore the source distribution no longer has the top level src directory

[cinder gazelle]

Yeah that was it. Looking ta other projects it seemed necessary

[umbral bridge]

I'm now questioning whether the inheritance for configuration is useful actually

I always err on the side of preemptively implementing potential feature requests but maybe this particular one was unnecessary

[tropic void]

Some measure of preventive yagni can safe a lot of extra work

[umbral bridge]

I am so incredibly frustrated with building distributions with PyOxidizer

it's the best option but still not quite there yet

no cross compiling support yet

Apple makes one go through enormous hoops to sign and notarize distributions

this all just sucks

I'm going to end up having to build an entire separate tool just to distribute Hatch easily for users

[half cedar]

If you don’t mind not having a single executable, Beeware folks mentioned they are interested in adding CLI support to Briefcase. Talk to Russell Keith-Magee if your goals align.

[low gyro]

this kind of struggles is exactly the reason behind Poetry not providing distro/os specific packages

[umbral bridge]

it's quite sad to me because I view this as the most important next feature

[half cedar]

Personally I’m looking forward to build this on pybi

[umbral bridge]

ironically (perhaps I'm being cynical) Windows is the platform where distributions work flawlessly and I have no complaints

[half cedar]

Windows does things differently, which makes some things difficult but also makes some others easy. Unix people like to shit on those difficult things, but this is one of the things Windows’s design makes simple

[river marten]

What needs to be notarized?

[umbral bridge]

everything included or else upon install gatekeeper displays a pop up blocking the install

Apple servers save data about what gets notarized

[river marten]

And ad-hoc "signing" doesn't work in this case?

[umbral bridge]

as far as I know it doesn't for our use case because we have special entitlements to allow running third-party code like random stuff from PyPI

[river marten]

I see

[umbral bridge]

I could test that I guess

[river marten]

I implemented ad-hoc signing for repairwheel, but certainly none of the notarization stuff since delocate doesn't use it

[umbral bridge]

after some sleep I just had an epiphany

I'm going to reach out to the maintainer of PyOxidizer and see if I can transfer some of my savings to him to fix like 1 or 2 issues that are blocking

my brain was fried yesterday, it's stupid to even think about reinventing this wheel when a project is so close

[tropic void]

@umbral bridge if you are still close to psf people on pycon , also crosscheck if the psf may be able to help

[umbral bridge]

I wasn't able to attend this year

had my proposal been accepted I would have made plans! but since it did not it's not worth it to schedule everything for transport, my caregivers, etc.

[tropic void]

i see, unfortunate - but i do wonder if any of the people that are at pycon may be able to help

[subtle bloom]

which proposal was this?

[umbral bridge]

to talk about Hatch providing the unified Cargo-like experience for Python

[subtle bloom]

is this a draft somewhere?

would love to know more

[umbral bridge]

I don't think I saved it, it was an online form

but my plan remains the same, there is a clear path forward

actually wait, I sent it to somebody and they gave feedback

I'll check my local files in a few minutes

[umbral bridge]

ha found it! https://github.com/ofek/talks/blob/master/hatch/proposal-pycon.md

Ruff's proposal also didn't make it, which from a technical standpoint is much more interesting, so I don't feel so bad 🙂

[subtle bloom]

I would have loved this talk fwiw 🙂 has your opinion changed on this #hatch message

[umbral bridge]

oh wow that was a long time ago

I don't know if you meant to link that particular comment but yes custom is what I will continue to call local plugins

if you meant the comment below that regarding Hatch not trying to be Cargo, then yes shortly after that I changed my mind

the main impetus was that I was embedding on a different team at work to help them with testing/tooling and they did everything in Rust

then I realized that I did not just 1 mistake but 2

the mistake that I knew about, which is why I rewrote Hatch, is being too opinionated with workflow stuff. it had a very particular test command, environment management, etc.

not really configurable or extendable at all and even I outgrew it

after embedding I realized my second mistake

I overcorrected: making every component a plug-in was indeed the right choice but where I went wrong was not having a recommendation for the things

Hatch documentation matched the general PyPA and is like "since all the standards are implemented you can choose/do whatever you want! mix and match tools"

I now realize that is a poor path and users need recommendations that are clear

[subtle bloom]

Hmm maybe discord bug*, yea thats the one.

Very nice. I love that. When i originally checked hatch out that was my initial requirement. It was checking my boxes, but it seemed too open ended and I wanted it to have opinions. Thats exciting.

[dull rapids]

Yeah, it’s a great architecture, but that might have played a role in the documentation part. Correct me if I’m wrong, but I’d guess that if it was a less pluggable codebase, only documenting the user facing parts would have been easier, so by the time you were done with it all, you’d have spent less time and energy. After documenting all the interfaces, I’d personally be a bit wiped out and would have a hard time motivating myself to also write an opinionated guide-style documentation, covering most of everything a second time.

But we don’t need that work to be done a million times. I think this is already a big help here, we just need to make things like this more visible: https://www.pyopensci.org/python-package-guide/package-structure-code/intro.html

[golden merlin]

Laziness question -- how does installing dependencies work for backends themselves?

More specifically how does it work for hatchling?

(Just curious -- hatchling doesn't pin some of its dep versions, so what decides which versions are used, are they vendored somehow as part of hatchling's release, or does it fetch them using itself or pip as part of bootstrapping itself, or what?)

[umbral bridge]

front ends like pip create an isolated environment and automatically installs build dependencies

[golden merlin]

Aha.

[fiery ether]

hi, I finally got around to trying out hatch, and tried searching around for answers on this but didn't see anything so ended up here. The default package name in a new project uses underscores and I'm wondering if there's any explanation about that cause it seems slightly weird.

[umbral bridge]

do you mean the directory that you import at runtime?

[fiery ether]

maybe? yes if that happens to be the directory nested under src by default

[low gyro]

@fiery ether what else would you expect?

[fiery ether]

feels like hyphens or no separators is more normal, I recall underscores getting used in other packages I've used though

[low gyro]

it's Python style by PEP 8 https://peps.python.org/pep-0008/#package-and-module-names

and I would say that 99% of Python code follows PEP 8 style

[fiery ether]

I had PEP 8 in mind specifically cause it says Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.

[patent tundra]

The directory under src are usually the one you like to import. The import statement doesn't allow - in the package name, because this is interpreted as a minus sign. This is why all folders that should be importable have _ instead of -.

[fiery ether]

ah ok, so really the preferable one is no separators, alternatively underscores

[low gyro]

yeah, and my_special_module is more readable than myspecialmodule

[wooden mist]

im trying to use hatch for a project which is 3.11+ but when i try create the env i get an error saying it requires a different version but i never mention 3.10 anywhere, hatch is installed on 3.11 as well

❯ hatch env create default
ERROR: Package 'kine' requires a different Python: 3.10.10 not in '>=3.11'

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "kine"
description = 'A react style gui framework in python'
readme = "README.md"
requires-python = ">=3.11"
license = "MIT"
keywords = []
authors = [
  { name = "Zomatree", email = "[e-mail]" },
]
classifiers = [
  "Development Status :: 4 - Beta",
  "Programming Language :: Python",
  "Programming Language :: Python :: 3.11",
  "Programming Language :: Python :: Implementation :: CPython"
]
dependencies = [
  "aiohttp==3.8.3",
  "toml==0.10.2",
  "typing_extensions==4.4.0",
  "pipdeptree==2.3.3",
  "msgpack==1.0.4"
]
dynamic = ["version"]

[project.urls]
Documentation = "https://github.com/Zomatree/kine#readme"
Issues = "https://github.com/Zomatree/kine/issues"
Source = "https://github.com/Zomatree/kine"

[tool.hatch.version]
path = "kine/__init__.py"

[[tool.hatch.envs.test.matrix]]
python = ["311"]

[tool.pyright]
typeCheckingMode = "strict"
reportPrivateUsage = "none"
reportImportCycles = "none"
reportMissingTypeStubs = "none"
reportMissingModuleSource = "none"

[tool.black]
line-length = 120
target-version = ["py311"]
``` pyproject file for reference

[umbral bridge]

Python 3.10 must be first in your PATH

if you use the test environment for example that will work since you are setting the version explicitly

this will be fixed in the next version

[wooden mist]

what is it looking for on my path?

python?

[umbral bridge]

yes

[wooden mist]

python resolves to 3.11 in my shell

[umbral bridge]

can you run the following in a new shell?

which -a python

[dull rapids]

I think if project.requires-python is >=3.11, hatch should default to python3.11 if python --version returns something < 3.11

There’s no situation in which a project with project.requires-python = '>=X' should ever be run in an environment with Python <X installed.

[umbral bridge]

yup

[wooden mist]

I forgot to reply yesterday but python was an alias to python3.11 which is probably why there where issues

I ended just removing 3.10 as I didn't need it

[balmy dock]

Does Hatch support installing dependencies from private GitHub repos?

I found pip install git+https://${GITHUB_TOKEN}@github.com/user/project.git@{version} online, which works, but stops working as soon as I try to add it to my pyproject.toml -- it looks like setuptools doesn't parse the environment variable
Is that something Hatch would be able to do?

[umbral bridge]

https://hatch.pypa.io/latest/config/context/#environment-variables

[balmy dock]

[balmy dock]

Oh I missed that. Not quite sure I understand though? Do I just uh....

- ${GITHUB_TOKEN}
+ {env:GITHUB_TOKEN}

[umbral bridge]

yes that should work

[balmy dock]

Alright, I'll give it a try this afternoon, thanks!

[umbral bridge]

another example https://hatch.pypa.io/latest/how-to/environment/package-indices/

[balmy dock]

I did see that one
I wasn't sure it would like me listing the same index twice
Though on second thought I don't see why that would matter

Oh it shows an example of environment variables there too

[gloomy flicker]

FYI installing hatch with --only-binary :all: backtracks to 0.23.1, which is presumably the earliest version to not depend on pyperclip which does not publish wheels

I installed hatch with pipx and I was wondering why nothing worked until I checked --version. PIP_ONLY_BINARY=:all: was set in the environment

[umbral bridge]

thanks for the heads up

I tried vendoring it but people were rather displeased so I closed the PR

[tropic void]

i#ll try to reach out to the author

gave him a shoot on twitter

[cursive lion]

https://github.com/asweigart/pyperclip/issues/213 is one of the upstream issues requesting a wheel

[tropic void]

im aware its been there since a while, hence the different platform reach out, it pretty much looks like the maintainer is just not working on it atm

[cursive lion]

Yep, he's busy on a book right now

Ideally would be good if he could add another maintainer

[proper tide]

How do I set the wheel build-tag?

[umbral bridge]

do you mean the entire tag or specifically just the optional build number component https://peps.python.org/pep-0427/#file-name-convention

[proper tide]

Just the build tag

[umbral bridge]

yes with build hooks https://hatch.pypa.io/latest/plugins/builder/wheel/#build-data

you can set the tag manually but there is no option for an explicit part of the tag

example https://github.com/facelessuser/Rummage/blob/d040a070eb1a98a4c0e7b7df25d4bb5cac883a15/hatch_build.py#L63

https://github.com/ofek/hatch-mypyc/blob/5a198c0ba8660494d02716cfc9d79ce4adfb1442/hatch_mypyc/plugin.py#L340

also, since someone recently asked me this and I asked them the same thing, what is your use case?

and why can't the project version contain that information instead?

[proper tide]

Rebuilding wheels for python 3.12 after the final release if the ABI changes

Without having to upload a new version

Pillow did this by remaining the
.whl files which produced invalid wheels

[umbral bridge]

how would you prefer to set the option? environment variable?

from the perspective of the builder there can be no concept of increments because building comes before wheel generation and therefore there is no name to use as a reference

[elfin crater]

i forgot all about build tags

[proper tide]

Ideally a -C flag to build

[elfin crater]

do they actually work

[proper tide]

Yeah pillow did it

[elfin crater]

neat

[umbral bridge]

I would love it if we could remove that from the spec lol

[proper tide]

Didn't do it properly either, just renamed the
.whl

[elfin crater]

why?

[proper tide]

#build message

[umbral bridge]

wait, why does renaming not work?

[proper tide]

Pillow-9.4.0-1 and Pillow-9.4.0-2 don't have the Build: 1 in their WHEEL file

But does anything read that?

[elfin crater]

uhh

I mean if it worked for pillow probably not, but seems like a bad thing to rely on people not reading that field

[proper tide]

https://inspector.pypi.io/project/pillow/9.4.0/packages/5c/2a/72b80cd8a35fac89142afb35aabab6ce2631a3261043b6216664c9137b29/Pillow-9.4.0-1-pp39-pypy39_pp73-macosx_10_10_x86_64.whl/Pillow-9.4.0.dist-info/WHEEL

Maybe something for twine check

[umbral bridge]

why

[umbral bridge]

@proper tide I'll add this ability with an environment variable. If you have time can you please open a feature request for tracking?

[cursive lion]

Yep, maybe technically invalid, but we did the rename for the Pillow wheels for macOS, they were the latest available from January to April this year and will have been installed millions of times, no problems reported. We've probably done it before too for other platforms.

[golden merlin]

I want to dynamically write out some JSON file at build time when hatch is building my thing. What's the easiest way to do that?

(write it out presumably into the distribution so it's accessible at runtime)

[empty flower]

Look at something like hatch-vcs as an example?

[umbral bridge]

FYI I'm going to do a release with binaries soon but starting right now every commit to master has artifacts that can be tried e.g. https://github.com/pypa/hatch/actions/runs/5239516133

in case you're wondering how, it uses an embedded wheel https://ofek.dev/pyapp/latest/config/#project-embedding

[tropic void]

@umbral bridge would it be feasible to have deps of hatchling use hatch if there was a bootstrap helper that could produce all wheels correctly by creating fake editable installs that use metadata from pyproject.toml for etrypoints/distribution objects instead of installed ones (or a hack around having dist info ready for importing and jsut using sys.path hacks)

[umbral bridge]

that would work technically I think

[tropic void]

@umbral bridge did i gather it right, that the metadata plugin allows to render the version of the current package into metadata (context is examining if hatch could be used for toga/toga_*) all of those packages come off a monorepo, and do single version interlocked release of multiple packages at once

[umbral bridge]

which metadata plugin, this? https://hatch.pypa.io/latest/plugins/build-hook/version/

[tropic void]

i meant the metadata hooks, sorry for the term mixup

hmm, ah, no its not able to :/

@umbral bridge bascially the intend is that if i have a package foo and its version turns out ot be 1.2.3 - i want to be able to depend on foo_somethin_fun==1.2.3

[umbral bridge]

you can use a custom hook

example of some custom stuff https://github.com/facelessuser/Rummage/blob/d040a070eb1a98a4c0e7b7df25d4bb5cac883a15/hatch_build.py#L31

[dull rapids]

@umbral bridge when are the metadata plugins run? I’d like to retrieve the list of package directories after package discovery is run. How would I do that?

[umbral bridge]

metadata plugins run before builds occur since metadata is required at that stage. I suppose I could make file discovery its own stage that happens before builds and run metadata hooks after that but that would be very complex and also the metadata hooks wouldn't have access to every included file because some are created by build hooks

[dull rapids]

I see, thank you! But your levels of abstraction are pretty nice to work with so WheelBuilder(self.root).config.packages seems to be quite enough to make it happen

https://github.com/flying-sheep/hatch-docstring-description/blob/7501ccc041a61f904f35fd5c6a1c1c3a4b623baa/src/hatch_docstring_description/read_description.py

[umbral bridge]

oh nice

[tropic void]

@umbral bridge wondering if there is any sensible way to combine metadata and vcs source - for vcs-versioning i'd like to have branch, commit date, upstream repo paths and more obtained from the scmversion, as well as putting them somewhere so that a sdist would have a snapshot

[umbral bridge]

can't you just put that in the same file that is written to if the user configures it?

[tropic void]

unfortunately not safely - and i want to avoid a own metadata sidechannel - instead i want hatch mtadata hooks to handle it

[winged cargo]

has extension modules been implemented yet

[umbral bridge]

no that work is entirely dependent on @errant timber

[tropic void]

@umbral bridge whats the recommended way to have a local (custom) plugin at configuration to hatch-vcs atm ?

[umbral bridge]

what are you trying to achieve?

[tropic void]

i want to enable people to pass custom local versions for version_schemes in hatch-vcs

[umbral bridge]

I'm sorry can you explain maybe a specific workflow so that I can understand it better?

[tropic void]

@umbral bridge bascially i want to enable the equivalent of

from setuptools import setup
def my_version_scheme(scm_version):
   return "13.4.5"
setup(use_scm_version={"version_scheme": my_version_scheme})

[errant timber]

Not familiar with Hatch's config system internals, but curous: how hard would it be to generate a json-schema for it (so it could be used in validate-pyproject)? A strict-config option would also be nice. See https://github.com/scikit-hep/awkward/pull/2604/files#r1283467154. I've just added json-schema generation & validate-pyproject support to scikit-build-core in https://github.com/scikit-build/scikit-build-core/pull/447 - it was easy due to scikit-build-core's declaritave dataclass-based config.

[umbral bridge]

it would be very difficult but I was actually just thinking about this last week

I want to switch Hatch config to Pydantic models which are generated from json-schema

I couldn't do that for build/Hatchling config however because that would introduce a dependency on Rust

[proven bronze]

let me know if you need help with the metadata pep

[umbral bridge]

thanks! I will be adding a lot to it this weekend since they want it to stand on its own rather than reference 722

my approach is going to be enumerating use cases that users have expressed and go down each one explaining how it satisfies that better than the alternatives

I was thinking about this a lot today

[proven bronze]

that sounds great. i think one thing that will be important is structuring it so that when you read it through, it's clear why it's better than 722 even if you don't care about other use cases.

so maybe something like a "Comparison to PEP 722 for running single file scripts" section followed by a "Additional extensibility offered over PEP 722" section

i think necessary to preempt a "722 is simple" and "723 is complicated and YAGNI" false dichotomy

[dull rapids]

I have started an effort here: https://github.com/SchemaStore/schemastore/pull/3060

Having it in schemastore will result in VS Code autocompletion by way of even-better-toml: https://github.com/tamasfe/taplo/issues/432

[errant timber]

FYI, you can implement this with dataclasses & zero dependencies pretty easily, I did that in scikit-build-core. The "ground truth" is the dataclass, then you can generate the schema from it. There's also https://github.com/Fatal1ty/mashumaro which does this too.

[umbral bridge]

updated https://discuss.python.org/t/pep-723-embedding-metadata-in-single-file-scripts/30979/13

[radiant cloak]

FWIW, I think it's useful to have the PR get landed and then initiate discussions. I made this mistake with PEP 704 and I'm seeing it repeated in these two PEPs as well. The PEP editors do a good job of helping flesh out the PEPs and avoid certain bits of confusion. By posting drafts, we're skipping that stage before heading into discussion.

[umbral bridge]

oh interesting, do you mean asking for it to be merged and then addressing all feedback afterward?

[radiant cloak]

Well, not putting it up for feedback until after the PR is merged.

That ship sailed obviously but mentioning this for the future.

I almost wrote this in dpo when the PEP 722 thread was posted, and didn't. Same for the 723 thread. And, well, I had a flight and I landed back to see a bit of a trash fire around the process and didn't want to poke a different aspect of it. 🙈

[cursive lion]

Yeah, technically, PEP 1 also says the PR should only be created after the sponsor decides it's ready; and then the discussion thread created after merge. But tbh this doesn't always happen.

[minor lance]

Can I mix setuptools and hatch to get extension modules working

I want the cool hatchling stuff but I just cant get it to build for all platforms

[umbral bridge]

yes https://hatch.pypa.io/latest/meta/faq/#interoperability

[tropic void]

@umbral bridge is there a canonical way to have a script invoke a python command of the same name?

[umbral bridge]

do you mean like you have a script configured as foo and you have an entry point of the same name that you wish to call inside the environment?

[tropic void]

Exactly

[umbral bridge]

I would rename the script or try python -m foo

unless you're saying there is no conflict situation but rather you want an easier way to run the commands, in which case I guess I'm confused because you can just do hatch run <ENTRY POINT>

[tropic void]

@umbral bridge renaming the script is unfortunate, i wish there was something like the cmd builtin of shells which goes for run this actual command instead of aliases

[orchid birch]

@umbral bridge I've been continuing with my hatch experiment on a proposed monorepo at $work. The snafu I have is that some subpackages depend on unreleased versions of other subpackages, and we want to do testing across the entire monorepo to ensure all subpackages changed in a PR or at HEAD still pass all their tests.

So where I have a dependency like mypkg.core in the project.dependencies section for when I build and publish the whl/sdist, I need to rewrite that as mypkg.core @ {root:uri}../mypkg.core for testing purposes and set tool.hatch.metadata.allow-direct-references = true to enable that feature. Of course I don't want to publish my artifacts with those settings though.

My thinking is that I can use a hatch metadata hook to rewrite the dependencies during testing (perhaps keyed off of an environment variable for local and CI testing, or perhaps the environment script I am running, e.g. hatch run test at the monorepo root. I don't know if I can implement that hook in the monorepo and get hatch's pluggy to discover it there, or need to create a whl/dist of my plugin and get that installed into the environment for testing, linting, etc. I'm also not positive that MetadataHookInterface will give me access to everything I need.

Before I go down a rabbit hole, I thought I'd reach out and ask if you have any other thoughts that I should be aware of. TIA!

[umbral bridge]

I'm sorry I haven't implemented workspaces yet, the TL;DR is I was given more management tasks at work which is taking up a lot of time.

You can definitely use a metadata hook without publishing a package by using the built-in custom one and having each project define the path to the central script that contains the implementation https://hatch.pypa.io/latest/plugins/metadata-hook/custom/#options

at work everyone is also moving to Hatch for various things so I've also been helping one on one, lmk if you want to hop on a call at some point https://github.com/DataDog/dd-trace-py/pull/6442

[orchid birch]

Awesome, thank you @umbral bridge . Let me experiment (I only have a couple of blocks of coding time a week myself) and will ping with updates, etc.

[orchid birch]

That looks like it will work. One thing I ran into is that I can't toggle allow-direct-references in the hook. I'm not sure if it's worth trying to set it to true only for local/CI builds and tests or just to leave it set to true in the pyproject.toml. I definitely don't want to leak those into the built package metadata, but it doesn't look like the custom metadata hook gives me access to that setting. I would have to conditionalize the rewriting of the dependency off an environment variable perhaps.

Actually, maybe I have to do that anyway, unless there's a way for the metadata hook to know if hatch build is being invoked (?).

[umbral bridge]

I guess I could add reading from an environment variable here https://github.com/pypa/hatch/blob/7dac9856d2545393f7dd96d31fc8620dde0dc12d/backend/src/hatchling/metadata/core.py#L1498

[orchid birch]

Possibly, although it seems to work with my hook conditionalizing on an internal environment variable. Basically the thing is I cannot do the dependency rewriting when I'm building the sdist/wheel artifacts, but in any other case where hatch needs to install a dependency from another subpackage in the monorepo, I need to do the rewrite.

[orchid birch]

@umbral bridge is there a way to conditionally set or unset an environment variable based on the command being run? The solution I've come up with for the dependency rewrites are something like the following in my metadata hook.

• If an environment variable like MYPROJ_MONOREPO exists (regardless of value), I rewrite the dependencies to be myproj.core @ {root:uri}/../<subdir> when the dependency name startswith('myproj').
• In hatch run build I have to explicitly cd libs{matrix:lib} && unset MYPROJ_MONOREPO HATCH_ENV_ACTIVE && hatch run build so that dependency rewrites do not happen for the build command (because that would leak the rewrites into the artifact metadata). We previously talked about why HATCH_ENV_ACTIVE has to be unset for recursive hatch invocations.

[umbral bridge]

yeah let me pull up the docs link

[orchid birch]

Thanks. BTW, I did look at Environment variable overrides but didn't see how that would work

[umbral bridge]

https://hatch.pypa.io/latest/config/environment/advanced/#environment-variable-overrides

oh lol

[orchid birch]

😄

It looked like that was a way to change hatch settings based on env var settings, not kind of the other way 'round

[umbral bridge]

there is also https://hatch.pypa.io/latest/config/environment/advanced/#required-environment-variables

[orchid birch]

Let me try that after this meeting that just started :p

[orchid birch]

@umbral bridge Again thanks, but unless I don't see it, I think that doesn't help. What I (think I) want is to be able to selectively set and unset environment variables based on the name of the script or env:script being run.

[umbral bridge]

okay then yes I suppose not supported at the moment

[orchid birch]

Cool, thanks. I have a pretty good workaround I think so it's not a big deal. I should post my approach either here or on Discuss. It looks like it'll work but I want to at least get it through my internal review process in case I'm missing something. Then I'll sanitize out the $work details and share it.

[tropic void]

@umbral bridge it seems that hatchling imported from the venv instead of the checkout when running the dev mode on hatch (i presume sideeffect of not being a workspace) - going to pause now (i replaced plugin hook registration with something deferred in both hatch and hatchling, it only works in sync tho - i may need to add compat later (got an idea for that)

[umbral bridge]

I'm not sure precisely the issue you are talking about but Hatchling doesn't do anything fancy it just has access to whatever is on Python's path

[tropic void]

when i run the testsuite of hatch/hatchling, it runs against a old version installed in site instead of a editable install

[umbral bridge]

what command?

[tropic void]

sorry iw as caleld into a mgt

$ hatch run dev
============== test session starts ============
...                                            
__________ ERROR collecting Projects/pypa/hatch/tests/project/test_config.py ________________
ImportError while importing test module '/home/ronny/Projects/pypa/hatch/tests/project/test_config.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib64/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/project/test_config.py:6: in <module>
    from hatch.plugin.manager import PluginManager
src/hatch/plugin/manager.py:2: in <module>
    from hatchling.plugin.manager import DeferredClassRegister
E   ImportError: cannot import name 'DeferredClassRegister' from 'hatchling.plugin.manager' (/home/ronny/.local/share/hatch/env/virtual/hatch/eaydgHWI/hatch/lib/python3.11/site-packages/hatchling/plugin/manager.py)
========================== short test summary info ================================
ERROR tests/project/test_config.py
!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!
======= 1 error in 0.67s ===========

it seems like the post install command was not done

hmm, ok, another purge solved it

[umbral bridge]

yes since workspaces is not yet implemented I rely on that post install command so you must hatch env prune

[tropic void]

ok, i'll open a mr shortly to enable discussion

can docker be optional ? i dont have it on my systems :/

@umbral bridge https://github.com/pypa/hatch/pull/972 - this prepares to remove the getattr hack for lazy properties, its not completely shifted around yet

the basic idea is to defer the pluggy based plugin manager even more, and enable entrypoint based lookup

for all default plugins its then possible to migrate from pluggy to entrypoints - and if no pluggy based plugin is avaliable via the entrypoint, then pluggy doesnt even have to be imported

@umbral bridge depending on your time, i'd love to have a quick chat abotu this after my next mtg (which ends in about 40-60 minutes)

(the idea bein to use voice chat for ease)

[umbral bridge]

definitely! I can after 12 noon New York time, the next two hours I have back-to-back meetings. the one in three minutes is with the folks from https://prefix.dev to talk about lock files for PyPI

[tropic void]

oh, then perhaps in the evening, in about 2 hurs i'l be back home with the kids

[cursive lion]

Hatch covered in the latest PythonBytes: https://pythonbytes.fm/episodes/show/353/hatching-another-episode

[umbral bridge]

that's awesome, I love hearing about happy users

[tropic void]

@umbral bridge thoughts on having a hatch config option for envs to force reinstall of a editable/package on every run (its literally the only thing keeping me on tox)

[umbral bridge]

no plugin is required, that feature is called workspaces (I'm copying Cargo) but I have not had time to do that yet unfortunately

after this next release that becomes high-priority

[tropic void]

@umbral bridge excellent, in that case i'll just migrate my stuff now ^^

[umbral bridge]

just to be clear because my words were not precise there (sorry) I have to implement that feature, I was just saying that I know the need exists

I promised Barry that feature I think 8 months ago...

[tropic void]

my work also has a potential need for this (we have acceptance tests for microservices, and really want workspaces that autosetup)

but i cant make any time alocation promises from my side to help yet

also yeah man, life happens, setuptools scm 8.x was planned like 8 months ago as well and look at al lthe landmine 🙂
(in addition the baby figured how to use chairs for stepping onto high things)

[umbral bridge]

@tropic void https://github.com/pypa/hatch/issues/976

[orchid birch]

And one job ago 😂 - but I have already migrated to hatch for $work monorepo. The hack is that I needed a plugin to rewrite dependencies during full-repo testing to point into file URLs rather than PyPI URLs dependency names when I detect an intra-repo cross dependency. Used a MetadataHookInterface subclass. It works well, but I haven't yet integrated other repos into my monorepo.

[umbral bridge]

I need more time in the day, just 4 or 5 more hours would be perfect

alternatively, if there was some magic pill I could take in order to not require sleep... that would also work

[elfin crater]

https://giphy.com/gifs/rhoc-shannon-beador-alaska-thunderfuck-5000-FHlMJHSx5sGBi

[orchid birch]

I've been trying to finish building my clone army for like 2 decades.

[golden merlin]

I've asked about this I think, but how can I diagnose what I assume is a hatch bug where it's not including files that are tracked in VCS?

Specifically, without these 2 lines: https://github.com/bowtie-json-schema/bowtie/blob/main/pyproject.toml#L13-L14 that file is not included in the wheel (but is included in the sdist) even though it's being tracked in VCS

[umbral bridge]

the issue is one of documentation which I will fix soon

basically you should not use global build config and rather specific targets should be configured

this I thought was a good idea, inheritance of config is nice for hypothetically less verbosity, but in practice I should have never implemented that

[golden merlin]

You mean I should configure the wheel target rather than having a section for all of hatch.build?

[umbral bridge]

yes

[golden merlin]

But I don't understand why the section is needed in the first place

Doesn't hatch say "if you have files and they're tracked in VCS they get included in your distribution"?

Why do I have to tell it to include those files?

[umbral bridge]

https://hatch.pypa.io/latest/plugins/builder/wheel/#default-file-selection

your project name is different than the directory name

[tool.hatch.build.targets.wheel]
packages = ["bowtie"]

[golden merlin]

Trying -- though I guess I don't see from that page how I should know that's relevant

I'm hitting case 5 there no?

What thing is supposed to tell me that that means I should expect different behavior about what files are included from within the package?

That indeed seems to work even if I now remove the force-include, thanks.

I'm definitely still confused on where I should have learned that though

[umbral bridge]

you were hitting 5 but it has to be read precisely 😉 :

Otherwise, every Python package and file that does not start with the word test will be included

[golden merlin]

I don't see what in that sentence says what you're telling me I'm afraid.

[umbral bridge]

just Python files

[golden merlin]

I expect that to mean "that's the same behavior as if stuff did match"

Yes but that's not the behavior if things do match right?

How would someone know that the behavior is different there?

[umbral bridge]

right outside of 5 pattern matching is the same everywhere

I mean, how would you recommend I reword 5 to be more explicit that is only Python files?

[golden merlin]

I guess I'd ask why the behavior is different first

Can't it just be the same?

[umbral bridge]

no. if there is no good heuristic to determine what is included and there is no explicit inclusion, you don't want your entire repository shipped in a wheel

[golden merlin]

surely a directory containing 1 python package is meaning to ship that python package, no?

I'd definitely have been ok if hatch didn't try to guess which package was mine and threw an error, and then I had to specify it, as long as I then got the same behavior

This seems worse to me personally

[umbral bridge]

what if there are two directories at the root that contain Python files?

[golden merlin]

I would throw an error personally.

[umbral bridge]

hmm, that is interesting

I suppose I could change 5 to simply error

can you please open an issue if you have time so I can track that change?

[golden merlin]

Cool! (Glad I wasn't totally missing what was happening, thanks for the patience). Will do.

https://github.com/pypa/hatch/issues/992 -- hopefully I captured that right

[cursive lion]

hello! hatch-vcs question: I'm trying to switch to hatch + hatch-vcs in https://github.com/python/cherry-picker/pull/95 and we have tags like cherry-picker-v2.2.0, but it's not picking the version up properly

I've added:

[tool.hatch.version]
source = "vcs"
# Change regex to match tags like "cherry-picker-v2.2.0".
tag-pattern = '^cherry-picker-(?P<version>[vV]?\d+(?:\.\d+){0,2}[^\+]*)(?:\+.*)?$'

but it's setting the version like 2.0.1.dev50. What am I missing?

[umbral bridge]

the option is correctly getting passed to the underlying library but the issue is that the option doesn't do what we think it does unfortunately, I ran into that as well. you have to do this https://github.com/DataDog/integrations-core/blob/87e3a4500b82b497508f0d245769e2a55f06db09/ddev/pyproject.toml#L62

[cursive lion]

hmm, with this, I still get cherry_picker-2.0.1.dev49.tar.gz:

 [tool.hatch.version]
 source = "vcs"
-# Change regex to match tags like "cherry-picker-v2.2.0".
-tag-pattern = '^cherry-picker-(?P<version>[vV]?\d+(?:\.\d+){0,2}[^\+]*)(?:\+.*)?$'
 
 [tool.hatch.version.raw-options]
+git_describe_command = ["git", "describe", "--dirty", "--tags", "--long", "--match", "cherry-picker-v*"]
 local_scheme = "no-local-version"

do I need to use another option as well?

[umbral bridge]

use every option that I use in that example file

[cursive lion]

same thing with (note single . needed for root):

[tool.hatch.version.raw-options]
root = "."
version_scheme = "python-simplified-semver"
local_scheme = "no-local-version"
parentdir_prefix_version = "cherry-picker-"
git_describe_command = ["git", "describe", "--dirty", "--tags", "--long", "--match", "cherry-picker-v*"]

[umbral bridge]

oh wait, what version are you expecting?

[cursive lion]

these are the tags, so at least 2.2.something?

cherry-picker-v2.0.0
cherry-picker-v2.1.0
cherry-picker-v2.2.0

[umbral bridge]

I'm not sure if you have used it before but changes are expected to induce dev versions

the state of your branch is not the tag

[cursive lion]

does it use the last tag as a base?

[cursive lion]

yep, happy hatch-vcs user in other projects 🙂

[umbral bridge]

hmm

[shut mantle]

Oh cool, cherry-picker 2.2.0 has been released!

[umbral bridge]

@tropic void do you know?

[tropic void]

unaware, taking alook now

@umbral bridge @cursive lion no amount of config is going to save the project from

$ git describe --tags --long
cherry-picker-v2.0.0-45-g38fcdad

@umbral bridge @cursive lion squash merge of release branches with tags is a bad idea

@cursive lion https://github.com/python/cherry-picker/issues/96

squahs merge for squashing your hopes

[umbral bridge]

wow nice debugging

[cursive lion]

Thanks! So it should be fine with the text prefix versions, but we should make sure to tag main "normally"

Actually, we're switching to Trusted Publishing and tags will be applied via GH UI so all should be good. Thanks again!

[tropic void]

is there a recommended plugin for hatch when wanting to try rust based extensions

[umbral bridge]

not yet, I am prioritizing CLI UX right now and will get back to build improvements in Q1

also, I am reluctant to do much on that because I am still kinda waiting on Henry to have free time

https://github.com/PyO3/maturin/issues/1419

my plan is still to push for #extensionlib throughout the ecosystem. it's the correct approach and the people that are saying otherwise are doing so because of my inability to express to all audiences why it is the correct approach

[low gyro]

on Poetry side we are also waiting for some standard to appear, extensionlib seems like a right direction

[tropic void]

@umbral bridge so no external known, and internal intended to use extensionlib - not ideal but better than a bad hack ^^

[cursive lion]

Hatchling issue: https://github.com/pypa/packaging.python.org/issues/1316

[umbral bridge]

thanks, commented

[dull rapids]

Ah, I know the feeling

[umbral bridge]

big feature coming https://github.com/pypa/hatch/pull/1002

[winged cargo]

how do you do the bracket dependencies thing with hatch? like pip install mylib[foo] will install different stuff than pip install mylib

[umbral bridge]

yeah that works, everything gets passed to pip in the end

[winged cargo]

wait but like how do you do it with hatch

[umbral bridge]

please explain with more context

[winged cargo]

how do you tell hatch to do that. like how do you have different dependencies based on the [key] thing

so if i pip install mylib[foo] i get x dependencies and if i pip install mylib[bar] i get y dependencies

[umbral bridge]

you're saying you want your package to have features like that?

[winged cargo]

yeah

is that what its called

[umbral bridge]

https://hatch.pypa.io/latest/config/metadata/#optional

[winged cargo]

thank you

sorry i just had no idea what its called

[sick jolt]

Hey all - I'm a hatch + mypyc n00b - How does one get hatch to try to use mypyc for black?

• I'm trying to make docker install compiled black ... and my doc parsing is failing me 😦
  https://github.com/psf/black/pull/3965

[umbral bridge]

it's not immediately obvious to me that Hatch is causing an issue here

[sick jolt]

I don't think it is, I believe I am just not driving it right

Am I correct in thinking I can ask hatch somehow to try and run it's mypyc plugin and give me a compiled wheel or am I wrong there?

[umbral bridge]

https://github.com/psf/black/blob/882d8795c6ff65c02f2657e596391748d1b6b7f5/pyproject.toml#L162

[sick jolt]

O, so it's env var driven? cool

[umbral bridge]

yeah

• https://github.com/psf/black/blob/882d8795c6ff65c02f2657e596391748d1b6b7f5/pyproject.toml#L117-L118
• https://hatch.pypa.io/latest/config/build/#conditional-execution
• https://hatch.pypa.io/latest/config/build/#environment-variables

[sick jolt]

Thanks - Will go play ...

[sick jolt]

Got it. Many thanks 🙂

[umbral bridge]

anytime!

[sick jolt]

CPUs around the world will thank you 🙂

[umbral bridge]

automatic Python downloads coming soon https://hatch.pypa.io/dev/plugins/environment/virtual/#python-resolution

[tropic void]

Wondering do those work on nix

[umbral bridge]

nixos?

[tropic void]

Yup, I use that on my personal servers/systems

[umbral bridge]

wdyt? https://github.com/pypa/hatch/pull/1015#issuecomment-1781628747

[golden merlin]

I am unfortunately mostly offline due to a sudden death in the family (we're ok.) But what's in the description there/ what the PR changed looks incorrect to me, the reason I reported that in the first place was non py files not being included and me finding that surprising compared to the "normal" hatch behavior.

I can have a closer look next week when I'm back online.

[proven bronze]

is this issue related to hatch's macos-max-compat setting or am i barking up the wrong tree? https://github.com/psf/black/issues/3981

[umbral bridge]

I responded

[umbral bridge]

FYI in the documentation I'm going to begin discouraging the use of pyenv now that we have these features:

• https://github.com/pypa/hatch/pull/970
• https://github.com/pypa/hatch/pull/1002

it causes too many issues

[low gyro]

somehow I feel like this adds another point to this:

[umbral bridge]

not quite, I think. I deliberately chose to make management of Python very easy to reason about

you can install yourself and add them to PATH, or if there is no version that is compatible with your project it will be downloaded for you and managed internally (with options to control that behavior https://hatch.pypa.io/dev/plugins/environment/virtual/#python-resolution)

no shims, etc. which don't work on Windows anyway

I would very much appreciate if people could test the new features, I'm hoping to add 2 or 3 others and then releasing by mid-December

specifically the Python manual management and environment automatic Python downloads are features I have always desired personally and now it exists 🙂

[low gyro]

I mean, it's still yet another way to install Python

I would love to have a cross-platform standard for that

windows has pep 514 because it's broken by design, but what about mac and linux? wild west

[fast surge]

Linux already has standardization for where things should be installed to and discoverability of them that python should adhere to, so no python on linux specific spec needed https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html

[umbral bridge]

I would push back on such a standard if you're talking about an interface like the py launcher as I think that is unnecessary

I would advocate for a recommendation on a shared location for installed distributions (Hatch defaults to ~/.pythons) and also a standard spec for relocatable distributions like the ones that I and Bazel and others use https://github.com/indygreg/python-build-standalone

I have a very strong preference for promoting those distributions to official and hosting them on the main website

[low gyro]

PyBI + some shared location would be a perfect solution imo

[umbral bridge]

I have knowledge of those but still would prefer the ones I linked as there is actually industry use already

[low gyro]

ok, I confused https://github.com/indygreg/python-build-standalone with PyBI

I thought those 2 were the same

[umbral bridge]

the build command now supports other backends https://github.com/pypa/hatch/pull/1018

only a few more features until release!

[dull rapids]

I thought about using it, saw it was written in bash, and decided I’d look elsewhere. I don’t think it’s possible to write maintainable software in any POSIXy shell. The fact that there’s individuals who do just that doesn’t generally invalidate my statement, it just means they have learned to keep 381 footguns in their head at all times and I pity them.

[dull rapids]

Yeah, I think that’s something that needs to happen: A standard for storing and finding Python installations from standard locations (as provided by platformdirs, don’t put e.g. cache files into locations that aren’t standard cache locations)

Until that happens, yes, hatch is yet another way to do this, but it’s an optional feature. On Linux, I use Pythons managed by my system package manager, and let Hatch use those.

On MacOS, I use a pretty clean setup thanks to Hatch’s new functionality

• nix for user-wide CLI tools like topgrade, rustup, pandoc, and pipx
• nix-installed pipx to manage user-wide Python CLI tools like pre-commit, build, Hatch, and jupyter lab (would include twine if I used it outside of CD)
• pipx-installed Hatch to manage project environments
• nix-installed topgrade to update everything

[tropic void]

@umbral bridge is there a recommended builtin mechanism to have certain build hooks trigger from vcs but not from sdist?

[umbral bridge]

no

[umbral bridge]

https://github.com/pytest-dev/pytest-html/issues/747#issuecomment-1789700531

[errant timber]

I don't see how to pass through args to the script in hatch (tox's {posargs} / nox's *session.posargs) AFAICT, hatch run test -- -v does collect it (no complaint), but don't see in the docs how I'd use it in the script line.

Ahh, it's {args}. Found by trial and error. Does not seem to be in the docs.

[umbral bridge]

https://hatch.pypa.io/latest/config/environment/advanced/#context-formatting

[dull rapids]

Took me a while to find that as well. Would probably be better to have an example with {args} that people run into sooner, together with this link.

[umbral bridge]

big day! finally landed this https://github.com/pypa/hatch/pull/1062

https://hatch.pypa.io/dev/config/static-analysis/

[tropic void]

The absolute Imports only creates some issues as for anything that wants to support being painlessly vendored

[umbral bridge]

I think that's a fine trade-off, folks can just ignore that rule for the vendored directory

absolute imports are easier to reason about and are required for things like Mypyc

[tropic void]

Can you elaborate on why mypyc needs them? Afaik they are supported

[umbral bridge]

no only relative imports in the current directory are supported, not anything outside the parent

[orchid birch]

Oy, this is a debate seems like I always have. I personally only use absolute imports, but some folks strongly prefer relative imports, both for vendoring and internal refactoring, although I point out that relative imports don't really make refactoring any easier. Relative imports are just harder to reason about, IMHO and I don't have much sympathy for the vendoring argument. I really, really, wish we had a more principled solution to the problems leading to vendoring, but I don't have any ideas on how to accomplish that.

[half cedar]

Very OT I really wish Python has something similar to Rust that makes relative imports more sane. Relative to the current module is nice sometimes but some sort of package-top-level-relative is more generally useful

[golden merlin]

I don't know this to be a hatch (and not a pip) problem yet as I haven't minimized the problem but is it a known issue that editable installs (i.e. PEP 660 I guess) don't work with namespace packages?

I know this used to be a problem in the Old Days™ pre-PEP 660

And I definitely observe they don't work with my package, but yeah haven't yet figured out whether that's hatch not doing something or pip not doing something by creating some smaller namespace package -- PEP 660 from what I see doesn't mention them

I guess I need both a minimal play package and also to see whether this works with any build tool not named setuptools

[umbral bridge]

should be fine since we use them at work

[umbral bridge]

can you show anything about the project that might help me diagnose?

[golden merlin]

Hm interesting!

The package in question is https://github.com/python-jsonschema/referencing-loaders

I decided to watch TV so didn't minimize yet

But yeah let me know if you see the same behavior I guess

[golden merlin]

A fuller reproducer seems like it's:

parent='git+https://github.com/python-jsonschema/referencing#egg=referencing'
child='git+https://github.com/python-jsonschema/referencing-loaders#egg=referencing-loaders'
echo "NOTHING EDITABLE: " && rm -rf venv && python3.12 -m venv venv &&
  venv/bin/pip install -q --disable-pip-version-check $parent &&
  venv/bin/pip install -q --disable-pip-version-check $child &&
  venv/bin/python -c 'import referencing.loaders; print(referencing.loaders)';
echo "BOTH EDITABLE: " && rm -rf venv && python3.12 -m venv venv &&
  venv/bin/pip install -q --disable-pip-version-check -e $parent &&
  venv/bin/pip install -q --disable-pip-version-check -e $child &&
  venv/bin/python -c 'import referencing.loaders; print(referencing.loaders)';
echo "ONLY PARENT EDITABLE: " && rm -rf venv && python3.12 -m venv venv &&
  venv/bin/pip install -q --disable-pip-version-check -e $parent &&
  venv/bin/pip install -q --disable-pip-version-check $child &&
  venv/bin/python -c 'import referencing.loaders; print(referencing.loaders)';
echo "ONLY CHILD EDITABLE: " && rm -rf venv && python3.12 -m venv venv &&
  venv/bin/pip install -q --disable-pip-version-check $parent &&
  venv/bin/pip install -q --disable-pip-version-check -e $child &&
  venv/bin/python -c 'import referencing.loaders; print(referencing.loaders)';
NOTHING EDITABLE:
<module 'referencing.loaders' from '/Users/julian/Desktop/venv/lib/python3.12/site-packages/referencing/loaders.py'>
BOTH EDITABLE:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'referencing.loaders'
ONLY PARENT EDITABLE:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'referencing.loaders'
ONLY CHILD EDITABLE:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'referencing.loaders'

(the namespace package only imports when neither it nor the root package are installed editable)

[umbral bridge]

sorry so what does this indicate?

[golden merlin]

something isn't working when installing packages editable no?

when either package is installed -e, referencing-loaders isn't properly installed

(sorry -- I guess it's not clear, but referencing.loaders is a module provided by referencing-loaders and not referencing)

[tropic void]

Referencing is not a namespace package, failure is expected,the installed variant worked by virtue of being a dirty mess

Aka when installed, the module is dropped in and work by virtue of being correctly located

However editable install needs correct namespaces

[golden merlin]

sorry, why not?

[tropic void]

There's a dunder init file

[golden merlin]

oh geez, the root one also can't have one?

silly

(ok, sorry for wasting time, I guess I'll move loaders.py to loaders/ and try again then.)

[tropic void]

I strongly recommend using entry points over namespace packages

[golden merlin]

yeah I knew I committed to memory "never use namespace packages" for a reason

ok let me remind myself how to do that

oh, no, I don't want this

All I want is to have a temporary package install some extra stuff, not a permanent plugin system

but I guess I'm gonna do the even more simple thing and just have it be totally separate.

[tropic void]

I shortly liked namespace packages,then I learned how the implementations worked,then came pep420

Sometimes I wonder if the number was chosen to indicate a drug incident

[golden merlin]

they were always complete pain IME, I thought I was being cute and saving myself time (which I'm sure I've thought 4 times before and each time found some silly thing)

idk, I'm probably just complaining at this point for not reading carefully again

[tropic void]

The namespace for complaints cannot be referenced outside of complaining

[umbral bridge]

good catch Ronny, thanks for saving me debugging time later today!

[orchid birch]

Let's just say that the number was chosen deliberately.

A bit of backstory: I was struggling with a particularly sticky Debuntu packaging problem, where to put the __init__.py for packages which contributed to umbrella packages. The zope.* ecosystem was a particularly rich and difficult one at the time. You can't put zope/__init__.py in any distro package because it's shared by all of them. This affected Redora ecosystem too. So Eric and I (we lived just a couple of miles from each other before I moved to CA) got together in person to tackle the problem. IIRC, it took like 3 completely fresh attempts to get something workable, both in design and implementation and we were going crazy trying to get it to work. 420 seemed appropriate given how much it hurt our heads, but we did end up with something that solved the problem in a nice way.

So yes, it was mostly a joke to pick that number.

[umbral bridge]

another fun PEP fact while we're at it: the standard way to define dependencies in pyproject.toml PEP 631 which got incorporated into the main spec is actually the number of my house that I built that same year and it was a total coincidence

• https://ofek.dev/house/
• https://peps.python.org/pep-0631/

[tropic void]

stairs at the entrance?

[umbral bridge]

yeah that part couldn't be modified really so I go through the garage

[tropic void]

what a pain

[umbral bridge]

although that also couldn't be modified very much lol they couldn't lower the door inside and I didn't want to have a superlong ramp so I have an elevator that takes me from the garage 3 feet up to the main floor and also goes up to the upstairs level

[turbid bane]

I thought stairs are pretty common

one of my neighbours has 10 steps leading up to their front door 🥴

tbf we live on an incline/the side of a hill

[umbral bridge]

oh yeah definitely I would normally only expect stairs but Ronny is referencing the fact that I'm in a wheelchair

[turbid bane]

Oh, I forgot about that 😅

[tropic void]

the leveling of thong looks slightly confusing - i dont get why the ground level cant be flat

my wife is in a wheelchar too . pain stairs are a no go for us

[umbral bridge]

stairs are only my number 2 enemy, my worst enemy is bugs because I can't do anything about them myself

[low gyro]

good test suite and a debugger should help 😄

[umbral bridge]

I would sacrifice a large chunk of my salary every year to never see a spider in real life again

[low gyro]

(joking of course)

[balmy dock]

sure, sure

[tropic void]

sounds like a job for a gigawatt laser sentry

[low gyro]

spiders (while not looking nice) are actually natural debuggers. as long as you leave them in peace, they will lure and catch all the flying bugs

[umbral bridge]

yeah that's true of course, to be fair I've never been bitten by a spider they just creep me out when I see them

[tropic void]

im fortunate in the ense that any spider whose main body is smaller than a hand is not scarry to me, and we dont have black widows over here

[low gyro]

yeah, kinda the same

[umbral bridge]

I just discovered such a creative use of standards https://github.com/dmlc/xgboost/blob/7196c9d95e706566424ae0b7ea0046fe9c5a3849/python-package/pyproject.toml#L1-L6

https://github.com/dmlc/xgboost/pull/9021#issuecomment-1507917599

built on top of hatchling, which led to modular code with minimal amount of boilerplate
❤️

[cursive lion]

Paul Everitt is getting into Hatch: https://fosstodon.org/@pauleveritt/111518734654999326

[orchid birch]

Nice! Paul's one of the OG 20: https://legacy.python.org/workshops/1994-11/pictures/PaulEveritt.gif

[umbral bridge]

so close now. updated list of users https://github.com/pypa/hatch/pull/1090

next is writing the release blog post, then releasing Hatchling followed by Hatch and then announcing everywhere 🙂

scratch that, lol https://github.com/beeware/toga/issues/2226#issuecomment-1842015758

first I'm going to finally write that page about the benefits versus various tools

[cursive lion]

I've heard criticism (sorry, don't remember where) that the community page looks like all those projects use Hatch, but some only use Hatchling. Perhaps that could be clarified?

[tropic void]

im responsible for another 100 internal repos or so at Redhat that converted to hatch

now that pytest dropped its setup.py i wonder i i should switch it over as well

[umbral bridge]

any blockers for this?

[tropic void]

Nothing technically, I'll do it soon

[balmy dock]

(I still don't even know what the difference is)

[cursive lion]

Hatchling is a build backend, similiar to Setuptools, Flit, PDM.

Hatch is a larger suite of project management tools. You can use Hatch for setting up envs, testing, building, publishing, formatting. So if you prefer to use a single tool, Hatch could be good for you, so you don't need to use other tools like tox, twine, Black etc (even if it might use some of those under the hood)

[balmy dock]

[umbral bridge]

are there any recommendations for making this distinction more clear? I have unified docs just like Poetry and PDM and I don't know if those projects get the same feedback about their CLI vs build backend

[umbral bridge]

the first part of the value proposition page is complete https://hatch.pypa.io/dev/why/

[umbral bridge]

as far as I can tell it is unbiased except for subjectivity in the ease of extensibility part

I love doing open source, people are so pleasant almost all the time https://github.com/sincekmori/tixent/pull/1

[golden merlin]

Has anyone written a hatch plugin to replace cffi's builtin setuptools one? (i.e. for building cffi projects)

or if not is someone maintaining a cffi project that uses hatch and has whatever I can copy paste to automatically trigger my build script

[umbral bridge]

not to my knowledge, I only know of a Cython one. do you mind explaining how that one works?

[golden merlin]

the setuptools one you mean?

from an end user perspective it looks like this: https://github.com/Julian/avif/blob/main/setup.py#L3

you get forced into using setup.py

and then you add a cffi_modules key which basically is a module spec -- and then it calls that function during the build process

(and the output of that is your .so / .dylib which it includes)

[umbral bridge]

that would be super easy to implement actually

[umbral bridge]

am I correct in understanding that to output the desired files, there is nothing special that setuptools does but rather you just have to call that compile method?

[golden merlin]

Yes I believe you are! (Though I've left my computer -- at some point I recall reading the source code for cffi's setuptools plugin and remember some non trivial stuff in it -- I'll look again later to try to remember)

But I think I remember a lot of that being because it's hard to customize build_whatever commands in setupyools

[umbral bridge]

actually it looks like either my memory does not serve me well or they have refactored to remove tight integration with setuptools, it looks like that literally builds the files and now their extension hooks onto that

please confirm when you come online

• https://cffi.readthedocs.io/en/stable/cdef.html#ffibuilder-compile-etc-compiling-out-of-line-modules
• https://github.com/python-cffi/cffi/blob/520e5f74cb6e92f5931c9e99452c5df61bfe23b7/src/cffi/api.py#L710
• https://github.com/python-cffi/cffi/blob/520e5f74cb6e92f5931c9e99452c5df61bfe23b7/src/cffi/recompiler.py#L1520

if true, this would be super easy to implement!

[umbral bridge]

nah, still coupled

[umbral bridge]

value proposition for environment management added, again I think this is unbiased but I'd like a spot check https://hatch.pypa.io/dev/why/#environment-management

[golden merlin]

Ah :/ fair enough thanks a ton for having a look anyhow!

[empty flower]

@umbral bridge well, there are 4 different modes, with different levels of coupling. I don't think cffi requires anything from setuptools, setuptools was just the only integration they created for building it

[umbral bridge]

do you know how to make it compile without any invocation of setuptools? for example the screenshot from above is this file which I would have assumed had nothing to do with setuptools: https://github.com/Julian/avif/blob/main/avif/_build.py

[empty flower]

Basically you just need to call the build script. The path to the build script is currently specified in setuptools metadata, you'd need a new mechanism to describe it. See: https://github.com/python-cffi/cffi/blob/main/src/cffi/setuptools_ext.py

[umbral bridge]

do you mean basically make a temporary setup.py?

if so, then yes I could do that which is how hatch-mypyc works actually. I just wish there was an easy mechanism

everything that has to do with compiling depends on internals of distutils/setuptools unfortunately (except for newer backends like maturin)

it's okay though, next year is the year I (along with help from Henry) start fixing that situation

yeah digging deeper CFFI definitely relies on internals: https://github.com/python-cffi/cffi/blob/520e5f74cb6e92f5931c9e99452c5df61bfe23b7/src/cffi/ffiplatform.py

[empty flower]

Oh, right, all of that. It uses that to drive a compiler

I imagine they'd appreciate contributions to migrate away from distutils there. But I don't know of any other best-practice API they could use

[low gyro]

that's the problem I guess. distutils are deprecated, but (aside from setuptools, which is practically the same) there is no alternative

[dull rapids]

extensionlib wants to be that, right?

[low gyro]

Yeah, but from what I have seen it's just a protocol for now without any implementation

[dull rapids]

I assume @umbral bridge referred to it here

[umbral bridge]

added value proposition for the new Python management functionality, I think I am content with this page for now and will start the blog post for the release https://hatch.pypa.io/dev/why/#python-management

[umbral bridge]

before releasing I would really appreciate people looking at that page to make sure I don't have anything wrong and that it is as unbiased as possible

for context: many people have been asking for such a page for a long time so I finally did it. I didn't actually want to compare verbosely to other software but I think the benefits do outweigh the risk of potentially annoying folks who maintain those alternatives

[cursive lion]

Optimized by default: The CPython distributions are built with profile guided optimization and link-time optimization, resulting in a 10-30% performance improvement depending on the workload. These distributions have seen wide adoption throughout the industry and are even used by the build system Bazel.

10-30% improvment compared with what?

how do they compare to the python.org ones?

[umbral bridge]

it would be the same as if you compiled with link-time optimizations and ran profile-guided optimizations using the PROFILE_TASK environment variable

the point is that they are built with all of the optimizations

benchmark: https://github.com/docker-library/python/issues/160#issuecomment-510272316

another one I read: https://developers.redhat.com/blog/2020/06/25/red-hat-enterprise-linux-8-2-brings-faster-python-3-8-run-speeds

the number I keep seeing is 10-30 so I wrote that, I can't actually find a table with improvements that show with and without the build optimization flags

[cursive lion]

that's good. and just now I read the first line that answers my question 🙂

"Here we compare Python management to that of pyenv."

I mean, answers this question: "10-30% improvment compared with what?"

[umbral bridge]

homepage is prettier now, here's a preview https://hatch.pypa.io/dev/

[umbral bridge]

social cards now! https://hatch.pypa.io/dev/install/

[umbral bridge]

halfway done the blog post, still on target to release tonight or tomorrow morning

[cursive lion]

btw there's a problem with the canonical link for pages like https://hatch.pypa.io/latest/environment/ pointing to https://hatch.pypa.io/latest/1.7/environment/ which is 404

See also:
https://developers.facebook.com/tools/debug/?q=https%3A%2F%2Fhatch.pypa.io%2Flatest%2Fenvironment%2F

[umbral bridge]

can you explain how to reproduce? what page did you start on?

I think the next release will fix it, the developer docs seem okay https://developers.facebook.com/tools/debug/?q=https%3A%2F%2Fhatch.pypa.io%2Fdev%2Fenvironment%2F

[cursive lion]

I googled for environments and went to https://hatch.pypa.io/latest/environment/

If you view source, look for the canonical link. iirc it was a relative link, so doesn't play well mixing latest + version

[cursive lion]

That looks OK, will have to see when it shifts to the non-dev URL scheme

1.7 at https://hatch.pypa.io/1.7/environment/

Also has a 404 canonical: https://hatch.pypa.io/latest/1.7/environment/

https://developers.facebook.com/tools/debug/?q=https%3A%2F%2Fhatch.pypa.io%2F1.7%2Fenvironment%2F

[umbral bridge]

I forgot to post the release announcement here! https://hatch.pypa.io/latest/blog/2023/12/11/hatch-v180/

[umbral bridge]

FYI there are two fixes I have to get in the next Hatchling release and I'm trying to get that out tonight. However I had an injection in my spine today and it's causing a massive headache so I don't know if I will be able to finish. Also FYI I think I will yank 1.19.1 due to a pretty bad bug introduced while fixing an edge case

[umbral bridge]

actually one bug is not a bug, release will happen tonight https://github.com/pypa/hatch/issues/1130#issuecomment-1853296562

[umbral bridge]

https://github.com/pypa/hatch/releases/tag/hatchling-v1.20.0

[umbral bridge]

thanks to Paul I have a short time documentation strategy that will help a lot https://github.com/pypa/hatch/issues/1123#issuecomment-1852328847

[gusty yew]

Hi, I am working in a somewhat constrained environment where I’d like to use Hatch but I cannot do system installs or install pipx and I can only install Python via conda. Is there a good pattern for using Hatch in this case?

[umbral bridge]

yes Hatch is available on Conda and also there is a plugin for environments https://github.com/OldGrumpyViking/hatch-conda

[gusty yew]

Thank you! Does installing Hatch in a separate conda environment via pip and then calling it directly sound like a reasonable setup given my constraints?

[umbral bridge]

I think so but I'm not intimately familiar with that ecosystem anymore

[gusty yew]

Thanks, no red flags is enough for me 😀

[golden merlin]

Just to point out, even though I don't use pipx (I use a tool I wrote which is essentially the same), you presumably don't need to "install" pipx to use it

[gusty yew]

Could you elaborate? I think I’m missing the distinction

[golden merlin]

I assume when you say you can't install things that you mean as root, or via your package manager, etc

But tools generally don't care about that, you can install pipx into a venv, or a conda env, or just use it as a pyapp IIRC.

[gusty yew]

That’s good to know! I’m dependent on an internal pypi mirror that was missing pipx, but if I can get it approved, I’m going to try that!

[tropic void]

@umbral bridge i observe: ```$ hatch version
[12/18/23 16:51:13] WARNING pyproject.toml does not contain a tool.setuptools_scm section setuptools.py:119
WARNING pyproject.toml does not contain a tool.setuptools_scm section setuptools.py:119
23.12.07.1.dev4+gd932260e.d20231218

the project is using hatch already - and the error disappears when installing hatch/hatch-vcs directly

any recommended action for debugging?

[umbral bridge]

I'm not sure where that error is coming from, perhaps setuptools?

[tropic void]

the log can only be triggered by setuptools, but thats not in use

@umbral bridge the log as given can only happen if rich is installed and someone tries to load setuptools_scm config files - which is the hist venv, any build system venv wouldnt have it

@umbral bridge when i try to bubble the error, hatch hides it in ```
Environment default is incompatible: pyproject.toml does not contain a tool.setuptools_scm section

its not happening on master, seems to relate to a version pin of hatchling, digging thedetails now

still happening on the latest releases,

@umbral bridge identified the order of items - _find_existing_interpreter would trigger it via virtualenv calling distutils

@umbral bridge as master fixes it, i'll simply ensure we install all deps so hatchling version can be used instead

[umbral bridge]

fix will be released today

[umbral bridge]

actually can you please show me the exact warning that should be suppressed?

[tropic void]

its not a warning, its a log

[umbral bridge]

https://hatch.pypa.io/latest/blog/2023/12/18/hatch-v190/

[umbral bridge]

I have such a dilemma

I can increase responsiveness by 40 ms if for dependency checking I do no normalization and consider for example FooBar as different than foobar and thus trigger a cache miss which could then check the actual state of the virtual environment

basically lazily importing packaging.requirements because it is apparently very slow to import

or I could introduce a super hack and have two layers of dependency hashes, the one I have now based on the normalized dependency definitions and a new one that would come before that

[half cedar]

Doesn’t normalisation only need packaging.utils and is fairely performant?

[umbral bridge]

not in this case because I also make sure it's a valid Requirement

[tropic void]

Can upstream be fixed?

[umbral bridge]

https://github.com/pypa/packaging/pull/754#issuecomment-1871698628

[tropic void]

Amazing 👍👍

[slim cairn]

What's the simplest way to override the wheel tag?

[umbral bridge]

set up a local build hook and use the tag option https://hatch.pypa.io/latest/plugins/builder/wheel/#build-data

[slim cairn]

Hm that sounds not simple haha 😄

[umbral bridge]

You'll need to define a custom build hook https://hatch.pypa.io/latest/plugins/build-hook/custom/ in order to set wheel build data https://hatch.pypa.io/latest/plugins/builder/wheel/#build-data:

pyproject.toml:

[tool.hatch.build.targets.wheel.hooks.custom]

hatch_build.py:

from hatchling.builders.hooks.plugin.interface import BuildHookInterface

class CustomBuildHook(BuildHookInterface):
    def initialize(self, version, build_data):
        build_data['tag'] = '...'