#hatch

1 messages · Page 4 of 1

errant timber
#

It's a cli toolkit, like click and argparse, but class based

#

plumbum is basically a collection of tools to help you write command line scripts. It predates good packaging, so it bundled stuff that really should have been seperate packages

#

The reason I'm thinking about it now is I want see if I can get Python 3.14's color --help output into a mkdocs project. 😉

#

colorlib has the ability to read and write ansi and write html. So I think I could convert with it. (I'm trying Rich first, but it adds a lot of stuff around the html)

umbral bridge
#

I forget the name but for color have you tried that package Hugo maintains? it's been around for a while but I think ownership was semi-recently transferred so it's in a better shape now

errant timber
#

termcolor, it seems, and doesn't seem to support html.

errant timber
tight lagoon
#

FWIW, I would love to have the opportunity to install Plumbum without those sub-packages. I'm a fan of modularity for its own sake. (And, well, you can imagine how I feel about Numpy as a result ;) )

#

re import time, my understanding having them namespaced would imply importing plumbum.__init__ before the sub-packages if it exists; if you can design the sub-packages to not depend on the main package, then the cost of the namespacing should be trivial (just chasing the path further and creating an extra empty module object and setting its attribute)

#

that sort of thing is much faster than running top-level code in a module (which I can confirm can take single-digit milliseconds per module; worse of course if there are transitive imports)

#

(for me, time python -c 'import plumbum' is about 80ms more than time python -c 'pass')

errant timber
#

Those packages are just a bit of pure Python without side effects (I think I delayed the color protection atexit). I need to check with -X importtime, but I think they don't add too much if anything. This idea wouldn't allow that, but maybe more parts could be pulled out. I really dislike plumbum.path since it's pre-pathlib, but it's remote version makes it both unreplacable and tangled with the local and remote submodules. And plumbum has a lot of top-level objects.

I was wondering if hatching can do it, and if there might be a problem doing it. Pretty sure the first answer is yes, but not sure how involved it might be.

#

You must love SciPy. 🙂

umbral bridge
#

oh I was wondering why you were asking in this channel, my bad I should have interpreted it as a question about packaging lol

#

yeah that's totally possible with Hatchling

#

not sure it would work out-of-the-box without config, let me check

#

so you would have to do manually for each package:

[tool.hatch.build.targets.wheel]
packages = ["src/plumbum"]
errant timber
#

Okay, I think I have an idea of how to do it. Will research uv's workspace feature to try to keep them in one repo.

#

I'm fine with (and expect) config for somethign like this 🙂

tight lagoon
#

oh it's about building multiple wheels from the same source tree, I get it. Yeah, if all else fails you can script "re-edit pyproject.toml and call the build frontend again"
(and NumPy's weight is fine when you need the full SciPy experience and that's part of it. When you have a couple hundred LOC to fill a bytes and just want to interpret it as a 3d array and slice it, maybe not so much. I don't even know how to evaluate whether my code will actually touch the blas/lapack libraries, actually.)

umbral bridge
#

oh thanks for the clarification. changing the name dynamically is not allowed so yes you would need multiple pyproject.toml files

tight lagoon
#

(maybe I misunderstood? and sorry about the off-topic part)

errant timber
#

Yes, that's a big part of it; I'm thinking I'll make two subdirectories for the two namespace packages. I think uv's workspaces (IIRC Hatch doesn't have them yet?) might be able to manage them. Then I can set them up with the config above. (And for SciPy, I was referring to the fact it's basically a big collection of unrelated modules that were bundled together since packaging was bad at the time, and the small collection of Fotran makes it really hard to compile the whole thing)

umbral bridge
#

correct no workspaces yet, I've been extremely demotivated

tight lagoon
#

I've never been a fan of all-in-one tools of that sort, but my hat's off to anyone who makes a multi-project "workspace" concept for Python that catches on. I wouldn't know where to start with the design.

slim cairn
#

We do have it in uv!

#

It's pretty early days though, so I'd love to hear more about whether it works or not for you.

tight lagoon
#

I don't know that I have the kind of project where it would help, but I suppose I should at least try to play around with it. (Actually, the tools I'm "currently working" on seem like the best candidates...)

inland hemlock
slim cairn
errant timber
#

Getting off topic, but I'm hoping to support uv workspaces in cibuildwheel when we add a native uv frontend (we have build[uv] currently). But that's pushed off till 3.1. I'm hoping doing it with hatchling in Plumbum will be a good way to learn about them (not binary ones, but hopefully close enough)

#

If hatch still doesn't support dependency-groups, I might try to contribute that in the future. 😉

errant timber
#

BTW, I noticed Hatch is reporting hatch-test.py3.14 -> cannot locate Python: 3.14 though I've installed 3.14 in GHA.

umbral bridge
#

show config/branch

errant timber
#

macOS is fine. Linux and Windows are not able to find it. I'm using the free-threading Python, since GHA's regular Python is broken on Windows for bulding (and Hatch needs a build with cffi)

#

I wonder if it happens to work since macOS has both versions (even though I request/activate the free-threaded one), but the others don't. I bet that's it.

#

Is there a flag I can use to make a missed Python an error? Nox sets the flag for you if CI is defined, since it's an easy footgun on CI.

umbral bridge
#

can you show the branch?

errant timber
umbral bridge
errant timber
#

I just happened to notice I wasn't actually testing what I thought I was testing, CI was passing but 3.14 was mostly skipped.

umbral bridge
errant timber
#

I can't ask for 3.13t, etc? I was rather planning on trying python = ["3.14t", "3.13t", "3.13", "3.12", "3.11", "3.10"]

umbral bridge
#

no

#

is that an actual standard syntax/identifier?

errant timber
#

It's the syntax of setup-python, uv, pyenv, virtualenv, tox, and nox, at least.

umbral bridge
#

I guess I have to support it then but it's a bummer because it's literally temporary until it becomes the default

errant timber
#

It seems to work already, actually

#

I tried it before you told me about the envvar

umbral bridge
#

like say you have 3.15 and freethreaded is the default, how would a user request the option to not have that, you know? I don't think that's something folks have figured out with that syntax which is why I consider that an option for an identifier rather than a new identifier

#

what works?

errant timber
#

I bet there's going to be a 3.16g or something (I'd be pretty surprised if the default flips in 3.15, I think working on a thread-safe JIT will be the focus of 3.15)

#

But they are living together for at least a few more years, and testing on both at the same time is really nice.

umbral bridge
#

I don't recall supporting that syntax, so I'm not sure how it's working

errant timber
#

It might be picked up from virtualenv/uv? It even gets the session name right. Trying it now with everything.

umbral bridge
#

oh I guess since virtualenv supports that syntax and Hatch passes the name for resolution then it would work as a byproduct, however if it wasn't already installed Hatch would not be able to download it

errant timber
#

Works on macOS, and I realized I can install hatch with 3.13 and test on 3.14, so both Windows works too. It looks like yaml won't import on 3.14t on ubuntu (only) due to a circular import (strange), so hatch-test.py3.14t fails there.

#

But as long as they are pre-installed, it works well 🙂

dull rapids
#

Hio, would be great to get some context for hatch shell’s parameters: https://github.com/pypa/hatch/issues/1972

They’re undocumented, and I don’t know if their behavior is expected or not (like, is name only for display or is it supposed to select a shell by name but that part doesn’t work?)

tropic void
#

@umbral bridge whats the recommended way to run hatch tests against master of pluggy - im reworkin downstream testing in pluggy and took note that the hatch run dev command is no longer working, probably never was (as its not editable isntalling the pluggy)

tropic void
umbral bridge
tropic void
hoary wraith
#

Going to do my best to word this and it not be confusing, what is the recommended way in Hatch if I need to ensure a plugin is installed by all users before any other installs can happen? I need to make sure my users are using a plugin that will give them a multi-package virtual environment and setup to point to a proxy rather than public pypi. What I mean by multi-package is not multiple packages in the same git repo but rather multiple git repos under the same parent directory.

tight lagoon
#

... is there a reason the plugin can't just be declared as a dependency?

dusty parcel
#

I believe the idea is the plugin affects dependency resolution during build time? If so that would require either declaring it in the buildsystem requires or disable build isolation I think

tropic void
dusty parcel
tropic void
#

that mapping is different form the env mappings

hoary wraith
#

The requirement is pre-install of any environment this plugin is installed. But I think Ronny got me what I was looking for as I think I can use EnvironmentInterface here to get what I need to do. And yes this is a very weird thing but unfortunately how things work at Amazon where the primary structure of work is polyrepo based.

karmic glacier
#

well, finally migrating a package to use hatchling, two years later.

inland hemlock
#

for build hooks, is there something similar to force_include? In particular, if you add to force_include then the package will be automatically recognized without the user needing to explicitly specify tool.hatch.build.targets.wheel.packages -- but when a user does an editable install, the files are added to the wheel and copied to site-packages. The .pth file points at the original source so it works as expected -- but it's really weird that the files were copied to site-packages.

I tried just adding to artifacts, but that didn't resolve the issue.

dull rapids
#

That might be because the correct way to access files shipped with a package is importlib.resources, which might try to load things from site-packages. But I’m not sure.

#

I use artifacts so files are included in the wheel that are .gitignored. I think it’s only for that and doesn’t influence anything other than “don’t ignore this file”

umbral bridge
inland hemlock
#

Nope, I actually want the inverse of force_include_editable -- only include things when it isn't building an editable install

merry sable
#

hi everyone 👋 I'm dropping in here as I noticed that the Hatch repository in PyPA hadn't had much activity in the past 5 months. We have tutorials and have been teaching using Hatch. I wondered what is going on with the project. @umbral bridge i also DM'd you in slack. I'm just a bit concerned and wondered if the project has plans to move forward / be maintained. It's such a great community project and we want to support community projects. Thank you all for any insight!

umbral bridge
#

For posterity: we discussed that I am looking for a co-maintainer, the project is not abandoned, and almost all functionality requested in issues is supported by plugins

merry sable
karmic glacier
#

I saw that pylock.toml has been accepted, and I know that hatch was waiting for a standard to implement lockfiles, is there an eta or pr in the works for pylock.toml lockfiles?

umbral bridge
#

no, but luckily I think I might have found a co-maintainer two days ago so when I find out more details about their/their team's level of commitment I will be able to offer more insight into the roadmap 🤞

karmic glacier
#

oh neat! thank you ofek!

hoary wraith
#

👋 I thought I would give a proper introduction as my name starts to pop up more. I am Cary, I am a Python Ecologist at Amazon, I help maintain the Python ecosystem internally for Amazon. Some of you who have attended PyCon might know my coworker Jeff Edwards. My previous open source contributions have been primarily in Airflow and poetry_plugin_pypi_proxy. I am going to be trying to spend more time contributing back to hatch starting with workspace support which had an absolutely amazing foundation from Ofek already started. Once I get through some of that work I can certainly chat with Ofek about implementing lockfile support once that happens.

hoary wraith
#

I want to get opinions on workspace support, Right now what I have put in will allow defining the names of the workspace packages under "tool.hatch.workspace" in pyproject.toml as an array of names. I think that part is the right approach, how much of workspace support should include CLI commands such as workspace remove, workspace add, workspace update? I want to make sure that what comes out meets the communities needs. 🙂

umbral bridge
#

I'd say that we introduce the baseline support first and then in a subsequent PR we can add those commands and potentially ones to manage dependencies as well (or that can wait until lock file stuff is added).

slim cairn
#

We've gotten pretty far without workspace-specific commands in uv

umbral bridge
#

Oh, I didn't realize there weren't any. @hoary wraith Let's not do that for now then 😄

hoary wraith
#

Oh good to know, I did not realize there were not any either. Then I think we can definitely hold off and focus on the lockfile piece. I should have time this weekend to be much closer to a PR for workspaces.

slim cairn
#

uv add <path> just adds it as a workspace member if it's in the current project

#

and uv init <path> in a workspace does the same

tropic void
#

@umbral bridge aware of any hatch plugin that uses hookspecs ?

im coming up with a complete removal of the pluggy dependency thats still backward compatible (putting my 3 month free claude max gift in use)

umbral bridge
tropic void
#

@umbral bridge hookimpls are used by all of them - i have compat forthat directly in place - but basically as all the hooks you have only ever return types and never take contextual args, its safe to jsut call them - so i can invoke all the pluggy hooks without pluggy - its just that legacy plugiosn would get warnings

low gyro
#

@tropic void is pluggy a deprecated package?

tropic void
#

@low gyro nope, but hatch uses it in a way thats ludicrus - all its doing with it is looking up classes without ever doing anything fun (like pytest does)

#

as all thats done is look up classes, its not sensible

low gyro
#

ok, makes sense. thanks for clarification

umbral bridge
#

I see, yes thanks, I don't think any plugins use specs

tropic void
#

@umbral bridge im going cazy with your custom massive explictit ruff config that breaks in different places and envs

umbral bridge
#

what breaks?

tropic void
#

different rules are missing on co, local, and after update that changed -

#

i'll undo all changes to the ruff rules file and then figure how the local one is supposed to run

#

interestingly it nworks now without a hickup - i suspect i might have had a poisoned cache for the fmt env

#

ah, still a behaviour difference

#

this is precisely why i use pre-commit - drift is so tricky

umbral bridge
#

I dislike precommit very much

#

although I know many folks have the opposite view

tropic void
#

i can understand the dislike - its very specific design choices for stability dont match the expectaions of some folks - of some parts im also not a fan - but its rock-solid and good enoug

tropic void
umbral bridge
#

That looks pretty good! I plan to land workspaces first (effectively frozen until @hoary wraith is finished) and then merge yours afterward.

upper hound
#

I’m curious - what is the zstd decompression in hatch for? Are there private wheel indices with zstd-compressed wheels or something?

low gyro
#

python from python-build-standalone is compressed with zstd

#

IIRC Hatch supports installing those Python distributions

hoary wraith
hoary wraith
#

I have marked the PR for workspace support ready for review, I am still working on what the changes need to be to dogfood it within CI but the primary code is ready. https://github.com/pypa/hatch/pull/2073

dull rapids
tropic void
#

@umbral bridge hatch is still hiding errors in version sources by cutting off traceebacks to be missing it all

hoary wraith
hoary wraith
tropic void
#

i dont recall if we filed one when i ran into this last time

umbral bridge
balmy dock
#

User reports: can't import my package
Just removing all Hatch config fixed it: https://github.com/fmorton/Birdbrain-Python-Library-2/pull/1

The dist-info folder was being created
/usr/local/python/3.12.1/lib/python3.12/site-packages/birdbrain_python_library_2-0.9.28.dist-info
But the folder with the actual code was not
/usr/local/python/3.12.1/lib/python3.12/site-packages/birdbrain_python_library_2

I've seen this before, but last time the build config just excluded all code from the wheel, so the wheel was just empty
Here, I still see all the code in the wheel on Inspector, so I'm not sure why it's not getting installed
https://inspector.pypi.io/project/birdbrain-python-library-2/0.9.26/

umbral bridge
#

The packages config should never be used with the sdist build target.

balmy dock
#

(why not?)

umbral bridge
balmy dock
#

👍 ,all good, take care of yourself, I'm just curious

umbral bridge
#

basically as the docs say it corrupts the path of the source distribution such that when you build the wheel from the unpacked source distribution the directory structure doesn't match the repository

karmic glacier
#

Ahhhh

#

I came here because I was having problems of my own migrating but that clears it up

hoary wraith
#

If you have a hatch plugin that utilizes the EnvironmentInterface, can you please test out changes from https://github.com/pypa/hatch/pull/2073 Ofek and I would like to make sure that these changes do not have a large negative impact on existing plugins, there have been some pretty significant changes to the EnvironmentInterface to make way for workspaces.

turbid bane
#

We can and should debug it further on pip's end, but if there is a potential issue on hatchling's side, I thought you'd want to know.

umbral bridge
#

commented

quiet pebble
#

Hi

tight lagoon
#

https://hatch.pypa.io/1.13/cli/about/ only gives general information about using Hatch's CLI. On the main page, it's linked as "CLI reference"; so I expected to see a list of all the subcommands that can be typed after hatch on the command line.

hoary wraith
tight lagoon
#

ah, that's the link labeled "root options"?

#

also, the "config file" referred to in documentation for hatch config... is that pyproject.toml or just what

#

ah no, I guess that's what hatch project is for.

hoary wraith
dull rapids
#

I’m stoked for dependency group support! What’s the plan for the next release?

hoary wraith
#

The next release is happening next week and will include dependency group support, workspace support, adding 3.14 and dropping 3.9, and support for writing sboms to wheels.

orchid birch
#

That is awesome

hoary wraith
#

I expect the release to happen tomorrow or Wednesday. I will share this here because it is going out in the blog post associated with the release. For anyone that did not know, I am taking on the responsibility of co-maintainer along with Ofek. I am here to hopefully reduce the burden on Ofek as much as I can. After this release, we will be working on something in the form of a roadmap for additional features to come including a typing command similar to hatch fmt and certainly implementing any additional PEP standards that might be missing from hatch and hatchling. I also want to thank everyone for their patience as I held up some changes due to the nature of the workspace changes and how big they were.

upper hound
#

I’ve got it on my list to work on free-threaded CI for hatch this week now that the zstandard dependency PR is merged

humble smelt
#

For anyone that did not know, I am taking on the responsibility of co-maintainer along with Ofek.
I didn't know explicitly, but somehow, I sort of guessed... 😜
Fantastic to hear! Fewer single points of failure, hopefully less stress and burnout risk, etc. All the good things! 🎉

cursive lion
hoary wraith
orchid birch
hoary wraith
hoary wraith
#

Hatch 1.16 and Hatchling 1.28 have been released! Thank you to @ofek for all the help on the release and reviewing all the code I put out. Thank you also to the other contributors that have helped add features and changes!

umbral bridge
umbral bridge
#

Heads up that starting this evening I'll be off the computer for several days on a trip and will return Sunday evening.

hoary wraith
#

Enjoy the time away!

umbral bridge
# hoary wraith Enjoy the time away!

Thanks Cary! It's difficult to express how appreciative I am of you unblocking development for us. Nobody knows this really but the state of the project has imposed a certain cognitive overhead that I feel at every moment of every day. It's like when you know you have to respond to emails, but so much worse lol. Now I feel better and am super excited to continue development 🙂

hoary wraith
#

Bug fixes are out for Hatch with the release of 1.16.1 to address regressions that occurred around context variable usage in dependencies, special character paths and adding support for dependency-groups to option overrides.

dull rapids
#

Fastest fix in the west ❤️

errant timber
#

And it’s on conda-forge now. 1.14.1 -> 1.16.1. 😳 Now finally using a ratter-build recipe, too.

umbral bridge
upper hound
umbral bridge
errant timber
#

Quick bug report: it seems hatch 1.16 is not installing local extras, both in classic and dependency-group from. In repo-review, I had an environment that was self-depending on "repo-review[cli]", but hatch 1.16 stopped including the cli extra. Moving to dependency-groups didn't change anything. Adding the dependency what is part of that extra manually to the dependency-group does fix hatch's environment.

umbral bridge
#

Which version of Hatchling?

errant timber
#

Last working build was last week (with 1.15.1), so I'd guess 1.26.3 vs. 1.28, though I think this would be in Hatch?

# 
[tool.hatch.envs.docs]
# duplicated since hatch doesn't support groups yet
dependencies = [
  "repo-review[cli]",
  "sphinx >=4.0",
  ...
]

Basically stopped installing the cli extras when building docs.

umbral bridge
#

Part of the changes that were pending on master for over a year include migrating some utilities from Hatchling so that it isn't required at runtime (not there yet) and I'm not sure if this would be related to config parsing over there or now Hatch

hoary wraith
#

Does this happen in 1.16.1 and 1.16.0? Just want to confirm which version as I have a potential idea of the issue but it would likely only exist in 1.16.1 if it is what I am thinking it might be.

errant timber
#

I can try 1.16.0

#

Locally, 1.16.0 also fails.

hoary wraith
#

I think I have this isolated to a change that was made for workspace environments that might be incorrectly filtering out these extras for the local extras case.

umbral bridge
hoary wraith
dull rapids
#

Is this released? Seems severe

umbral bridge
umbral bridge
dull rapids
#

Thank you so much!

errant timber
#

I'd like to set up downstream testing for hatch and/or hatchling in packaging. I need to be able to run pytest directly in a nox environment. It would be nice if hatch had dependency-groups, so I could just do -e. --group=test for it. And I'm not sure what plugins are actually required for hatch, since some of them are not listed expliclity in extra-dependencies. And a lot of failures are popping up. How should I go about that?

    elif project == "hatch":
        session.install("-e.", "pytest", "filelock", "flit-core", "trustme", "editables")
        session.run("pytest")
hoary wraith
errant timber
#

Uhoh, hatch noticed ._version has been removed.

#

Changing the internal _version tuple then reassigning it is very much not supported! Version objects are not mutable. That's part of why ._version used to exist.

#

I think the other failures are issues with the setup, but this one very much is something that will break with packaging 26.0

elder lotus
#

Is there an issue you found?

errant timber
#

I'm not sure how bad the breakage is, but it breaks some of the tests in hatch.

#

I've got a backwards compatibility patch, not sure if I should introduce warnings now or in the next release, though (in packaging)

elder lotus
#

I've played around with this, this breaks the hatch version CLI, that is used to bump versions of a project

#

Look like the new __replace__ method would be a great fit for the code here

#

I don't see any reason to wait on adding a warning, only delay situations we don't know about from finding out it's going to be removed

errant timber
#

I can run the tests directly from hatch (not sure what I need to do to get it working in our downstream version), with the patch, I think the rest of the failing tests are due to the space with @

#

I was assuming something in hatchling would break, since it's in the hatchling source, that's good if it's only a CLI breakage

#

Yes, this is basically an implementation of __replace__

elder lotus
errant timber
#

@elder lotus Do you know why the license file helper needed whitespace changes? I think I noticed that but didn't look into why.

elder lotus
hoary wraith
dull rapids
errant timber
#

I think I can just limit packaging for the tests to get this in, but it would be really nice if the tests could pass with packaging 26 🙂

hoary wraith
elder lotus
hoary wraith
sturdy agate
#

it would be valuable for us to have more voices on the proposal than just uv and pip

hoary wraith
#

I am going to gather my thoughts on the PEP this weekend from the hatch perspective. I think Jonathan and others know my primary stance already as Henry, myself and some Nvidia folks at Scipy 2025 had discussions around security concerns at the time of arbitrary code execution and that does still seem to be present in some ways.

sturdy agate
#

highly appreciated! based on that feedback in the first round, we've redone the security model in this iteration: the tl;dr is that untrusted code must never run without opt-in, while tools vendor trusted providers (say having GPU detection for a pypa-org library, something to hash out after the technical proposal) so that opt-in is not necessary for let's a <tool> install torch (avoiding security fatigue). there seems to be some confusion about what this means for tools, so we appreciate feedback on what we need to explain better (or what you'd rather do different).

hoary wraith
#

There is a lot to digest with that PEP so these are just initial thoughts while I work on a more coherent reply to go into DPO after Ofek and I align.

I think my concerns around provenance being potentially broken by encouraging that tools vendor plugin providers have mostly been already surfaced by others in DPO. I think I am most ammenable to VariantLib providing the list of trusted providers through it as declared dependencies. Vendoring these things breaks provenance. This is less of a hatch issue here and more of me putting on my security hat as the person that is responsible for ensuring a healthy Python ecosystem at Amazon. Vendoring risks provenance for me in one key area and that is tainted code and hidden changes. If code is copied into the vendor directory without clear documentation of its origin (e.g., version number, commit hash, original repository), it becomes hard to verify if the code has been altered or contains malicious additions.( I am happy to have Seth tell me I am wrong here if I am) And I do not see the PEP taking a clear stance to say if a tool chooses to vendor that they must maintain provenance. Perhaps the PEP is not the right place for taking that stance but that just sticks out to me as where I see some concerns.

From the build backend perspective I do not see anything that sticks out as problematic. From the hatchling perspective we are likely going to push

They SHOULD also query providers to determine whether variant properties requested by the user are valid, though they MAY permit skipping this verification and therefore emitting variant wheels with potentially unknown properties.

as being build hook plugins that a user can choose to use. There are too many variables for this for hatchling to provide good defaults out of the box. This would also remove hatchling from the middle of the trust relationship here. Providers here should be who the users trust on checking for valid properties.

I do agree that there needs to be a balance to avoid security fatigue occurring when it comes to the opt-in mechanism of provider plugins.

A lot of what is in this PEP are things that we generally defer to package managers already on the install side for hatch. So I would defer to you and the other uv folks as well as Paul and others on the pip side to have more meaningful input on that side of the PEP.

One thing of note, and this was provided as feedback on https://github.com/pypa/hatch/pull/2037 and my hope here is that the intention is that these are going to live in VariantLib. Otherwise I have a lot of concerns here about drift between tools.

sturdy agate
#

thanks, i've shared the feedback with the other PEP authors

hoary wraith
#

Thanks! Hopefully it all makes sense, I know the DPO has sort of gone in a bunch of different directions and some people are really digging in on certain aspects. That is not my intention with my feedback just where I see some potential concerns. I do think the security aspect might be getting blown a little out of proportion in DPO because the reality is that there really should only be a small number of trusted publisher plugins in terms of namespaces as those should be based on hardware vendors. Which is why I think having them as dependencies of VariantLib is an appropriate move and then for any untrusted plugins those can be opt-in by users that need them. It is a model that works well with us for hatch.

hoary wraith
dull rapids
#

Happy to! I think in the future, the main way to do lockfiles shoule be declaratively, using a per-env locked = true setting or so. This makes it possible to keep the PR almost as it is, but I’d probably rename the -o/--output option to --export and make it mandatory for now so it’s clear that hatch doesn’t use that lockfile yet when creating or syncing environments. More details in my comment on the PR.

hoary wraith
#

Thanks! That was the other direction I was leaning was being declarative per environment because i do think that is more flexible so i think with that feedback that is the right way to go now. And that also allows this to happen during different commands, while working on this draft PR the thing that kept coming to mind is having a sync command for environments.

dull rapids
#

The missing sync command was weird, I always did hatch run <envname>:true or so

hoary wraith
dull rapids
#

yay! thank you!

hoary wraith
#

I know, my fork got into a weird state so I had to do that to get the release out. I still have all that code locally and can get it onto the new fork pretty easily. I should have a new PR for that out this week as I would like to get lockfile support ready to go.

rare spear
#

hey

#

the PR for the fix with virtual env has failed on what looks like an unrelated issue

#

@dull rapids

dull rapids
#

yeah, I think we need @hoary wraith or @umbral bridge to restart them or fix them on main.

rare spear
#

what timezone are they in?

dull rapids
#

idk, but @umbral bridge is online rn

hoary wraith
#

I am in Pacific Time and just getting my morning started. I have restarted the tests and will be reviewing the PR shortly. I have been working on a fix for those tests already but put them on hold while we were making changes to the release process and releasing the latest release.

umbral bridge
#

Hey, let me know how I can help

rare spear
#

looks like it just merged. Thanks a lot!

hoary wraith
#

PR is merged. I have. a few meetings this morning and then I can get to a release

hoary wraith
#

Release is happening now. Sorry for the delays but now the issue with the publish tests failing intermittently is fixed 🎉

errant timber
#

I'm seeing some odd behavior when making a copy of a repo (in a test): building an SDist in that repo copy doesn't respect the .gitignore anymore, uv.lock is being included in the SDist. Git is working correctly and does not show uv.lock. I don't see any difference between the two directory trees other than some extended attributes didn't get copied.

#

(I'm trying to fix the test suite for check-sdist to work in parallel)

errant timber
#

It has something to do with being in the pytest temporary directory

errant timber
#

Found it with GPT-5.3-Codex. It reads the .gitignore, then starts processing the ignores, and comes across var/ pattern caused hatchling to think the entire project dir is being ignored. Then it throws out the rest of the ignores after that, since now it thinks everything is being ignored, and goes back to the default ignores.

#

I don't think it should be checking absolute paths, but only relative paths to the repo. Git itself works fine if you ignore var/ but have a var in the path to the repo itself.

#

The same thing could be triggered by putting the repo in a build directory, since build/ is also a common ignore.

#

Unrelated: hatchling seems to be normalizing version everywhere, when it should only be normalized for wheel/sdist names. The metadata should contain the exact version the user inputted, like 2026.03.03, so PyPI displays the correct version, just like the way names work now (normalized name for SDist/wheel, unormalized name for metadata and PyPI display). We might be adding a property to packaging.version.Version to make it a little easier, but when I was looking around for this bug, it seems like strings are passed around most places, so it should be easy to fix, I think.

#

(People have been complaining about git tags not matching PyPI versions for CalVer, and I'm pretty sure this would be enough to fix that)

#

(I can open issues but I haven't worked that latter one out fully to know if it does get displayed correctly if the metadata is fixed)

#

The first problem could be a security issue when making SDists, actually. It means I can change what files are included in the SDist simply by moving the repo into a path with a specific name, removing the ability of the author to control what files get excluded in the SDist.

hoary wraith
#

@errant timber I will try to take a look at these this week. Feel free to open issues even if the latter one is not fully worked out. I will have enough context here to at least know the shape of the problem to dig into it.

dull rapids
tropic void
#

psa: i beleive im finally finished with the iteration on setuptools-scm 10 and vcs-versioning 1 - my next step on that front would be to add more api in vcs-versioning to make hatch-vcs work nice and idrectly with it (no more setuptools)

tropic void
dull rapids
tropic void
dull rapids
#

Awesome. Of course the packager had a bug and I needed to push a new release, so make sure to try 0.1.1!

errant timber
#

Does hatch not work if there's not a python command? I've only got python3 on this mac, just tried brew and uv tool installs for hatch, both crash with FileNotFoundError on python. I'm trying to do hatch run docs:serve, and my command is just sphinx-autodocs (it's failing setting up the dependency-groups). Maybe I'm doing something wrong, though.

hatch run --help's formatting is wrong for one of the toml blocks, too.

#

Opened an issue and PR for the latter

#

uv run hatch run types:check is fine. Thinking it might be a cache issue

# 
hatch clean
/Users/henryfs/Library/Application Support/hatch/env/virtual/repo-review/bd7TtdJB/hatch-build/lib/python3.14/site-packages/vcs_versioning/overrides.py:609: UserWarning: No GlobalOverrides context is active. Auto-creating one with SETUPTOOLS_SCM prefix for backwards compatibility. Consider using 'with GlobalOverrides.from_env("YOUR_TOOL"):' explicitly.
  return get_active_overrides().subprocess_timeout

(That's from the hatch-vcs plugin and setuptools-scm's newst release, I think)

#

This is the command it's trying to run:

['python', '-W', 'ignore', '-'] {'capture_output': True, 'input': b"import os\nimport platform\nimport sys\n\nif hasattr(sys, 'implementation'):\n    info = sys.implementation.version\n    iver =
'{0.major}.{0.minor}.{0.micro}'.format(info)\n    kind = info.releaselevel\n    if kind != 'final':\n        iver += kind[0] + str(info.serial)\n    implementation_name = sys.implementation.name\nelse:\n    iver
= '0'\n    implementation_name = ''\n\nenvironment = {\n    'implementation_name': implementation_name,\n    'implementation_version': iver,\n    'os_name': os.name,\n    'platform_machine': platform.machine(),\n
'platform_python_implementation': platform.python_implementation(),\n    'platform_release': platform.release(),\n    'platform_system': platform.system(),\n    'platform_version': platform.version(),\n
'python_full_version': platform.python_version(),\n    'python_version': '.'.join(platform.python_version_tuple()[:2]),\n    'sys_platform': sys.platform,\n}\nsys_path = [path for path in sys.path if
path]\n\nprint({'environment': environment, 'sys_path': sys_path})\n"}
#

I think this is from src/hatch/utils/env.py:23. It's hardcoded to "python". Works better if I change the default to "python3" or sys.executable. Works if I change that.

hoary wraith
#

PR for the help portion is merged. I will take a look at a fix for the python command issue.

tropic void
hoary wraith
#

I have just merged lockfile support in hatch per environment, I would love for others to test it out and provide feedback for it before it is released!

errant timber
hoary wraith
dull rapids
dull rapids
hoary wraith
#

I wanted to make it easy to find me and chat at PyCon my schedule is
Arrive Tuesday afternoon and am free until Thursday for anyone who is early that wants to chat.
Thursday - Typing Summit and opening reception
Friday - Morning at AWS booth, afternoon Packaging Summit
Saturday - Morning at AWS booth, afternoon Maintainers Summit
Sunday- attending talks and open to discussions
Monday - dev sprint for Hatch

tropic void
#

oO did i miss the dev sprint for hatch

humble smelt
#

There was; though I didn't actually do anything while I was there other than build goodwill with Cary. 😅
I didn't realize you were at PyCon -- another missed opportunity (for me at least). Such things happen.

orchid birch
#

Same! It was nice to see @humble smelt and @hoary wraith at the sprints. I didn't get as far as I'd hoped with workspaces on my project, but that's okay. My TL;DR was that it makes no sense to have a pyproject.toml at the top level, so I'm working on converting that to a hatch.toml. Definitely something to think about on the journey for standardizing workspaces.

humble smelt
#

I always imagined that we'd have pyproject.toml at the root, FWIW, but with no [project] table. (I'm not sure if that's where we'll really end up.)

hoary wraith
#

So after chatting with Zanie, that is what they do actually in uv is that you just do not have a [project] table for the top level pyproject.toml I think that is probably the best way to go Barry. I would be curious if when you do that, does the hatch build --all work for you.

I indeed did host a sprint on Monday and we got some first time contributors. It was great to hang out and chat with @humble smelt @orchid birch and others. I am very excited for the rest of this year in the packaging space with hatch.

We expect a new release in the next week! What is in the release? We will be delivering pylock.toml support per environment, fixing a few bugs, and the one I am most excited about is we are reconfiguring our commands. We are introducing hatch check as a command and will in the future retire hatch fmt. The check command will have sub commands for type checks with pyrefly, and splitting out the linting and formatting commands with ruff. This will allow us to be more resilent with adding future sub commands. For those that want mypy or ty that is still possible with creating your own type checking environment but we believe that pyrefly offers a good experience without requiring us to maintain a list of configurations to apply. Minimally we only need to tell Pyrefly the search paths.

orchid birch
hoary wraith
slim cairn
#

There are use cases for having the workspace root be an actual project too 🤷‍♀️ it varies.

#

(uv supports either way)

hoary wraith
#

We support the workspace root being an actual project right now but we do not support the other way right now of not having the workspace root as a project.

#

I will be getting back to fixing that maybe tomorrow. I already have a lot of other irons in the fire tonight with refactoring to the hatch check command group.

orchid birch
#

I can see it either way. In the use case I've been playing with, the workspace root is not a project; both projects live in sub-packages. I may not get back to this until next week but it sounds like with the upcoming hatch release for check that'll be fine.

hoary wraith
#

I pushed changes Barry that should fix the issue, I have added several tests to make sure that it should work the way you have things setup. It should now support both with and without the top level pyproject.tom; having a [project] table defined. 🙂

umbral bridge
elder lotus
#

That's how I've constructed nab, I didn't spend a lot of time thinking about it though, just seemed a natural fit

cursive lion
#

appropriate that #hatch is opened on Easter! 🥚🐣

umbral bridge
dull rapids
#

woo!

#

One question: How does hatchling detect packages? if you have your package as src/project_name, it’ll just work™, but what if the PyPI name and module name diverge? What if one doesn’t use the src layout but just has the package next to pyproject.toml?

umbral bridge
mint bramble
#

One thing I find could be improved is the separation between hatch and hatchling in the documentation 🤔

#

For people who want to use hatchling but prefer e.g. tox for environment set-up and management

umbral bridge
#

I should add a tutorials tab, ty I'll work on this

umbral bridge
golden merlin
#

Congrats on the release.

tropic void
#

@umbral bridge awesome, wish I was further along with setuptools_scm, it may take me mid May before I can mix the vcs plugin up

mint bramble
#

@umbral bridge do you plan to propose hatch for adoption to pypa, considering the existence of this chat 😂

umbral bridge
#

ha, I wouldn't mind

#

would ppl want that?

mint bramble
#

I'd +1 it, especially the hatchling part of it 💪

sick jolt
#

Approved Ofek Lev to pypa-committers

umbral bridge
#

thanks!

mint bramble
#

Technically shouldn't we only do that past the vote to pass? 🤔

#

PEP-609 says at least so

sick jolt
#

Can always remove

umbral bridge
#

oop I pressed subscribe, didn't realize it was a special list

sick jolt
#

If hatch does not pass …

mint bramble
#

I voted for it, so I'm not against it, just noted that the bylaws were not followed 😂

umbral bridge
tropic void
#

@umbral bridge aware of anything to ease migrating a setuptools using project to hatch?

umbral bridge
#

python hp.py <PATH>

tropic void
#

@umbral bridge could this be integrated into hatch init --new

#

(i have about 70 repos i want to migrate from setuptools+setuptools_scm to hatch

umbral bridge
#

it's not perfect (cannot be) so I'm hesitant to

tropic void
#

what im looking for is not perfect, but safe 10-15 minutes transcoding setup.cfg to pyprject.toml then fix the rest (im happy with a command name that indicates its only trying, practically most metadata should transfer relatively safely

#

plus its fair game if it prints a red "warnining, best effort, validation necessary" ^^

umbral bridge
#

ok, I will

tropic void
#

👍

umbral bridge
#

mind a separate entry point? hatch-migrate <PATH>

tropic void
#

thats fine as well

umbral bridge
#

nice, ty

tropic void
#

@umbral bridge oh, and for something completely else - would it be thinkable/possible to have a editable install of the current package for builds, im trying to figure self-bootstrap of setuptools_scm

umbral bridge
# 
[build-system]
backend-path = ['src']  # or ['.']

?

tropic void
#

@umbral bridge but if hatch + hatch-vcs was in use - wouldnt it pull in the transtive setuptools_scm dependency of hatch-vcs as well

#

(btw, im still in the middle of full git archive support there - i introduced a regression there that im fixing now)

umbral bridge
errant timber
#

Looks like it's time to move this channel up a category. 😉

mint bramble
#

Indeed 😊

errant timber
#

Did that work? I tried dragging it up

mint bramble
#

yes

#

@umbral bridge you'll need to liason with @dusty obsidian, @elfin crater or @sick jolt AFAIK to make the transfer

sick jolt
#

Yeah I don’t haz those powers unfortunately

mint bramble
#

Dustin, Donald and Paul Moore are the only ones then

umbral bridge
#

transferred!

mint bramble
#

Cool

umbral bridge
#

yay CI secrets persisted

umbral bridge
elfin crater
#

That was fun trying to remember how all this stuff was setup 😛

turbid bane
#

... is any of it documented? I presume the answer is no :p

elfin crater
#

No

#

at least not that I'm aware of

umbral bridge
radiant cloak
radiant cloak
elfin crater
#

well there used to be a Fastly thing

#

that is still setup

#

but not receiving traffic

#

so that waylaid me 🙂

radiant cloak
#

Hehe

sick jolt
#

Maybe bullet point out the process and we store it somewhere …

umbral bridge
#
  1. invite owner of repo to pypa so they can transfer (if no longer wishes to maintain transfer to member of pypa)
  2. transfer to pypa (might take a few minutes)
  3. make teams:
    @<PROJECT>-team -> Triage
    @<PROJECT>-committers -> Write
    @<PROJECT>-maintainers -> Maintain
    @<PROJECT>-administrators -> Admin
  4. for projects that were using GitHub Pages, on https://github.com/pypa/<REPO>/settings/pages set custom domain to <REPO>.pypa.io (for RTD no idea)
umbral bridge
#

@mint bramble I'm implementing substitutions similar to tox like {env:KEY:DEFAULT} today... it's not very fun lol

mint bramble
#

Nesting/recursion can be fun to handle

umbral bridge
mint bramble
#

tox uses regexes

#

Not sure if this string formatting supports nesting 🤷‍♂️

#

I'd be surprised if it does

umbral bridge
#

@mint bramble it does!

karmic glacier
#

someone should update the channel topic lol

umbral bridge
#

@karmic glacier what would you suggest?

karmic glacier
#

pypa/hatch :P

umbral bridge
#

it is, no?

karmic glacier
#

(or maybe PyPA/hatch, idr)

#

... is this a caching bug on discord?

umbral bridge
#

likely

karmic glacier
#

can you send a screenshot of it, I can report this as a bug

umbral bridge
#

ok

umbral bridge
#

@tropic void did you happen to start writing that plugin?

tropic void
umbral bridge
tropic void
#

still pretty much delayed - stil lhavent ifinished the git archive full support pr and the upcoming changes

umbral bridge
mint bramble
#

👌 looks alright, even though I only use hatchling for now 😅

umbral bridge
tropic void
tropic void
# tropic void Yeeha, going to try right now

@umbral bridge first error is Could not automatically migrate from setuptools: [Errno 18] Invalid cross-device link: '/tmp/tmpxvbrbdlv/repo/pyproject.toml' -> '/home/ronny/Projects/${redacted}/pyproject.toml'

umbral bridge
#

😦

#

I'll fix os.replace after lunch...

#

I guess:

def replace_file(src, dst):
    try:
        os.replace(src, dst)
    # Happens when on different filesystems like /tmp or caused by layering in containers
    except OSError:
        shutil.copy2(src, dst)
        os.remove(src)
tropic void
#

yikes, that one is a really painful

#

why not use move?

#

shutil.move implements what you wrote

#

@umbral bridge actually, why not use use copyfile from shutil, the temporary directory will go away anyway

umbral bridge
#

oh, yes to both. ty

tropic void
#

@umbral bridge what type of system do you run that this didnt break the world? (for all my systems, tmpfs for tmp would break normal operations, but a pytest tmppath caveat exists

umbral bridge
#

Windows

tropic void
#

hmm, 😱

#

;P

umbral bridge
tropic void
#

@umbral bridge works, i'll add some extra setuptools_scm detection next week, 👍

umbral bridge
#

@tropic void it does that a bit already 🙂

tropic void
#

its missing raw tool.setuptools_scm sections as far as i can tell

#

(i just tried it in apipkg)

tropic void
#

@umbral bridge would it be possible toset hatch up, so that it could generate signed tags which include verifiable signatures for the expected wheels

the idea being that when a tag for a release is added by a user, it could be added in a way that includes the signatures of reproducible artifacts that will publish,, and then a publish would fail if the signature wont match

umbral bridge
#

like ship in wheel?

tropic void
#

@umbral bridge no- more like when i tell it to tag a release, build a wheel for that release - and then use git tag --sign -m "pypkg-version.whl gpg-signature", so that on the ci, hatch build can fail if the signature on the tag doest match

umbral bridge
#

Hatch/Hatchling doesn't do any tagging

tropic void
umbral bridge
umbral bridge
tropic void
#

hmm, would it be too evil to make zerg memes around hatch?

gloomy flicker
#

congrats on v1 :)

dull rapids
#

Ah, it’s very nice! Compared to flit, the only things I have to mind are

  1. Specify tool.hatch.build.targets.wheel.packages = ['src/mypackage'] to avoid having accidental inclusions in the wheel (like tests/*, docs/conf.py)
  2. I have to manually set description, which means the same sentence now needs to go into pyproject.toml, __init__.py, README.rst and the GitHub repo description (one place more than before)

It would be great if the first one would be visible prominently in every tutorial (@umbral bridge @errant timber), I’d like to avoid seeing PyPI packages shipping their tests folder and my PyCharm jumping into some random package’s tests folder when I start debugging instead of my tests.

dull rapids
errant timber
#

Tests and docs should not show up in wheels, but they should show up in SDists. It's a really easy and really bad mistake to make, so some default filtering is almost always applied. The very least (no non-package paths in the wheel) is okay, though tests often have an __init__.py.

It's important for me to make sure all tutorials do the right thing by default without configuration so that one tutorial (scikit-hep.org/developer, packaging.python.org) will work for all backends. Also so that scikit-hep/cookie will work with all 11 backends with minimal per-backend fiddling. Also for user experience.

#

Setuptools used to have issues with including the tests folder, I think it is manually ignored nowadays

umbral bridge
errant timber
#

That sounds right, @dull rapids are you sure docs and tests are getting added to wheels?

dull rapids
#

The docs, and other stuff, because this is hit:

  1. Otherwise, every Python package and file that does not start with the word test will be included

because the package name doesn’t match the distribution name.

#

But of course that lessens my concerns a lot since for most packages, names should be the same. Sorry for being alarmist 😅

tropic void
#

with a src layout its reasonably safe i would say

errant timber
#

docs doesn't usually have an __init__.py, tests is the one that might have one

#

And yes, matching package & distribution name should be the default for most

umbral bridge
tropic void
#

@umbral bridge it might be nice to error out with unknown setup.py sections (like flake8 or isort) i missed the drop of one of those much to the complaints of the ci system ;P - its probably something i can work on later this week

umbral bridge
#

what dropped?

tropic void
umbral bridge
#

@tropic void setup.cfg doesn't get written to

tropic void
#

hmm, then i suppose its human error on me all the way 🤦

golden merlin
#

Complete laziness question, but is there a migration guide somewhere

#

I.e. a page that says "here's how you migrate a project to hatch"

umbral bridge
golden merlin
#

Ah amazing, thank you!

golden merlin
#

Is there a setuptools-scm equivalent for hatch

spare vortex
golden merlin
#

Thanks!

umbral bridge
golden merlin
#

(Even including that I hadn't been using PEP 621)

#

Do I still want to use build or should I use hatch to build in CI.. I guess the former still, I want to make sure folks can build with whatever thing.

umbral bridge
#

build is fine

tropic void
#

@umbral bridge whats the recommended way to run more than one commands of a script section of a environment?

umbral bridge
#

example?

tropic void
#

i have a env that has a mypy and a pytest script - i want to run both of them in a single command

umbral bridge
tropic void
#

@umbral bridge it would help the docs to have that example, i went into my shell way of trying to use a ; in the string ^^

umbral bridge
#

2nd snippet

tropic void
#

hmm, ok, this indicates that my brain juice for today is utterly depleted - i was searching for that there for a while and completely missed it,, calling it a night

umbral bridge
#

np 🙂

tropic void
#

thanks, and great work, im slowly replacing all my personal tox usages with much nicer hatch usage

umbral bridge
#

aww thanks! I'm actually quite proud of what I came up with for environments so that means a lot

golden merlin
#

Why / how?

#

I was just considering the same thing (so legitimate question)

#

(+1 on hatch being really nice though, just pushed the first release of jsonschema which uses it so T-X seconds till I get the first user complaining they can't find a setup.py file)

#

But yeah how are you replacing tox?

tropic void
#

@golden merlin hatch envs as linked above are basicall tox environments on steroids (it can do pretty much all the things tox couldnt do nice because of ini files and matrix expansion > factors

golden merlin
#

OK, I haven't read enough I suppose, I didn't know it can do the latter

#

(matrix)

#

Ah. I see it now.

umbral bridge
#

@golden merlin one big difference is that whereas in tox envs are "executable", in Hatch you aren't limited to a given command set. so:

tox -e style
tox -e fmt

vs 1 env:

hatch run lint:style
hatch run lint:fmt
golden merlin
#

aha, ok, nice

#

and there's I think I see a way that corresponds to running tox bare?

#

I.e. "run all of a set of environments"

umbral bridge
#

not an OOTB way no, the paradigm is just different

#

i.e. in CI I want steps broken into logical units

#

not 1 "do-all-the-things" step

golden merlin
#

hm, but then how does one do "run all the things that I want to pass to know this thing works"

umbral bridge
#

if you have an env or matrix for testing called test:

hatch run test:all

[envs.test.scripts]
foo = "pytest"
bar = [
  "foo",
  "baz",
]
all = [
  "bar",
  "...",
]
golden merlin
#

ok cool yeah that's what I meant by "I think I see something matching tox bare" it was yeah "put all that in one environment" 😄 -- good, will give it a shot, thanks

umbral bridge
#

I want to see which is faster probably

Hatch is, but haven't tested tox4

#

@golden merlin a cool test might be:

hatch run test:python -c "import sys;print(sys.executable)"

[envs.test.overrides]
env.HERMETIC_TESTS.type = [
  { value = "container", if = ["true"] },
  "virtual",
]

[[envs.test.matrix]]
python = ["37", "38", "39", "310", "311"]

then run again to see the speed

then pip install hatch-container & set env var HERMETIC_TESTS=true and run again

golden merlin
#

nice, ok will give that a shot tomorrow then, thanks for the tips!

tropic void
#

@umbral bridge any idea how to make pycharm aware o f hatch envs?

umbral bridge
mint bramble
umbral bridge
#

the dep sync check is faster in Hatch but dunno why

mint bramble
#

That shouldn't be the case in tox 4

umbral bridge
#

why? 🙂

mint bramble
#

Also in tox 4 you can switch to install the wheel over sdist Speeding up even more

#

What do you define exactly as dep sync?

umbral bridge
#

list of pep 508 strings is met + sync if not

mint bramble
#

tox 3 doesn't do that at all 😅 so guess it's faster

#

tox 4 does it though and uses prepare metadata for wheel to be fast-ish 👌

umbral bridge
#

? it does [testenv.deps]

mint bramble
#

Well guess does it sometimes, but not for project install and extras

#

Only for deps

#

And if there's a diff recreates from scratch in tox 3

#

tox 4 installs only the diff if the dependencies are append only, if there's a remove needs to recreate as there's no good way how to know what to uninstall (e.g. transitive dependencies are not handled)

#

To be fair even additions only are not necessarily save to just install afterwards, as adding a new dependency might in theory invalidate the previously resolved dependencies in the pip solver

#

But as it's likely good enough 95% of time it allows it

mint bramble
umbral bridge
#

Hatch does that tox4 approach too then

#

nice

mint bramble
#

How does hatch handles package building, as far as communication with the backend and build environment?

umbral bridge
#

just shells out to pip for env installs

mint bramble
#

I've invested a significant amount of time to make sure package building is fast as possible 😅 so wondering how hatch handles it

umbral bridge
mint bramble
#

Guess then that's an area where tox 4 might be faster as tox reuses build environments and uses a persistent process to minimise process start-up time especially hard hitting on Windows

umbral bridge
#

for builds i.e. hatch build Hatch does pickle-based RPC with Hatchling for colored text so Rich not needed

mint bramble
#

hatch can still gain advantage overall though due allowing env sharing, which tox doesn't do yet

#

90% of pip install time for the project will be pip recreating the isolated build environment and installing your build dependencies

umbral bridge
#

true but only happens once, afterward just the defined deps are checked

mint bramble
#

After every code change you need to rebuild the artifact unless you're installing in editable mode, no?

umbral bridge
#

no, user in charge of that

#

dev mode/editable is default

#

if disabled they chose that path

mint bramble
#

So for non editable mode, which is the true test env, you still need to rebuild/reinstall the project before every run, no?

umbral bridge
#

yes if they want to, Hatch doesn't manage that

mint bramble
#

IMHO editable mode differs to much from normal mode to be default to be a reliable validation env, I view it more a unit test, where normal mode is the true integration test

#

After all your lib users will do normal installs not editable 🤷‍♂️ guess there's a difference of philosophy between projects here 👌

umbral bridge
#

difference of philosophy 👍 I think in most cases it's the same

#

even for C ext

mint bramble
umbral bridge
#

esp. since the main old issue was setuptools config was just easy to mess up

mint bramble
#

Still it's probably still 90% of time probably close enough 👌

#

As long as you require hatchling as backend for hatch, and there's no fancy import hook logic active for the project

tropic void
#

@umbral bridge i think the issue is that pycharm would need a plugin to know hatch envs like tox envs or normal venvs

#

hmm

#

oh, lovely, hatch is in recent fedora

golden merlin
#

(the behavior I always want is "locally should always reinstall from scratch fully, CI should always reinstall from scratch but I want to restore a wheel cache each time")

umbral bridge
#

locally should always reinstall from scratch

for every change?

golden merlin
#

yes, think so, but will see what my workflow becomes for hatch

#

if I don't want that, I run my test runner

#

(if I'm running tox, or hatch, it's because I want to run everything, and not rely on some state)

tropic void
#

@umbral bridge any plans for parallel running of environments ?

umbral bridge
#

Ability to parallelize command execution in multiple environments, I want it to be pretty and have an interactive option to expand to full size output based on keypresses so you can check on the progress of any at will

tropic void
#

ah, lovely, i suppose this will need textual and subprocess protocols + ptys ? (and pain)

umbral bridge
#

yes, much pain

tropic void
#

@umbral bridge perhpas its a valid strategy to convince textual into providing tabs and sub terminals" then the parallel mode can open up a main tab, and hide the rest away in other tabs one can click on,

mint bramble
#

tox 4 also wants/needs to deal with that much pain

rancid mesa
#

hi! i just found out about hatch (out of left field!) and it got me salivating. it's like everything i want from poetry plus tox and vcs integration on top 😮 also cibuildwheel integration, woah. i tried to init it on https://github.com/fonttools/fonttools/ and it crashes though because it can't find fontTools/__init__.py for version determination. can't figure this one out... to repro, clone the repo and do hatch new --init && hatch env create

tropic void
#

also please create a draft pr with the changes so its directly visible

rancid mesa
#

i want that eventually, but for now i'm just playing around. the project currently uses the bump2version approach to versioning so i wanted to keep it until i'm more familiar with hatch. ok, will do after i have set something up 🙂

tropic void
#

@umbral bridge i have a feature request - can we create something that provides the expected content hashes of artifacts for a release - i want to tag releases with a signed annotated tag that includes the expected output hashes of the reproducible artifacts

umbral bridge
#

sure, what would that workflow look like exactly?

tropic void
#

@umbral bridge bascially i would locally ask hatch to build the pacakges of a future release and then obtain content hashes/signatures

afterwards the target tag would be created as a signed/annotated tag containing the expected signatures/content hashes of the artifacts

then when pushing it to the ci/cd platform, the release process could gate on the signatures + do actual signed uploads

tropic void
#

as far as im aware, apipkg will be the first package in pytest-dev to be turned hatch

rancid mesa
spare vortex
#

Did you try with Lib/fontTools/__init__.py on line 100 of pyproject.toml?

rancid mesa
#

yep, no dice 😦

umbral bridge
#

@rancid mesa are you sure you tried?

[tool.hatch.version]
path = "Lib/fontTools/__init__.py"
rancid mesa
spare vortex
#

I think I found the issue. hatch doesn't like that the line starts version = __version__ = (i.e. sets both variables). The regex expects __version__ to be at the start of the line.

umbral bridge
#

oh yup

rancid mesa
#

i tried setting the regex before but same error

#

changing the line to read just __version__ = "4.33.4.dev0" still results in the same error

umbral bridge
#

tox might've cached the sdist?

#

I'll clone your branch

rancid mesa
#

git clean -xdf && hatch env prune && hatch run test:test still gives the same error

#

also, funny: after the regex version thing fails, running hatch run test:test again results in "'test' is not recognized as an internal or external command, operable program or batch file.". i have to prune envs and try again

umbral bridge
#

there is no test script

rancid mesa
#

ah 😄 i haven't understood the tox/env stuff yet

umbral bridge
#

this does work

__version__ = "4.33.4.dev0"
version = __version__
rancid mesa
#

ah! it's the combination of Lib/fontTools/__init__.py plus your modification. thanks 🙂 very misleading error though

umbral bridge
#

& ty @spare vortex for finding out what the error was!

rancid mesa
#

oh duh. i now realize that i was suffering from traceback blindness and didn't realize that i got a different error after changing the path. duh. sorry for the noise 😬

radiant cloak
#

@umbral bridge is hatch using some sort of pretty traceback presentation (eg: rich's?)

umbral bridge
#

no but I want to

#

was pip's PR very tedious or not bad?

radiant cloak
#

pip's not using rich's tracebacks (except when you pass --debug which is mainly for debugging crashes). It's using rich-renderable error objects, which I should really factor out into a separate package at some point.

pip's PR was tedious because I needed to figure out how to plumb the rich exception presentation through logging without disrupting regular logging use. It was fine, otherwise. :)

#

Rich wasn't the hard part. :)

golden merlin
#

Is there something other than PEP 508 which specifies extras should be normalized?

half cedar
golden merlin
#

Aha, thank you! /me reads

#

And there's not really anything I can do in the meantime it'd seem, given I'm already specifying it with the form it was specified in before, hatch is just following the guidance from 685 before pip is ready for it?

#

Er sorry probably it's build doing that in this case

gloomy flicker
#

whatever's happening I doubt build is involved

#

if the extra names are being normalised, then that's hatch doing that

umbral bridge
#

yes I've implemented that pep

golden merlin
#

Ah bleh sorry it's confusing keeping it in my head

#

build builds my distributions

#

But it does that by calling out to hatch

#

So yeah thanks

#

(for unconfusing me)

#

@umbral bridge I ask not as a leading question but purely to inquire -- should hatch not do that until pip actually supports the PEP?

umbral bridge
#

maybe I should've gated it on core metadata v2.3

#

but PyPI still doesn't accept v2.2 even

golden merlin
#

yeah that section to me says it's recommending pip should win right?

umbral bridge
#

pip implementation should be prioritized yes

golden merlin
#

(filing an issue just for visiblity, hope that works?)

umbral bridge
#

sure

tropic void
#

@umbral bridge another feature detail came up - a good way to run commands of a environment in the current python to help replicate the test commands for packaging in say fedora which for example use tox --current-env to run the test commands against a installed package

umbral bridge
#

I'll add a way

rancid mesa
#

hi again! I'm struggling with excluding a certain feature for just pypy3 in a matrix. i have this snippet:

[tool.hatch.envs.test]
dependencies = [
    "pytest",
    # ...
]
features = [
    "interpolatable",
    "lxml",
    # ...
]

[tool.hatch.envs.test.scripts]
cov = "pytest --cov-config=.coveragerc --cov"
no-cov = "pytest"

[[tool.hatch.envs.test.matrix]]
python = ["37", "38", "39", "310", "pypy3"]

i want to exclude the lxml feature when running under pypy3 because it just slows things down. i'm not sure what to do from reading https://hatch.pypa.io/latest/config/environment/#option-overrides... :/ or do i have to make a separate env?

Environments - Hatch

Modern, extensible Python project management

tropic void
#

@umbral bridge how would negative conditions apply as far as i understand the ask here is for something like

features = [
  { value= "lxml", platform = [{ not = "pypy"}]}
]

my dictionary programming language sense is mismatching braces

rancid mesa
#

(i don't want to put the platform anti-specifier into the feature itself because maybe someone really wants to use the lxml feature on pypy for some reason, i just don't want to run the tests with it)

rancid mesa
#

are there any examples building cython modules from hatch and having them tested in envs?

umbral bridge
#

no support, but you can continue using setuptools with Hatch envs

rancid mesa
#

ok, thanks 🙂

umbral bridge
#

how would negative conditions apply
@tropic void no support, yet

excluding a certain feature for just pypy3 in a matrix
@rancid mesa for now use https://hatch.pypa.io/1.2/config/environment/#overwriting:

[[tool.hatch.envs.test.matrix]]
python = [..., "pypy3"]

[tool.hatch.envs.test.overrides]
matrix.python.set-features = [
  { value = [ ... ], if = ["pypy3"] },
]
rancid mesa
#

that works, thanks

rancid mesa
#

what's the best way to use pinned deps for testing? i understand that a lockfile format is being fought over and the faq-listed pip PR is not merged yet. i'm primarily interested in tests not randomly breaking with updates when i haven't touched a project for a while (i also understand that this won't save me from having to move with my deps)

umbral bridge
#

I'm not sure yet 😦

umbral bridge
#

wow os.stat is literally broken on Python 2 on Windows

umbral bridge
tropic void
#

@umbral bridge im thinking of surrounding hatch/setuptools_scm with a set of github actions + plugins to facilitate

  • proposed releases with something like towncrier/changelog integration + output of reproducible artifact hashes (intended use on pytest)
  • triggering release proposals on milestone completion and/or bugfix merges/branches (also intended use on pytest)
  • proposing commands that will ensure the target release tags are signed and include artifact signatures (i want to go all in on supply chain security, going to use this for hopefully > 100 internal and external projects)

i'd appreciate any input before i get to it starting mid week hopefully (first setuptools_scm regressions)

umbral bridge
#

sounds cool, lmk if I can help at all

tropic void
#

@umbral bridge before starting i'd like to get some input on possible issues you see, and how to best work with repeatable builds to suggest artifact hashes in pipeline + also what place to best check them

#

(the diea being that to do a pypi release a maintainer pushes a signed tag, which also includes artifact signatures, for the wheels, possibly even the signature that would go into the record

umbral bridge
#

all of that seems possible with plugins and scripts

tropic void
#

@umbral bridge ok, then im starting hopwfully this month ^^

tropic void
#

@umbral bridge is there a way for hatch to build both mypyc wheels and non-mypyc wheels that i missed in the docs? (except for running twice with the different env vars

umbral bridge
#

except for running twice with the different env vars
no

tropic void
#

i have 2 use-cases here - a speed compare a mypyc lib versus a non-mypyc lib and publishing both types of wheel for broader consumption , im fine with a little script tho

tropic void
#

@umbral bridge can environments be configured so different build settings are used ?

umbral bridge
#

builds use separate env

tropic void
#

i see, so matrix is probably not possibly easily and sanely,, so script it shal be

umbral bridge
#

I mean, you could matrix + install/use #build in each

umbral bridge
tropic void
#

@umbral bridge could build variants be used as input to the option?

umbral bridge
#

pep517 config_settings? no, env vars yes

tropic void
umbral bridge
#

sorry, example?

tropic void
#

@umbral bridge i was wondering if those could be used as way to enable/disable specific build hooks, also bbl, toddler needs me

tropic void
#

@umbral bridge basically - its not clear to me how the versions thing of build targets works and i do wonder if i could use them as part of a if expression to enable/disable mypy building

umbral bridge
#

no

tropic void
#

@umbral bridge i see, with that part it makes much more sense - the example versions in the versions docs dont exactly help to make this clear, maybe a link to this or a example may help

umbral bridge
#

will do

umbral bridge
tropic void
umbral bridge
tropic void
#

@umbral bridge i'll try to get pytest in there once the fedora packaging tooling is hatch compatible

#

with the caveat that i may be out of commission for 2 months for having a daughter starting in the second half of august ^^

umbral bridge
#

congrats!

turbid bane
#

congrats! enjoy the sleep while you can get it :)

tropic void
#

hmm, my sleep schedule dearly wishes i would heed your words

umbral bridge
#

oh the responsibility!

e.PNG p.PNG
radiant cloak
#

^.^

umbral bridge
#

I want to rename the default publishing plugin name/ID pypi to something more generic. thoughts? cc @elfin crater

#

index? standard? keep as-is?

elfin crater
#

If you come up with a good name let me know, because I hate calling something the PEP 503 API, since its' really the PEP 503+592+691+othernumbersiforget

umbral bridge
#

going with index

#

like devpi

#

& flit

proper tide
umbral bridge
#

too long to type

proper tide
#

smpl-rpstry

umbral bridge
#

haha perfect

umbral bridge
tropic void
#

@umbral bridge are there any plans for project local plugins for hatch/hatchling (to enable stuff like setuptools_scm self bootstrapping)

umbral bridge
tropic void
#

@umbral bridge thanks, after taking a closer look, it seems oblivious, it may be a good idea to link to some practical examples in a non reference section of the docs

umbral bridge
#

will do 👍

golden merlin
#

GitHub Actions' setup-python has native support for caching via pip, pipenv and poetry

#

Does anyone know if there's a way I can use this with hatch?

#

Where does hatch store concrete lockfiles, I guess I should be able to answer that myself in the docs

tropic void
#

hatch aint dealing with that - just use the normal pip cache

#

oh wait, i misunderstood the tool

#

so currently hatch is not supported, but enabling caching of pip caches may speed up things

golden merlin
#

I think (but haven't tried yet) that it won't do much unless I manage pinning versions, for the same reason mentioned there, that pip will still go look for latest versions

#

I'm aware hatch doesn't fully manage this, just trying to see how to complete the workflow

#

like, I think I need to use something like hatch dev show requirements, but then pip-tools comes back in the picture

tropic void
#

hatch will eventually support pep based lock files, currently it doesnt, and it may help to use pip-tools for the meantime

golden merlin
#

ok

#

I think pip-tools itself doesn't support hatch at the minute

#

so I guess I get to pipe into it

umbral bridge
umbral bridge
mint bramble
hybrid thorn
umbral bridge
#

awesome 🙂

tropic void
#

oO - isnt that basically going to make pip-tools be able to disengage fromthe apis?

hybrid thorn
# tropic void oO - isnt that basically going to make pip-tools be able to disengage fromthe ap...

It's addressing these kind of use cases, yes, together with pip inspect that I also hope to land in pip 22.2. https://github.com/pypa/pip/pull/11245

GitHub
Add pip inspect command by sbidoul · Pull Request #11245 · pypa/pip

This follows from the discussions in #11223 and #11097. Inspired from pip list and the installation report format.
This is fairly straightforward (although the underlying logic in pip may not be as...

radiant cloak
proper tide
errant timber
#

Is there a good way / best practice for pulling out part of a README for the description? In converting some packages, I run into things looking for markers and just pulling out parts for PyPI in the setup.py.

umbral bridge
karmic glacier
#

interesting, does hatch have a feature which replaces tox?

umbral bridge
#

actually just presented my RFC to switch at work today

#

we should be done migrating by the end of Q3

karmic glacier
umbral bridge
#

lmk if I can help

karmic glacier
#

tbh I should probably explore other options first

#

as we currently use setuptools, that is the main thing to shift away from

#

Its also a library, so I'm not terribly concerned about not having lockfiles; I don't think that libraries should use them

#

the build tools I know of are flit, hatch, poetry, and setuptools

karmic glacier
umbral bridge
#

of tox/hatch/nox or setuptools/hatchling/flit?

karmic glacier
#

both would be cool but mainly interested in the latter, as we'll probably stick with nox for a period of time even if we switched to hatch

#

let me check when we switched to nox

#

we added usage of nox on june 3rd

umbral bridge
karmic glacier
#

thanks, i'll take a look!

umbral bridge
karmic glacier
#

you did, haha

umbral bridge
#

I'm not too familiar with nox but the size/verbosity alone scares me lol

karmic glacier
#

I wonder if hatch supports everything we're doing with nox

#

this is the list of our current sessions:

- docs -> Build and generate the documentation.
- lint -> Check all files for linting errors
- slotscheck -> Run slotscheck.
- pyright -> Run pyright.
- test-3.8(extras=[]) -> Run tests.
- test-3.9(extras=[]) -> Run tests.
- test-3.10(extras=[]) -> Run tests.
- coverage -> Display coverage information from the tests.
- setup -> Set up the external environment.

the important things which I would be interested if hatch supports are the things below:
the first session, docs, invokes a different dependency based on if we're running in CI or an interactive session, plus different arguments
the test sessions call the coverage session once they are done running
the setup session installs all of the dependencies to the local environment, checking that the developer has a venv activated before doing so (this one is almost certainly implemented when using hatch for dependency management)

umbral bridge
#

different dependency based on if we're running in CI or an interactive session

CI/env vars yes, interactive session I think no atm. you mean like detecting TTY?

karmic glacier
#

i actually don't know how nox does it 😅 but I think it detects the TTY or a forced environment variable

umbral bridge
karmic glacier
#

er, what is hatchling?

#

oh, the build backend

#

does hatch new --init not send anything to stdout/stderr intentionally?

umbral bridge
#

yes

karmic glacier
#

what is the reason?

umbral bridge
#

I figure if no files are created then no need for showing a tree

karmic glacier
#

ah, but i found it odd to wait for ~20 seconds with no indication of what was occuring

#

just read through the pyproject.toml file, quite impressive

umbral bridge
#

oh good point, mock running python setup.py could take a bit, I'll add text

#

ty 🙂

karmic glacier
#

does hatch ever write to the pyproject.toml file other than hatch new --init?

#

because when it did, it changed some formatting and removed any existing comments

umbral bridge
#

no only time

#

there's a lib tomlkit that tries to preserve formatting but it's buggy

karmic glacier
#

I haven't noticed bugs with tomlkit, but I have noticed it is somewhat slow

#

and one issue where it sometimes deletes or adds a newline in one specific place, which may be poetry's doing

#

(I use poetry for my applications)

karmic glacier
#

hm... the publish command is a bit scary

#

tested with a private index, and it would be heavily appreciated if confirmation was per-file, or a list of files to be uploaded would be shown.
Because of how the rest of hatch works, I partially expected hatch to only upload files that match the current version

unknown.png
#

Ah, I did just find hatch clean but I feel like I would forget to clean it right before making a release

umbral bridge
#

upload files that match the current version

no b/c often you'll publish artifacts in a separate step that doesn't even have the project

karmic glacier
umbral bridge
#

usual steps:

  1. build artifacts, upload to cache
  2. download artifacts from cache without cloning repo, upload to pypi
karmic glacier
#

huh, where is the step to test the wheel/sdist?

#

I haven't made an automated release workflow yet but I've intended to

umbral bridge
#

my main point is that the final upload step often won't see the project

karmic glacier
#

ah okay