#hatch

1 messages · Page 3 of 1

slim cairn
#

I did something infinitely more disturbing

#

I copy the universal wheel to a new file name

#

😉

low gyro
#

why?

slim cairn
#

Uhh I want a bunch of tag names for testing resolution, there are no real platform requirements

umbral bridge
#

that isn't correct because you also have to change the WHEEL file inside

slim cairn
#

Hm true. I wonder when that file is inspected.

low gyro
#

on wheel installation

slim cairn
#

Well the tags are in the name of the file, why parse them from there too?

low gyro
#

from what I have noticed, neither pip nor installer actually read the tag from WHEEL file, so you can get away with just changing the file name

umbral bridge
#

maybe true but I wouldn't recommend not conforming to spec

#

for example, IIRC the *.dist-info/RECORD also isn't used but I think it would be bad to not have that be correct

low gyro
#

installer verifies hashes based on RECORD file

#

I remember because we had some issues with that when we introduced installer-based installer

umbral bridge
#

oh, that's awesome. I think pip does not

umbral bridge
errant timber
#

Is Hatch getting PEP 723 support at some point? I seem to remember you mentioning that somewhere.

umbral bridge
#

yes it will be in the next minor release

dull rapids
#

@blazing orbit I was about to open a feature request in VS Code Python about Hatch env discovery,
as people from the PyOpenSci community start to embrace Hatch and found this to be a pain point.

I then found “Reach out to Hatch about environment discovery proposal; @brettcannon” in VS Code Python’s “Iteration Plan for June 2022”.

Did anything come out of this? Is there a PEP planned for Python venv discovery? Would supporting Hatch specifically in VS Code be welcome until then (IIRC, Poetry has special support)

low gyro
#

Poetry has special support in VS code? Interesting

umbral bridge
#

yes so does pipenv

#

if I was good at TypeScript I would have contributed the feature a while ago

dull rapids
blazing orbit
# dull rapids <@377660079509340163> I was about to open a feature request in VS Code Python ab...

The plan/hope is to come up with a workflow API this year and then move out all special support for anyone not pip and venv to separate extensions (and we will ask projects if they want to own those extensions before we publish them). And the "environment discovery proposal" Ofek knows of (and likes 😉) as it's the idea I have in the Python Launcher for Unix's discussions section on that topic

low gyro
blazing orbit
low gyro
blazing orbit
# low gyro yeah, ship with Python

Not at the moment. I'm enjoying the fact I can still iterate on it. I would rather get relocatable binaries built and made available on python.org so that py can download and install them. When that's possible then it might make sense to make it the thing to download from python.org

low gyro
#

makes sense

umbral bridge
dull rapids
#

Not yet. I’m busy with work, but I’ll try to finish it up at the end of the week or next week

dull rapids
#

Actually I just checked and it does work! It’ll find Hatch envs that are already created.

In the future, we could maybe enhance the extension so it lists all defined environments, and creates one that is being switched to but that doesn’t exist yet. For that, maybe Hatch might want to grow a “create environment, but don’t sync dependencies yet” command.

dull rapids
#

That PR is now ready for review, so hopefully it can be merged soon!

umbral bridge
#

that's so awesome, thank you!

old grotto
#

🥚

golden merlin
#

I have a data directory with some static data that I want installed into the package as resources when it's installed (the reason it doesn't live inside the package in the source layout is because it's shared with other things which do not get installed) -- force-include with data = "mypackage/data" seems to do "the right thing" there -- is that the intended option to use for this use case?

#

I guess I need to check that does the right thing for editable installs too first, hopefully so, /me checks

umbral bridge
#

yes that is correct usage but editable I think would not work

golden merlin
#

would symlinking it work

#

yeah I found that ok good, glad to know the backup plan

umbral bridge
#

I guess I could promote that to actual option rather than requiring a hook but I didn't know how useful it would be

#

the existing build data option was a feature request

golden merlin
#

I think this kind of thing might not be unheard of, basically it's a backend and a frontend that need to share some data, but I also wouldn't think it's super high priority to work on if there are some ways to do it

#

some day I'll understand importlib.resources

#

the force-include way is doing weird things even for non-editable installs

#

but I think because of importlib.resources, not hatch

#

specifically it's finding paths in cwd rather than the installed virtualenv, what...

#

I can see the file is in the right spot, it's in venv/.../site-packages/bowtie/data, so hatch did its job, yeah.

#

ok I'll shut up and fight with importlib and stop vomiting on the channel 🙂 will come back with build hook questions if I get stuck, thanks as usual

#

oh geez that will be because of dumb sys.path-automatically-prepending-. issues presumably, ok now I'll stop

golden merlin
#

I thought I was clever and just fixed editable packages by relying on having the source checkout available (and not using importlib.resources in that case, just opening the file directly), but somehow force-include breaks twine building the package via the sdist

#

not sure what's up there (nor did I diagnose yet, so I'm being quite lazy even mentioning it until I do 🙂

umbral bridge
#

would the target path be different depending on wheel vs source distribution?

golden merlin
#

Whoops, didn't see your response sorry -- but I fixed it just by moving the force-include down

#

(to tool.hatch.build.wheel.force-include rather than the global one)

#

Having the data layout for sdists match the repo layout is fine / probably intended anyhow, so think all works well!

golden merlin
#

Now to see how the heck to reproduce that.

golden merlin
slim cairn
#

I'm encountering some very weird behavior with hatchling

#

In two checkouts of a repository, one builds a wheel without the src/<pkg> bundled and the other doesn't

# 

# Checkout "a"
❯ git clean -fdx
❯ which hatchling
/Users/mz/eng/src/astral-sh/uv/.venv/bin/hatchling
❯ git rev-parse HEAD
68c75bd0cb1b4346f2d8b3ed44e2e680384357bb
❯ hatchling build
[sdist]
dist/uv_private_pypackage-0.1.0.tar.gz

[wheel]
dist/uv_private_pypackage-0.1.0-py3-none-any.whl

❯ tar -xvf dist/uv_private_pypackage-0.1.0.tar.gz
x uv_private_pypackage-0.1.0/src/uv_private_pypackage/__init__.py
x uv_private_pypackage-0.1.0/.gitignore
x uv_private_pypackage-0.1.0/pyproject.toml
x uv_private_pypackage-0.1.0/PKG-INFO


# Checkout "b"
❯ git clean -fdx
❯ which hatchling
/Users/mz/eng/src/astral-sh/uv/.venv/bin/hatchling
❯ git rev-parse HEAD
68c75bd0cb1b4346f2d8b3ed44e2e680384357bb
❯ hatchling build
[sdist]
dist/uv_private_pypackage-0.1.0.tar.gz

[wheel]
dist/uv_private_pypackage-0.1.0-py3-none-any.whl
❯ tar -xvf dist/uv_private_pypackage-0.1.0-py3-none-any.whl
x uv_private_pypackage-0.1.0.dist-info/METADATA
x uv_private_pypackage-0.1.0.dist-info/WHEEL
x uv_private_pypackage-0.1.0.dist-info/RECORD
#

If I unpack the build sdist it looks like a gitignore is getting generated for me?

❯ cat uv_private_pypackage-0.1.0/.gitignore
.vscode
.venv
.python-version
.envrc
.direnv


❯ cat .gitignore
cat: .gitignore: No such file or directory
#

and in the bad build, the gitignore is * for some wild reason

❯ cat uv_private_pypackage-0.1.0/.gitignore
*%
dusty parcel
#

To be clear, it's the same as the git repo, but with all instances of "public" replaced with "private"?
Could it be that you didn't update the package path under the package key in pyproject.toml?

slim cairn
#

Yeah. It builds fine in one of the clones of it, the path is correct.

#

It must have something to do with the way it's cloned? A fresh clone with git builds fine but building from libgit2's clone ends up injecting this bad gitignore

#

I'm not sure what's generating this gitignore file

dusty parcel
#

Weird!

slim cairn
#

I wonder if it's pulling my global personal gitignore in

#

In the one that works, it's in a subdir of my home directory so it finds it

#

In the other, it's in a temporary directory and defaults to some horrible "*" ignore

tender summit
#

@umbral bridge are any details of the proposed lockfile standard public?

umbral bridge
umbral bridge
slim cairn
#

Isn't that a big problem for sdist builds in other environments?

umbral bridge
#

the ignore file gets shipped in source distributions (if the aforementioned option is not disabled)

slim cairn
#

If building a source distribution from a git repo (e.g. via pip install git+...) this will use the users ignore file which can break the package

#

I think you should stop at git boundaries so you don't traverse up out of the repository when searching for an ignorefile

umbral bridge
#

hmm, yeah that would be a good fix

#

would you mind opening an issue?

slim cairn
umbral bridge
slim cairn
#

Thanks again!

#

That one cost me so many hours haha

dull rapids
#

Oh nice, didn't expect that to be merged without doing more busywork. It's very refreshing to see that they're pragmatic and don't love their new testing standards so much that they let them delay the PR.

(I said I don't have much time and therefore can't pursue a higher testing standard than older integrations)

dull rapids
#

Ah, I misunderstood them! They just wanted manual testing instructions, not me to do many more unit tests. Easy!

So yeah, it's merged and it works.

umbral bridge
#

working on the test command now, it's going to be really nice

tropic void
umbral bridge
#

yeah a hatch test like I mentioned would be coming in that blog post

orchid birch
#

I have a question about overriding dependencies, in the context of a monorepo. I have a namespace package that doesn't have any code, just metadata. It has a project.dependency array with a pinned dependency on one of its subpackages. I need to bump that pinned dependency in a PR that prepares things for release, but that pinned dependency version isn't released yet, so normally that would break because hatch couldn't find this yet-to-be-released child package on PyPI.

I have the same problem when I build the namespace package, but there I have a tool.hatch.metadata.hooks.custom which rewrites the dependencies to use file URIs and remove the version pin. This works fine for most cases.

However, it breaks when it wants to install the project in development mode. Apparently my metadata hook does not get called when the environment first gets created (i.e. from a cold start clean clone). I see Installing project in development mode and then I get the failing error because it can't find the pinned dependency.

What's odd though is that if I run the same command again, it all works. I see Checking dependencies and Syncing dependencies and my hook debug prints prove the hook is getting called. My hook implements MetadataHookInterface.update().

So the question is: how can I hook into this Installing project in development mode step to rewrite the dependencies to use file URIs in the same way? I don't know if such a hook exists, or if there is a better way to handle this dev mode problem.

Worse comes to worse, I think I can hack my CI to ignore the error the first time, and run the command again. Not ideal, but it might work.

TIA

orchid birch
#

I think the root problem is that I have to be able to modify the project's dependencies (i.e. to remove the pin and point to the file URIs) when the project is installed in development mode. Is there a hook that allows me to do that?

orchid birch
#

Looks like for now I can add skip-install = true but that's not a long term solution.

tropic void
#

@umbral bridge i mised the post, can you link it?

umbral bridge
#

I'm so close now, the command is completely finished and tested, only documentation remains

image.png
orchid birch
#

@umbral bridge I'm not sure you saw my above question: #hatch message

It's the one thing that's hanging me up with monorepo. I mention this because your blog post above references monorepo. Could you confirm that I'm missing something that already exists, or that it's a gap that needs a bug report, which I'm happy to do. Thanks.

#

@umbral bridge BTW, I finally completed the conversion of some of my personal projects (not monorepo) to hatch and ruff. Gotta say, I really appreciate all the work you've done on hatch, and I think you've really found the right UX and functionality. I was able to drop pdm and tox and a bunch of hacks I had on top of all that to make things work. Everything is simpler now, easier to understand, better organized. Just want to give you major kudos!

umbral bridge
#

@ofek I'm not sure you saw my above

#

ah 🙂 thank you for your kind words! I'm very happy to hear that and I plan to add more cool stuff regularly, work has been busy recently

umbral bridge
#

it took a lot more time to get it polished than I thought but the test command is merged!

#

I can't wait to start using it in all my projects

umbral bridge
#

FYI making two breaking changes in the next minor that are fixes:

  • the run and shell commands will no longer switch to the project root if already somewhere inside
  • the shell command arguments are being replaced by flags and the only argument will be the name of the environment to enter. everyone was surprised by the previous behavior, including myself lol
umbral bridge
#

(just merged both of those changes today)

umbral bridge
dull rapids
#

Happy to help!

umbral bridge
#

PEP 723 support coming later today!

image.png
umbral bridge
half spire
#

how to do run hatch shell on a specific environment ? like if I normally do hatch shell it'll spin up default env. how can I spin other envs which are already creadted ?

umbral bridge
half spire
umbral bridge
#

UV support coming sometime in the next week and then I will release the next minor version!

image.png
half spire
#

How to make hatch publish use my pypi API token?

hybrid thorn
low gyro
#

I would understand hatch using uv if it had a dependency management capabilities, but this sounds more confusing than like a good feature

dull rapids
#

I agree. Hatch should use uv for dependency resolution, but hatch run pip should still run pip.

#

The algorithm for selecting a command should stay simple: “if a Hatch script is defined in pyproject.toml or hatch.toml, use that, otherwise use a binary in the env’s PATH”

umbral bridge
#

this is all behind a configuration option, not the default behavior. when UV is enabled pip will expand to uv pip to prevent mistakes when users run pip and there happens to be an executable outside of the virtual environment. in virtual environments created with UV there is no pip

#

that script override can be disabled with another option

#

this also makes it easier for users to keep using the same scripts e.g. if one of their post-install-commands runs some pip command but some environment in their matrix they override to not use UV (maybe a dependency doesn't resolve properly) then the command wouldn't have to change

umbral bridge
#

at the same time, if 3 people say to not do that then I probably shouldn't...

#

okay I'll just put a tip in the documentation that adding such a script might be useful

dull rapids
#

I mean, I think we all didn’t know that it’s opt-in, in which case it’s a conscious decision to enable it

#

In general, I think it is unfortunate that a lot of Python distributions come without an EXTERNALLY-MANAGED file (which when present causes pip to refuse to operate when there is no virtualenv to operate on)

#

Maybe Hatch should come with a pip (and pip3) shim that just displays an error message when there is no pip inside of the virtualenv? That would both be helpful while also not replacing one tool with another.

Then we could also by default not install pip and setuptools into the venv (like rye does: virtualenv --no-seed)

So basically, when doing hatch run, PATH would be

  1. <venv>/bin (might contain pip or not)
  2. <hatch>/shims (contains a pip shim that warns people that they’re not operating on the venv)
  3. default stuff
umbral bridge
dull rapids
#

Beautiful! Love the examples, those really show how to become a power user!

umbral bridge
#

FYI I'm very sick and won't be on the computer much the next few days so will have to wait a little bit for the next minor release. just started antibiotics, steroids and decongestant

orchid birch
#

Oh no! I hope you feel better soon @umbral bridge

orchid birch
#

@umbral bridge How are you feeling?

umbral bridge
#

I'm hoping to release tomorrow

#

and then prepare the presentation for PyCon

slim cairn
orchid birch
#

Great to hear you're feeling better @umbral bridge . Looking forward to meeting you in Pittsburgh!

umbral bridge
#

trying to find a package to showcase the test command

record.gif
umbral bridge
#

Rich might be a good candidate, just a bummer about the test suite causing that warning

record.gif
orchid birch
#

You're welcome to use one of mine. They're small and not amazingly popular 😄 but they're already hatchified. I actually started to switch over to the test command but then realized it wasn't yet released (and then got distracted).

umbral bridge
dull rapids
#

uv and the test command work together really well! I switched some of my stuff and it makes things really convenient!

umbral bridge
#

@errant timber at work folks are seemingly most excited about your build plugin, or at least I've gotten a few messages only about that lol good job!

errant timber
orchid birch
#

Congratulations @umbral bridge - now I'm just waiting for the new version to show up in homebrew (I know, I know, pipx :D)

umbral bridge
umbral bridge
#

@orchid birch I just thought of something, do you want to see something cool? If so, what platform & arch are you on?

orchid birch
#

I love cool things! macOS and have both Intel and Apple Silicon

umbral bridge
# orchid birch I love cool things! macOS and have both Intel and Apple Silicon

you mentioned homebrew and then pipx, which makes me think you might not have tried the binaries. UV has made the first time installation a lot faster

curl -LO https://github.com/pypa/hatch/releases/download/hatch-v1.10.0/hatch-1.10.0-aarch64-apple-darwin.tar.gz
tar xzf hatch-1.10.0-aarch64-apple-darwin.tar.gz
mv hatch-1.10.0-aarch64-apple-darwin hatch
./hatch
orchid birch
#

That is pretty cool. Do you have any sense of relative popularity of the various distribution channels? I'm guessing you're not collecting any telemetry on that, but I thought I'd ask.

Do you have technical details about how this works?

umbral bridge
errant timber
#

Homebrew is out, finished a bit ago. Doing conda-forge now

umbral bridge
#

there is also a PKG for macOS (which just ships the binary) but I thought for testing didn't want to modify your system

orchid birch
#

Yeah 😄

umbral bridge
#

for the next minor release I'm planning on signing the Windows release artifacts but I don't yet know who to talk to, Ee helped me set up the PSF certificate for the macOS stuff

errant timber
umbral bridge
orchid birch
#

Yep, I guess also conda.

orchid birch
errant timber
last cobalt
#

Someone called?

umbral bridge
#

Ee assisted with the macOS certificates but from my recollection they only had purview over Apple stuff

last cobalt
#

Are you using the same certificate we use for CPython? Or did Ee get you your own?

umbral bridge
#

a new one was created on that account

last cobalt
#

Ah, because with Apple you pay for one developer account and can create multiple certs

#

Windows is different - you pay for certs from third parties such as DigiSign. Ee ought to be able to arrange one for you, but it probably requires the PSF to approve it

umbral bridge
#

okay I'll ask thank you. then my next question is, how? lol I've never done Windows signing before actually

last cobalt
umbral bridge
last cobalt
#

If you've got an Azure subscription you can use, absolutely. Wouldn't surprise me if there are tools that work with AWS/GCP equivalents, but I doubt there are many that support those and support Windows binaries 😅

umbral bridge
#

cool, and is only one certificate required? for Apple it was really weird, you had to have one type to sign binaries and another to sign the distribution artifact

last cobalt
#

Yeah, one cert for everything on Windows. Works on .exe/.dll/.pyd/.msi directly, and a few other file types with the right additional tools (I even have an addon to sign .py with them, but that's not public 😉 )

umbral bridge
last cobalt
#

No, they're two unrelated things. The Azure subscription is just secure storage, but the cert comes from another company and is unique to you (I've tried for years to get a Let's Encrypt type feature for Azure Pipelines/GitHub Actions, but we can't make it work... maybe with OIDC someone else can do it?)

#

It might be possible to let you use the one I have in storage for CPython, but the risk is that if it gets out, we have to revoke the cert and all existing installs of Python will start raising red flags.

umbral bridge
#

let's not do that

last cobalt
#

Last I looked the cost is less than $100 for a 3 year cert

#

It's just that anything greater than zero is more complicated than zero

#

Oh, it looks like DigiCert is charging more, but they also offer secure hosting and remote signing.

dusty parcel
#

AFAIK it is currently free but will be $10/mo at some point

last cobalt
#

I have a feeling that service wouldn't use root trust certificates for normal machines, it was meant for enterprises that could generate their own root certificate and install it on all their machines

#

Though it does say "public trust" as an option

umbral bridge
dusty parcel
#

Yeah, I heard about this from someone who uses certs for signing an OSS application

last cobalt
#

That repo is barely two months old, so that's why I haven't seen it before 😄

umbral bridge
#

so... how can I verify that trusted signing does what we require for binary distribution?

last cobalt
#

Looks like it's invite-only right now, and they're keeping to trusted companies. But the root CA has been rolled out and will be trusted everywhere

dusty parcel
#

I think it will go GA in June from what I read

last cobalt
#

So yeah, this would be a viable option when it's accessible. I can try to get the PSF onboarded, but I suspect they are keeping it tight before someone figures out nasty hacks

umbral bridge
#

this sounds awesome and I don't think I would have tried tinkering with signing before June anyway so I'll just wait until it's GA

umbral bridge
last cobalt
#

I'll drop the team a message and see what they think

umbral bridge
#

great, thank you for answering all my questions, I have a much better idea of what Windows signing entails and thanks @dusty parcel for that tip

dusty parcel
#

Sure thing! Saw the discussion and remembered what my friend mentioned, glad it will be useful

orchid birch
#

@umbral bridge if you have some time, I'm just starting to play around with hatch test and have a few questions

umbral bridge
orchid birch
#

Translated now to:

# Default hatch-test matrix is currently 3.8-3.12
[tool.hatch.envs.hatch-test]
default-args = ['tests', 'docs']
extra-args = [
    '--cov=public',
    '--cov-report=term',
    '--cov-report=xml',
    '-p', 'no:doctest',
]
extra-dependencies = [
    'diff-cover',
    'pytest-cov',
    'sybil',
]
#

So first thing is that with those extra-args I think it breaks hatch test --cover because the coverage files are already collected.

umbral bridge
#

that final argument is two arguments

orchid birch
#

Ah, good catch

umbral bridge
#

you can do your own coverage wrangling but I would recommend getting rid of that plugin and using coverage directly with or without the test command flag

#

I was a big proponent of the plugin for years until I realized how everything worked and now it's just a hindrance (and mostly unmaintained)

orchid birch
#

Was thinking the same thing, though is it possible to make --cover (and for that matter --all) the defaults for hatch test?

umbral bridge
#

no those flags must be passed

#

because they are not so much configuration as modifying test execution

orchid birch
#

I can see that. At least for this package the tests run very quickly, so I always run everything and I always like seeing the console coverage report.

umbral bridge
#

if you're doing crazy stuff like using a different test framework or something

orchid birch
#

Yep, saw that. Haven't tried it yet.

#

Other questions is about diff-cov. I'm not quite sure where to plumb that in, maybe in the cov-report script override?

#

Funny thing is you have to use - diff-cov because you can't run this if you're not in a branch other than main.

umbral bridge
orchid birch
#

That's what I thought; thought I tried it but let me confirm

#

Oh, now here's an interesting observation. hatch test --cover --all gives me the coverage report after all the python matrix tests run, so I get one console report. In my previous incarnation, each python matrix run gets its own console coverage report.

Why this could matter: if your tests are Python version dependent, then you might have different coverages for different runs. I've definitely seen that in some projects.

#

My guess is that to get the behavior I want, I'd have to change the default scripts

#

The cov-combine script runs after all tests complete when measuring code coverage, as well as the cov-report script

#

Are you sure about the [tool.hatch.envs.hatch-test.extra-scripts] there? Seems like it has to be [tool.hatch.envs.hatch-test.scripts] - and you have to define all 4 default scripts. I don't think extra-scripts overrides just a single script (e.g. run)

# 
[tool.hatch.envs.hatch-test.scripts]
run = 'pytest{env:HATCH_TEST_ARGS:} {args}'
run-cov = 'coverage run -m pytest{env:HATCH_TEST_ARGS:} {args}'
cov-combine = 'coverage combine'
cov-report = [
    'coverage report',
    'coverage xml',
    '- diff-cover coverage.xml',
]
#

^^ solves the diff-cover question, but note the hatch test --cover --all question.

umbral bridge
orchid birch
umbral bridge
orchid birch
umbral bridge
#

the run command happens once for every environment when using that flag

#

so you would want each run to test and then run some command afterward right?

orchid birch
#

Ah yes, I think I see.

#

Seems to do the trick:

[tool.hatch.envs.hatch-test.scripts]
run = [
    'coverage run -m pytest{env:HATCH_TEST_ARGS:} {args}',
    'coverage combine',
    'coverage report',
    'coverage xml',
    '- diff-cover coverage.xml',
]
run-cov = 'hatch test'
cov-combine = ''
cov-report = ''
umbral bridge
#

Why this could matter: if your tests are Python version dependent, then you might have different coverages for different runs. I've definitely seen that in some projects.
I'm a bit confused here, isn't that the whole point of combining coverage at the end? I don't quite understand the benefit of this approach

orchid birch
#

What if you had a line that got covered under 3.11 but not 3.10, and maybe that was unexpected (i.e. you expected every line to be covered under every Python version). Combining at the end would hide that issue.

umbral bridge
#

I see, I understand now but I would say for differing functionality the tests themselves should fail rather than relying on what is essentially metadata

#

i.e. if someone proposed that on my team I probably would require them rewriting as a test assertion

#

also possible that I don't actually understand and this complexity is necessary 🙂

orchid birch
#

Could also be that the test passes and the missing coverage silently gets ignored. It would be pretty difficult to spot. I know I've seen this in the past, but admit I can't come up with an example atm.

#

It could also be my paranoia 😄

#

Anyway, I appreciate the ear. Maybe worth some time to let it simmer (for both of us)

umbral bridge
orchid birch
#

Thanks! Next up hatch fmt conversion.

orchid birch
last cobalt
#

@umbral bridge @dusty parcel FYI, I was pointed to the actual docs for Trusted Signing, though they're kinda thin. https://learn.microsoft.com/en-us/azure/trusted-signing/overview Sounds like getting the PSF and/or individuals into the preview is possible, just a matter of identity verification. I'll chat with the team next week and get more details

dusty parcel
#

That's great to hear!

dull rapids
# umbral bridge you can do your own coverage wrangling but I would recommend getting rid of that...

I went on the same path as you. Then I was told that vanilla coverage.py doesn’t handle pytest-xdist well: https://github.com/nedbat/coveragepy/issues/1341#issuecomment-1302863172

Is there something you do that makes that work?

A clear disadvantage is that coverage.py doesn’t support removing the working directory from PYTHONPATH. I can’t get the PR through that fixes it since I don’t have windows: https://github.com/nedbat/coveragepy/pull/1700/

umbral bridge
dull rapids
#

That’s awesome! Would be great to have that independently as well …

tropic void
#

@umbral bridge with all the test command and format details - hatch is closer to what tox set out to be than tox is ^

orchid birch
tropic void
#

@orchid birch it's a venv manager and command runner initially created by the creator of pytest to facilitate local first ci/CD fooling

orchid birch
#

"fooling" [sic] 😄

#

@umbral bridge is there an option to get hatch to log or output every command it's calling? HATCH_VERBOSE=3 doesn't seem to give me any additional output for e.g. hatch fmt --check

umbral bridge
#

not yet

errant timber
#

Nox does this on almost every command, and it's one reason I like it 🙂

orchid birch
#

Ok. I was thinking about debugging the "why is it moving that import" and I wanted to see how hatch is actually invoking ruff.

umbral bridge
#

I can fix that, good catch

low gyro
errant timber
orchid birch
errant timber
#

yes, though I was hoping to see a way to serve with autobuild and regenerate the API docs. I can start there though

#

One task can't depend on another like in pixi, right?

#

Also the ability to target multiple backends, not just html. linkcheck and man specifically are important. But maybe those can each be scripts.

orchid birch
#

I never really spent much brainpower on it because I mostly rely on RTD, so I can't wait to see what you come up with.

I don't know about task dependencies, but I think not. What I do sometimes is build higher level scripts/envs that collect a bunch of operations: https://gitlab.com/warsaw/public/-/blob/main/pyproject.toml#L60

errant timber
#

Ah, a script can call other scripts, intersting!

orchid birch
#

Yes! There can be some oddities though. I don't remember the details, but at $work, there was a case where I had to unset HATCH_ENV_ACTIVE env before calling the sub-script from a script. The outer script was calling hatch build so that might be where the conflict came from.

errant timber
#

You can set both features and dependencies, right? Having a missing command issue I'm debugging

#

Ahh, my fault, nevermind

#

No, I take it back, it's not installing the item I have in dependencies when I have features

umbral bridge
errant timber
#

I have:

[tool.hatch.envs.docs]
features = ["docs"]
dependenices = ["sphinx-autobuild"]
scripts.serve = "sphinx-autobuild -n -T -b=html docs docs/_build/html {args}"

Then when I run it with -v after a purge, there's no sphinx-autobuild in the install and it's not available as a command

umbral bridge
#

dependenices typo

errant timber
#

Ahhhh, yes, thank you

#

Much better 🙂

errant timber
# 
post-install-commands = ["$HATCH_UV pip install -e tests/test_utilities"]

Is this the best way to do this?

umbral bridge
#

yes

#

until after PyCon when I implement workspaces

errant timber
#

I did try dependencies = ["-e tests/test_utilities"] but that wasn't supported

umbral bridge
#

yes it's not like tox/nox where it's literally just arguments to pip but rather they must be actual dependency definitions

errant timber
#

I'm not sure workspaces would solve it, I do want both the main package and the example plugin

#

It doesn't rerun the post install if I edit (or add) it, have to env prune every time I fiddle with it

umbral bridge
#

yes and that's not great but workspaces will solve that

#

essentially, anytime you want to do -e on another directory, that is what workspaces is for

errant timber
#

Great 🙂

#

What's the best/fastest way to install hatch on CI?

# 
      - name: Setup uv
        uses: yezz123/setup-uv@v4

      - name: Install hatch
        run: uv pip install --system hatch

?

errant timber
#

The default test matrix doesn't respect reqiures-python. That would have been nice

errant timber
errant timber
# orchid birch I never really spent much brainpower on it because I mostly rely on RTD, so I ca... 
[tool.hatch.envs.docs]
features = ["docs"]
dependencies = ["sphinx-autobuild"]
scripts.linkcheck = "sphinx-build -b=linkcheck docs docs/_build/linkcheck {args}"
scripts.html = "sphinx-build --keep-going -n -T -b=html docs docs/_build/html {args}"
scripts.serve = "sphinx-autobuild -n -T -b=html docs docs/_build/html {args}"
scripts.man = "sphinx-build --keep-going -n -T -b=man docs docs/_build/man {args}"

[tool.hatch.envs.api-docs]
skip-install = true
dependencies = ["sphinx"]
scripts.build = "sphinx-apidoc -o docs/api/ --module-first --no-toc --force src/repo_review"
#

The nox one supported arbitrary builders, this doesn't (due to needing to include -b=<X> in the directory name), and it always installs sphinx-autobuild (could have inherited an env instead), but I think otherwise it's simliar to what I had.

umbral bridge
half spire
#

How can we use hatch publish in github CI/CD to automate package publishing ?

tropic void
#

@umbral bridge any consideration for having hatch also act as pre-commit hook , in particular wrt linting and formatting

dusty parcel
#

Wouldn't you want those to be local hooks because each command is unique other than "hatch"? And their versioning would be tied to the pyproject.toml

dull rapids
#

Wouldn’t Hatch still run on pre-commit.ci? I’d break pre-commit’s env caching, but it would simply run create its own envs there.

tropic void
#

i'd use thus on a few hundred repos in a gitlab instance, so that type of caching is not that much of a problem, also pre-commit.ci has to reject that feature on principle

umbral bridge
#

generally speaking I very much dislike pre-commit

tropic void
#

how come, its pretty handy (in combination with formatters/linters)

umbral bridge
errant timber
#

I usually would have hatch call pre-commit

tropic void
#

hmm, the inboard one pretty much lists stuf thats actually valid and ought to be handled in thedownstreams, disappointing

as for the sentiment, i consider your opinion on the topic "wrong" based on the documented design choices of the tool that is multi language not python only

umbral bridge
#

sure, I'm not saying that Hatch is a valid replacement, I'm saying that the downsides and unexpected behavior with pre-commit does not justify its use for me personally

errant timber
#

Pasted from a DM: By the way, a lot of those arguments about pre-commit are due the fact it's a general check runner, not a Python check runner. I don't mind it not using pyproject.toml because it's not python specific. I use it for Ruby, C++, LaTeX, and other languages too. It doesn't know how to pull Python deps or tool config for affected files because it's not specific. And it's not awful in pre-commit mode and doesn't slow things down too much. I don't use it in pre-commit mode all the time, but don't mind it that way. IMO quick feedback is a better experience than git commit being lightning fast.

umbral bridge
#

I haven't had time to create my own official GitHub Action for Hatch but it's on my list for summer actually!

orchid birch
half spire
#

after I did hatch version micro my terminal is fozen. I have closed vscode as well. is this a bug ?

image.png
tropic void
umbral bridge
tropic void
#

@umbral bridge im wondering if it would be possible for a ci action, to make hatch use the current virtualenv/python env instead of creating own ones

umbral bridge
tropic void
#

oh, that looks lovely - im hoping to fix some bits up with that wrt test commands

umbral bridge
tropic void
half spire
sleek storm
#

@umbral bridge hey! am i reading this right that hatchling is using eval(...) to compute/normalize an SPDX license expression? https://discuss.python.org/t/pep-639-round-2-improving-license-clarity-with-better-package-metadata/12622/112

(not to do impromptu code review since nobody likes that, but IMO that might be worth changing -- i think this would allow someone to sneak arbitrary python expressions into the SPDX identifier as long as it eventually evaluates to something valid)

Discussions on Python.org
PEP 639, Round 2: Improving license clarity with better package met...

It’s hilarious that it works: I implemented it in the days when people would complain about any additional dependency. People still do but now I don’t care so much. I’m definitely interested in a dedicated, small package that only does parsing.

umbral bridge
sleek storm
umbral bridge
#

yeah a security person at work also tried to induce bad behavior and was unsuccessful fortunately

errant timber
#

Hmm, hatch fmt produces a bunch of:

tests/test_adler32.py:1:1: INP001 File `tests/test_adler32.py` is part of an implicit namespace package. Add an `__init__.py`.

Even though I've already configured ruff and don't have INP selected. (And this is a terrible suggestion, test folder's don't need __init__.py's 🙂 )

#

I've only set extend-include, so maybe it's setting a (different) base include?

tropic void
umbral bridge
#

yeah as Ronny said that's actually catching a real issue

tropic void
#

Btw, there's a plan to fix that which I hope to land in this years pytest sprint

umbral bridge
#

there is an option in pytest to change the import mechanism but I don't recommend that

tropic void
#

The importlib mode has a different share of problems

errant timber
#

Historically that meant that some backends (setuptools at one point) would try to ship tests as a package. I've found you shouldn't import anything from tests, and use a test/utilities folder and pythonpath="test/utilities" in your pytest config if you really need to import.

umbral bridge
#

good idea is using Hatchling then 🙂

tropic void
#

Unless importlib mode is used, it's easy to end up with a import mismatch then

errant timber
#

But that doesn't explain why this is appearing when I already have a ruff config. I was trying to help out someone and incrementally add things, not slam them with a bunch of issues, so I have a very minimal ruff config mimicing what they currently are using.

#

(I do also like adding importlib mode, I like avoiding pytest's weird custom importer)

umbral bridge
#

Hatch has default settings that are used which the user must extend/override

errant timber
umbral bridge
errant timber
#

That's pretty rough for a default, I use relative packages all the time, it expecially useful if you support vendoring. But given it's recommended in PEP 8, guess it's acceptable as a default. :/

#

I've tried setting 120 char width before, and had others yell at me because they had tiled windows less than 120 wide. So I now mostly stick to the default 88 or so.

#

If I set config-path = "ruff_defaults.toml" does it write out to that path?

#

Or does it read from that path and I need to generate it somehow?

#

Ahh, found it (--sync)

umbral bridge
#

I'm about to go to sleep but FYI everything can be overridden by your config, Hatch just provides defaults

errant timber
#

config-path = "" doesn't work. Was a little bothered by setting it to "none" when that's an path input. --sync would write out none. But I guess no one would know to try --sync.

#

That's fine, I think I'll be setting it to none in this specific case. This seems like a reasonable system in leu of something fancier from Ruff (which hasn't happened yet)

#

(I'm also looking into updating scientific-python/cookie, and I think I also want none there so it integrates with the manual settings)

umbral bridge
#

I would be very interested in hearing about how the defaults are for your repositories, it sometimes just requires a few options to get what you want (and often not even that)

orchid birch
balmy dock
#

... do right-to-left languages have longer lines?

errant timber
umbral bridge
#

no it wouldn't make sense there, it would need to be a dedicated package for that particular purpose similar to trove-classifiers

errant timber
#

spdx has been untouched (the repo) for 8 years and was archived 4 years ago when the last release was made (2.5.1), I wonder if we could request the PyPI name

umbral bridge
errant timber
#

I assume a parser package would also have a copy of the licenses, though? That seems like the "hard" part, the parser's just a pretty generic one

#

I think you could swap out the eval for a syntax tree based approach really easily

umbral bridge
#

yeah true I guess it would have both

#

although I view the parser as the "hard" part, continuously updating a Python file automatically from an external source seems more straightforward to me

radiant cloak
#

I won't mind having this in packaging TBH.

#

(the list of licenses for spdx, with the need for regular updates)

umbral bridge
low gyro
#

we have that setup in Poetry if you want a reference point

radiant cloak
#

I don't like having standards related discussions on a platform that you can't read without an account/additional access. 🙈

radiant cloak
umbral bridge
radiant cloak
#

I don't know enough about either the hatching parser nor have a fully formed opinion yet on this topic. So, I can't really answer that beyond: do the hatching one, but we might iterate on that or go a different way if necessary?

errant timber
#

The hatchling parser has an eval in it? I'm pretty sure we could get rid of the eval pretty easily, it's just a simple language with parens and and/or AFAICT?

umbral bridge
#

I'll introduce the generation of the enumeration as a first PR

umbral bridge
winged cargo
#

hatch doesn't support extension modules yet, right? - would adding a c compiler interface in the standard library help jumpstart it? (or alternatively, just forking the old distutils.ccompiler for hatch)

low gyro
#

there is setuptools that support extension modules

umbral bridge
winged cargo
umbral bridge
winged cargo
#

ive used hatch-mypyc before, it's nice, but none of those allow you to build raw c code

umbral bridge
#

the first one does

#

uses CMake

#

it's the most powerful extension module builder that currently exists

winged cargo
#

oh, TIL

umbral bridge
winged cargo
#

in the future then, perhaps a much simpler native version could be added to hatch - bringing cmakelists into it can be a bit overkill for some projects

umbral bridge
#

I agree with that, I actually do have a plan to add native support later this summer/fall but I don't want to talk too much about it in case it's not as awesome as I think but it will allow for cross compilation if all goes well

winged cargo
#

awesome - again, look into using the old distutils.ccompiler if you end up struggling to get cross compilation working

#

it's luckily more or less independent of distutils

umbral bridge
#

I'll be using Zig which is actually shipped on PyPI

winged cargo
#

oh right, theres a zig cc command right?

umbral bridge
#

yes exactly

winged cargo
#

that will be interesting to see, good luck!

errant timber
#

You can also do it yourself in a local plugin pretty easily, though it needs some addition to infer-tags that I’ll be working on

umbral bridge
#

oh crap I have to review that PR of yours, sorry been very busy, I'll do that today

tropic void
#

zig is most curious

low gyro
#

zig is C, but designed in 21st century

half cedar
#

I kind of feel Zig is more like C designed for 2010. Still way better than C but already showing some age.

errant timber
#

Zig's ability to cross-compile V/C++ is the really interesting bit, at least to me

half cedar
tropic void
tropic void
#

@umbral bridge I just stumbled upon rye workspaces and im wondering about the plans for hatch workspaces

umbral bridge
#

it'll be based on my experience and a hard requirement to satisfy Barry Warsaw and Apache Airflow maintainer who I can't remember at the moment

umbral bridge
#

taking a cursory glance it looks like what Rye refers to as virtual packages is just an extremely limited form of Hatch environments

slim cairn
umbral bridge
#

@tropic void I'm designing workspaces in my head currently and I thought of something. Say 1000 packages all use hatch-vcs in the same monorepo and are part of the same workspace, what are your thoughts on a caching mechanism so that the build environment for each doesn't have to make the same Git subprocess calls 1000 times?

#

when synchronizing dependencies I have to get the dependencies from the build environment and that requires hooks running every time

tropic void
half spire
#

Hi, if we set the matrix platforms like this

[tool.hatch.envs.hatch-test.overrides]
matrix.feature.platforms = [
  { value = "linux", if = ["foo", "bar"] },
  { value = "windows", if = ["foo"] },
  { value = "macos", if = ["bar"] },
]

will the tests run on each of the platforms or just we're specifying that if the platform is Linux then do this and so on & depending on which platform we're running either manually or in the pipeline the test will run accordingly

umbral bridge
#

the incompatible platforms will not run tests and will output a nice message at the end saying so

half spire
umbral bridge
low gyro
#

doesn't this model make install script versioning harder?

umbral bridge
low gyro
#

usually action have org/name@version scheme, which allows for action versioning. your setup makes it that you are pointing to a branch and prevent users from locking the versions

umbral bridge
#

true, which I'm going to document in a few minutes, one must use the commit hash

#

I think the benefit of being in the same repository outweighs that minor inconvenience (although at large companies security folks usually enforce pinning to commit hashes anyway)

balmy dock
#

They do
But those are normally the hashes of release commits
Which is how Dependabot can update the comment next to it -- the comment is just the tag

#

Pinning to the commit hash in the main branch for the app itself is just going to cause tooling to trigger update PRs for every single commit that you ever make, regardless of whether it's actually updating the action or not

umbral bridge
umbral bridge
umbral bridge
half spire
#

I'm getting NotADirectoryError: [Errno 20] Not a directory: '/usr/bin/ruby' while doing hatch shell

umbral bridge
#

Thread

half spire
umbral bridge
#

nothing official but for discovery I would recommend going to each plugin type in the docs and on top there is a list that I try to maintain for each

half spire
#

how can we use uv as default installer for all of the envs ?

umbral bridge
#

the reason it's not the default is because I'm planning on upstreaming stuff to pip which would make it comparably fast (or at least significantly closer) but that won't be for a long time

turbid bane
#

what are those patches?

umbral bridge
# turbid bane what are those patches?

cache as unpacked wheels mostly, and if it's warranted rewriting the resolver based on pubgrub (but I don't know how much that would help versus the current implementation)

#

for the former that would of course be coupled with new installation options like symbolic/hard linking

turbid bane
umbral bridge
#

something that would definitely help but I don't know if it's merged yet is the parallel downloads PR

turbid bane
#

the networking stack is not fully thread safe (being so old as you're aware) AFAIU

umbral bridge
#

hmm, what do you mean? doesn't it just use a vendored version of requests?

#

if you mean requests itself is not thread safe then yeah lol I can imagine because I've encountered issues with that in the past

turbid bane
umbral bridge
#

I'm trying to get rid of requests everywhere I can in favor of urllib3 or HTTPX

turbid bane
umbral bridge
#

my secret hope is that I don't have to do anything because by the time I get to it it will already be done by Nvidia's new dedicated pip/warehouse maintainer 🤞

turbid bane
#

oh?

#

I'm on the pip team and I haven't heard anything about this.

umbral bridge
#

hmm not sure if it's public sorry let me check

#

yes okay good it's public, Barry Warsaw 🙂

turbid bane
#

Ah okay. Neat!

half spire
#

so will it get automatically enabled for the rest 2 along with test ?

#

or it has to be explicit ?

half spire
dull rapids
half spire
#

That is what I was asking 😁

half spire
#

I'm seeing that some times hatch shell doesn't actually start the virtualenv. Saw this issue with pdm as well as poetry in the past. maybe an python venv issue? using zsh

umbral bridge
urban marten
#

Hello! I am trying to migrate a setuptools project to hatch, and have a few questions... Perhaps I can start with this one - Can I make hatch only create / use a single virtual environment for everything (the default one)? By everything I mean linting, testing, etc?

#

Is this not the hatch channel?

low gyro
#

my fault, Discord was showing me that I was on #general

urban marten
#

I should probably expand that the reason for my question is that Im struggling to find how/why I should work with multiple environments to start with, and what is the reason for their existence for simple projects like mine. It seems to me that a single virtual environment kind of setup might provide for a straightforward way to get some people (like me) to switch over easily...? Happy to be enlightened otherwise, but although Ive been browsing the documentation for the past day, I have not yet stumbled on the reasoning behind this scheme... 🙏

umbral bridge
umbral bridge
urban marten
urban marten
dull rapids
urban marten
#

Hello! Can I make hatch either use the platform name to name virtual environments or use a different parent dirs.env.virtual based on the platform? 🙏

umbral bridge
half spire
#

Hi I'm getting Error importing plugin "pydantic.mypy": No module named 'pydantic' error while running pre-commit hooks. I tired runnig from different env as well. what am I missing ?

image.png
low gyro
half spire
# low gyro show your pre-commit spec 
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
  rev: v4.5.0
  hooks:
  - id: trailing-whitespace
  - id: check-added-large-files
  - id: check-merge-conflict
  - id: check-shebang-scripts-are-executable
  - id: check-docstring-first
  - id: debug-statements
  - id: check-ast
  - id: check-json
  - id: check-toml
  - id: check-xml
  - id: check-yaml
    args: ['--unsafe']  # needed for !! tags in mkdocs.yml
  - id: end-of-file-fixer
  - id: mixed-line-ending
    args: ['--fix=auto']  # replace 'auto' with 'lf' to enforce Linux/Mac line endings or 'crlf' for Windows

# Ruff replaces black, flake8, autoflake, isort and more
- repo: https://github.com/charliermarsh/ruff-pre-commit
  rev: 'v0.3.4'  # make sure this is always consistent with hatch configs
  hooks:
    - id: ruff
    - id: ruff-format
      args: [--check, --config, ./pyproject.toml]

- repo: https://github.com/pre-commit/mirrors-mypy
  rev: 'v1.9.0'  # make sure this is always consistent with hatch configs
  hooks:
    - id: mypy
      args: ["--install-types", "--non-interactive"]
      additional_dependencies: [types-tabulate, types-cachetools]
low gyro
#

cause pre-commit installs all those in separate env

half spire
half spire
#

Is there a way to specify which tests should be run depending on the platform. Ex. Say Run all tests(unit + integration) if running in linux & only run unit if running from windows ?

#

apart from specifying it in the [tool.hatch.envs.test.scripts] section

umbral bridge
umbral bridge
winged cargo
#

is there an example repo for scikit-build-core with hatch

dull rapids
#

I’m a bit stumped: shouldn’t this work?

[project]
name = "scanpy-tutorials"
version = "0.1"
requires-python = ">=3.9"
dependencies = ["sphinx"]

[tool.hatch.build.targets.wheel]
bypass-selection = true  # This is not a package

[tool.hatch.envs.default]
python = "3.12"
installer = "uv"
[tool.hatch.envs.default.scripts]
build = "sphinx-build -M html . ./_build {args}"

$ hatch run build
/bin/sh: line 1: build: command not found

I’m pretty sure this worked before. Why doesn’t hatch run run the script and instead tries to find some executable?

merry sable
#

QUESTION: Using Hatch scripts / envts to replace pre-commit

hey everyone - i'm working on an example package using hatch and wondered if anyone has migrated away from pre-commit here --> hatch (and could provide me an example of what that might look like with tools beyond ruff, etc. If i am asking this in the wrong channel please direct me to the right place! many thanks 👐

umbral bridge
#

I’m a bit stumped: shouldn’t this work?

#

hey everyone - i'm working on an example

tropic void
#

is there any recommended hatch plugin for building c extensions with hpy ?

umbral bridge
#

I'm still working on workspaces, that's all I'm doing in my limited free time

tropic void
#

kk, i'll defer trying to figure how to make pluggy support hpy speedups for now then

#

@umbral bridge i jsut realized i need to use \\instead of just \to split a hatch script comand into multiple lines - that one is quite a bit puzzleing (as i couldnt find it in the docs)

# 
scripts.generate = """
   datamodel-codegen \\
     --input=openapi/openapi.yaml \\
     --input-file-type=openapi \\
     --output=model_codegen.py \\
     --output-model-type=pydantic_v2.BaseModel
"""

works

umbral bridge
tropic void
#

andi think i asked this before - but whats the correct way to reffer to a command thats the same name as the script

tropic void
umbral bridge
tropic void
#

ah, you meant literal strings 🤦‍♂️

umbral bridge
tropic void
#

I'm currently using env

#

But that's not portable

umbral bridge
undone forge
#

Hello everyone!

I am currently trying to setup hatch in my repository.

The thing is, the name of the repository, and of my main module inside my code is eve
But the metadata name of my project if sdl-eve.

When I try to perform hatch env create, here is my error:

❯ hatch env create
error: Failed to download and build: `sdl-eve @ file:///home/me/ws/sdlrep/eve`
  Caused by: Package metadata name `eve` does not match given name `sdl-eve`

Here is my build configuration:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[build]
exclude = [
  "coverage",
  ".eve",
  ".walle",
  "tests",
  "scripts",
]

[build.targets.sdist]
only-include = ["eve"]

[build.targets.wheel]
only-include = ["eve"]

When I was using poetry, just doing this solved the issue:

[tool.poetry]
packages = [ { include = "eve" } ]

By chance, do you have any clues?

Thanks !!

umbral bridge
#

update: workspaces is coming along nicely but I didn't realize that a prerequisite would be better built-in support for the standardized build hooks so that's why it's taking a bit longer than I thought (and lack of free time). after this is over build will no longer be a dependency

undone forge
#

Hi @umbral bridge ! Hope you're fine.
I was looking at an alternative for our poetry.lock
I thought it would be great to exploit the UV_EXCLUDE_NEWER environment variable.
I was wondering if I could dynamically set this environment variable value for hatch with the latest commit date of the pyproject.toml or hatch.toml.

Do you know if it is possible to set dynamically this kind of setting ?

Maybe instead I should override the command used by hatch when calling UV in order to add this flag dynamically at each command ?

Thanks!!!

umbral bridge
undone forge
#

And not bother with dynamic value computing

undone forge
undone forge
#

Seems to be working great! Wonderful

tropic void
#

@umbral bridge aware of any trick to make pycharm auto-find the hatch environment?

umbral bridge
tropic void
umbral bridge
undone forge
#

When using hatch version, can we do some dry-run of the command ?
In order to get the next version proposal ?

undone forge
#

Don't think so 🙂
Can be a good first contribution for me 😄

barren rampart
#

👋 Hello all. Im working on porting a project from setuptools/setup.py to hatch/pyproject.toml and Im wondering if there is an equivalent to scripts from setuptools to copy arbitrary scripts to the bin dir of the installed (v)env?

Ex. our project bundles a couple of python scripts for handling parsing gprof output

    ...
    scripts=['scripts/geordi', 'scripts/gprof2dot'],
    ...
)```
umbral bridge
barren rampart
#

Awesome! Thank you

tropic void
umbral bridge
#

yes definitely, the only valid use case now in my opinion is distributing binaries like Ruff

barren rampart
umbral bridge
tropic void
# barren rampart Im not sure I know what that means, got an example?

the general suggestion is not to use actual script scripts, but rather have the cli bits be python modules with a entrypoint function, then using project.scripts section to specify those (that allows for the binaries to be more nicely managed by the installer, which gets you better platform support for the odd ones out (like windows)

barren rampart
#

Ahh the downside being any shell scripts will need to be refactored/wrapped?

umbral bridge
#

it's mostly for the aforementioned cross-platform compatibility issue

tropic void
barren rampart
#

Same! haha

barren rampart
#

@tropic void did you used to work with bitbucket btw? Your name looks familiar

tropic void
#

Never worked there, but back before they broke mercurial we had pytest on it

barren rampart
#

I work on it now, but I must have come across your name in public tickets or something

tropic void
#

Likely

#

But I consider them responsible for the end of mercurial and stay far away, I'm really salty about it even after All these years

barren rampart
#

I think that is a perfectly reasonable take and Im privy to all of the inside baseball

#

From a "business" perspective, there simply werent enough paying customers using it to warrant the required R&D to continue supporting both libraries. But those lovely business calculations never seem to account for uhhh destruction of goodwill in the community

#

And the people in charge at Atlassian now are sooooo far removed from what Bitbucket used to be that it doesn't even register to them

tropic void
#

When bitbucket was started,the founder was in the mercurial irc channel and bitbucket was on top of updating things

#

But enough with that, this is the hatch channel

umbral bridge
#

definitely the best part of open source 💜

image.png
balmy dock
half spire
#

but still it's not visible

balmy dock
#

Why do you have an asterik on the left side?

#

The path starts with .config, doesn't it?

half spire
#

yah tried without that

#

still same

balmy dock
#

I think we'll have to get someone smarter than me

umbral bridge
half spire
#

did something like this but still the same

half spire
#

ok I was able to solve the issue by doing this

# Build configs
[tool.hatch.build.targets.sdist]
include = [
  "src/**/*.py",
  ".configs/*"
]

[tool.hatch.build.targets.wheel]
packages = ["src", ".configs"]
#

thanks

orchid birch
tropic void
umbral bridge
#

okay, the massive build refactor has been merged, now workspaces is unlocked once again

half spire
#

Is there a way to stop redefining deps for each env ?

this is my default env

[tool.hatch.envs.default]
installer = "uv"
dependencies = [
    "click",
    "rich"
]

& I need these for all of my 3 other env & I need the deps from lint env into test as well.

umbral bridge
half spire
half spire
umbral bridge
#

no

half spire
#

I'm getting this error

    from ape_utils._cli import main
ModuleNotFoundError: No module named 'ape_utils'
# 
# Cli tool name & options
[project.scripts]
ape_call = "ape_utils._cli:main"
# 
.
├── CONTRIBUTING.md
├── LICENSE
├── Notes.md
├── pyproject.toml
├── README.md
├── src
│   └── ape_utils
│       ├── __about__.py
│       ├── _cli.py
│       ├── __init__.py
│       ├── __pycache__
│       │   ├── _cli.cpython-311.pyc
│       │   └── __init__.cpython-311.pyc
│       ├── py.typed
│       └── utils.py
└── tests
    └── __init__.py
umbral bridge
#

what command are you running?

half spire
#

the name I specified for the tool

#

tried hatch run ape_call as well same error

umbral bridge
#

you shouldn't ever set build paths for wheels unless there is an error, the default behavior is correct

#

then hatch env prune and try again

half spire
winged cargo
#

how can i tell hatch to install an optional dependency inside an environment?

#

e.g. i want this to install myproject[full] and not myproject

[envs.test.scripts]
no-cov = "pytest --memray --native"
cov = "coverage run pytest --memray --native"
half spire
#

take this example

# 
[tool.hatch.envs.default]
installer = "uv"
dependencies = [
    "myproject[full]",
]

[tool.hatch.envs.lint]
extra-dependencies = [
    "mypy",
    "ruff",
    "myproject"
]
#

so for the lint env hatch will install the default deps as well as the extra ones defined in lint

#

If I define the deps inside the default env when I'm going to install the package using pip the default ones will be project deps right ? to get the lint deps we have to do pip install package[lint] right @umbral bridge(sorry for the ping) ?

winged cargo
#

well, that’s a bit inconvenient

#

i already have the dependencies specified in project.optional-dependencies, it’s a bit silly to redefine all of them

#

it also adds for more maintenance, i.e. if i add another optional dependency i have to add it in two places not just one

umbral bridge
half spire
#

how does hatch publish auth creds looke like?

# 
hatch publish -u __TOKEN__ -a $TEST_PYPY_SECRET -r test

is giving me 403 Username/Password authentication is no longer supported. Migrate to API Tokens or Trusted Publishers instead

umbral bridge
#

the default behavior assumes a token and should work if you remove the username option; it looks like you capitalized the word token when it needs to be lowercase

half spire
half spire
#

that's why I disturb you the most 😵‍💫

umbral bridge
#

what should be moved?

half spire
#

what do you think ?

umbral bridge
#

there is a further resources section at the end

#

the pages at the start of the docs are meant to be a brief showcase of working with Hatch

half spire
umbral bridge
half spire
#

maybe it can be mentioned here ?

image.png
#

or inside > ?

half spire
#

is there a way to include deps which has prereleased deps ?

#

"eth-ape>=0.8.9" module has this dep for which I can't add it to my projects. have to use older version. Don't know this is an issue with uv or not

umbral bridge
#

there's probably an environment variable that can control the behavior

#

or whatever environment you have that configured in, disable UV, I just happened to link the testing environment docs

half spire
#

yup it's an issue with uvswitching to pip works just fine.

half spire
umbral bridge
#

not yet, passing arguments is a an open feature request

half spire
austere relic
#

can i force hatch to use the downloaded pythons rather than available pythons?

umbral bridge
umbral bridge
austere relic
half spire
undone forge
#

Hello, can we define the publish index in a repository config file ?

#

@umbral bridge More generally, i want hatch to use a local repository config file without specifying it every time in the CLI.
Otherwise, I would want to configure publish.index.repos url directly in my pyproject.toml.
is it possible? thank you very much!

undone forge
#

@umbral bridge Hello! Hope you're fine?
I think I found a bug:
When scripting my release mechanism, I call hatch version from a subprocess.run.
As you can see, the result contains some hatch ouput (it's inside the command of course):

CalledProcessError: Command '('git', 'log', '--pretty=format:%an|%s', '--no-merges', 'vSetting up build environment for missing dependencies\n0.15.1..HEAD', '--')' returned non-zero exit status 128.

Then of course, I can use hatch --quiet
But when I use it:

🛑 Command '(PosixPath('/home/jcarlens/ws/sdlrep/eve/.virtualenvs/hatch/bin/hatch'), '--no-color', '--data-dir=/home/jcarlens/ws/sdlrep/eve/.virtualenvs/hatch', '--config=/home/jcarlens/ws/sdlrep/eve/config.toml', 'version')' returned non-zero exit status 1.

Setting up build environment for missing dependencies                                                                                                                                                                                                                                                                                                                                      
error: the argument '--quiet' cannot be used multiple times                                                                                                                                                                                                                               Usage: uv venv [OPTIONS] [NAME]                                                                                                                                                                                                                                                              For more information, try '--help'.

It seems there is some conflicts with UV flags.
I also tried to use the env var HATCH_QUIET, same issue

image.png
#

(The image at the end is not relevant, uploaded by mistake)

low gyro
#

looks like subprocess hell

#

you subprocess to hatch, hatch subprocesses to uv

undone forge
#

oh, maybe..

slim cairn
#

Sounds like a bug in hatch though

orchid birch
#

@umbral bridge I wanted to get your take on supporting Trusted Publishing natively in hatch. I might find some time to contribute something but I don't want to reinvent the wheel, duplicate existing work, or build something you wouldn't be interested in. I'm aware of the GH action on the PyPA org, but I want support for GitLab. Any thoughts?

umbral bridge
orchid birch
umbral bridge
#

for prototyping I would recommend a plugin first so you know it works and then we could incorporate into the main code

orchid birch
#

Awesome, thanks. If/when I get to this, I'll open an issue on GH.

umbral bridge
#

something else to note is that although there is an option to pass arguments it's not that pretty and actually at work I'm prototyping something with a different tool that has plugins to allow a msgspec struct/class to transform into Click options and expose that dynamically in the help text

#

I'm confident I'll get it to work however so whenever we have success internally I'll incorporate into Hatch

#

unless Pydantic is able to drastically improve model build times it looks like Hatch will be using msgspec for configuration for everything

orchid birch
#

Gotcha. And yep, I haven't actually used msgspec but I've been hearing good buzz about it.

cursive lion
orchid birch
#

Basically, I'm thinking (long term) about hatch publish doing that recipe natively.

winged cargo
#

i'm having some trouble with environments, it seems setting dev-mode to false prevents the project from being installed in scripts

umbral bridge
#

i'm having some trouble with

tropic void
#

@umbral bridge is there a way to make command templates - im hitting a case where i have to run a similar command for a mapping - and it seems like i need to start using a build tool instead of hatch

umbral bridge
#

what do you mean by template?

tropic void
#

@umbral bridge kind of like shell aliases or functions - i moved that to a python script now as its a bit too complex for hatch scripts

umbral bridge
#

I see, can you please give an example?

tropic void
#

like have a command like openapi-code-generator .... --input = something{name} --package-name=something{name}and then calling it with different values for name

umbral bridge
#

use a matrix with variables?

tropic void
#

its for a part of the scripts

#

in the same env

umbral bridge
#

actually in your particular example you can just do {args} at both spots

#

@tropic void does that suffice?

tropic void
#

@umbral bridge not quite - im having more than one thing i need as a variable in the end

umbral bridge
#

probably nothing more can be done unless you want to make your commands Python scripts e.g.:

foo = '''python -c "\
import stuff
use {args}
...
"
'''
tropic void
#

i made an actual python script - hatch shouldnt invent a way to make "scripts" simple "functions"

umbral bridge
#

that sounds like the better way

orchid birch
umbral bridge
orchid birch
undone forge
#

Hello, currently what is the normal workflow when you update a dependency and you're working on your IDE ?
For example with poetry, a notification on VSCode remind you that you should run a poetry command to update the environment, for hatch, I am performing hatch shellcommand manually, and hatch --env=tests shell to also update other environments for example.
I was wondering if for now it was the normal flow of doing this ?

#

Don't think there is a command to update all environments right?

austere relic
#

is there a non enviroment variable flag that tells hatch to install the latest pip onto a env?

tropic void
winged cargo
#

how far away are we from getting editable installs for the scikit-build-core plugin?

umbral bridge
#

how far away are we from getting

umbral bridge
#

I'm sorry I've had minimal free time for the past few months but I swear workspaces is really close, the first test is passing

image.png
#

also, overall, development will be more rapid for two reasons:

  1. I introduced a new CLI at work where I did a significant amount of experimentation and a lot of it was successful so will be incorporated and cleanup some internal stuff I've been displeased with
  2. I'm no longer discouraged by the speed of UV development and have accepted that I cannot compete with a dedicated team and understand that there are many benefits to Hatch in comparison which I will talk about in one of the upcoming release blog posts
cursive lion
#

I really enjoyed your PyCon US talk, it was cool to see all the good UX stuff in Hatch

umbral bridge
umbral bridge
austere relic
umbral bridge
austere relic
umbral bridge
austere relic
umbral bridge
# 
[[tool.hatch.envs.hatch-test.matrix]]
python = ["3.13", "3.12", "3.11", "3.10", "3.9", "3.8"]
[[tool.hatch.envs.hatch-test.matrix]]
python = ["3.13"]
gil = ["freethreaded"]
[tool.hatch.envs.hatch-test.overrides]
matrix.gil.env-vars = [
  { key = "HATCH_PYTHON_VARIANT_GIL", value = "freethreaded", if = ["freethreaded"] },
]
austere relic
#

ah, was about to ask again how to modify the one i had, which is

sorry, i struggle with your pyproject.toml it isn't intituitive to me

[[tool.hatch.envs.main.matrix]]
python = ["3.13", "freethreaded", "3.12", "3.11", "3.10", "3.9", "3.8"]
matrix.version.env-vars = [
    { key = "HATCH_PYTHON_VARIANT_GIL", value = "freethreaded", if = ["freethreaded"] },
]
#

so we have to duplicate tool.hatch.envs.hatch-test.matrix seems weird

umbral bridge
#

it's not duplicating, it's adding another series

austere relic
#

i get this with that

last cobalt
# umbral bridge did they ever respond about giving the PSF access?

Yes and the PSF now has access. I've been setting up the CPython releases to use it (been away for a few days so it's not quite finished yet, but this week). There'll likely be a discussion needed (with Ee, initially) about providing access to other projects, but technically it can now be done.

umbral bridge
austere relic
#

is this behavior correct?

umbral bridge
#

yes that shows what is available

austere relic
dusty parcel
#

I'm reading through the docs for build hooks as I'm interested in adding files to the list of includes or not based on libmagic data. But if I'm reading the docs correctly, that isn't possible if I want to use the wheel target?

umbral bridge
#

(afk for dinner)

dusty parcel
#

(updated above s/update/initialize, it's a build hook not a metadata hook 🙂 )

dusty parcel
umbral bridge
dusty parcel
#

hatch build -t wheel

#

I put repro commands in the readme of the repo

umbral bridge
#

oh, you're trying to ship directories without files?

#

that's not supported, although I'm interested in hearing why you think that should be supported

dusty parcel
#

Ah, I think I just didn't realize it was failing only for directories

umbral bridge
#

actually that's not supported by the spec as far as I can tell, you can't derive metadata to put inside the RECORD file

dusty parcel
#

It might be good to raise an error in that instance then

umbral bridge
#

that wouldn't make sense in this case because the directory does exist

#

an error already is thrown when a forced inclusion does not exist

dusty parcel
#

But I don't currently see an error when I build the above

umbral bridge
#

did you make a modification or are you saying you don't see an error how your reproduction repository currently exists?

dusty parcel
#

I do not see an error with the state in the git repo

umbral bridge
#

yes, that is correct behavior because the directory exists

#

add a foo to the key in the forced inclusion mapping and then try again

dusty parcel
#

That does error. But shouldn't it be an error to try to force_include a directory that is empty?

umbral bridge
#

no

dusty parcel
#

Why? If it doesn't get included in the wheel I expect that would be surprising

umbral bridge
#

it is included in the wheel, there just happens to be nothing in the directory

dusty parcel
#

oh I think you misunderstand, the empty directory doesn't end up in the wheel

# 
$ unzip -l dist/demo_hook-1.0-py2.py3-none-any.whl
Archive:  dist/demo_hook-1.0-py2.py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
      370  2020-02-02 00:00   demo_hook-1.0.dist-info/METADATA
      105  2020-02-02 00:00   demo_hook-1.0.dist-info/WHEEL
     1095  2020-02-02 00:00   demo_hook-1.0.dist-info/licenses/LICENSE.txt
      307  2020-02-02 00:00   demo_hook-1.0.dist-info/RECORD
---------                     -------
     1877                     4 files
umbral bridge
#

yes I do understand that is the correct behavior. you're defining configuration that says "the contents of this directory should be mapped to this path inside the wheel" the directory does exist and everything is working properly but it just so happens that the directory is empty

#

if the directory didn't exist that would be an error

dusty parcel
#

Oh, I see, I didn't realize force-include when referring to directories included their children. That makes much more sense now. I thought it was merely specifying that path x should be placed at path y in the wheel

#

Thank you for the explanation!

umbral bridge
#

ah I see, happy to help!

dull rapids
#

Is there a way to make hatch run/hatch test execute a build plugin every time? (Or better every time a file from a set of file changes)

I use some codegen in one of my projects and I’d like to make sure that my dev environment stays synced with changes.

Currently I need to remember to do hatch env prune/hatch env remove ... whenever the codegen changes.

umbral bridge
#

you can change your scripts to always have an initial command that does what you want

dull rapids
#

I just got told that my “wonderful” hatch plugin doesn’t even work.

Trying to make it work results in infinite recursion, because I currently evaluate WheelBuilder(self.root).config.packages, which in turn calls the metadata hook again.

Any idea how to fix this more elegantly that inspecting the stack? https://github.com/flying-sheep/hatch-docstring-description/pull/73

umbral bridge
void jasper
#

@umbral bridge, first off, I wanted to say nice to meet you. I've been a user of hatchling, hatch, and pyapp (which is my personal favorite) for some time now, and I enjoyed your session at this year's PyCon. PyApp really solves the packaged application distribution in an elegant way.

To stay up to date, I've been moving a few PDM based projects to uv, but hit some differences in the generated source and wheel files. These difference have left me wondering which is correct.

Here is the specific question I opened on the UV repo: https://github.com/astral-sh/uv/issues/8833

After re-reading the hatchling docs and re-testing with hatch. It seems like the behavior you've documented is correct. In summary, I can use this:

[tool.hatch.build]
dev-mode-dirs = ["."]
sources = ["."]

[tool.hatch.build.targets.wheel]
artifacts = ["**/public/*"]
packages = ["app"]

instead of this:

[tool.hatch.build]
dev-mode-dirs = ["."]
ignore-vcs = true
sources = ["."]

[tool.hatch.build.targets.wheel]
packages = ["app"]

Is my understanding of artifacts correct here?

umbral bridge
#

the difference between tool behavior is due to your use of the global [tool.hatch.build] table which is used as the default for every target. in hindsight this was a big mistake for me to implement or at least document which is why the docs are updated to never do that

#

you should never use the global table. what is happening is that the table is configuring the build for source distributions and build and UV by default build a source distribution and then build the wheel from that but Hatch does not (and I'm assuming based on your description that neither does PDM)

void jasper
#

OK, let me make sure I got it. I need to refactor my setup so that I no longer tool.hatch.build in my pyproject? Also, now that I revisit the docs, I don't see a reference for this there. I guess that's what you mean by "docs are updated"? 🙂

umbral bridge
#

yes exactly configure the targets individually

#

basically with UV (and future Hatch default behavior) you have to think about what will be included in the source distribution and what you configure for the wheel will be based on its content e.g. if you exclude a file from the source distribution then it won't be present to include in the wheel

void jasper
#

Got it. Thanks for the quick reply.

Is this also the right place to ask pyapp specific questions? I don't see a specific channel for it.

umbral bridge
#

anytime! and yes you have to scroll to the bottom in the other projects section

void jasper
#

I see it now 🙂

umbral bridge
inland hemlock
errant timber
inland hemlock
#

I'd think it would be good to cross-post

lethal birch
#

Is it possible to define a version plugin via hatch_build.py alongside the rest?

I need to generate a dynamic version from git, but from a submodule not the current repository, so versioningit doesn't work OOTB. But when I try to define a version plugin in hatch_build.py and set its PLUGIN_NAME as tool.hatch.version.source, build tells me it's an unknown source.

vcs and versioningit define an entry_point, but I don't think that can work for the project trying to use the thing (and it certainly doesn't seem to work to define a project.entry-points.hatch whose value is set to "hatch_build"

tropic void
lethal birch
tropic void
# lethal birch custom builder?

its what makes the build hooks work for the hatch_build plugin - but i jsut took note that theres no variant for version sources as far as i can tell

umbral bridge
#

that should be able to allow version modifications but you won't get interoperability with the hatch version command

tropic void
#

ah, so the metadata hook provides version in that case?

lethal birch
umbral bridge
#

you would have to link your code

lethal birch
# umbral bridge you would have to link your code

Aha, thanks, I think when I tried metadata hooks the previous time it might have failed to work because I didn't get the registration stricture correct (I think I missed that I had to add a [tool.hatch.metadata.hooks.custom] in addition to the [tool.hatch.build.hooks.custom], the metadata hook does seem to work and correctly set the dynamic version field once it's actually correctly hooked in 🫡

errant timber
#

I haven't tried all the combinations Enrico has but I did verify that the folder is missing from SDists.

errant timber
void jasper
#

I've got a question re: the pre-built python that is installed by hatch.

I've found a few Xeon based machines where the python build does not work. I am assuming this is is because of the CPU variant configuration it was built with.

In PyApp, you can set PYAPP_DISTRIBUTION_VARIANT_CPU to get a specific compiled version. Is this possible with hatch python install?

errant timber
#

Hatch doesn't support dependency groups yet, correct? It would be really nice for mixing a bit of uv and hatch.

umbral bridge
#

correct not yet, what's your use case?

half spire
#

Hi, how can we install the deps from default env into another env like test without writing them again ?

half spire
#

test env deps

image.png
#

default env deps

image.png
# 
[...]
# Default env dendencies
[tool.hatch.envs.default]
dependencies = [
  "pydantic",
  "google-cloud-kms",
  "eth-account",
  "eth-utils",
  "eth-keys",
]
installer = "uv" # or "pip"
post-install-commands = ["pre-commit install"]

[project.optional-dependencies]
dev = [
  "ipython",
  "web3"
]

[...]

# Testing env dendencies
[tool.hatch.envs.test]
dependencies = [
  "pytest",
  "pytest-cov",
  "eth-account"
]

[tool.pytest.ini_options]
addopts = [
    "--verbose",
    "--strict-markers",
    "-ra",
    "--cov=src/google_cloud_hsm/",
    "--cov-report=term-missing",
    "--cov-fail-under=80",
]
markers = [
    "integration: marks tests as integration tests",
    "performance: marks tests as performance tests",
]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]

what am I doing wrong ?

tropic void
half spire
half spire
#

Hi, everyone Merry Christmas. I was using pycharm and saw that it doesn't detect the vritual env created by hatch then I get to know about this workaround

[tool.hatch.envs.default]
type = "virtual"
path = ".venv"

So it got the thinking what's the reason behind this choice of design ? I can definitely see a upside is when we're out of storage we can easily clean all of the envs from one place without going to the individual projects much like conventional virtual envs.

umbral bridge
#

I don't want my working trees cluttered and would prefer centrally managed data directories

#

additionally, beginners are prone to checking in data that is undesirable like virtual environments or log files

half spire
#

I like this design more TBH.

#

too bad pycharm is not supporting it yet

half spire
half spire
#

only the default one as far as I understood. and docs and other deps will still be inside the default location

umbral bridge
#

all options should respect inheritance I think, perhaps try re-creating?

tropic void
#

@umbral bridge hatch test is incorrect by forcing the no args pytest arg to be tests - its quite a pain - default testpaths in pytest have a config var that pytest cannot use if one incorrectly passes a path by default

also the common pattern is to use testing as folder name as tests actually shadows stdlib package

umbral bridge
tropic void
#

all pytest related pacakges use it and afair we also document it in the docs

umbral bridge
#

fair, but I think even in a few decades everyone is still going to name the directory tests and I have no desire to change that default

tropic void
#

just dont pass anything and let pytest decide

umbral bridge
#

no Hatch just needs to learn how to read the arguments for pytest and if there are none default to the test directory

#

without a default the performance is suboptimal because pytest has to scan everything in the current directory

#

that's actually literally the only reason I made a point to have a default, for performance and accidental test recognition of user code

tropic void
#

then at least prominently docment how to return pytest to its default behavior correctly as part of the test docs

#

bbl, family is back

umbral bridge
#

@tropic void enjoy the holiday with your family! when you return I'm curious about an idea I just had, what if pytest added a new option that would only consider a single top level directory that appeared to be for tests e.g. named test, tests or testing? if there was such an option I would completely remove the default argument logic from Hatch

tropic void
umbral bridge
#

no I mean to have a new flag which would make pytest be smart about which top-level directory to target based on the name

#

and depending on usage I would imagine that flag being enabled by default after a while because that is better as I mentioned above

for performance and accidental test recognition of user code

tropic void
#

Pytest supporst tests in packages tests at the toplevel ans some more

That fla eill be a pain in the neck but most likely worth it

umbral bridge
#

do you know of an open feature request to change the behavior/add the flag? if not I can open one for you

tropic void
#

Theres none open for the case you made

half spire
#

Also just noticed mkdocs is using the cached response from the first build. even after doing a build with --clean and removing the sites dir completely it's returning the pages from the very first build. Only work around was to remove the venv complety by doing hatch env prune

#

added this line. and no change here

image.png image.png image.png
#

after env prune now it's there

image.png
half spire
# 
[tool.hatch.envs.default]
# https://hatch.pypa.io/1.12/config/environment/overview/#dependencies
extra-dependencies = [
    "pydantic",
    "pydantic-settings",
    "google-cloud-kms",
    "eth-account",
    "eth-utils",
    "eth-keys",
    "ecdsa[gmpy2]",
    "cryptography",
    # "rich", # typer will include it
    "typer"
]

when the field was dependencies only I think it was fine. but changing it to extra-dependencies they are not getting installed along with module itself

#

My usecase. My project is using the above deps. And I want them to be installed in my other envs like docs & test and don't want to define them manually for all of the envs.

#

This config is working fine but when I'm trying to install the project the deps are not installed .

tropic void
half spire
#

bcz last time I used hatch defining deps in the dependencies section of the default env was enough to inherit other envs

#

or I'm horribly wrong ?

tropic void
#

Extra dependencies are specifically to extend similar to ruff and other tools enabl expending on defaults

tropic void
#

@umbral bridge any opinion on having hatch fmt default to invoking pre-commit in case its configured?

umbral bridge
#

I wouldn't mind an option but that will never be the default

half spire
#

Gm everyone. I was wondering if we are going to see hatch add any time soon ?

umbral bridge
#

probably not, I don't have time

#

unless I'm given more time at work

half spire
umbral bridge
#

use dependabot

half spire
#

afaik hatch takes the latest version but sometimes it takes the latest compatible version of the deps to avoid dep collitions right ?

half spire
umbral bridge
#

please describe exactly what you're trying to do, I don't really understand what kind of workflow you're talking about

half spire
#

ex. here these are my project deps right. so when we use other tools like pdm, rye they add a minium version number like say

pydantic>=2.14 or whatever the latest but in hatch we don't have that. we have to manually add the deps into dependencies section. sometimes it becomes very hard to keep track of the version of say 10,20 deps. So I was wondering if threre is a better way

image.png
umbral bridge
#

are you developing a library or application?

half spire
umbral bridge
#

then you want the dependency range to be as wide as possible and only limit based on what you know to not work

#

the other tools that you mentioned, or anything related to "locking", are mostly geared toward applications

half spire
umbral bridge
#

you can use automation like dependabot or write a script, what do the other tools do that you're trying to emulate? I'd like a concrete example of what command does what in some other tool

half spire
umbral bridge
#

so in that case you would limit the range, that sort of matches what I said about known issues right?

half spire
# umbral bridge you can use automation like dependabot or write a script, what do the other tool...

So the main library I'm working with is web3.py which is in the dev deps section of my module.

To extend it's functionality. But web3.py has some breaking changes from release 0.7. Many people are using 0.6 still. So people can use the module I'm building along with web3.py to build theirs. But the problem will be if they use version 0.6, pip will try to pin the version to 0.6 also for my module which will break it's functionality. If that makes sense

umbral bridge
#

can you please give me an exact example of what another tool like Poetry does that you wish Hatch did?

half spire
# 
[project]
...
dependencies = [
  "web3>=0.7",
]
#

hatch can do this but we have to define the versions manually

#

rye add web3 will add something like this

dependencies = [
    "web3>=7.6.1", # latest version as min.
]
half spire
umbral bridge
#

I see. so even if Hatch gained such a command it would not do what you're asking because that wouldn't be good for libraries, I would add exactly what one types at the command line

half spire
#

unless people specify the version like hatch add web3==0.6 or something else

#

that would be nice

umbral bridge
#

sorry let me rephrase to be more precise: Hatch will never automatically set the minimum to the latest version because that is bad for libraries

#

say you add dependency foo which requires the latest version and now somebody uses your library, if they also use foo they are now restricted to the latest version

half spire
umbral bridge
#

I'd be totally fine with a flag to the add command but it would never happen by default

#

do you understand my reasoning regarding the applications vs libraries bit I mentioned?

half spire
half spire
winged cargo
#

is there any way to get hatch to build a dependency via a (relative) path in an environment? i'd like to use hatch test for testing, but I have a specific extension module (using setuptools) that is needed for my test suite. it would be nice if I could do something like:

[envs.hatch-test]
extra-dependencies = [
  "./tests/extension_module",
]
umbral bridge
austere relic
#

does hatch support setting the version tuple here?

[tool.hatch.build.hooks.version]
path = "codeflash/version.py"
template = '''
__version__ = "{version}"
__version_tuple__ = (0, 0, 0)
'''
# 
[tool.hatch.build.hooks.vcs]
version-file = "dir/version.py"
[tool.hatch.version]
source = "vcs"
tag-pattern = "release/(?P<version>[0-9]+\\.[0-9]+\\.[0-9]+)"

[build-system]
requires = ["hatchling", "hatch-vcs"]
build-backend = "hatchling.build"

i think i've got it Hmm_Thumbs_Up

turbid bane
#

man, the amount of logic that exists simply for handling versions magically... is nuts

#

it's cool, but I've lost track of all of the different ways to handle the version

dull rapids
#

Hatch currently uses @tropic void’s setuptools-scm for that, which he plans to split into multiple packages. Once that’s happened, it think it’s possible to integrate hatch-vcs’s documentation more cleanly with the new backend package, reducing confusion.

tropic void
#

Well it all started as clone of the logic behind mercurial setup.py about 15 years ago

Then came feature requests

austere relic
#

if i have a set enviroment called test;
how can i make another enviroment; say plus, inherit all the depencencies of the test enviroment?

umbral bridge
austere relic
#

oh it's template isn't it

austere relic
austere relic
#

not sure if i should be asking here or in the uv server

[tool.hatch.build.hooks.vcs]
version-file = "codeflash/version.py"

[tool.hatch.version]
source = "vcs"
tag-pattern = "(?:^|^v|^release/)(?P<version>[0-9]+\\.[0-9]+\\.[0-9]+)$"


[build-system]
requires = ["hatchling", "hatch-vcs"]
build-backend = "hatchling.build"

i'm using hatchling as the build backend, are there any utilities for bumping versions for release?

slim cairn
#

If the source is VCS?

#

I think we'll support this via a uv command which emits the bumped version, which you'd pass to your VCS

austere relic
slim cairn
#

Tracks this

#

I presume Hatch has support

austere relic
cursive lion
# austere relic not sure if i should be asking here or in the uv server ``` [tool.hatch.build.ho...

there's a few different ways. I'm using hatchling+hatch-vcs (for example) and then use Release Drafter to bump the version tags.

I add Keep a Changelog labels to PRs, and Release Drafter then knows which SemVer tag to use next: if there's a Fixed, it's patch. if there's an Added, it's minor. if Removed, it's major.

and it creates a draft release, I only need to publish the release, that does the tag and GH Release and uses Trusted Publishers to upload. so I can release from my phone from wherever

feral sentinel
#

Does hatch(/hatchling) support building two distribution packages from the same source tree, with different files included?

I'm looking to convert a package (phonenumbers) to use pyproject.toml. From the same source tree, we need to generate the main package, and a 'lite' version that strips out ~2MB of data files. My first attempt was using a custom build script that swapped out pyproject.toml files, but I wondered if a more fully featured backend like hatch would support this.

#

The handful of resources I've found online seem tailored to a monorepo, where you would distribute different subfolders that contain full sub-packages, rather than excluding individual modules/datafiles

umbral bridge
#

I think since the spec says the name of the project cannot be dynamic that Hatchling wouldn't allow you to override that dynamically using a hook

#

what about a different directory you build that just contains a pyproject.toml?

feral sentinel
#

Sorry, yes, the idea is that we create a subdirectory to contain a pyproject.toml for the 'lite' package

#

I just haven't found a configuration that works!

umbral bridge
#

show config?

feral sentinel
# 
[build-system]
requires = ["hatchling", "hatch-fancy-pypi-readme"]
build-backend = "hatchling.build"

[project]
name = "phonenumberslite"
dynamic = [
    "readme",
    "version",
]

[tool.hatch.build]
exclude = [
    "phonenumbers/carrierdata/*",
    "phonenumbers/geodata/*",
    "phonenumbers/tzdata/*",
]

[tool.hatch.build.targets.sdist.force-include]
"../phonenumbers" = "./phonenumbers"
"../HISTORY.md" = "./HISTORY.md"
"../LICENSE" = "./LICENSE"
"../../README.md" = "./README.md"
"../tests" = "./tests"
"../testwrapper.py" = "./testwrapper.py"

[tool.hatch.build.targets.wheel]
packages = ["phonenumbers"]

[tool.hatch.version]
path = "../phonenumbers/__init__.py"

[tool.hatch.metadata.hooks.fancy-pypi-readme]
content-type = "text/markdown"
fragments = [{ path = "../../README.md" }]
#

The wheel seems to be correct, but the sdist includes the datafiles I want to exclude.

#

I haven't found a way to get the README inclusion to work (as project.readme = "../README.md" is invalid), so I've resorted to dynamic generation via fancy-readme

umbral bridge
#

it depends on how you build at that point, if you build using Hatch then the wheel builds from the source directory whereas if you use build or UV then you build from an unpacked source distribution

feral sentinel
#

Right, I'm trying to make it build-tool agnostic

#

For either building from sdist (like build) or building a wheel directly (like pip install)

#

force-include works as currently set up if I use build, but building a wheel directly is broken (the build succeeds but contains only metadata files)

umbral bridge
# feral sentinel Right, I'm trying to make it build-tool agnostic

I thought about this for a few hours and took a look at other cases of folks running into this and issues and I think the solution is for the wheel target to gain new duplicate file inclusion options suffixed by -from-sdist that only take precedence when the project directory has a PKG-INFO file

#

what do you think?

feral sentinel
umbral bridge
#

no I mean introduce options that are literally duplicates of the current options but with the suffix

feral sentinel
#

Ah!

#

so e.g. force-include-from-sdist, include-from-sdist, exclude-from-sdist?

umbral bridge
#

yes

feral sentinel
#

I think this would solve the 'building a wheel directly' case, but I don't think it solves the problem of the sdist itself including too much data

umbral bridge
#

up to you whether you think -from-sdist or the inverse -from-source is better

#

oh I may have missed a requirement, what do you mean?

feral sentinel
#

The config above has ../phonenumbers = "./phonenumbers" in the force-include section. I then also have the three modules/subdirectories I want to exclude in the exclude section.

The force-include config seems to take precedence over exclude, meaning the entire directory is included. I tried using a negative glob in force-include but (very reasonably) that config only allows literal paths

#

So building a distribution package with the above config leads to the wheel being the right size (~0.2MB) but the sdist containing the extra 2MB of content we want to exclude from the 'lite' package

#

Sorry if this was unclear, my mistake.

umbral bridge
#

I would recommend never ever using the global [tool.hatch.build] config, it was a mistake on my part because it has caused a significant amount of user confusion

feral sentinel
#

Let me try with the same config but build.targets.{sdist,wheel}

#

Ok, I've removed the global table and tried with the targets tables. Only the targets.wheel.exclude array seems to have any effect, the sdist is consistently the same size

#

I've tried both using ./ paths (for after the force-inclusion has set up the sdist virtual source tree) and ../ paths (to reflect the actual source tree filesystem)

umbral bridge
feral sentinel
#

I added the below to try every path combination at once, but doesn't seem to work:


[tool.hatch.build.targets.sdist]
artifacts = [
    "!../phonenumbers/carrierdata/*",
    "!../phonenumbers/geodata/*",
    "!../phonenumbers/tzdata/*",
    "!phonenumbers/carrierdata/*",
    "!phonenumbers/geodata/*",
    "!phonenumbers/tzdata/*",
    "!./phonenumbers/carrierdata/*",
    "!./phonenumbers/geodata/*",
    "!./phonenumbers/tzdata/*",
    "!/phonenumbers/carrierdata/*",
    "!/phonenumbers/geodata/*",
    "!/phonenumbers/tzdata/*",
]
umbral bridge
#

okay yeah then there must be a branch just for forced inclusions that it ignores everything else

#

you could do a custom build target that would inherit from the wheel builder and modify a method I think to get the desired behavior

feral sentinel
#

Do custom build targets work with third party frontends (e.g. build or uv build)?

umbral bridge
#

ah, no

#

let me think of a way...

feral sentinel
#

I'm very concious I'm contributing this to a project I'm not the primary maintainer for, so I want to keep things as simple as possible

umbral bridge
#

taking a step back, how would you fix this i.e. what would your preferred UX be for this scenario?

feral sentinel
#

Currently packaging for the project is done as ./setup.py / ./setup.py lite -- the file checks sys.argv at execution time!

feral sentinel
# umbral bridge taking a step back, how would you fix this i.e. what would your preferred UX be ...

As you said earlier, project.name can't be dynamic, so I think there must be two pyproject.toml files. Thinking out loud:

  • I think I would expect to have pip install . 'just work' in the main project directory, so
  • the 'lite' version of the package needs to have its own metadata in a sibling or child directory.
  • I'd want to avoid symlinks as much as possible (e.g. they are difficult on Windows), so replicating the project layout in the 'lite' directory with symlinks is not an option
  • The modifications from the 'full' package only relate to file selection -- all other metadata remains the same (save the name)
  • Building the package directly as a wheel must result in the same output whether from an sdist or the directly from the source tree
  • Using third-party build frontends should be possible

I am a big fan of Hatch's path-rewriting via force-include, as it allows creating a 'virtual' filesystem for the distribution package that avoids actual file operations

#

A slightly radical suggestion:

The specification notes that:

"The lack of a [project] table implicitly means the build backend will dynamically provide all keys."

Would it be possible to have something like:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

# no [project]!

[tool.hatch]
inherit-pyproject-from = "../pyproject.toml"
[tool.hatch.override]
name = "phonenumberslite"
[tool.hatch.override.build]
exclude = ["replacement-excludes"]

This would take the metadata from the "inherited" pyproject.toml, and then supplant it with some additional metadata -- including allowing changing the name, as there is no [project] table (the inherit functionality and the project table would be mututally exclusive)

#

A more gradual proposal (instead of pyproject inheritance!) would be adding some more granularity to the path-rewriting logic, and perhaps e.g. resolving symlinks and replacing them with the files that they link to (as currently done by setuptools).

umbral bridge
#

my instinct is to go with the radical approach because using a single line in a spec as a loophole sounds hilarious to me but let me sleep on that and circle back tomorrow. thank you for brainstorming here!

feral sentinel
#

It may also be worth bringing this discussion to e.g. Discourse / somewhere more public if there is more merit to it than just my case at hand -- sorry for opening the can of worms!

half spire
#

Hey I'm having issues specifying the python version in hatch

#

I'm trying to use python 3.6

# 
[tool.hatch.envs.default]
python = "3.6"
installer = "pip"
type = "virtual"
#

specified like this and getting the error RuntimeError: failed to find interpreter for Builtin discover of python_spec='None'

#

$HOME/.pyenv/shims/python3.6 is the path of python3.6

#

is python 3.6 not supported ?

tropic void
#

more context is needed, also please note that python 3.6 is end of life since years now - its possible that a number of tools simply cant support it anymore

#

the oldes not end of life version of python is 3.9 - and it wil lbe eol in about 7 months
if you need to use python 3.6 - then you wil lhave to ensure compatibility for everything you use as most software these days simply doesn't even support 3.8 in their latest versions

dull rapids
#

scientific python is on min 3.11 these days

half spire
low gyro
#

Most likely

tropic void
dull rapids
#

I didn’t know that the dependency groups PEP has been accepted, nice! @umbral bridge what do you think of https://github.com/pypa/hatch/pull/1922?

I feel like it would definitely do the trick, but maybe dependency groups and environments should be tied together more tightly? Maybe dependency groups should be an alternative way to specify dependencies for epynomous environments or so?

[dependency-groups]
hatch-test = ["pytest", "foobar"]
umbral bridge
#

the PR seems reasonable but I just don't have time

dull rapids
umbral bridge
#

indeed, the most joyous of days!

left bluff
#

oh so they turned the new uv release obsolete ?

#

cuz they advertised the very same feature iirc

low gyro
#

I don't see any mention of PEP 751 support anywhere in the recent uv releases...

balmy dock
tight lagoon
#

Nice! So I definitely won't have to hesitate about starting to implement it for my own project.

dull rapids
umbral bridge
#

I just responded with an example

dull rapids
#

Thanks!

umbral bridge
dull rapids
#

We’re making a cookiecutter template that generates projects that people should be able to modify, without too much complexity, so I think for that purpose, not really.

umbral bridge
#

can you provide a concrete example of how that relates to ignoring a particular environment based on the value of multiple matrix variables?

dull rapids
#

Ah I see you replied in the GitHub discussion! Thanks! Copying your response:

[[tool.hatch.envs.hatch-test.matrix]]
deps = ["stable", "pre"]
python = ["3.13"]

[[tool.hatch.envs.hatch-test.matrix]]
deps = ["stable"]
python = ["3.10", "3.11", "3.12"]

[tool.hatch.envs.hatch-test.overrides]
matrix.deps.env-vars = [
  { key = "UV_PRERELEASE", value = "allow", if = [ "pre" ] },
]
umbral bridge
dull rapids
#

Thank you so much for the continued work.

I think in this case, it's totally on me. I just haven't thought about the fact that it's an array at all beyond the first time reading the docs

umbral bridge
#

no worries! it's basically a requirement to have a series of matrices rather than just one, quite like the format of GitHub actions jobs

#

or if you're familiar with tox, that list of strings on top with each one being a generator of environments

winged vale
#

does hatch/hatchling support dependency groups?

umbral bridge
#

not yet, there is an open PR

#

Hatchling wouldn't support that because it's not package metadata, it's part of Hatch environment config

winged vale
#

yeah I figured

#

waiting anxiously for the next pip release which should support them

cursive lion
feral sentinel
winged vale
tropic void
#

@umbral bridge any oppinion on adding a sdist mode to hatchling that would createa dist with the metadata included and no need for any dependencies for install - im thinking of creating a meta build backend that allows sdists with vastly smaller footprint (just including dist-info and the metadata to make the whell for pure python packages

umbral bridge
tropic void
umbral bridge
#

so you want a source distribution but the only difference is to strip out the required dependencies from metadata?

tropic void
umbral bridge
#

can you please describe how this source distribution should differ exactly?

tropic void
#

it would be a dist info directory + some json to map the files, full backend being opt-in but low level backend being usable

#

(the build process beign reduced to makign a zip fil and then adding the files an the dist info)

tropic void
#

im goign to bring back my gumby elf package as the meta build backend
i think i'll add a hatch plugin to provide the required metadata - the goal being that the wheel build from the meta backend being reproducible just like the ones from hatch, but the meta backend only composing the wheel

umbral bridge
#

I'm trying to understand but I can't quite grok the requirement here, I'm willing to make it so Hatchling supports your use case just I don't get what you want exactly

tropic void
#

@umbral bridge i want to create a meta backend for bootstrapping that wont have any additional dependencies for building from sdist - the plan is to have a plugin for the real backend set up the metadata so this build bakcend can be a really dumb thing thats just illing in a wheel

and it shal have jokes about a gumby brain surgery elf packaging things up

half spire
#

Hi, is this the correct syntax to pull a module from github and specify the branch ?

dependencies = [
  "web3-ethereum-defi @ git+https://github.com/Aviksaikat/web3-ethereum-defi@feat/gmx"
]

the branch is feat/gmx

umbral bridge
#

yes that syntax is correct

half spire
#

might be an uv issue

umbral bridge
#

I would need more info to diagnose

half spire
#

guess we can fix linux PCs by rebooting lol

errant timber
umbral bridge
errant timber
#

Thanks!

errant timber
#

Can I make 3.14 conditional in the matrix so I can exclude it on Windows? Hatch doesn't support 3.14.0b1 on Windows (on GitHub Actions, anyway)

umbral bridge
errant timber
#

Yes. I just need to skip 3.14 on Windows in CI, so I could override it on the CLI if I can't do it there.

umbral bridge
#

can you show the CI config?

errant timber
#

It's just the non-free-threading build that is broken

errant timber
#

I've got a package, plumbum, and I'd like to pull out two packages as namespace packages (plumbum.colorlib and plumbum.cli), with the original package remaining a normal package that depends on those two namespace packages. Does someone need to talk me out of it? 🙂

umbral bridge
errant timber
#

Ability to use color/cli without depenencies (pywin32, not available for 3.14 yet, as it's binary only) (and plumbum installs a background thread atexit function that isn't needed for color/cli)

umbral bridge
#

if you want to do separate packages then do that but without namespacing, I personally dislike the increased import time

#

I don't use that package though so I'm not sure if it's a factor for you

errant timber
#

The nice thing about this (possibly bad) idea is that it keeps the original package as-is since it becomes a normal package as soon as you install the parent package. But it lets you install parts if you just want parts.

#

How bad is the import time penalty?

umbral bridge
#

single-digit milliseconds

#

if I remember correctly

#

would a user actually import the CLI? if not then I don't understand the benefit of caring about the import path