hm, it would be nice to have automatic pyenv integration
#hatch
karmic glacier
umbral bridge
how so?
tropic void
I presume for finding /adding pythons
umbral bridge
making a PR to switch over Django
it seems Hatchling is currently the only non-setuptools build backend that supports building it ๐
umbral bridge
I explain why in https://github.com/django/django/pull/15874
tropic void
@umbral bridge is great to see hatch go places, i also started to get feely on the pytest side, but probably wont be able to deep touch that the next quarter
umbral bridge
I can open a PR to pytest if you want
proper tide
I can open a PR to pytest if you want
have you seen the twisted setup.cfg?
umbral bridge
haha not yet ... ๐
umbral bridge
oh actually pip supports self-referential extras nowadays
proper tide
but it makes different wheels
umbral bridge
oh?
umbral bridge
like
[project]
name = "pkg"
[project.optional-dependencies]
all = ["pkg[a]", "pkg[b]"]
a = ["..."]
b = ["..."]
Yeah what comes out in the wheel when you do that vs writing it out?
umbral bridge
Provides-Extra: a
Requires-Dist: foo; extra == 'a'
Provides-Extra: all
Requires-Dist: pkg[a]; extra == 'all'
Requires-Dist: pkg[b]; extra == 'all'
Provides-Extra: b
Requires-Dist: bar; extra == 'b'
Yeah so we want the
Requires-Dist: foo; extra == 'all'
Requires-Dist: bar; extra == 'all'
@umbral bridge can you solve sudokus with just self referential deps?
karmic glacier
how so?
what ronny said, it would be nice to be able to find pythons that way.
Right now, hatch completely ignores pyenv and just uses the system python, which is not the python or python3 on my path.
Partially related, hatch made a venv for me that was not supported by the project being installed, and failed when running the install step rather than before even creating the venv.
hatch created a python 3.9 venv when the project required 3.10 and then failed when trying to install the project
umbral bridge
you can choose if they're on your path https://hatch.pypa.io/latest/config/environment/overview/#python-version
karmic glacier
but that would need to go in my pyproject.toml file, wouldn't it? I don't see a command line option
umbral bridge
python 3.9 venv when the project required 3.10 and then failed when trying to install the project
I could add a check for that
but that would need to go in my pyproject.toml file, wouldn't it? I don't see a command line option
yes or env var
karmic glacier
The version of Python to find on your system and subsequently use to create the environment, defaulting to the HATCH_PYTHON environment variable, followed by the Python executable Hatch is running on. For more information, see the documentation.
Ah, this is where using the current path would be nice, or even pyenv integration, which would allow to not have all of the versions activated at once
I installed hatch with pipx using a python 3.9 interpreter, but I use the lowest supported python version when developing for disnake which is 3.8
I'd like to make hatch use 3.8 when that's the active python version, or whatever the active version is
pyenv integration would be nice because i wouldn't have to have all of the desired versions on my path at the same time -- nox does this so it would be a step back for us to use hatch for that as well
umbral bridge
originally, Hatch defaulted to creating envs with python on PATH but an issue was opened which made me change it to sys.executable
karmic glacier
๐
umbral bridge
can't find it but essentially came down to on Linux/macOS python points to py2
karmic glacier
ah
umbral bridge
I'll investigate now, possible I made the wrong choice there ๐คทโโ๏ธ
umbral bridge
wait if python existed but was just py2 why did I change it? Hatchling supported py2... ๐ค
oh lol I see
pip gained pep 660 support after dropping py2
so default case would fail env install on unix
which in my mind would look bad for adoption by new users of Hatch
umbral bridge
@mint bramble well done maintaining tox config all this time. I'm adding a way to alter options based on regex matching env names and I'm having difficulty understanding how Hatch's current code is implemented ๐
mint bramble
thanks ๐
tropic void
@umbral bridge as the programmign in toml gets more and more intense, i wonder if it wouldnt make sense to have a custom hook for overrides isntead of mangleing toml more and more (its terrifying)
umbral bridge
example?
tropic void
@umbral bridge regex, conditions, overrides, its starting to seem more and more like a programming language, - i wonder at which point its just more effective to have a better way to hook it into a local plugin
i dont have a example hany, but i'll try to make one up
umbral bridge
oh for envs? yeah I could
tropic void
[[tool.hatch.envs.test.overrides]]
when = { condition.hook = "custom", name.matches = "^0" }
env-vars = "TESTING_UNSTABLE=true"
perhasp as starting points where a hook decides if to apply them, with a default hook handling common conditionals/matchers
umbral bridge
@turbid bane I was re-reading your Mypyc posts. are Click-based CLIs unsupported?
I was planning on compiling Hatch
low gyro
<@429445826251587616> I was re-reading your Mypyc posts. are Click-based CLIs un...
I think black is compiled and it is using click
turbid bane
<@429445826251587616> I was re-reading your Mypyc posts. are Click-based CLIs un...
They work just fine with mypyc, but there are a few things to note:
- docstrings are currently stripped so if you're using docstrings as the command's help, you're going to have to duplicate it / move it into the
.command(help="...")decorator call - mypy[c] won't catch any issues with invalid type annotations since click's decorators are too magical
turbid bane
I think black is compiled and it is using click
the blog posts are about compiling Black with mypyc :)
umbral bridge
turbid bane
nope. that seems unlikely given mypyc shouldn't be compiling decorated functions using decorators it doesn't know
lemme check
hmm, actually mypyc might still compile functions into native ones even if they're decorated.
Python 3.8.5 (default, Sep 9 2020, 23:45:44)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import black
>>> black.COMPILED
True
>>> black.__version__
'22.6.0'
>>> black.main
<Command main>
>>> black.main.lolwhatever = 1
>>> black.main.lolwhatever
1
>>> black.main.__dict__
{'name': 'main', 'context_settings': {'help_option_names': ['-h', '--help']}, 'callback': <function main at 0x7f6cd11025e0>, 'params': [<Option code>, <Option line_length>, <Option target_version>, <Option pyi>, <Option ipynb>, <Option python_cell_magics>, <Option skip_string_normalization>, <Option skip_magic_trailing_comma>, <Option experimental_string_processing>, <Option preview>, <Option check>, <Option diff>, <Option color>, <Option fast>, <Option required_version>, <Option include>, <Option exclude>, <Option extend_exclude>, <Option force_exclude>, <Option stdin_filename>, <Option workers>, <Option quiet>, <Option verbose>, <Option version>, <Argument src>, <Option config>], 'help': 'The uncompromising code formatter.', 'epilog': None, 'options_metavar': '[OPTIONS]', 'short_help': None, 'add_help_option': True, 'no_args_is_help': False, 'hidden': False, 'deprecated': False, '__doc__': None, 'lolwhatever': 1}
not sure what mypyc is doing to be honest, but it seems like the function __dict__ is still available and usable
>>> black.strings.has_triple_quotes
<built-in function has_triple_quotes>
>>> black.strings.has_triple_quotes.lolwhatever = "a"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'builtin_function_or_method' object has no attribute 'lolwhatever'
OK, so yeah, decorated functions still support __dict__, but regular ones don't after compilation
umbral bridge
^ linked example hard fails for me though with the exact error from that issue if ordered differently
turbid bane
with mypyc? that's interesting
umbral bridge
yup
turbid bane
I can reproduce, how fun
comparing the generated C, I can't see a reason why it'd break
umbral bridge
I haven't compiled Hatch b/c ^ scares me lol
turbid bane
gosh this is like a nightmare to debug
why do you want to compile hatch with mypyc?
import time reduction?
umbral bridge
yes
turbid bane
gosh this is like a nightmare to debug
the way click's decorators work is confusing af
turbid bane
I haven't compiled Hatch b/c ^ scares me lol
Yeaaaah, mypyc is still a young project. If you're willing to deal with weird issues (like me for black) then go for it, but if you don't want to, then don't. There are still quite a few rough edges that need to be worked on.
And even for Black, it's pretty likely I'm going to be the only one maintaining black's mypyc integration forever since I'm the only one who has enough experience to work with it ๐
umbral bridge
if you figure it out and open an issue I'm willing to write fix, whether in Mypyc or Click
turbid bane
sounds good, I'm failing to understand how click's decorators even work though, seems to be like callback hell or something ๐
I should probably look into supporting docstrings in mypyc if it's possible, would make things nicer for libraries (and click CLIs)
import click
@click.version_option(version="1.0.0")
def main() -> None:
print("daylily")
FTR this is also crashing on import with the same issue. I feel like I'm missing something on how decorators work in Python since according to your hatch example, this should work fine
OH WAIT, I forgot that decorators are applied last to first
that's not embarrassing or anything ๐
tropic void
thats such a common pain point ๐
tropic void
@umbral bridge can we make hatch-vcs deprecate the raw settings and make direct use of the setuptools_scm section (but ensuring write_to gets a warning/error) - right now hatch using sprojects break python -m setuptools_scm and its likely a longer while before i can get a new package name in a sensible manner
umbral bridge
breaks how?
tropic void
@umbral bridge the setuptools_scm cli cant be aware of the different option keys used by hatch-vcs
so python -m setuptools_scm will result in wrong data
karmic glacier
well, here goes: I'm about to start a new project of which I'll attempt to use hatch and see if there's any shortcomings I can find that would block a larger switch-over
karmic glacier
Some thoughts already
hatch newdoesn't ask for more information, like python versions or other identifiers. This is a bit annoying as the default for me was python 3.7, but I support 3.8 and up so I had to manually update each place.- The documentation at https://hatch.pypa.io/latest/config/project-templates/#src-layout isn't very helpful IMHO, referring to reading a blog post rather than putting the important information in the documentation
- hatch-vcs doesn't have all of the options I would like. I don't want the date here, (but i'm open to convincing), but I do want the commit hash and the other numbers that are provided.
package-0.1.dev1+g7a989c0.d20220731. I would also like to base if off of an internal version if it existed (like we currently do, and then append the necessary metadata)
karmic glacier
hmmm
AttributeError: module 'hatchling.build' has no attribute 'prepare_metadata_for_build_editable'
happens when i try to make an editable install
umbral bridge
^ is not the real error
dapper willow
we should probably get pip to remove that context from the error
karmic glacier
^ is not the real error
I can give you access to the repo with the code/dm you the pyproject.toml if you would like to see it
I think its something with using hatch-vcs, but that would be a bit surprising
umbral bridge
yes please
karmic glacier
thank you btw 
umbral bridge
I've spent so much time this weekend trying to improve publishing
then trying to use self-signed certs with devpi
epic fail. now trying trustme + nginx, seems ok
umbral bridge
tried tomlkit, still don't like default style so not switching yet. no time to manually style using its api
karmic glacier
ah yeah, I could perhaps take a look at that if I have time
Its not terribly important, as it only occurs on the very first run (and never again)
karmic glacier
I'm probably not doing something right, but I have to reinstall the program every time I modify my cli one bit; it seems like the cli interface is getting the old version installed somewhere. There is also not an hatch install command or related (that I could find), so I've been using pip install -e .
as soon as I sent that I found a workaround
it'll be fixed when the editable bug is fixed ๐
this line was the problem: https://canary.discord.com/channels/@me/1003379454183223436/1003383301811216434, i've temp renamed the project to remove that line
karmic glacier
... is there anything against a Hatch install command which reinstalls the lib in the activated env, or is pip install -e . the recommended way to do that?
karmic glacier
why re-install?
mostly changing the package configuration or needing to update the installed package metadata
karmic glacier
hm, that arises with a different question I have: is there a way to configure Hatch to use the activated venv, or to use a venv in-project, without forcing all devs to use that?
I think there is but I can't find the place in the docs
umbral bridge
it will by default
or env var (good for docker or shared venv) https://hatch.pypa.io/latest/plugins/environment/virtual/#options
umbral bridge
this weekend I'm adding the ability to build projects using any backend
might save myself work and use #build
oh it depends on colorama, nvm ๐ข
low gyro
Is colorama bad?
karmic glacier
I am curious too
can't find any anti-colorama results from a tiny search on duckduckgo
umbral bridge
anything that globally modifies stuff is bad
example from literally today https://github.com/PyCQA/isort/pull/1961
umbral bridge
yup
low gyro
oh right colorama replaces sys.stdout and sys.stderr iirc
Oh for everything sacred, who came up with thatโฆ. This is wrong on so many levelsโฆ
karmic glacier
:)
On Windows, calling init() will filter ANSI escape sequences out of any text sent to stdout or stderr, and replace them with equivalent Win32 calls.
On other platforms, calling init() has no effect (unless you request other optional functionality, see "Init Keyword Args" below; or if output is redirected). By design, this permits applications to call init() unconditionally on all platforms, after which ANSI output should just work.
hm i should update colorama
half cedar
who came up with that
People who are used to POSIX systems and think they can โfixโ Windows; there are much many more people like this than youโd imagine, albeit most things they build are not as widely used as colorama. (Itโs getting a lot better these days as well, like 10 years ago youโd be laughed out of a project for Windows support)
shut mantle
On modern windows (anything that hasn't reached standard EOL aside from Windows 8.1), you can just enable VT support for currently running app with winapi which is what rich does. I think it still supports the older Windows versions that don't allow that using the same methods as colorama but at least it keeps it to itself instead of overriding sys.stdout/err. colorama does have a way to not override sys.stdout/err and just returning wrapped stream instead but most people probably don't use it
This is especially annoying when libs do it.
looks at click
Actually maybe it wasn't click... I recall one dep that didn't depend on colorama but when it was there, it initialized it so click's hard dependency on colorama would be enough to cause it lol
Yeah, I think click actually only wraps so probably it was that other dep
Either way, very annoying
dull rapids
*looks at click*
Click sure does wrap standard streams (https://github.com/pallets/click/pull/1784), and this is not the only global mucking that Click does, see this page (which formerly talked about โPython 3 limitationsโ): https://click.palletsprojects.com/en/8.1.x/unicode-support/
Apart from that, I never got Click anyway. It seemed to get less useful and intuitive the more I used it. I then went back to argparse and suddenly things were simple again.
Thereโs more people who donโt like it: http://xion.io/post/programming/python-dont-use-click.html
tropic void
anything that globally modifies stuff is bad
That makes me remember the terrifying issues we will eventually have to resort to for pytest support of rich
umbral bridge
pretty cool https://github.com/hynek/hatch-fancy-pypi-readme
umbral bridge
errant timber
Does hatch add the file listed in build.hooks.vcs.version-file to the include list even if it's .gitignore'd (which it should be)? I was assuming it would, but it seems like maybe it doesn't?
umbral bridge
errant timber
I'm seeing it in my wheel and SDist too (along with node_modules, which is listed in my .gitignore, odd Ahh, forgot the "dirty" part changes each time, was looking at the same file. :)) - so not sure what the user was doing ( they are working on making a freeBSD port of a package)
karmic glacier
hmm, I'm trying to convert and simplify the list of dependencies we have for development but I don't see a good way to have multiple environments with the same dependency without declaring that dependency twice
one way if it doesn't already exist would be to declare development groups and then be able to include those groups in environment definitions
@umbral bridge does this seem doable/make sense? To me, it feels like the main blocker for our move to hatch
this would replace both setuptools and nox, plus having to maintain multiple environments manually and more
karmic glacier
hmmm, is there any way to provide arguments to a script at runtime? eg hatch run lint:lint -- flake8
specifically i have a docs script that i want devs to be able to provide arguments to and modify on their own machines
umbral bridge
one way if it doesn't already exist would be to declare development groups and then be able to include those groups in environment definitions
@umbral bridge does this seem doable/make sense?
yes
umbral bridge
is there any way to provide arguments to a script at runtime?
argshttps://hatch.pypa.io/latest/config/environment/advanced/#context-formatting
karmic glacier
> one way if it doesn't already exist would be to declare development groups and...
so is it possible to implement this into hatch? I'd rather not use extras/features for the dependency groups
karmic glacier
why not use templated environments + inheritance?
can't inherit multiple environments
this is my current configuration https://github.com/DisnakeDev/disnake/compare/master...onerandomusername:disnake:feat/use-hatch
karmic glacier
i don't think so
umbral bridge
so, we need a way to reference config from other envs like tox can?
karmic glacier
possibly
what i was thinking of was a way that additional extras could be defined but they wouldn't be included in the built package
like dependency groups or something like that
umbral bridge
oh I can, sure
mint bramble
so, we need a way to reference config from other envs like tox can?
just use tox and sorted? ๐
umbral bridge
lol
karmic glacier
just use tox and sorted? ๐
rn we use nox for our ci but I really don't want to have dependencies defined in multiple places, which is what we'd have to do if we use hatch
umbral bridge
btw for envs I recommend hatch.toml https://hatch.pypa.io/latest/intro/#configuration
karmic glacier
can both config in pyproject.toml and hatch.toml be used at once?
umbral bridge
yup
karmic glacier
๐
umbral bridge
Top level keys in the latter file take precedence when defined in both.
I'll try to get to dependency groups over the weekend
karmic glacier
nice!
umbral bridge
I have a feature that I need to implement first that is blocking my Master Plan โข๏ธ
karmic glacier
oo i can't wait
umbral bridge
@errant timber fyi whenever I open Hatchling PRs like https://github.com/urllib3/urllib3/pull/2708 if affected I'm always adding now:
We should probably remove the Python upper bound constraint as upper bounds for libraries are heavily discouraged by the community, see https://iscinumpy.dev/post/bound-version-constraints/
๐
umbral bridge
and another https://github.com/deepset-ai/haystack/pull/3079#pullrequestreview-1079714294
everyone is open to it so far, we just need to keep sharing that blog post around
cursive lion
curious about this upper bound in the first but not second PR:
[build-system]
requires = ["hatchling>=1.6.0,<2"]
build-backend = "hatchling.build"
the first replaced flit with an upper bound, the second replaced setuptools with no upper bound.
but generally, does the upper bound advice apply to this config too?
tropic void
@umbral bridge I'd like to discuss an upcoming package name for the setuptools_scm replacement, as I'd like hatch vs to use its tool settings ASAP
umbral bridge
<@334562134274998273> I'd like to discuss an upcoming package name for the setup...
sure ๐
umbral bridge
the first replaced flit with an upper bound, the second replaced setuptools with...
I keep upper if present, otherwise no need imo
umbral bridge
vcs-version(ing)
tropic void
Maybe, wondering about resurrection of anyvcs, while I'm at it, but posix fork is not avaliable for biological systems
umbral bridge
also, please add a function to set/tag the version ๐
tropic void
also, please add a function to set/tag the version ๐
Yeah, absolutely
mint bramble
@umbral bridge any reason why hatchling doesn't implement https://peps.python.org/pep-0517/#prepare-metadata-for-build-wheel ?
mint bramble
is not entirely clear to me what's the issue is here
would the prepapred metadata be 100% be identical to a potential follow-up I don't think so
but is that a problem? :-d
I don't think so, the goal of the prepare metadata is to "give me project description" without building a wheel ๐
umbral bridge
it goes against the peps
mint bramble
which part of the pep does it violate?
umbral bridge
identical metadata
mint bramble
which fields in the metadata would it be different?
shut mantle
That reminds me of https://discuss.python.org/t/nobody-is-following-the-metadata-directory-promise-in-pep-517
umbral bridge
WHEEL file mainly
mint bramble
not the entire file, just partially no? which parts in there?
umbral bridge
the tags
umbral bridge
That reminds me of https://discuss.python.org/t/nobody-is-following-the-metadata...
"Setuptools actively generates non-identical metadata" ha yup it does dynamic things too
shut mantle
It seems like all backends that support the hook (setuptools, poetry, flit), violate the PEP
umbral bridge
oh seems like I commented https://discuss.python.org/t/6964/17
umbral bridge
pyvers,plat,abi
mint bramble
perhaps we need a get_dependencies endpoint instead ๐ just because some info does not always matches up doesn't mean the entire feature is uselss
umbral bridge
I'd love a metadata api ๐
like returns project dict with dynamic resolved to static
tropic void
@umbral bridge couldn't build hooks also participate in generating the Metadata for build?
umbral bridge
oh true they can rewrite the final artifact
oh wow! I need to make a build hook to finalize wheels using auditwheel, delocate, and whatever the lib for Windows is
tropic void
Delocate?
Oh, at first glance delocate seems potentially terrifying (when applied to dynamic libs with state)
tropic void
@umbral bridge while taking a look at the docs setup, i took note, that hatch might be able make use of the new gh page deploy mechanism (from action instead of from branch)
umbral bridge
link?
tropic void
https://github.com/actions/upload-pages-artifact - i just took note of the public beta thing, gimme a few to find the blog post where i learned about it
tropic void
ah
tropic void
@umbral bridge would it make sense to rename from custom to project-local
umbral bridge
no, too verbose imo
iron quartz
maybe just "local"?
umbral bridge
the name should (imo) clearly indicate what it does implementation-wise
tropic void
@umbral bridge custom is very confusing tho IMHO
umbral bridge
it doesn't do "local" it does custom logic
also think of a user wanting their own small logic and they do a web search
"hatch custom _ plugin"
fyi I actually named these local first but renamed after thinking about it
iron quartz
"hatch custom _ plugin"
FWIW I wouldn't think of searching for that, any hatch plugin I would need to write would be custom kinda by definition e.g. if cython provided a hatch plugin directly I'd consider that custom.
An in-tree (/ in-project) plugin would be either that or local.
But maybe mention / allow both?
umbral bridge
FWIW I wouldn't think of searching for that, any hatch plugin I would need to wr...
you, Ronny, & I are quite experienced which is why we think of "project-local"/"in-tree" first in our minds. what I'm positing is that we're in the minority
tropic void
you, Ronny, & I are quite experienced which is why we think of "project-local"/"...
what im posing is - use long names that make sense to all and ave al lthe words different experience levels wouldsearch for instead of short names that leave people scratching the head
errant timber
For GHA Pages, see https://scikit-hep.org/developer/gha_basic#github-pages ๐
I've moved level-up-your-python (jupyer-book & jupyter-lite WebAssembly), codas-hep.org (pelican), https://github.com/klieret/everything-you-didnt-now-you-needed (slidev), and a couple more over if you have questions or need examples. ๐ (Also helped them fix a bug in the initial version, it picked the alphabetical first artifact instead of the one named "gh-pages". ๐คฆ)
tropic void
I've moved level-up-your-python (jupyer-book & jupyter-lite WebAssembly), codas-...
Is any of those a example where docs for multiple versions are deployed at once?
errant timber
Nope, just random examples of various systems. Though the deploy part is pretty simple. Are the multiple versions all built each commit, or are they stashed into an unchanging place? The latter is likely harder to implement.
umbral bridge
errant timber
You might be better off with a branch for a stashed output. (though it will grow with each release, etc) I guess you could attach the artifact to each GH release, and then pull those in the build otherwise.
tropic void
thats kinda how its being done atm
iron quartz
you, Ronny, & I are quite experienced which is why we think of "project-local"/"...
I mean I'm neither ronny nor you, and while I guess I'm experienced in general I am not so in packaging (let alone Python packaging). And I'm the person who literally had no idea that "custom" stood for package-local plugins over in #build.
subtle bloom
is it correct to assume hatch does not aim to be the cargo for Python? in the sense that it's intended to be useful in more ways than dependency and environment management? meaning it wouldn't aim to be the defacto method for doing things like code formatting, linting, etc.
umbral bridge
correct ๐
half cedar
it wouldn't aim to be the defacto method for doing things like code formatting, linting, etc.
I want to point out that Cargo does not handle code formatting and linting either, it just provides a hook to run an external tool (i.e. Clippy) against the correct target. Hatch does provide an equivalent (not exactly one-to-one by functionally the same) inhatch run.
subtle bloom
yea understood. that's kind of why i was asking. because in theory it's doable, but if it's not part of the philosophy behind hatch then that imo is the answer im looking for.
half cedar
Yeah I think the difference is mostly that Python people already have other preferred ways to do this (Nox, Tox, or pre-commit) so itโs easier to follow the trend instead of back-fit habits from another community
hybrid thorn
Hi, Is it possible to use a custom license for a project that uses hatchling? I'm stuck with "invalid SPDX expression" or "unknown license".
umbral bridge
does pep 639 allow that?
if not then don't define license and rely on just shipping the raw files
shut mantle
can't you just define license = { file = "LICENSE" } in the project metadata
shut mantle
shouldn't license from 621 get deprecated or something then
oh, you're talking about hatch specifically?
well, still, I guess it would make sense to deprecate if this is part of pep 639
umbral bridge
license as non-string will be deprecated
hybrid thorn
I'm using the PEP 621 license field with hatch. Just noticing PEP 639 wants to deprecate it.
shut mantle
I wrongly assumed that it's already accepted if you're talking about it but it's just that hatch implements these things before they get accepted
umbral bridge
there is no way 639 won't be accepted
umbral bridge
oh nice find ty
hybrid thorn
Thanks for the link. So IIUC hatch should add suport for LicenseRef-Public-Domain and LicenseRef-Proprietary ?
shut mantle
there is no way 639 won't be accepted
yeah but it's not part of the spec yet so it won't really work with other tooling while it's in discussion phase
umbral bridge
Thanks for the link. So IIUC hatch should add suport for `LicenseRef-Public-Doma...
ah yes ๐คฆโโ๏ธ I thought they were in https://github.com/pypa/hatch/blob/hatchling-v1.8.1/backend/src/hatchling/licenses/supported.py
I'll add this weekend, ty
umbral bridge
@hybrid thorn https://github.com/pypa/hatch/pull/451
tropic void
@umbral bridge is there any equivalent to package_dirs in hatch or is that intentionally left out (at least i couldn't find it)
tropic void
@umbral bridge also unrelated - is there a example on how to use project templates outside of hatch - either by puling them in somewhere or by referring them as package?
tropic void
<@334562134274998273> is there any equivalent to package_dirs in hatch or is tha...
nm, i just found path rewriting
umbral bridge
project templates will be a plugin but not high priority
mint bramble
how do you include modules instead of packages? ๐
umbral bridge
like a top-level single file?
umbral bridge
any option works. include, packages, etc.
mint bramble
packages is what I tried and did not ๐
umbral bridge
paste?
mint bramble
ah nm! ๐
umbral bridge
what was the issue?
mint bramble
[tool.hatch]
build.targets.wheel.packages = ["app"]
didn't work ๐ค but removed the entry and then the vcs variant did
umbral bridge
app is a directory pattern
umbral bridge
yup
mint bramble
emit a warning if a pattern does not match? ๐
umbral bridge
all non-matching patterns? ๐ค
mint bramble
for newbies could be helpful to check if there's a file with that name ๐ค and suggest ot change it to that in the warning
mint bramble
all non-matching patterns? ๐ค
I'd say so, if a include pattern does not match should be either removed ๐ or something is wrong, no?
umbral bridge
that will likely never happen for perf reasons. atm all patterns are joined rather than looping over each
mint bramble
how can I extend the include list? ๐ค or if I specify include can I say keep vcs?
umbral bridge
paste?
mint bramble
[tool.hatch]
this includes my app.py, and config.py however misses my static (JS) folder, when I add
[tool.hatch]
build.include = ["static"]
Now only the static folder is shipped ๐
both app.py and static is VCS commited
umbral bridge
first block missing?
mint bramble
first block is just use defaults, e.g. select from vcs
mint bramble
yeah that works
just wanted if would be possible to use the default selection logic and add on top of it rather than have to enumerate all ๐ explicitly
because I have like 12 files at root ๐
so that list gets rather large
but doing the include as above works for now
umbral bridge
extending seems tricky to me
@mint bramble oh btw https://github.com/dependabot/dependabot-core/issues/3290#issuecomment-1238417795
you and others get your wish soon lol
mint bramble
soon
umbral bridge
this weekend likely I'll do it
mint bramble
umbral bridge
mint bramble
This will make my tox 4 project builds significantly faster ๐
though I want to add a pure pyproject.toml metadata support in tox 4 as that would make it even faster, but in the meantime this helps a lot
cursive lion
I'm switching from setuptools + setuptools_scm to hatchling + hatch-vcs, so far so good! I noticed this difference:
I don't have .idea in .gitignore and it's also not committed it
the old method doesn't include .idea in the sdist
the new one does include it
it's easy enough to add it to .gitignore, but is this intentional behaviour?
errant timber
I've really liked this, as some of my .gitignores are now better. ๐
cursive lion
thanks, will update my .gitignore
umbral bridge
umbral bridge
I love how plugins are unlocking the switch for many projects now https://github.com/squidfunk/mkdocs-material/blob/master/pyproject.toml
tropic void
@umbral bridge would you consider exposing versions as Version instance if https://github.com/pypa/packaging/issues/590 lands and measures good enough
umbral bridge
like a new attribute on the internal metadata class?
tropic void
like a new attribute on the internal metadata class?
Nah, in version files
umbral bridge
oh like an option to skip validation/parsing?
tropic void
oh like an option to skip validation/parsing?
in a sense, not skipping, but rather directly constructing the post-parsing value and having neglect-able import cost, i remember you taking issue with that for version files
umbral bridge
can you explain with an example?
tropic void
bascially in _version.py
verison = "1.0.0"
version_tuple = (1, 0, 0)
version_obj = __import__("packaging.version").Version(release=(1,0,0), ....)
ok, & what should Hatchling do with version_obj?
tropic void
ok, & what should Hatchling do with `version_obj`?
It's for exposure in published packages,
umbral bridge
gloomy flicker
Pydantic has also switched to hatch I noticed earlier
umbral bridge
oh yup, reminds me I have to make a PR there
umbral bridge
golden merlin
Any advice on how I diagnose why a file isn't being included in artifacts
Specifically I don't understand why https://github.com/python-jsonschema/bowtie/blob/main/bowtie/schemas/io-schema.json isn't getting included
It's not ignored in my VCS (clearly :P)
Huh, it's in the wheel...
OK it's missing only from the sdist (and shiv).
cursive lion
I ran into the opposite, a VCS-tracked data file included in the sdist but not the wheel, and needed force-include:
[tool.hatch.build.force-include]
"em/emojis.json" = "em/emojis.json"
https://github.com/hugovk/em-keyboard/blob/556828a3008c5e0e42e830e39f0131a73af76f01/pyproject.toml#L67-L68
docs: https://hatch.pypa.io/latest/config/build/#forced-inclusion
umbral bridge
@golden merlin is it a symlink?
mint bramble
@umbral bridge where does pathspec come from?
umbral bridge
hatchling
mint bramble
but can't seem a file
for it
even though claims to self package https://github.com/pypa/hatch/blob/master/backend/pyproject.toml#L4
(I'm trying to DPKG hatchling for our corp env, and fails when trying to import pathspec)
umbral bridge
mint bramble
ah, has no requires but has wheel_requires
umbral bridge
when < 3.11 is EOL then I can put config in pyproject.toml
mint bramble
5 years down the line ๐
golden merlin
<@315721698135638016> is it a symlink?
no
but it has a symlink to it if that affects things
golden merlin
I ran into the opposite, a VCS-tracked data file included in the sdist but not t...
will try this
relatedly, it's slightly surprising that include replaces the default include list I think
yeah adding it to force-include indeed makes it included
mint bramble
@umbral bridge do you have any plan for keeping the hatchling build env graph acyclic? aka to only take on dependencies that don't use hatchling themselves... DPKG doesn't like cyclic graphs ๐
I see currently tomli uses flit, while the rest of the deps uses setuptools, but wonder what will happen if one day one of them moves to hatchling ๐
umbral bridge
yes no dep must ever use us
umbral bridge
actually that's mandated by pep 517
umbral bridge
I'd tell them "plz stahp" lol
mint bramble
do they know? ๐
umbral bridge
an issue would open, like https://github.com/hynek/hatch-fancy-pypi-readme/issues/14
https://github.com/hynek/hatch-fancy-pypi-readme/blob/main/CHANGELOG.md#2270---2022-09-12
The price of building packaging tools is to not use packages.
ouch, true
mint bramble
cool
low gyro
What about vendoring the packages?
tropic void
@umbral bridge any idea what to do about readthedocs and hatch usage - its a semi-blocker for pytest-dev related projects (they have a hardcodded setup all)
golden merlin
(I too have no issues with RTD + Hatch)
tropic void
https://github.com/pradyunsg/furo/blob/main/.readthedocs.yaml for example
๐คฆโโ๏ธ i wish i RTFMd, will do exactly that now thanks!
golden merlin
If I want to write a Hatch plugin to compile C code, I want to look at https://github.com/ofek/extensionlib right?
(I want to compile not just one specific C extension, it's a plugin to expose to packages which want to do so)
umbral bridge
umbral bridge
oh, I'm going to NYC tomorrow for a work trip for a week. between using the work laptop rather than my desktop, transitioning to this other wheelchair, and life being generally Very Different โข๏ธ, I won't be doing any Hatch work nor will I be in Discord (still on email & Twitter though)
just fyi ๐
waxen tangle
hey, so I'm trying out hatch
and i'm coming from a project with a setup.cfg that has
[options.data_files]
share/runpanel/configs =
data/config_stubs/*.yaml
src/runpanel/system_config.yaml
share/applications =
data/runpanel.desktop
share/icons =
data/runpanel.svg
how can I make that hatchable?
i tried hatch new --init and that didn't seem to catch any of this
tropic void
and i'm coming from a project with a setup.cfg that has
```
[options.data_files]...
I believe data files are unfortunately uncommon, in addition i believe that the particular feature you are using is only supported by setuptools install, not by wheels
Unless im mistaken a new pep is needed for that kinda installation
waxen tangle
The above goes into my wheels without issue. I find that being able to put non-python files into non-python places is extremely useful!
I'm really having trouble following all the changes in the world of python packaging
What's the defacto standard now? is it hatch?
I was thinking it might be the standard after reading this https://packaging.python.org/en/latest/tutorials/packaging-projects/
is the data_files feature being dropped?
cursive lion
I ran into the opposite, a VCS-tracked data file included in the sdist but not t...
@waxen tangle does [tool.hatch.build.force-include] help? ^
waxen tangle
From https://hatch.pypa.io/latest/config/build/#forced-inclusion "The force-include option allows you to select specific files or directories from anywhere on the file system that should be included and map them to the desired relative distribution path."
Looks like that will really help @cursive lion!
They should call it data_files ๐
waxen tangle
Is there a hatch config option that I can use to build/package resources that I can later load at runtime with importlib.resources.read_text()?
In my setup.cfg I could do that with
[options.package_data]
runpanel = ui.glade
then in my program I could do glade_file = importlib.resources.read_text(__package__, "ui.glade") to access my package data
waxen tangle
ah ha! [tool.hatch.build.targets.wheel.shared-data] is the secret sauce!
umbral bridge
I'm back ๐
tropic void
@umbral bridge are there any examples of plugins that generate python files in specific ways ? i intend to add a hatch plugin to apipkg that takes in pyi files and generates a init.py file with lazy loading plugins - goal being to have type-safe lazy loading
umbral bridge
not .py but hatch-mypyc might help as an example
tropic void
ok, i'll give that a spin then, thanks
umbral bridge
vscode support coming https://github.com/pypa/hatch/pull/524
tropic void
absolutely fabulous
umbral bridge
we have blog support now ๐ https://github.com/pypa/hatch/pull/530
umbral bridge
this is a hack, but I'm proud of it lol https://github.com/pypa/hatch/pull/534
I truly loathe the prepare_metadata_for_build_* methods
umbral bridge
releases + first blog post coming soon https://github.com/pypa/hatch/pull/538
umbral bridge
rancid mesa
hi! how do i tell hatch run to use a newer py for the default env? i have a project that requires >= 3.10 but my default python3 is 3.8; i have newer ones installed but hatch wants to see a matrix to let me select them or something? the default env doesn't have a matrix though
rancid mesa
yes, modifying the file works, but i mean from the CLI?
umbral bridge
rancid mesa
ah, kthx ๐
umbral bridge
my team has started the migration to Hatch envs from tox https://datadoghq.dev/integrations-core/meta/status/#hatch-migration
here's a good example of how concise and readable Hatch config can be https://github.com/DataDog/integrations-core/pull/13152/files
umbral bridge
@karmic glacier I haven't forgotten about your environment dependency groups feature request lol
umbral bridge
^ only blocker?
karmic glacier
probably not 
I admire poetry's dependency management and it would play quite well with nox (our current tox-like runner)
I can explain more in a little bit, currently on mobile
umbral bridge
so really just need locking support?
karmic glacier
I can explain more in a little bit, currently on mobile
what i really appreciate is being able to use specific dependencies in each environment and not reinstalling them each time
i had written a system with nox and poetry which ensures that each local environment will have the exact dependencies it is supposed to have for every test
the issue is that I can't easily export the list of dependencies that poetry defines, so i'm re-exploring hatch
it would be nice to see something like hatch install --sync and then define groups of dependencies to install
gloomy flicker
here's a good example of how concise and readable Hatch config can be https://gi...
is there a technical reason why it's python = ["38"] instead of "3.8"? or did you just inherit the idiom from tox?
umbral bridge
both work, gets passed to virtualenv
@karmic glacier
what i really appreciate is being able to use specific dependencies in each environment and not reinstalling them each time
Hatch already does that
dull rapids
Deleting dots from versions
umbral bridge
@dull rapids https://github.com/pypa/hatch/pull/570
umbral bridge
fyi I'll be documenting "why to use" vs other things like setuptools, poetry, tox, etc.
I thought it might not be well received as, since it's part of the PyPA and offers so much, it might look like bashing on other tools or a recommendation for certain use cases
but I get asked so much that I need to write it somewhere
shut mantle
If you have a blog, you could post it there and link to it from the documentation. Or you could just ignore people not receiving it well I suppose (though yeah, it is a tough cookie with it being part of PyPA)
umbral bridge
I can't repro https://github.com/pypa/hatch/issues/562
rancid mesa
no only env var https://hatch.pypa.io/latest/plugins/environment/virtual/#option...
random thought: is there a way for hatch to know something requires a certain python version before creating the venv and use a newer version or at least print a message saying how to recreate it with the correct version? if a package provides a "foo" entry point and the user types "hatch run foo", it might know and can do the check up front? or it could take the py version constraint in pyproject.toml as a hint? anything to avoid head scratching
umbral bridge
sure, I will
umbral bridge
ha scary but awesome https://github.com/deepset-ai/haystack/pull/3484
tropic void
ha scary but awesome https://github.com/deepset-ai/haystack/pull/3484
Yikes, they made a auto running footgun
karmic glacier
@umbral bridge you have a script running checking for mentions of hatch or hatchling on GitHub, don't you? ๐
umbral bridge
I check on weekends haha
karmic glacier
ah lol
I'm not sure I've seen a pep 517/518/621/etc pull request where you didn't comment
cursive lion
A1 customer support! ๐
karmic glacier
does hatch and hatchling support using only hatchling?
would use a different package as the front end
Assuming that hatchling supports all of the normal apis (PEP 621, 517, 660, and 518), that shouldn't be an issue, but is there anything special to look out for?
shut mantle
Hatch isn't required when using hatchling backend
umbral bridge
winged cargo
will hatch have better support for c extensions any time soon
i dont see why something like this cant exist
[project]
extensions = [
{ name = "myextension", path = "mod.c" },
]
is it really that hard to access the systems compiler?
errant timber
Very much planning to work on a pep to support that. ๐
Well, sort of that.
There are several tricky things to supporting extensions. Like what file extension does it compile to? A regular one, an ABI3 one, or a pythonless shared lib. All of which affect the wheel tag too.
winged cargo
isnt there like a distutils.Extension thing
umbral bridge
the point is to get rid of distutils
dull rapids
For native extensions, you could try out maturin: https://www.maturin.rs/
errant timber
Or scikit-build-core or meson-python. ๐ The long-term goal is to have a nice way to build an extension to go along with a larger Python package that uses hatchling or Poetry or another Python build tool.
low gyro
It would be nice if there was an extension library (like what distutils was) that all buildsystems could integrate with
errant timber
That's the idea of the extensionlib PEP. I'm going to help with that once I get a little further on scikit-build-core, so that I have a good idea of the minimum requirements. I've already learned that being able to communicate sdist/wheel requirements is important. ABI3 and Pythonless wheels are also possibly important. Optional builds are important. I haven't tackled editable installs or caching yet. Once I have a good idea of what would be useful, I'll work on preparing a PEP and implementing a PoC with extensionlib.
Though it won't be exactly like distutils, it will only be the glue between a binary builder (like CMake for scikit-build-core, meson for meson-python, or Cargo for maturin) and a Python builder like Hatchling, Poetry, Flit, etc. Someone might come up with a native one, but building is really hard. Distutils is improved by setuptools, but still doesn't support nearly anything you need today to build, like C++11+, headers in a folder, etc. ๐
umbral bridge
@low gyro #extensionlib
umbral bridge
I'm convinced over half of users don't read docs
subtle bloom
I'm convinced over half of users don't read docs
Generous
umbral bridge
it really sucks b/c I've put so much effort into writing them
I do so much cross-linking in docs! makes me sad
subtle bloom
fwiw I think your docs are great
orchid birch
I'm convinced over half of users don't read docs
Warsaw's Fourth Law
Have you seen diataxis.fr ? That's the best framework for doc writing that I've seen.
umbral bridge
yup quite nice
turbid bane
@umbral bridge was there any further action taken on the mislabelled cross-architecture macOS wheels? I'm upgrading Black's toolchain and it reminded me that our release process is still broken.
If not, I'll open an issue since I'd like to get this fixed eventually.
umbral bridge
is that Hatch issue or setuptools?
turbid bane
IIRC hatch since setuptools generated the universal2 and arm64 wheels with the right platform tags
umbral bridge
which env vars are being set?
turbid bane
yeah hatch eventually ends up calling https://github.com/pypa/packaging/blob/fd4f11139d1c884a637be8aa26bb60a31fbc9411/packaging/tags.py#L346 AFAICT, which isn't correct when you're building with cibuildwheel and forcing a platform tag
according to Zsolt
https://github.com/pypa/cibuildwheel/blob/c85450161ca6dd6eae9c0ee33faf246a9026dc68/cibuildwheel/macos.py#L217-L229 ah yeah, presumably packaging completely ignores _PYTHON_HOST_PLATFORM
me.
I'm just pulling from an old discussion in the core group chat. I don't remember any of the context so give me a second.
turbid bane
which env vars are being set?
https://github.com/pypa/cibuildwheel/blob/2e6cbe2435ef410156319c98625002ed76e08785/cibuildwheel/macos.py#L219-L232 cibuildwheel sets _PYTHON_HOST_PLATFORM during macOS cross-arch builds here
umbral bridge
turbid bane
not sure why setuptools respected it while hatch does not
umbral bridge
@turbid bane ^ open issue in hatch-mypyc
also tbh I've no idea how to derive wheel name/tags from any compiler env vars
maybe @errant timber can help
turbid bane
Digging through the setuptools codebase, I'll guess that it uses sysconfig.get_platform() which looks at _PYTHON_HOST_PLATFORM here https://github.com/python/cpython/blob/504e12272b19395a3c59631a20a94e3323af78e5/Lib/sysconfig.py#L774
In setuptools' Wheel class, its egg_name() method calls distutils.util.get_platform() assuming it isn't any already.
https://github.com/pypa/setuptools/blob/c75928017022ac52464ed65e9fe3a553281372fc/setuptools/wheel.py#L94
distutils.util.get_platform() calls distutils.util.get_host_platform()
https://github.com/pypa/setuptools/blob/c75928017022ac52464ed65e9fe3a553281372fc/setuptools/_distutils/util.py#L60
distutils.util.get_host_platform() calls sysconfig.get_platform() which as noted, checks _PYTHON_HOST_PLATFORM
https://github.com/pypa/setuptools/blob/c75928017022ac52464ed65e9fe3a553281372fc/setuptools/_distutils/util.py#L48
Figuring out where the platform tags stored in .dist-info come from isn't nearly as obvious -- I barely know the setuptools codebase
turbid bane
<@429445826251587616> ^ open issue in hatch-mypyc
Fortunately, only the metadata is wonky. The wheels themselves are usable.
Editing the metadata of wheels by hand during a release felt a bit sketchy tbh haha. It happens :)
errant timber
ARCHFLAGS is how setuptools targets other archโs and cibuildwheel and conda set it.
umbral bridge
nice ty ๐
gloomy flicker
would it possible to extend hatch-vcs to use dunamai in place of setuptools-scm or would I be better off creating a new plug-in?
umbral bridge
new
then open PR to https://hatch.pypa.io/latest/plugins/version-source/reference/#known-third-party if you want
errant timber
Not sure if you've already considered this, but you get a much better error if you raise SystemExit(1) from None in a build hook vs. throwing an error. I've been using this is scikit-build-core.
umbral bridge
it's so fast omg https://github.com/pypa/hatch/pull/607
low gyro
it's so fast omg https://github.com/pypa/hatch/pull/607
I see you removed flake8 plugins. Does ruff come with all their functionality built-in?
umbral bridge
yup
umbral bridge
ruff sorts imports too?
gloomy flicker
yeah, ruff --select I0 --fix ...
umbral bridge
๐ ty
orchid birch
I want to help ruff support blue autoformatting
low gyro
blue?
crystal idol
`blue`?
Barry & co's fork of Black https://github.com/grantjenks/blue
cyan turret
Not sure if it is my overthinking or if there is some mechanism to make it work, but what if some dependencies of hatchling also use hatchling as the backend and the wheels are banned? Will it cause recursion?
umbral bridge
yes Hatchling deps must use other backends
umbral bridge
low gyro
why did you remove pytest-cov?
orchid birch
Actually, I think it makes sense (if I understand the PR correctly). I was a little surprised that built-in support for these tools was provided. Explicit is better than implicit?
umbral bridge
I only want to recommend workflows that have large buy-in or are not controversial and coverage via that plugin is often not the best nor easiest way for most users
umbral bridge
ok, added pure coverage
gloomy flicker
pure coverage doesn't collect coverage from subprocesses though, right? That comes up surprisingly often I think
cyan turret
why did you remove pytest cov
cursive lion
See also this proposal: https://github.com/pytest-dev/pytest-cov/issues/337
umbral bridge
ok, back to pytest-cov for subprocesses and xdist...
also now enforcing absolute imports for easy addition of Mypyc if the user wants to eventually
umbral bridge
dull rapids
Huh, I tried to use the hatch envs feature and I seem to be missing something: https://github.com/flying-sheep/fine-coverage/blob/4e010279830d61184d3a8b03fc91ab7f4939c4ad/pyproject.toml
When running hatch run test, pytest crashes with ModuleNotFoundError: No module named 'fine_coverage'
Why isnโt the project being installed into the env?
umbral bridge
@dull rapids have 3.11 on PATH?
dull rapids
yup!
umbral bridge
hatch env prune then retry
dull rapids
ah:
$ hatch run test
ERROR: Package 'fine-coverage' requires a different Python: 3.10.8 not in '>=3.11'
but:
$ which python3.11
/usr/bin/python3.11
fine-coverage needs Python 3.11+ but it seems to be trying use 3.10?
dull rapids
yeah, I specified
[[tool.hatch.envs.test.matrix]]
python = ['3.11']
and python3.11 is in PATH, but hatch seems to still create the env with python3.10 for some reason.
Ah, I guess it just creates the default env and runs pytest in there, and python = [...] applies to a test env thatโs never created. hatch new -i somename (hatch version is 1.5.0) creates the following:
[tool.hatch.envs.default]
dependencies = ["pytest", "pytest-cov"]
[tool.hatch.envs.default.scripts]
cov = "pytest --cov-report=term-missing --cov-config=pyproject.toml --cov=hatch_test --cov=tests"
no-cov = "cov --no-cov"
[[tool.hatch.envs.test.matrix]]
python = ["37", "38", "39", "310", "311"]
which seems wrong?
umbral bridge
@dull rapids https://hatch.pypa.io/latest/environment/#selection
dull rapids
Thanks! I think I get the intention now: hatch run cov would run test coverage in the default environment (once) while hatch run test:cov runs it for the whole matrix.
Maybe my specific issue could be fixed by making hatch refuse to create an environment if dependencies can't be resolved. I missed the message the first time, which caused hatch to create and reuse an invalid environment.
umbral bridge
good idea ty
@tropic void I got a FR at work for https://github.com/ofek/hatch-vcs#metadata-hook, in case you also find it useful
tropic void
@umbral bridge i saw that but i`m a little put off as its creating a initial parallel infrastructure and is easy to get wrong (like commit hashes in urls) - whats the use case it feeds and what can be done to integrate it more with setuptools_scm (which is in slow motion as we have a 4 month old in addition to the 5 year old ^^)
umbral bridge
use case is for tracing product, source code line numbers, etc. can't say more but concept seems useful
waxen tangle
should I expect hatch version minor to work?
I'm getting a NotImplementedError
just wondering if that's a problem on my end
umbral bridge
not if you're using hatch-vcs
waxen tangle
gotcha, thanks.
umbral bridge
@tropic void I could implement that, wdyt?
tropic void
@umbral bridge hmm, i wish that could land in setuptools_scm, unfortunately my bandwidth for hatch-vcs vs setptools_scm vs the new name for a shard project is zero well until the next year, and im a bit terrified that hatch-vcs grows in scope in complex ways that are a pain to reintegrated
umbral bridge
it'd basically say "only works with Git" until setuptools_scm or the new lib adds a way to set
tropic void
@umbral bridge i suppose thats a way to go - the more interesting use-case would be to not just create a tag, but creating a tag thats annotated with the expected signatures of the artifacts (so that build automation for releasing can verify when ci does the uploads + attach them)
low gyro
Do I get it right, that you plan on making setuptools_scm universal?
tropic void
Do I get it right, that you plan on making setuptools_scm universal?
In part it already is, most of the guts are tool independent, but the integration ain't easily shipped
tropic void
> signatures of the artifacts
the sdist & wheels?
Of sdists and wheels, the idea being to enable the upload of reproducible release builds with gpg signatures based on the Metadata the maintainer supplied in the tag annotation
tropic void
@umbral bridge btw, is there a recommended solution for using hatch for pluggy (it's a dependency of hatchling and would make a build recursion hell for potential downstreams
umbral bridge
no ๐ฆ no Hatchling dep can use Hatchling for builds
tropic void
@umbral bridge i mean it's potentially not a issue for build envs, since those just install from a preexisting wheel, but the missing bootstrap story is a bit of a pain
umbral bridge
Hatchling deps will never change I'd say https://github.com/pypa/hatch/blob/d3eeb62a0c094681d30284a084d6a7c50f8864a6/backend/pyproject.toml#L37-L44
4 forever deps
tropic void
@umbral bridge would you be onboard with setting up hatchling so that a hatchling_sumo can be released which vendors those deps so that that one could use hatching sumo for only those
tropic void
hmm
@umbral bridge i dont quite follow, my idea was to make a released artifact that would include the dependencies (just like pip)
umbral bridge
right I get that but my point is Hatchling doesn't need changing
tropic void
or are you proposing something that just vendors all the things, mangles sys.path to make it work like hatchling?
umbral bridge
yup
tropic void
โค๏ธ
so bascially config would be the same, just that the vendored hatch would be the input
@umbral bridge aware of any hatch plugin that could sort out the vendoring part nicely? i'd like to avoid copying pip/setuptools behaviours
umbral bridge
no, but cool idea
tropic void
@umbral bridge it might also be a nice to have for pytest - i want to trim down dependencies while at the same time splitting stuff out thats supposed to be shared with other runners
umbral bridge
I love seeing users that were locked in to dynamic setup.py finally have an alternative https://github.com/tlambert03/ome-types/pull/161/files#diff-d56d1b58c0ca28bdddd10c2cee2ab3cb4312766ea5de51f210dbc15a0fea42fa
turbid bane
Any updates on supporting cross-architecture builds? I'm wondering whether it'd be worth it creating a tool for fixing the platform tags.
umbral bridge
fyi I plan to add this to the default project template https://github.com/charliermarsh/ruff/issues/1255
umbral bridge
I was a bad maintainer/reviewer and assumed the contributors adding types were flawless
turns out their first PR accidently added an option that effectively disabled Mypy...
my weekend is gone, and the next
this is rough to fix
turbid bane
oh right sorry I'll fix hatch-mypyc tonight
thank you very much! and I'm sorry to hear about that :(
dull rapids
Ah damn, those things are so hard to spot. Tools donโt usually come with a xfail function.
umbral bridge
I wish someone more experienced with typing could just swoop in and fix in <1 hour
rather than days
I'm now thinking types are useless if code coverage is high
low gyro
What seems to be the issue?
umbral bridge
mostly subclasses
low gyro
I'm now thinking types are useless if code coverage is high
Types are useful for more than just code checking. They make it easier for people to work with existing codebase
tropic void
@umbral bridge maybe i can help, cna you link me the pr?
(im on a typing spree these days , even with good coverage they uncover all kinds of messes)
umbral bridge
getting close
@tropic void https://github.com/pypa/hatch/pull/677
linter runs on Python 3.9 on Linux
tropic void
oh, extra monsterous
crystal idol
Yeah, I find them useful even with good test coverage, and invaluable for integrating with other codebases (e.g. people writing plugins if you've typed your public APIs)
normal tundra
Sadly there some things are horrible to type. I love typing and use it almost everywhere, almost because for some constructs I just give up or not do it because it kills readability.
Callable and Generator anyone? ๐
So I am impressed if you fix the whole projects typing!
tropic void
Sadly there some things are horrible to type. I love typing and use it almost ev...
the real horror is that types are not computable / cant be expressions
tropic void
@umbral bridge could the version file hook be changed, so its only including the wheel/editable wheel as opposed to putting it in a workdir
tropic void
@umbral bridge needing to have a explicitly excluded file in the worktree is a handgun
@umbral bridge i ust took a look at the current way editables works, as far as i can tell, by virtue of pep420 namespaces, the editable wheel may be able to just ship as part of the wheel in both cases
umbral bridge
the current way editables works
that's opt-in https://hatch.pypa.io/latest/config/build/#dev-mode
dull rapids
<@233155068155658240> https://github.com/pypa/hatch/pull/677
seems like this might get easier once this commit is in a mypy release: https://github.com/python/mypy/commit/77dd4b4df5b8bcd716352144feeb78862427f4dd
Then we can just use -> Self for all the __new__ classmethods.
umbral bridge
the tricky part is that the Custom* plugins load from a file and return a user-defined subclass of that particular interface. they're just a glorified __new__
dunno how to fix other than ignoring
dull rapids
I guess with that level of dynamicity, one could just validate at runtime that the returned object is really a valid subclass and then bypass mypy if it doesn't understand that kind of type assertion.
mint bramble
@umbral bridge do you know why a venv.py file is not included? ๐
is this some bad virtual environment exclusion rule ๐
(for hatchling)
umbral bridge
is it ignored by Git?
mint bramble
nope
https://github.com/gaborbernat/virtualenv/blob/fix/src/virtualenv/create/via_global_ref/venv.py this file is not packaged
mint bramble
I know
but we still add it with git
shouldn't this be all added to git, plus files not matching gitignore?
there's no way in gitignore to say folders only ๐
tropic void
next version of execnet will use hatch ๐
tropic void
@umbral bridge aware of a ci system that plays well with hatch and can be self-hosted, i have a number of things that I want to free from github, and most ci systems seem to be yaml hell or drunk docker dunking
low gyro
YAML-free CI system is a dream...
tropic void
YAML-free CI system is a dream...
Jenkins is putting that ad absurdum with groovy (which at this point is the drunk lovechild of java and yaml)
patent tundra
YAML-free CI system is a dream...
It's on my list to try it out: https://dagger.io
tropic void
It's on my list to try it out: https://dagger.io
Slightly terrifying to see yaml be replaced with multi language sdks to containers behind graphql, while at the same time lacking artifacts and integration examples
umbral bridge
gitlab?
tropic void
using that in some work projects, never want to touch it for anything private based on the pain and mess - i havea gitea as forge for my personal repos but i want a ci against it
umbral bridge
btw why would you like to move away from GitHub?
shut mantle
woodpecker ci uses YAML so I guess it's of no interest to you
Cirrus CI supports configuration using one or both of YAML and Starlark (subset of Python) but they don't quite open source the whole stack. Their task runner is open source so you could supposedly use it with a different CI frontend but that seems like a hassle
tropic void
woodpecker ci uses YAML so I guess it's of no interest to you
im likely going to use woodpeeker for now, thanks
tropic void
btw why would you like to move away from GitHub?
its free as in shackles - im still going to use it for opensource stuff, but for my personal/private repos - i want a own forge and a own ci (currently likely gitea + woodpeeker)
umbral bridge
@turbid bane what's your strategy for type hinting stuff from imported modules that don't exist on certain platforms?
the most recently merged type hinting pull request made it so it now fails on Windows
turbid bane
I don't think I've had to do that before honestly, or at least I don't remember how I'd handle that. cc @lament cedar
lament cedar
mypy has some options around that, right?
so it basically treats those imports as Any
not ideal but still gives you some typing coverage
turbid bane
I'd probably guard that with assert sys.platform != "win32" (no idea if mypy supports OS checks with assert) or if sys.platform != "win32": ..., but a type ignore would be acceptable too IMO.
lament cedar
oh yeah, assert sys.platform would be best
turbid bane
it looks like the signal module does exist but the attribute does not
It is unix-only
umbral bridge
I've never done that to satisfy Mypy, do I put the assert statement right before the offending line?
turbid bane
yup
umbral bridge
nice, thank you both very much. I'm going to start running the lint step on every job. I was trying to save CI time by just running on one
turbid bane
typing is the only thing that should differ across OSes FWIW but that can't hurt either
umbral bridge
also I guess Ruff is so fast that it doesn't matter anymore without flake8 lol
turbid bane
other than mypy, is black now a bottleneck? haha
umbral bridge
yes it is ๐
turbid bane
whelp ๐
umbral bridge
Mypy will likely forever be its own thing but the maintainer of Ruff has expressed interest in formatting like Black
turbid bane
We're going to be put out of business
Anyway I can think of an optimization for cases where Black makes changes, but I haven't looked into optimizing a single pass only at all.
umbral bridge
I'm actually a bit concerned for about two years from now when we finally have a lock file that people will start complaining about slow dependency resolution
umbral bridge
either Mypyc in pip or I'll have to write my own logic in Hatch in Rust
turbid bane
oh, you mean Python might be the bottleneck ๐
turbid bane
either Mypyc in pip or I'll have to write my own logic in Hatch in Rust
I don't think pip will ever adopt non Python code. mypyc is an interesting prospect but the deployment story sounds like a nightmare
(also mypyc isn't close to being mature enough IMHO)
low gyro
Yeah, sadly
umbral bridge
I don't know if they would allow it but they could still ship a pure Python wheel
turbid bane
That reminds me, I should get back on working on mypyc actually -- been a while.. although Black also needs a lot of attention, ahhhh ๐
low gyro
Honestly, I don't think going with Rust-based code is a way to go too. I mean, if you don't like Python, go full Rust and don't bother with "slow" Python
My hopes are in faster Python itself.
umbral bridge
I am quite curious what other high-level languages do, like does NPM have some logic in C?
low gyro
https://github.com/npm/cli Nope, pure JS
umbral bridge
okay, pure Python forever then
turbid bane
I do wonder if resolvelib is compilable with mypyc though
umbral bridge
the benefits would never be felt though because pip vendors everything, right?
turbid bane
yeah :(
low gyro
Yeah. Bootstraping is the painful part
turbid bane
I also feel like enterprises will hate the fact pip would be now a compiled executable, not introspectable or auditable at all
umbral bridge
is that the important part or is the code being open source and the distribution channel secure the important part?
most things are compiled
turbid bane
I agree, I'm just saying for something as major as pip which has always been pure-Python, it would be a major shift that they might not be comfortable with
umbral bridge
btw @turbid bane https://github.com/pypa/hatch/pull/672#issuecomment-1367593882
turbid bane
the x86_64 wheels (no cross-compilation) worked fine for us even before the fix
umbral bridge
Yeah I have no idea what's going on there
turbid bane
I haven't tried building black with patched hatch, I can do that right now
wait a second, what the... https://github.com/psf/black/actions/runs/3658832698/jobs/6184104544
so building a native x86_64 wheel on x86_64 macOS doesn't work, but fixing the platform tag of a wheel built using a cross-architecture build targeting ARM64 does? what the hell
it worked before I swear, I'm so confused
ugh the logs for 22.8.0 expired
I guess 10_6 is the newest macOS version supported for 3.7 wheels?
turbid bane
OK so patched hatchling seemed to fix the ARM64 build, but the universal2 and more importantly the x86_64 builds are still broken.
hatchling from pypi: https://github.com/ichard26/black-mypyc-wheels/actions/runs/3803319125
crystal idol
I am quite curious what other high-level languages do, like does NPM have some l...
JS engines just have a massive amount of money invested in optimization, JITting, etc. because browsers
umbral bridge
OK so patched hatchling seemed to fix the ARM64 build, but the universal2 and mo...
how did it suddenly break?
turbid bane
I have no idea ยฏ_(ใ)_/ยฏ
umbral bridge
downgrade setuptools?
turbid bane
the x86_64 build for 22.10.0 produced wheels with reasonable platform tags (although they seem more strict than they need to be)
5 wheels produced in 21 minutes:
black-22.10.0-cp310-cp310-macosx_11_0_x86_64.whl 1,381 kB
black-22.10.0-cp311-cp311-macosx_11_0_x86_64.whl 1,363 kB
black-22.10.0-cp37-cp37m-macosx_10_16_x86_64.whl 1,314 kB
black-22.10.0-cp38-cp38-macosx_10_16_x86_64.whl 1,372 kB
black-22.10.0-cp39-cp39-macosx_11_0_x86_64.whl 1,380 kB
https://github.com/psf/black/actions/runs/3200959148/jobs/5228453985
turbid bane
downgrade setuptools?
uhhhh how would I do that?
umbral bridge
downgrade setuptools or packaging in deps
turbid bane
in build-system.requires?
umbral bridge
yes. new releases may have broken something
turbid bane
I'd put the blame on packaging 22.0.0 first before setuptools
I'll try downgrading that first
umbral bridge
well actually, that I think is only explanation because I haven't released Hatchling in a while
turbid bane
Yeah I noticed that too
Maybe mypyc started generating code that requires a newer macOS version, but that seems really unlikely (since it wouldn't be interacting with any system APIs)
I plan on also reverting the hatchling PR and triggering a test build to see what the platform tags should really be.
umbral bridge
Hatchling uses packaging to determine what the most precise tags should be so I'm assuming version 22.0 is the issue
turbid bane
https://github.com/pypa/packaging/pull/513 hmmm this might be the source of our problems
I wish I had a macOS machine available to me. I have so many questions that I don't have answers for
umbral bridge
yeah that makes sense since other operating systems are unaffected
umbral bridge
does setuptools vendor packaging?
turbid bane
I triggered a new run w/ hatchling from main and packaging==21.3: https://github.com/ichard26/black-mypyc-wheels/actions/runs/3804407857
let's see if it works, if so we have a suspect ๐
ugh, hatchling on main requires packaging 22.0 ๐
radiant cloak
Is this legacy versions or macos tags?
turbid bane
should I just downgrade hatchling and then use packaging 22.0 (so basically the inverse setup)?
umbral bridge
I'll fix that right now
turbid bane
Is this legacy versions or macos tags?
What do you mean? Sorry I don't work with macOS at all ๐ถ
We're trying to figure out why Hatchling is producing a different platform tag on x86_64 macOS that isn't even supported on the system building the wheel (from a few months ago).
umbral bridge
okay I just reverted the packaging upgrade
radiant cloak
Ah, so macos tags; cool cool.
turbid bane
macOS platform tags were already busted with hatchling during cross-compilation, but now even native x86_64 builds are breaking
so Black's CD is totally broken right now for macOS ๐
radiant cloak
nods.
turbid bane
https://github.com/ichard26/black-mypyc-wheels/actions/runs/3803441949/jobs/6469849507#step:5:281 here's the failure we're investigating
radiant cloak
If there's a backwards-compatibility change in packaging that's a breaking change other than legacy versions; plz file an issue. I need to go get some sleep. ๐
turbid bane
If there's a backwards-compatibility change in packaging that's a breaking chang...
We'll try to figure it out. Good night, sleep tight!
turbid bane
macOS platform tags were already busted with hatchling during cross-compilation,...
I should clarify the cross-architecture build failures were hatchling's fault, but the native x86_64 builds breaking is probably packaging's
turbid bane
okay I just reverted the packaging upgrade
Awesome thanks. Triggered a rerun: https://github.com/ichard26/black-mypyc-wheels/actions/runs/3804407857
- pip install '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-xa7s3tha/cp37-macosx_x86_64/repaired_wheel/black-22.12.1.dev29+g6450f2f-cp37-cp37m-macosx_10_16_x86_64.whl[d, jupyter]'
Yup packaging is the culprit, w/ packaging 21.3 this is the built wheel
now whether that is the right macOS version, I have no idea since well I know nothing about macOS, but it does at least install during testing.
umbral bridge
turbid bane
Yeah, I just expect setuptools to produce a different set of platform tags sooooo ๐
umbral bridge
yeah I'm quite curious what it produces, based on documentation Hatchling is doing the right thing
turbid bane
I'm working on switching Black back to setuptools so I can trigger another test build, but it's kinda a pain. The configuration got some changes post-hatchling so there's a million conflicts.
umbral bridge
this is basically how Hatchling derives the most specific tag:
'-'.join(next(packaging.tags.sys_tags()))
Using the most-specific tag isn't always the right thing for the build-backend.
umbral bridge
Yeah definitely it's behind an option that in this case the mypyc plugin enables
turbid bane
Using setup.cfg to only configure setup_requires is irrelevant today, right?
radiant cloak
What does python -c 'from packaging.tags import mac_platforms\nfor tag in mac_platforms(): print(tag)' print on that machine?
radiant cloak
Using setup.cfg to only configure setup_requires is irrelevant today, right?
Pretty much; you can do that but it's preferable to use pyproject.toml's build-system.requires directly.
turbid bane
Before hatchling we had setup.cfg and now I'm switching back to our old seutptools setup, I was wondering if I needed to copy that file over.
umbral bridge
can't you just reset to an old commit to test?
radiant cloak
^
umbral bridge
rather than manually doing stuff
radiant cloak
No need to backport changes if you're just testing things.
turbid bane
ah well, too late ๐
radiant cloak
XD'
turbid bane
I'm done now
radiant cloak
What does `python -c 'from packaging.tags import mac_platforms\nfor tag in mac_p...
FWIW, this would be useful info to know.
turbid bane
I'll put in the workflow before I trigger another run
radiant cloak
Especially if you can do another run with SYSTEM_VERSION_COMPAT=1 env var as well, and compare/contrast.
umbral bridge
I don't precisely know what's going on but I'd wager everything would work if pip vendored the latest version of packaging
packaging still needs to be fixed, I'm just saying that I think that would work
turbid bane
man I suck at CI
umbral bridge
@turbid bane I'm going to release this tonight https://github.com/ofek/hatch-mypyc/pull/27
turbid bane
FWIW, this would be useful info to know.
setuptools: https://github.com/ichard26/black-mypyc-wheels/actions/runs/3804627105/jobs/6471997448#step:5:29 w/ packaging 22.0 on macOS 11 (the debug step is using Python 3.7.9)
setuptools w/ SYSTEM_VERSION_COMPAT=1: https://github.com/ichard26/black-mypyc-wheels/actions/runs/3804658334/jobs/6472031376#step:5:25 w/ packaging 22.0
radiant cloak
I don't precisely know what's going on but I'd wager everything would work if pi...
Yea, pretty sure that is the fix here.
turbid bane
hatchling (from main) w/ SYSTEM_VERSION_COMPAT=1: https://github.com/ichard26/black-mypyc-wheels/actions/runs/3804683335/jobs/6472072376 w/ packaging 22.0
So the setuptools job worked, producing these wheels:
black-22.12.1.dev29+g19452b8-cp310-cp310-macosx_10_9_x86_64.whl 1,413 kB
black-22.12.1.dev29+g19452b8-cp311-cp311-macosx_10_9_x86_64.whl 1,398 kB
black-22.12.1.dev29+g19452b8-cp37-cp37m-macosx_10_9_x86_64.whl 1,345 kB
black-22.12.1.dev29+g19452b8-cp38-cp38-macosx_10_9_x86_64.whl 1,405 kB
black-22.12.1.dev29+g19452b8-cp39-cp39-macosx_10_9_x86_64.whl 1,412 kB
The setuptools job w/ SYSTEM_VERSION_COMPAT=1 crashed while building the 3.11 wheel though.
SYSTEM_VERSION_COMPAT=1 crashed the hatchling job too actually
Ugh, for some reason the setuptools jobs used packaging 21.3 during the actual wheel builds. pretty much means everything I just did is useless >.<
That's wrong, everything is fine. I have too many tabs open.
turbid bane
So the setuptools job worked, producing these wheels:
```
black-22.12.1.dev29+g1...
It's worth noting these tags are the same as the Black 22.8.0 wheels which used setuptools and packaging 21.x -> https://pypi.org/project/black/22.8.0/#files (minus the fact there's no 3.11 wheel here since 22.8.0 didn't support that yet)
I'm tired now. I'm sorry for taking forever to test and get back with all of the results. I'm not great at CI :(
If you need me to do some more testing, please let me know. I'm going to work on something else now.
umbral bridge
so is the issue that pip can't recognize 10_16?
turbid bane
it doesn't recognize 11_0 as a compatible tag on Python 3.7 (and 3.8 maybe?). AFAIK from reading related issues: 11_0 is equivalent to 10_16
previously w/ packaging 21.3, hatchling would select 10_16 for the Python 3.7/8 wheels which works fine, but w/ packaging 22.0, now hatchling selects 11_0 for those wheels which breaks because โฌ๏ธ
that's my understanding of the situation at least
umbral bridge
what would be your recommendation to fix?
turbid bane
Yea, pretty sure that is the fix here.
this
umbral bridge
darn
turbid bane
I don't precisely know what's going on but I'd wager everything would work if pi...
this (wrong message, sorry pradysung!)
umbral bridge
@turbid bane I'm going to release this tonight https://github.com/ofek/hatch-mypyc/pull/27
should I not do this then?
I was about to push the button lol
turbid bane
that's fine, since in theory pip vendoring packaging 22.0 should still recognize 10_16 as a compatible tag (unless it doesn't, in which case ... ๐ญ)
umbral bridge
okay I will release now
turbid bane
I mean, maybe pip w/ packaging 22.0 won't recognize 10_16 as a compatible tag, but that would probably break more than just hatch's mypyc plugin
umbral bridge
my question then is how long do I have to keep this pin?
since it takes people years to upgrade pip
turbid bane
you're kinda asking the wrong person, but I'd guess until pip can install wheels built w/ packaging 22.0
which would be a long while indeed ...
turbid bane
> <@429445826251587616> I'm going to release this tonight https://github.com/ofe...
I'm honestly not sure whether this is the right call longterm. I know almost nothing about macOS, but for now this will unblock Black's PyPI workflow (finally).
umbral bridge
yeah make sense. just to confirm, the Hatchling patch makes Black + archflags work right? if so I can release right now
turbid bane
https://github.com/ichard26/black-mypyc-wheels/actions/runs/3804407857/jobs/6471680692 yup, hatching with that ARCHFLAGS patch and packaging 21.3 builds successfully ๐
umbral bridge
cool, I'll begin right now
turbid bane
I can't believe I'm still hitting issues w/ our use of mypyc, it's been almost 2 years since 22.1.0
On the bright side, at least this should mean other people have a nicer experience when they use mypyc.
cursive lion
I wish I had a macOS machine available to me. I have so many questions that I do...
I've never used them, but there are some tricks to SSH into a GitHub Actions runner, for example https://www.codejam.info/2022/05/debugging-github-actions-workflow-ssh.html (I've used the builtin one on Travis before and it was really useful)
turbid bane
I have never used SSH ... yeah I know... what kind of "developer" am I ๐
I've heard of the idea before though, thanks for the link. I'll look into it later. Knowing my luck, it'll be useful time after time again down the road. :)
dapper willow
I tried to setup a mac VM once and gave up ๐
I just ended up getting a mac mini as a work expense, which I am thankful was an available option to me
but yeah, supporting systems we don't use is a PITA
turbid bane
I've considered applying for access to the GCC compiler farm (by suggestion of isidentical) mostly just so I have access to some sort of macOS machine, but I never did (for several reasons).
turbid bane
Is there anything else I should do @radiant cloak? I can open an issue for pip/packaging if it'd be helpful
radiant cloak
Is packaging 22.0 not generating tags that 21.3 used to?
turbid bane
AFAICT yes, I can check though
turbid bane
Is packaging 22.0 not generating tags that 21.3 used to?
https://github.com/ichard26/testblackaction/actions/runs/3808903657/jobs/6479867547 Well erm no, using packaging 22.0, mac_platforms() returns macosx_11_0_* tags on macOS 11 but packaging 21.3 does not. It's the other way around.
The difference only occurs on Python 3.7 and 3.8. On macOS 12, packaging 22.0 also returns the 12_0 series tags while packaging 21.3 doesn't return the 12_0 or 11_0 series tags at all. (Now whether this causes issues, I don't know, we aren't building Black with macOS 12)
@umbral bridge FYI I'm waiting for a new release of hatchling with the ARCHFLAGS patch so we can start building macOS wheels for Black. No rush though!
umbral bridge
oops sorry I'll do that right now. last night there was a minor regression with Ruff that is now fixed, but I forgot to release ๐
turbid bane
Thank you!
umbral bridge
turbid bane
โฏ git fix
[build-mypyc-wheels-for-macos-again e283974] Reenable mypyc wheels for macOS
Date: Fri Dec 30 15:11:55 2022 -0500
1 file changed, 6 insertions(+), 7 deletions(-)
awesome, great timing :D
https://github.com/ichard26/black-mypyc-wheels/actions/runs/3809064130/jobs/6480151828 whelp, banning packaging 22.0 on hatch-mypyc's side doesn't work actually
the environment holding the build backend (hatchling) is isolated from the environment used to perform the build?
hatchling (which is generating the tag) is using packaging 22.0 https://github.com/ichard26/black-mypyc-wheels/actions/runs/3809064130/jobs/6480151828#step:5:196 while hatch-mypyc is not (which doesn't handle tags at all)
I'm confused how pip wheel works..
umbral bridge
huh? ๐ค
umbral bridge
I'm confused how pip wheel works..
try build
turbid bane
"Installing build dependencies" and "Installing backend dependencies" are apparently two different isolated operations
turbid bane
try `build`
will do
umbral bridge
actually I'm not sure why #cibuildwheel doesn't default to build
turbid bane
they will one day https://cibuildwheel.readthedocs.io/en/stable/options/#build-frontend
annoying build doesn't log what specific packages it's installing
umbral bridge
-v
turbid bane
how do I tell CIBW that? should I up CIBW_BUILD_VERBOSITY?
umbral bridge
eh, just see if it works first
turbid bane
- python -m build /Users/runner/work/black-mypyc-wheels/black-mypyc-wheels --wheel --outdir=/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-so3_uei9/cp37-macosx_x86_64/built_wheel --config-setting=-v
it passed -v to the backend, but not build itself lol
umbral bridge
haha
about that, Hatchling will never support config settings. I think environment variables plus config file adequately satisfy every use case
turbid bane
damn it's still broken, clearly hatchling is calculating the platform tag while packaging 22 is still installed ... https://github.com/ichard26/black-mypyc-wheels/actions/runs/3809064130/jobs/6480332844#step:5:183
shut mantle
any reason you're not using 21.3 for hatchling as well?
turbid bane
we were expecting banning packaging 22.0 from hatch-mypyc's side would work, but yeah it looks like we'll have to ban it in [build-system].requires instead :(
umbral bridge
hmm wait
try hatch-mypyc>=0.14.1
what I'm thinking is that since the plugin deps are installed after Hatchling itself that packaging version 22.0 is installed then maybe dependency resolution sees that the latest plugin version disallows that version of packaging and installs an older version of the plug-in
turbid bane
`-v`
build doesn't come with a -v flag
I had to install it locally to confirm, that's bad for debugging ...
optional arguments:
-h, --help show this help message and exit
--version, -V show program's version number and exit
--sdist, -s build a source distribution (disables the default behavior)
--wheel, -w build a wheel (disables the default behavior)
--outdir OUTDIR, -o OUTDIR
output directory (defaults to {srcdir}/dist)
--skip-dependency-check, -x
do not check that build dependencies are installed
--no-isolation, -n do not isolate the build in a virtual environment
--config-setting CONFIG_SETTING, -C CONFIG_SETTING
pass options to the backend. options which begin with a hyphen must be in the form of "--config-setting=--opt(=value)" or "-C--opt(=value)"
this is all #build has
shut mantle
build is verbose by default
turbid bane
The conflict is caused by:
hatch-mypyc 0.14.1 depends on packaging!=22.0
The user requested (constraint) packaging==22.0
https://github.com/ichard26/black-mypyc-wheels/actions/runs/3809064130/jobs/6480449164#step:5:225 are you kidding me
turbid bane
build is verbose by default
on failure only
umbral bridge
what is setting that constraint?
turbid bane
build it appears?
ERROR Command '['/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-env-mo9_8dmf/bin/python', '-Im', 'pip', 'install', '--use-pep517', '--no-warn-script-location', '-r', '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/build-reqs-4_3a2ccm.txt']' returned non-zero exit status 1.
I meant that it always shows build backend's output
which seems verbose to me but maybe hatch isn't as verbose so it doesn't seem that way
I guess it doesn't show output from package installation
turbid bane
ah okay that makes more sense
shut mantle
when I run python -m build on setuptools-based project I get like 6k lines of output lol
turbid bane
somehow I can't reproduce the conflict locally
shut mantle
which is mostly it reporting on every file it copies :D
umbral bridge
I guess as @shut mantle mentioned you should set the desired packaging version range in the requirements of the build system
turbid bane
why?
Also I do actually have a script that designed to patch pyproject.toml during a build, looks like it'll come in handy for this lol
shut mantle
because cibuildwheel pins packaging for something too
maybe it happens to be in this step
though I guess build should be isolated anyway
turbid bane
I don't undestand how that'll affect build which shouldn't care one bit about the external environment, but sure I can try that.
umbral bridge
I guess if it pins 22.0 then that will exist in pip's cache and will be what the isolated environment uses by default
turbid bane
If this is the case, my gosh this is the most involved problem I've ever encountered in my life :D
shut mantle
it got through cpy37, that's progress, right
turbid bane
lol you're stalking my repo ๐
shut mantle
I only managed to open it mid-build
turbid bane
I was about to say it seems to be working somehow
shut mantle
it's stupid that you can't see GH workflow step's logs in full until it ends
it streams new lines but doesn't give ones that showed up before you opened the page :(
turbid bane
umbral bridge
I'm not familiar with its architecture but I can imagine that would require streaming access to permanent storage, which does actually seem tough
turbid bane
Successfully built black-22.12.1.dev29+g41572bf-cp38-cp38-macosx_10_16_x86_64.whl
looks like it's using packaging 21.3
shut mantle
every other CI platform I can think of does this fine is all I'm saying
turbid bane
Python 3.9+ isn't affected by packaging 22.0's changes to its macOS tag logic, only Python 3.7 and 3.8 are.
umbral bridge
so what was the final fix?
turbid bane
banning packaging 22.0 on hatch-mypyc's side, requiring hatch-mypyc>=0.14.1 and using CIBW 2.11.3 ๐ฅด
shut mantle
seems like this is the highest voted discussion matching my problem from the ones on GH community discussions: https://github.com/community/community/discussions/35363
turbid bane
well okay maybe requiring hatch-mypyc>=0.14.1 isn't necessary anymore but I'm scared to touch this to be honest :)
umbral bridge
lol try to remove those one by one to pinpoint what the actual fix is
turbid bane
I'll remove the hatch-mypyc pin first.
umbral bridge
using CIBW 2.11.3
I'm guessing it was this
it's a bummer that anyone using that plug-in that is not in discord will experience this
turbid bane
restoring the old hatch-mypyc lower bound (to >=0.13.0): https://github.com/ichard26/black-mypyc-wheels/actions/runs/3809368739/jobs/6480684712
banning hatch-mypyc==0.14.1 which bans packaging 22.0: https://github.com/ichard26/black-mypyc-wheels/actions/runs/3809371923/jobs/6480689866 (GHA you could've assigned the first available macOS worker to the x86_64 build... >.<)
shut mantle
you can cancel a single job
turbid bane
I do want to see if these builds fully pass so I don't really want to do that
turbid bane
I'll use these runs as proof when I file the Black PR reenabling the macOS mypyc wheels that everything works
so restoring the old lower bound on hatch-mypyc seems to work properly (presumably it's still installing 0.14.1 and therefore packaging 21.3)
currently waiting to see what happens when hatch-mypyc 0.14.1 is banned
turbid bane
would not be nice if something else broke and you didn't know
given all of the moving parts involved here, I would not be surprised if "something else" breaks ๐
turbid bane
> using CIBW 2.11.3
I'm guessing it was this
huh, hatch-mypyc!=0.14.1 actually worked. I'm surprised it's using packaging 21.3 AFAICT https://github.com/ichard26/black-mypyc-wheels/actions/runs/3809371923/jobs/6480689866#step:5:165
umbral bridge
@tropic void I just got the email that it is now designated as critical; I had no idea it was being depended on that much already!
turbid bane
What's the threshold again?
umbral bridge
top 5%?
turbid bane
Also you've been migrating projects to hatch and hatch-vcs, you probably can blame yourself for a big part of this :)
umbral bridge
turbid bane
(lol yay people accept my new 
emoji ๐
)
umbral bridge
1/3 of the downloads came in just the last month lol
turbid bane
:O
umbral bridge
huh, hatch-mypyc!=0.14.1 actually worked. I'm surprised it's using packaging 21....
what's the fix? other projects using it are broken for macos
turbid bane
apparently use cibuildwheel 2.11.3? I don't understand why this works though (caching?) since in theory packaging 22.0 should be installed
umbral bridge
turbid bane
Yeah, I might just continue to use my script to patch pyproject.toml to disallow packaging 22.0
I don't think it's possible for the build backend to avoid this issue unfortunately :(
(well unless build changes how it installs the build dependencies after the backend dependencies, but shrug)
umbral bridge
does setuptools vendor packaging?
or does it use custom logic?
I need to find a solution for users that doesn't involve them waiting for pip
turbid bane
does setuptools vendor packaging?
looks like it does https://github.com/pypa/setuptools/tree/main/setuptools/_vendor/packaging
trying to figure out where it calculates the platform tag ATM
umbral bridge
so you're saying in theory that when pip upgrades packaging that macos will be broken for everyone as well?
I would imagine that that is unacceptable lol so now I am thinking packaging should fix this
turbid bane
it shouldn't. I was worried that pip w/ packaging 22.0 won't recognize macox_10_16 as a compatible platform tag but it will still
turbid bane
the breakage is only unidirectional, packaging 22.0 recognizes more versions than packaging 21.3 as compatible (and the new ones are the more specific, hatch selects it and causes pip w/ packaging 21.3 to not recognize it as compatible)
turbid bane
when setuptools upgrades packaging
uhhh it seems like their platform tag logic is different
umbral bridge
uhhh it seems like their platform tag logic is different
๐ฆ link?
turbid bane
So the setuptools job worked, producing these wheels:
```
black-22.12.1.dev29+g1...
it produced these platform tags which doesn't look like the most specific tag?
tbh no idea how it decided on macosx_10_9
turbid bane
๐ฆ link?
still digging through the setuptools source to figure this out
https://github.com/pypa/setuptools/blob/f0e72c608b2925f302746236126826ef272b7328/setuptools/_distutils/util.py#L51 this is the entrypoint where it detects the platform tag
umbral bridge
ughh distutils
I guess I can copy the logic from distutils but then like what's the whole point of that packaging function that calculates the most specific tag?
turbid bane
I don't know how platform tags work well enough to offer any advice ยฏ_(ใ)_/ยฏ
distutils uses sysconfig.get_platform on macos
umbral bridge
which is also what packaging does based on my glance over the code recently
umbral bridge
I might hack around this
umbral bridge
@turbid bane I'm fixing this right now. does this affect every Python version?
turbid bane
At least Python 3.7, probably 3.8, probably not Python 3.9 and higher. Do you want me to verify?
Actually I'll just check now, I still have the GHA logs open.
turbid bane
The difference only occurs on Python 3.7 and 3.8. On macOS 12, packaging 22.0 al...
Looks like it should be Python 3.7 and 3.8 only, packaging 21.3 and 22.0 provide the same tags on Python 3.9+
umbral bridge
so just to clarify my plan then is to introduce an option that is true by default (that will become false sometime in the future) that on Python version 3.7 and 3.8 will check the version part of the tag and if it is >= 11.0 then it will be changed to 10.16
Is that correct?
(if the build hook has enabled that tag inference option of course)
turbid bane
That sounds like it'd work around the issue , but I'd appreciate one more pair of eyes.
The problem afterall is that pip doesn't recognize 11_0 (on macOS 11 and 12) or 12_0 (on macOS 12) since it's only treated as a compatible platform tag version by packaging 22.0 (which it does not vendor ATM)
umbral bridge
Looks like it should be Python 3.7 and 3.8 only, packaging 21.3 and 22.0 provide...
for my own understanding: by the same tags you mean 10.16 right?
turbid bane
I mean they provide the same most specific set of platform tags (11_0 on macOS 11, and 12_0 on macOS 12) which are the ones relevant to hatchling.
By most specific set, I mean at the top of the list (although I should double check the API documentation if this is a valid assumption)
umbral bridge
Oh wait sorry are you saying then that on Python version 3.7 and 3.8 that the tag has 10.16 but after that it starts using 11, 12, etc.?
turbid bane
hmm, this is for mac_plaforms() only, hatchling uses something else doesn't it? I mean whatever packaging API hatchling is calling, it probably uses mac_platforms() under the hood, but I'll double check
umbral bridge
I guess I'll just do this on all Python versions
turbid bane
Oh wait sorry are you saying then that on Python version 3.7 and 3.8 that the ta...
Yea, because it's the first tag provided by mac_platforms()
umbral bridge
thanks
turbid bane
sys_tags() calls cpython_tags() which calls platform_tags(), finally calling mac_platforms() https://github.com/pypa/packaging/blob/5c70ed4f983c4b27e67c34fe67266e1514484bf0/src/packaging/tags.py#L211
it looks like the first tag returned by sys_tags() should use the first platform tag returned by mac_platforms()
turbid bane
Oh wait sorry are you saying then that on Python version 3.7 and 3.8 that the ta...
And just to build on top of this, packaging 22.0 changes the macOS logic where the first tag returned by sys_tags() has 11_0 instead of 10_16 (on Python 3.7 and 3.8)
I hope that clears things up ๐
packaging isn't in the wrong AFAICT (unfortunately for us, the OS version is misreported as 10.16 when it's actually higher in certain situations), but by recognizing additional platform versions (as appropriate), there's a possibility for a mismatch where the consumers don't know those (newly recognized) platform versions are compatible depending on what version of packaging they're using
half cedar
packaging.tags is mostly for answering does this tag work here, and for the purpose of find the best tag for this platform you need different logic
A while ago I was wondering if wheel tags can be used to tag a Python interpreter and reached this conclusion eventually
turbid bane
hatchling currently lets packaging.tags.sys_tags() decide what's the best tag for the platform, it uses the first tag returned which is documented to be the "best-matching tag"
whether that's the best way of deciding the tag for a wheel during a build, I dunno
half cedar
Yeah I was doing the same as well and was essentially told thatโs not guaranteed to work, I think by Brett or someone else
umbral bridge
@turbid bane https://github.com/pypa/hatch/pull/699
winged cargo
is there any way to use distutils.build_ext from hatch
or at least just do stuff with CCompiler
umbral bridge
or at least just do stuff with `CCompiler`
winged cargo
yeah thats what im doing
i just cant figure out what to do for tags and whatnot
was hoping i could just get distutils to do it entirely
crystal idol
Does packaging and/or https://github.com/ofek/extensionlib not do what you need, as opposed to relying on ancient, deprecated and to-be-gone distutils?
winged cargo
truly, i have no idea how to use extensionlib, the docs arent very clear
i dont really know much about packaging, im just trying to get my project to build properly
hatch is still pretty new to me, ive always relied on the old setup.py
crystal idol
It would seem to me you may have an XY problem, where X is distutils and Y is what you actually want to do with it, which at least to me has not been made clear. If you could explain your needs further, it would likely help others here in giving much more useful and specific advice. On thing I certainly can say is your project won't build properly (or at all) in Python >=3.12 if you're still relying on distutils, of course, so you're going to need to migrate to something else very soon.
In fact, it may even be an XYZ problem, as Hatch may or may not even be the best backend to choose for your particular project's needs, given it appears to be heavy on building extension modules, which as far as I understand (correct me if I'm wrong @umbral bridge ) is not yet fully supported by Hatch directly https://github.com/pypa/hatch/issues/591 . For projects building extension modules, you might be better off with what I assume would probably be a much simpler migration to Setuptools for now, making sure your metadata and config are specified declaratively in pyproject.toml and pulling as much bespoke custom logic out of your setup.py as practical (since likely a lot of it isn't necessary under modern Setuptools), which will keep everything working in the short term while making a migration to Hatch, MesonPy, Scikit-Build (CMake), or another more modern tool much easier in the medium to longer term once they're fully mature.
winged cargo
It would seem to me you may have an XY problem, where X is distutils and Y is wh...
essentially, i just want to build some c code into a python module. through setuptools, this is done through setuptools.Extension. through some research, ive found that this originates from distutils.Extension, along with distutils.ccompiler.CCompiler. as of now, i have it building properly through setuptools' version of distutils (setuptools._distutils.ccompiler.CCompiler), but i cant get python to actually load it properly because i dont know what to put for the platform tags and such. i would like to continue using hatch, since i dont really like poetry and plain setuptools is a bit featureless. ive mixed hatch and setuptools before (this is done in this repo https://github.com/ZeroIntensity/pointers.py), but ive found that it ends up being pretty buggy.
umbral bridge