#importlib-metadata

1 messages · Page 1 of 1 (latest)

stiff pine
#

Hi, coming here from the discussion on https://github.com/python/importlib_metadata/issues/115. Trying to setup a developer env for importlib_metadata, are there some docs for this somewhere? Failed to find anything in the repo itself, so far, and my python3.11 -m venv .venv followed by pip install tox and tox is far from green.

worldly crypt
#

there aren't any docs unfortunately

#

yeah, the recommended way to run the test suite is via tox

#

if it's green, then it should be good

#

then you want to add a test in test_main.py

#

instead of RECORD, you'll want to have your source

stiff pine
#

Thanks a lot for your pointers. Very useful!

Would be interested to know how you set up your development environment for a project like this. For example, given a new laptop, roughly what steps would you follow to get to a green tox run?

worldly crypt
#

eh, I usually just run pytest directly

#

and tox if I am having trouble for some reason

#

to make sure it's not my environment that's the issue

stiff pine
#

I dug a little further into my tox failure. Haven't gotten to the bottom of it yet, but I think there's a problem when combined with the latest Mypy release (1.1.1): tox -- -vv tests/test_main.py fails on tests/test_main.py::mypy-status with:

importlib_metadata/__init__.py:435: error: Incompatible return value type (got "Message", expected "PackageMetadata")  [return-value]
importlib_metadata/__init__.py:435: note: Following member(s) of "Message" have conflicts:
importlib_metadata/__init__.py:435: note:     Expected:
importlib_metadata/__init__.py:435: note:         def [_T] get_all(self, name: str, failobj: _T = ...) -> Union[List[Any], _T]
importlib_metadata/__init__.py:435: note:     Got:
importlib_metadata/__init__.py:435: note:         @overload
importlib_metadata/__init__.py:435: note:         def get_all(self, name: str, failobj: None = ...) -> Optional[List[Any]]
importlib_metadata/__init__.py:435: note:         @overload
importlib_metadata/__init__.py:435: note:         def [_T] get_all(self, name: str, failobj: _T) -> Union[List[Any], _T]
Found 1 error in 1 file (checked 1 source file)

But with this diff the same tox command passes:

diff --git a/setup.cfg b/setup.cfg
index f5c7280..52eef95 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -45,6 +45,7 @@ testing =
        pytest-mypy >= 0.9.1; \
                # workaround for jaraco/skeleton#22
                python_implementation != "PyPy"
+       mypy < 1.1
        pytest-enabler >= 1.3
 
        # local
worldly crypt
#

yes, I noticed this earlier

#

you can ignore that error

stiff pine
#

Thanks. Running tox across the entire project, I ran into another failure in exercises.py::exercises::py:cached distribution. It failed while trying to run this:

subprocess.CalledProcessError: Command '('/home/jherland/code/importlib_metadata/.tox/python/bin/python', '-m', 'pip', 'install', '-t', PosixPath('/run/user/1000/pip-run-aofstq26'), 'importlib_metadata.git[perf]@git+git@github.com:python/importlib_metadata.git')' returned non-zero exit status 1.

The importlib_metadata.git[perf]@git+git@github.com:python/importlib_metadata.git specifier/URL does not make sense, and I traced it back to the origin URL for my git config:

$ git config remote.origin.url
git@github.com:python/importlib_metadata.git

I originally cloned this project using the Git URL suggested on the GitHub repo page. If I run with the suggested HTTPS URL instead:

https://github.com/python/importlib_metadata.git

the same command fails with a slightly different specifier/URL: importlib_metadata.git[perf]@git+https://github.com/python/importlib_metadata.git

Finally, removing the .git suffix from the HTTPS URL by hand, (git remote set-url origin https://github.com/python/importlib_metadata), I can get that test to pass. It seems somewhat brittle to have the tests rely on the origin URL in this manner?

worldly crypt
#

hummm, interesting

#

I haven't run into that before

#

I wouldn't care about it much though, that's just the performance tests

#

to make sure we are aware if there's any significant performance regression

stiff pine
worldly crypt
#

thanks! looks good to me overall, only thing that jumps out to me is that you shouldn't really need all that metadata in the tests

#

but I'll do a proper review later when I am at the computer

stiff pine
#

Thanks, I mostly copied from the example above, and I agree that a lot of the metadata can be dropped.

stiff pine
#

(refreshed the PR with unnecessary details removed from the tests, as well as some black fixes that escaped the first iteration)

stiff pine
#

I think I found a problem with https://github.com/python/importlib_metadata/pull/432 which was merged to main recently. I added a comment to the PR, but I don't know if that is enough to raise people's attention, so I mention it here as well.

GitHub
Add missing modules to packages_distributions by FFY00 · Pull Reque...

This now considers:

Modules that don't have the .py prefix (eg. native modules, .pyc-only modules)
Namespace packages without any .py files inside

Which I think should be all possible modules...

signal topaz
#

looks like it's missing a str.isidentifier check. I'm also seeing .. and __pycache__. The latter shouldn't be added in the RECORD I thought?

worldly crypt
#

yep, I overlooked that and it didn't get caught in the review

main turtle
#

@sand arrow whats the correct way to "override" a installed distirbution with a dnyamic metadata finder

i would like to have something that when given a editable install, it replaces whatever uses only the dist.info - and also uses the pyproject.toml and setuptools_scm/vcs_versioning to get the metadata of a editable package (aka autoupdate entrypoints and/or version)

sand arrow
sand arrow
#

You can extend them by adding a MetadataDistributionFinder to sys.meta_path (or insert at the beginning for priority). Entrypoints will de-duplicate based on normalized distribution name, so you'll only get entry points for the first one discovered.

#

So for entry points, extending the finders will effectively override.

main turtle
#

So having a "enhanced" distribution discovered first is the implied mechanism

sand arrow
main turtle
sand arrow
#

I'm unsure how well tested the expectation of "primacy of duplicate distributions" is. I wouldn't mind if someone wanted to add tests to importlib_metadata to properly capture an expectation of one distribution superseding another.

main turtle
#

Another question i have, how does one best add fake namespace packages to sys.Modules i want to add worktree testroots to the import system in a manner that allows pep420 aware test collection in testing/tests folders without adding their parent to sys.path

main turtle
sand arrow
#

It's been forever since I've worked on the namespace packages implementation. As much as possible, namespace packages are meant to appear just like regular modules with as small as possible variance for their purpose.

#

So probably, you create a types.ModuleType(name) and then assign mod.__spec__.loader = NamespaceLoader() with loader._path set to some list of paths served by that namespace and you'll have something that renders as a namespace package. You may have to initialize other attributes too, like __origin__=None.

#

The loader._path is probably [parent], where parent is the parent of the testing/tests folders.

#

I hope that helps.

main turtle
#

@sand arrow thanks for the links, it seems i just need a finder that returns a spec for the folder either as namespace or as package (depending on whether __init__.py is added

main turtle
#

@sand arrow any oppinion on adding full type annotations directly to importlib_metadata - the current lack is a minor pain in other fully typed codebases and im under the impression its doable in a complete manner

main turtle
worldly crypt
#

yeah, we should do that for the public API at least

#

we already do it for importlib_resources

signal topaz
sand arrow
#

I'm supportive of it. The rejection in 342 was mainly because adding the typing required changing the logic. I'd start with uncontroversial changes that don't require altering the behavior.

#

It'll also be easier once some of the deprecated interfaces are removed.

#

You're also welcome to start from 342 and address the comments in the PR.

worldly crypt
#

I think a good first step would be to just type the public interfaces, and then go from there if it makes sense. https://github.com/python/importlib_metadata/pull/342 attempted typing everything right away, which unfortunately meant that the most beneficial contribution, having the public API typed, got blocked together with the rest of the PR.

GitHub
Complete type annotations by andersk · Pull Request #342 · python/i...

Add type annotations to every function, and correct some existing annotations, so that we pass mypy in strict mode. This makes the annotations more useful to downstream projects and improves our co...

main turtle
#

Yikes, that one had multiple breaking changes

#

Based on the detail /approach it seems like 342 is not a suitable base 😱

main turtle
#

anyone aware of examples for remote passover of importlib metadata metadata? im trying to add a dict/json meta path importer that allows send in of a distribution for bootstrapping purposes

main turtle
#

anyone aware of a potential solution for https://github.com/pypa/setuptools-scm/issues/1013#issuecomment-2814014441

my current understanding (unverified) is that due to version numbers in dist info it seems that with a sufficiently messed up pythonpath importlib-metadata will find entrypoints of incompatible versions that will fail to import as packages shadow, but dist-infos have unique distinc names

GitHub
8.0.4 fails to install with Python 3.8/3.9 (but works fine with Pyt...

Is it even supported for Python <= 3.9? It fails for me: ---> Configuring py38-setuptools_scm ---> Building py38-setuptools_scm Executing: cd "/opt/local/var/macports/build/_opt_local...

#

as far as i understand any ports like system is affected due to pythonpath interaction

worldly crypt
#

I think this is a setuptools, not ours

#

I commented on the issue

main turtle
#

thanks

main turtle
raven cosmos
#

I’ll happily make a CPython docs update to fix this once I know what’s actually true

worldly crypt
#

but yeah, I agree the CPython documentation is very lacking

sharp steeple
#

there are also significant drifts between the stdlib version and the "upstream" one. Is there a plan to unify the development into the stdlib version only?

grand vector
#

(aside: so I then ended up on that 2016 pep517/pyproject-hooks rabbithole, and then on setuptools and then distutils and somehow I was reading py3k peps on this journey of curiosity... But I digress)

worldly crypt
#

yeah, there are a couple places that happens, but it's very rare

#

it should only ever be the case for optional things that would be almost impossible to implement without using 3rd party packages

#

other example is datetime accepting the tzdata package as a source of the tz database

sand arrow
# sharp steeple there are also significant drifts between the stdlib version and the "upstream" ...

The drift should be small and definitively indicated in https://github.com/python/importlib_metadata/compare/main..cpython .
To the extent that there are changes downstream in stdlib, those need to be ported to importlib_metadata.

GitHub
Comparing main..cpython · python/importlib_metadata

Library to access metadata for Python packages. Contribute to python/importlib_metadata development by creating an account on GitHub.

raven cosmos
rain valve
sand arrow
#

@mossy kayak How about here?

#

(I just created the channel)

#

I replicated your issue:

~ $ pip-run backports.entry_points_selectable -- -c "import importlib.metadata as md; md.distribution('backports-entry-points-selectable')"
Collecting backports.entry_points_selectable
  Using cached backports.entry_points_selectable-1.1.0-py2.py3-none-any.whl (6.2 kB)
Installing collected packages: backports.entry-points-selectable
Successfully installed backports.entry-points-selectable-1.1.0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/metadata.py", line 524, in distribution
    return Distribution.from_name(distribution_name)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/metadata.py", line 187, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: backports-entry-points-selectable
#

Confirmed - that issue was fixed in importlib_metadata 2.1.

# 
~ $ pip-run backports.entry_points_selectable 'importlib_metadata<2.1' -- -c "import importlib_metadata as md; md.distribution('backports-entry-points-selectable'
)"
Collecting backports.entry_points_selectable
  Using cached backports.entry_points_selectable-1.1.0-py2.py3-none-any.whl (6.2 kB)
Collecting importlib_metadata<2.1
  Using cached importlib_metadata-2.0.0-py2.py3-none-any.whl (31 kB)
Collecting zipp>=0.5
  Using cached zipp-3.5.0-py3-none-any.whl (5.7 kB)
Installing collected packages: zipp, importlib-metadata, backports.entry-points-selectable
Successfully installed backports.entry-points-selectable-1.1.0 importlib-metadata-2.0.0 zipp-3.5.0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-satw542a/importlib_metadata/__init__.py", line 558, in distribution
    return Distribution.from_name(distribution_name)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-satw542a/importlib_metadata/__init__.py", line 215, in from_name
    raise PackageNotFoundError(name)
importlib_metadata.PackageNotFoundError: No package metadata was found for backports-entry-points-selectable
# 
~ $ pip-run backports.entry_points_selectable 'importlib_metadata==2.1' -- -c "import importlib_metadata as md; md.distribution('backports-entry-points-selectable
')" && echo works \o/
Collecting backports.entry_points_selectable
  Using cached backports.entry_points_selectable-1.1.0-py2.py3-none-any.whl (6.2 kB)
Collecting importlib_metadata==2.1
  Using cached importlib_metadata-2.1.0-py2.py3-none-any.whl (10 kB)
Collecting zipp>=0.5
  Using cached zipp-3.5.0-py3-none-any.whl (5.7 kB)
Installing collected packages: zipp, importlib-metadata, backports.entry-points-selectable
Successfully installed backports.entry-points-selectable-1.1.0 importlib-metadata-2.1.0 zipp-3.5.0
works \o/
mossy kayak
#

Yeah, but in the above case importlib.metadat is pulled from python3.9 I think

sand arrow
#

And since importlib.metadata on Python 3.9 is equivalent to importlib_metadata==1.4, that explains why the issue occurs on Python 3.9.

mossy kayak
#

😐 this leaves us in a weird place 😄 do I need to not use importlib-metadata from stdlib on 3.9? :-~

sand arrow
#

You can. And would that be so weird?

#

If you want a feature that's available in Python 3.10 on older Pythons, you use a backport.

#

And importlib_metadata is that backport.

mossy kayak
#

shouldn't backports-entry-points-selectable handle this automatically

sand arrow
#

That backport was only created for a specific feature that needed special handling (selectable entry points). It doesn't back port anything else.

#

Still, I would recommend not using manglednormalized names.

#

Is there a way that the routine that's encountering this issue could instead simply use the official names for the package (backports.entry_points_selectable instead of backports-entry-points-selectable)?

#

The latter is used to de-duplicate for PyPI (PEP 503) but it's preferable to use the canonical name, which communicates more information than the normalized name.

mossy kayak
sand arrow
#

That seems like best approach, at least for compatibility.

glass glen
#

is there a way to read the metadata from a zip?

slim tundra
#

Something like this

dists = list(Distribution.discover(path="path/to.zip"))
glass glen
#

i get no errors but the result is an empty list

worldly crypt
#

then use zipfile.Path which I linked you in the other channel

#

importlib.metadata.PathDistribution(your_zip_path)

#

it should discover

#

example

signal topaz
#

*assuming you're using Python 3.8+

worldly crypt
#

there's a backport so

signal topaz
#

you can also shorten it to list(distributions(path=["path/to.zip"]))

slim tundra
glass glen
#

thanks so much for the help @worldly crypt @signal topaz @slim tundra

worldly crypt
#

no problem

slim tundra
#

What is the best way to load a Distribution object from a wheel zip? It’d be best if the loading function also checks the name and version match, but that I can do myself if importlib-metadata does not do it

worldly crypt
#

it doesn't, it's very annoying

slim tundra
#

:(
So I guess manually fetching out the dist-info directory’s name first (using ZipFile) and instantiating with PathDistribution(zipfile.Path(zf, dist-info-name)) is the best possible approach for now?

worldly crypt
#

yes