#pkgconf-pypi

1 messages · Page 1 of 1 (latest)

pliant imp
rigid snow
#

@graceful ether This is what you were looking for before, I think, a redistribution of pkg-conf for PyPI?

graceful ether
#

oh it doesn't provide binaries itself but helps with building packages that do? the package I mentioned I think is still what I need because it provides the binaries for Windows

rigid snow
#

provides a pre-built pkgconf binary
I took it this to mean it had binaries too.

graceful ether
rigid snow
#

I think it has entry points. Unfortunately that means the binaries are "wrapped" in Python. ninja is moving to providing the native binary instead of Python entry points. I don't think cmake can, since it needs to be able to access a support directory.

pliant imp
#

yes, it has entrypoints, which wrap the pkgconf binary and allow packages to register their own .pc files

pliant imp
#

@rigid snow what's the issue in the binary being wrapped in an entrypoint?

#

we also have a pkgconf.get_executable() to access the original executable

#

the idea moved from originally just publishing the binaries to also adding a Python integration layer to allow Python packages to register their pkgconf/pkg-config paths

rigid snow
#

You can still put the executable in /scripts and then access it from site-packages. It just doesn't work the other way. The issues are you have to pay the startup cost of Python to launch the executable, and sometimes the fact that it is wrapped leaks through, such as issues with file descriptors being closed in https://github.com/scikit-build/cmake-python-distributions/pull/303 and I think I've seen another wrapping issue that hasn't been fixed yet (forget which package it was for)

#

Also, by adding more stuff but also being a redistrubiton, you are going to eventually get all the package managers complaining. Fedora is currently writing out fake .dist-info files for cmake and ninja to get package checks like the one in build to pass, but the package isn't actually installed and there are no Python components. 🤦

#

(Those aren't huge drawbacks, and we do pay them for cmake, since it needs to access files. We don't pay them for ninja anymore)

pliant imp
#

I made sure to make its path available via the Python module though

pliant imp
left sorrel
#

What's the status of this project @pliant imp ? I'm working on migrating my robotpy projects to meson (and getting frustrated by it's limitations wrt codegen, lol), but I think pkgconf-pypi is likely a good solution for making wheels that can link to each other (and I have made proof of concepts that work); I'm still working refactoring my own project internals, but at some point I'm going to want to depend on this project. If it's no longer useful for you and you don't want to maintain it, I'm happy to take it over or alternatively start publishing my own fork

pliant imp
#

sorry, just been a bit overwhelmed lately

#

the project is still being maintained, but I need to catch up

#

I assume you want the loop issue resolved, right?

left sorrel
#

especially that one

#

like I said, I'm still doing everything local right now, so it's not a big deal yet, but at some point in the next month I expect it will be

left sorrel
#

I'm going to fork this @pliant imp since I don't expect that you'll give me commit/release access on your repo (though I will point out that since it's unlikely anyone is using this, it's not that big of a risk). My hope is that eventually you'll take my changes and integrate them, so I'm not going to change the entrypoint or the package name (hopefully nobody installs yours and mine at the same time!). I haven't decided on a pypi name yet, unfortunately pkgconf is the most obvious name and other obvious choices are taken, and pkgconf2 would be annoying if development starts again on pkgconf.

pliant imp
#

ugh, sorry for the delay, I have been a bit overwhelmed

#

I'll put this on top of my priority list

#

but I think we can add you to the project

gloomy dust
#

completely off-topic but I hope things get better for ya soon(-ish?) 🫂

#

I know I mentioned you like 3 days ago on a pip PR, it is absolutely not important

pliant imp
#

yeah, I purposely didn't reply because there weren't any concerns from our part and I wanted to avoid interacting with Eli (used to be involved in arch but got removed due to repeated CoC violations)

#

but yeah, back to pkgconf-pypi, I have a working prototype of PEP 739 in Meson, so getting pkgconf-pypi up to speed was one of the followups to that, as it'll make cross-compiling much easier, which is the overall goal of the work I am doing

#

so with that mostly dealt with, I am happy to bump pkgconf-pypi up the priority list @left sorrel

#

FWIW, we are planning to use this in Numpy and SciPy

#

this particular project just got backlogged until I dealt with the other pieces

left sorrel
#

lots of pieces everywhere

#

I have another improvement I'll push up in a PR in a bit

ebon shale
#

Oh, never mind. Looking a bit closer at the GitHub repo makes it clearer.

pliant imp
#

it could still be better clarified

#

also, here's an example scenario that someone asked about on discuss where pkgconf-pypi could be helpful, https://discuss.python.org/t/shipping-common-libraries-in-a-dedicated-wheel/76407/2

Discussions on Python.org
Shipping common libraries in a dedicated wheel

Numpy might serve as a sort-of example. It doesn’t use a separate wheel but if does use a separate package and supports being linked against. You’re unlikely to find clean, officially endorsed solutions to this. The inability to link across wheels is one of the reasons why conda exists (and has to dismantle and reassemble so much to make it hap...

#

@left sorrel I went over the PRs and also invited you to the repo

left sorrel
#

thanks. My turn to disappear, we're heading on vacation for 8 days starting tomorrow morning. I'll be bringing my laptop, so I might work on things, but might not.

pliant imp
#

@left sorrel Ralf should be inviting you to the org soon, I just added you directly to the repo temporarilly because I don't have permissions for the org

left sorrel
#

I'm hoping to get back to hammering on all of this in the near future; FRC season is upon me, so I've been doing more bugfixing for my existing projects rather than the build-system refactoring I want to finish.

#

I am interested to see where you end up with your dynamic-library stuff, since it's effectively the same problem I've been solving for robotpy-build for the last 4 years, but my existing solutions aren't particularly elegant

left sorrel
#

coming back to robotpy-build refactor. The refactored build system will be a series of hatchling plugins, some of which register pkgconf entry points, eventually ending up in hatch-meson which consumes them. So I think I need some way to add the registered entry point in a way that pkgconf can find it before/during the wheel building. Thinking to add a convenience function to pkgconf that will just update os.environ['PKG_CONFIG_PATH'] with the new .pc locations so they can be found in-build.

#

Not 100% sure that's what I actually need, we'll see how far I get

left sorrel
left sorrel
#

Hm. pybind11 requires python headers, so the file I added technically isn't correct. However, it would also fail to find python3 because it's not on our pkgconf path. I wonder if we should add pythons PC files to the path so it could be used this way.

#

The counter argument is that python is going to be special anyway, so you're probably already linking to it via your build system

#

.. leaning towards adding it, seems more consistent

pliant imp
#

is the python in your test setup statically linked?

#

because I am struggling to see why it would find libpython.so, but not python3.pc

#

but yeah, in cases where there may be installation configuration mismatches, it would be good to ensure the correct python3.pc is found

#

that said, we should put it at the very end of the PKG_CONFIG_PATH, so that .pc files in /usr/lib/pkgconfig for eg. don't take precedence over files in our environment

left sorrel
#

In my test setup I was just using meson's extension_module to compile, so it would add the python include paths automatically. I was running into problems when I switched my header parser from pcpp's preprocessing over to gcc's... which was when I realized that pybind11's pc file should probably depend on python.

#

I think that's a really good point about not wanting to add the system pkg_config_path, so maybe we need a special case here then. What if instead we copied the .pc to a temporary directory... nope, that would break any pc-relative stuff in the .pc file.

#

ok. we could add a special case for python3 and only add it when it's asked for, but then we would need to parse pkgconf's command line, which would be slow.

#

hm, I guess that pybind11 should actually not depend on python3.pc, because that would break people wanting to embed python

#

will have to think about it

#

something that doesn't feel too terrible but probably would be rejected by pkgconf upstream is extending the use of PKG_CONFIG_PATH such that if it contained individual .pc files that pkgconf would use that file without other things in its directory

left sorrel
#

I think for the moment I'm just going to file an issue and ignore this. I have a local workaround that I can do for now. adding the global PKG_CONFIG_PATH definitely seems worse.

pliant imp
#

ah, those were the changes that broke compilation on older systems 😅

#

I had to fix it and backport it for the wheel-cross job to pass

left sorrel
#

I made a PR to pkgconf to support files in the environment variable. It seems pretty straightforward, not sure what side effects it would cause. Not sure if they'll take it or not, guess we'll see.

pliant imp
#

yeah, thank you for that

#

we'll see

left sorrel
#

hm, something in 2.4.3-1 broke me locally

left sorrel
#

Bedtime, but made an issue. Should be straightforward to make a test package to reproduce tomorrow evening.

ashen niche
left sorrel
left sorrel
#

so this isn't something that I personally care about right now (though I did .. wow, almost 10 years ago! https://github.com/opencv/opencv-python/issues/22), but eventually pkgconf-pypi could be a useful part of making it so that people can publish wheels that link to opencv (this comment isn't really aimed at anyone in particular, but rather at passers-by who might have an interest in that)

GitHub
Issues · opencv/opencv-python

Automated CI toolchain to produce precompiled opencv-python, opencv-python-headless, opencv-contrib-python and opencv-contrib-python-headless packages. - Issues · opencv/opencv-python

left sorrel
#

I've finally finished all the refactoring of my robotpy-build projects, and now they all use pkgconf for finding headers/libraries. In particular, https://github.com/robotpy/mostrobotpy builds... 22 different pypi packages/wheels that build off each other to build/load libraries and such. Working on macOS, Windows, and Linux, and cross compiled for Linux/Arm

left sorrel
left sorrel