#extensionlib

1 messages · Page 1 of 1 (latest)

final dune
#

That would be up to the builder, yes. (work is restarting on this)

steady flame
snow widget
#

So it will be awesome when I can deprecate that code and move things to a Python standard

void vale
snow widget
#

Yes!

void vale
gusty path
#

hey @void vale, can we coordinate regarding PEP 739 ?

#

right now I am working on a real-world implementation by adding support for it in Meson, but it should probably be easier enough to implement it in extensionlib too

void vale
#

hey! I haven't read that thoroughly, how do you suppose they would interop?

steady flame
gusty path
#

I think the most impactful use-case for the Python packaging environment is cross-compilation, but everything else that currently needs to introspect Python installations might benefit from this

gusty path
#

that said, I want to followup with a PEP that provides the environment-specific installation

steady flame
gusty path
#

yes

spare ridge
#

One cross-platform annoyance PEP 739 could help with in the general case is finding the Python executable to run. It has a .exe extension solely on Windows, and whether it's in the base directory (the one containing lib), a bin subdirectory, or a Scripts subdirectory can also vary.

The PEP would be nicer if paths could be relative to the JSON file instead of always being absolute, though (it's annoying enough that pyvenv.cfg and script shebang lines have to use absolute paths, we don't want to add more mechanisms that hinder Python installation and environment portability across machines)

gusty path
gusty path
gusty path
# gusty path PEP 739 currently does not specify the interpreter, but I am thinking it may be ...

Expanding a bit more on the Python installation/environment discoverability, the current plan is for PEP 739 to specify all data needed to build Python extensions and all other use-cases where we need to introspect the build configuration of the Python installation, then I will push another PEP introducing a JSON file containing the data needed to introspect a environment (this will include paths to the interpreter, site-packages, stdlib, etc. and a path to the PEP 739 build details file), after that, I don't have a specif plan yet, I would like to standardize a way do discover Python installations (this could be a set of paths where you can place a environment description file, or some kind of pointer to one).

#

the reason for the build and environment separation is that environments (eg. virtual environments) often outlive Python installations/builds

#

with this approach, you can create a virtual environment, update the Python installation, and still have the tools be able to rely on the introspection JSON data

#

if the build and environment data were together in the same file, you could never rely on the build details being correct, as the Python installation might have been updated

void vale
#

still need to write tests

alpine vortex
#

Is the idea that libraries that do the actual compilation register themselves with extensionlib, and I (as a backend) can ask extensionlib to run that builder without knowing anything about the builder itself?

void vale
#

yes precisely

bleak nest
#

@void vale looks interesting, i presume this will superseed/power hatchs mypyc/cython/extension modules integrations

void vale
#

yes

snow widget
#

This would be great. Are you looking for collaborators?

sonic scroll
#

I think yes. At least testers and contributors.

lethal magnet
#

Hi @void vale thank you very much for working on this.
I went through the documentation and it looks very good.

If the idea is to use this library as a proof of concept for a future PEP targeting interoperability between backends, I would have the following (unsolicited 😝, sorry for that, please feel free to simply ignore) suggestions:

  • How about making the ExtensionModules interface a protocol that implementations must adhere to instead of using inheritance?
    I think this is more in-line with the spirit of a PEP and also would free backends from having to add another dependency (which is always complicated given the limitations of PEP 517 in terms of cycles).
  • How about having 2 separated methods (or properties), one for the "input files" and one for the "output files"?
    This way a build backend may decide which files should be added to the sdist (input files) and which files should be added to the wheel (output files).

Additionally, when talking about a possible follow up PEP, I would assume the following:

  • The ExtensionModules interface is the only thing specified by the PEP. The BuildRunner interface is an implementation detail.
  • To be "library-independent" spec, the PEP could replace [[tool.extensionlib.*]] with a similar [[project.extension-modules.*]] table (or any other name really that is not related to a single implementation).
    • If that is the case, maybe it would be required to add a configure(self, options: dict) method to the ExtensionModules interface, so the build-backend can pass information used to derive the values of input_files and output_files. Or simply make a spec for __init__(self, options: dict).
    • The options argument, being an JSON/TOML-serializable dict, whose keys are a combination of arbitrarily defined by each implementation and that are written to the [[project.extension-modules.*]] table.
void vale
#

@lethal magnet thanks!

How about making the ExtensionModules interface a protocol that implementations must adhere to instead of using inheritance?
I can write that in the PEP, but I think the recommendation should be this lib. What isn't shown is that it will have some utilities that pretty much every implementation would end up needing (like sysconfig stuff). @final dune will help with that, it's not my expertise.

How about having 2 separated methods (or properties), one for [...] the sdist (input files) and [...] the wheel (output files).
I immediately love this idea! Though, do you have an example of something currently generated that is intended to be shipped in the sdist?

The ExtensionModules interface is the only thing specified by the PEP. The BuildRunner interface is an implementation detail.
Yup!

To be "library-independent" spec, the PEP could replace [[tool.extensionlib.*]] with a similar [[project.extension-modules.*]] table (or any other name really that is not related to a single implementation).
Yup, that's my exact plan! This is just temporary until there's at least one PoC builder.

If that is the case, maybe it would be required to add a configure(self, options: dict) method to the ExtensionModules interface, so the build-backend can pass information used to derive the values of input_files and output_files. Or simply make a spec for __init__(self, options: dict).
That's part of the build method https://ofek.dev/extensionlib/interface/#extension._interface.ExtensionModules.build

The options argument, being an JSON/TOML-serializable dict, whose keys are a combination of arbitrarily defined by each implementation and that are written to the [[project.extension-modules.*]] table.
That's https://ofek.dev/extensionlib/interface/#extension._interface.ExtensionModules.config

bleak nest
#

@void vale wrt generated in sdist - cythonize comes to mind (when wanting to avoid cython as dep on the consumer)

void vale
#

ah interesting

bleak nest
#

borgbackup does that for example, the sdist for distros gets cythonized sdists that only need a c compiler

#

(its setup.py is horrendously painfull due to these details)

lethal magnet
#

Yes, Cython is a good example. But I believe that other regular builds also have this property right?

For example, if someone writes an extension in C or C++, they probably want the C/C++ files to be in the sdist but not in the wheel.
If the backend can query for input_files, it will be able to exclude them from the wheel.

void vale
#

So is the idea here that you can only call the files method after you run build?
no anytime

sonic scroll
#

Perhaps this should also live under pypa? 🤔 especially if hatchling will use it

void vale
sonic scroll
#

Oh, I'd have accepted it from day 1, but does need a vote 🤣

lethal magnet
sonic scroll
#

I think is fine to develop projects under pypa if we voted to accept it 😄

#

But seems Paul seems to disagree

lethal magnet
void vale
#

btw even though I'm ok with

The BuildRunner interface is an implementation detail.

the runner options, imo, are required

lethal magnet
# void vale btw even though I'm ok with > The `BuildRunner` interface is an implementation ...

In the docs you mentions as an example [[tool.extensionlib.spam]] and then says that the table is named after the plugin, which is the extension_modules entry-point, right? Then in your example of entry point you define an ExtensionModule as plugin.

So does it make sense to call these options runner options? Aren't they the options that the runner will pass through the ExtensionModule class anyway? In my mind on top of the enable-by-default and the force-rebuild options, these tables would also contain other parameters, like flags that can be passed to the compilers and (why not) files that will actually be compiled. which brings me to the following question

lethal magnet
void vale
lethal magnet
#

So the __init__ method is also part of the API for ExtensionModules, and the config gets passed through it. Thank you!

void vale
#

my bad for not documenting __init__

#

I will

lethal magnet
#

Thanks @void vale, I was just thinking that (if there is no requirement for inheriting from extensionlib), it might be possible for setuptools to export such an entry-point based on the existing code for Extension and build_ext... I have not investigated this deeply though.

final dune
#

I'll be catching up on this next week (PyCON till tomorrow afternoon was one solid mess of talks, meetings, and deadlines), but I'm strongly in favor of a Protocol over ABCs. If you look at UHI, that's a Protocol plus some helpers. Most users like the helpers and use UHI, but you don't have to, and I think extensionlib should be the same way.

void vale
#

pushed an update: big refactor + 100% test coverage

void vale
blazing spindle
#

Hi, I just found out about this project, and the idea seems really interesting! Is there any way I can get involved and contribute? Thanks!

void vale
snow widget
#

Do you think you'll infer whether an extension is a library based on its name or should there be [[project.libraries.foo]]?

void vale
#

what do you mean by library?

snow widget
#

If you want to build a dynamic library instead of a normal extension you can import Library from setuptools.extension and use that in ext_modules instead of extension.

void vale
#

that would likely be up to each builder

#

but then again that's a question for @final dune who knows more about that