#scikit-build

1 messages · Page 1 of 1 (latest)

tawdry rover
#

Hi, I have some questions about migrating from setuptools setup to a scikit-build-core, but I don't see a channel specific to the -core version. Is this the right channel for my questions or is there somewhere else?

vernal hazel
#

It's probably fine here, you can always create a thread for it!

#

We aren't too formal here, just ask your questions so you get the answers you need faster :)

tawdry rover
#

Okay, so currently I'm building a Python wheel which uses Pybind11 to make bindings for a C++ project, and this is all managed through setup.py and CMake. It currently links to the C++ library which is built with CMake from a CMakeLists.txt in a parent folder, and the bindings are in a subfolder (bindings/python/...).

What I did to make this work was add code to build and run a few CMake commands in the setup.py by adding to a build_ext in the cmdclass of the setup() call, much like how it is done in this SO post: https://stackoverflow.com/questions/42585210/extending-setuptools-extension-to-use-cmake-in-setup-py

This way I can cmake -S ../../ ...(etc) to build the dependency using the CMakeLists.txt in the parent project's folder, then run a cmake --build . --target pyExample --config Release to do the build of the bindings, all within a single call to python setup.py bdist_wheel giving me a wheel file. The intermediary .pyd is moved using -DCMAKE_LIBRARY_OUTPUT_DIRECTORY_RELEASE and -DCMAKE_LIBRARY_OUTPUT_DIRECTORY so the wheel can be built with it.

I'm running into some issues trying to move this whole process to a python -m build --wheel call with scikit-build-core, specifically with how to run cmake on the parent source folder to build the C++ library to link against. Is there a standard approach for how to do this, or any advice you all may have? Maybe some examples or another project that is doing this? Build systems are definitely not my strong suit and I could use some help, thanks in advance

tawdry rover
#

Just checking in to see if anyone has any ideas on this, maybe @urban turtle ?

urban turtle
#

Ahh, I guess I thought by making the channel I’d be notified about posts here!

#

Scikit-build and scikit-build-core both go here. Same thing as the GitHub discussions, there’s just one for both. Eventually scikit-build will use scikit-build-core too.

#

Do you have a repo or anything I can see? I think what you want is pretty standard. Main thing different from the “setuptools” examples is you’ll want to use the install step too

#

Why you have might be similar to pybind/cmake_example, you want something like pybind/scikit_build_example.

tawdry rover
tawdry rover
#

I realized if I just move the pyproject.toml to the root directory I can kick everything off there and handle the directory issue. I feel kinda dumb for not realizing that earlier, thanks for all the help

urban turtle
#

Yes, you can specify a subdirectory for your CMakeLists, but things are much easier if you have pyproject.toml at top-level, especially if you are using files from other places in the repo.

tawdry rover
#

I'm trying to migrate a scikit-build-core based install to a build server which has no internet connection, but I'm failing when installing the packages in the venv:

ERROR: Could not find a version that satisfies the requirement scikit-build-core (from versions: none)

Is there a way to specify a local directory to use for the installation in this case? @urban turtle

urban turtle
#

Do you mean when pip installing it or when doing the build? If the latter, then no build isolation is the easiest way usually. But you can actually still use build isolation if you specify a local wheel dir via env var. I do that for our tests, actually.

#

But need a tiny bit more context to provide recipes. 🙂

zenith relic
#

was trying out scikit-build-core because i'm looking to replace my hacks with editable.rebuild=true

#

it's working pretty well so far, but i think editable.rebuild is racy when running multiple processes, probably needs a filelock somewhere

tawdry rover
tawdry rover
#

"But you can actually still use build isolation if you specify a local wheel dir via env var. I do that for our tests, actually"

Do you know which env var? could you link me to the tests, @urban turtle ?

urban turtle
tawdry rover
#

Is there a way I can specify a specific cmake executable to use for a build?

#

I could make a custom wheel and use that from a local directory, but I'd rather avoid that if I can

urban turtle
#

Currently it looks for the installed cmake package, then cmake3 and cmake in the path. We don't currently have an envvar for that.

tawdry rover
urban turtle
#

That code is triggered from get_requires_for_build_wheel, which is a hook in scikit-build-core called by build during the install phase. If you have a sufficent version of cmake or cmake3 on your path, it will not request cmake the package.

#

(Unless you list cmake in your package requirements, which you shouldn't)

tawdry rover
#

Great info, thanks again

verbal charm
#

Hope you don't mind a repost on this channel since in #cpython it did not get any response:
I am using a design like:

ctypes.cdll.LoadLibrary(Path(bundled_lib))           
import _python_binding

In this example, the _python_binding module is built with target_link_libraries to the one in bundled_lib. As far as I have tested, it works fine, but I got a report that on alpine docker (muslinux specifically I think) this one failed, i.e. the library in bundled_lib is not detected by _python_binding. Anyone got an idea of that?

#

I am trying to get in touch with relevant upstream about this issue. Any idea who/where to ping about this, other than cpython repo?

austere niche
verbal charm
#

Yeah, I've posted there as well

Discussions on Python.org
Loading dependency of python binding manually

Basically I have a snippet that does: ctypes.cdll.LoadLibrary(Path(bundled_lib)) import _python_binding where the _python_binding module is built with link_libraries to the one in bundled_lib. As far as I have tested, it works fine on manylinux, but I got a report that on alpine docker (muslinux) it fails. Anyone got an idea of what...

tawdry rover
#

Does scikit-build-core provide a way to use different cmake args/build flags when running on different environments?

For example:

[tool.scikit-build]
cmake.args = [
    "-DLLD_DIR=C:/..."

Could I specify different cmake.args if building on say, linux vs windows?

urban turtle
#

I started to implement it but then figured out how to do conditional Limited API / Stable API nicely without it, so it hasn't been a priority yet. Though it looked pretty straightforward, so it's probably not too far off.

#

I've done it once before, in cibuildwheel

urban turtle
urban turtle
tawdry rover
#

That was fast, thank you so much @urban turtle ! Already using the feature 💯

urban turtle
#

Do you have a link where I can see it in use? Curious to see it in practice. 🙂

tawdry rover
#

Sadly no, it's not a public project (not my code base) 😦

I'll likely use it on something open source though, and will ping you when/if I do!

urban turtle
#

That's okay, and thanks!

tawdry rover
#

np, I made a pull request for an additional override choice that I'm using if that's any consolation

urban turtle
#

Monthly Scikit-build Community meeting starting in 25 mins. https://meet.google.com/dvx-jkai-xhq

Meet

Real-time meetings by Google. Using your browser, share your video, desktop, and presentations with teammates and customers.

tawdry rover
#

I'm doing a build of a C/C++ tool that has python bindings, and it's built via cmake and python -m build with a scikit-build-core backend.

In the isolated build environment, the compiled .so is bundled into the wheel and saved on disk. Is there a way I can also save the .so, not just the .whl? I could post-process the wheel and extract the .so, but I'd rather save it directly if I can.

#

I believe the solution would be something to the effect of what build_ext was

urban turtle
#

We are the 15th most popular build backend with 138 projects on PyPI (counting leaving it empty as a separate backend) out of 127,368 projects with a pyproject.toml:

  0 setuptools.build_meta: 52481
  1 poetry.core.masonry.api: 35852
  2 unknown: 12989
  3 hatchling.build: 9039
  4 poetry.masonry.api: 7082
  5 flit_core.buildapi: 4107
  6 maturin: 1373
  7 setuptools.build_meta:__legacy__: 932
  8 pdm.backend: 873
  9 poetry_dynamic_versioning.backend: 630
 10 pdm.pep517.api: 584
 11 flit.buildapi: 519
 12 jupyter_packaging.build_api: 184
 13 scikit_build_core.build: 138
 14 mesonpy: 87
dusty moth
pine forum
#

i mean there's inspector.pypi.org

urban turtle
mystic linden
#

🥳 Happy that that post is being put to good use 🙂

urban turtle
#

Also see this msg which looks at tool.* sections. #cibuildwheel message - I'll be using that soon to see what the average usage of our settings looks like.

I also have seen the exact number of packages that have a CMakeLists.txt in them (around 2200 IIRC, would have to rerun the query). Probably are other interesting things to look at too.

urban turtle
#

I can ask really interesting questions now, like what is the most popular setting for tool.black.line-length (120). For our settings, here's the breakdown:

wheel: 99
cmake: 85
sdist: 56
metadata: 51
build-dir: 46
minimum-version: 36
ninja: 19
logging: 13
experimental: 12
generate: 5
install: 4
strict-config: 2
overrides: 1
#

The two projects that set strict-config set it to True (the default), interestingly.

heady pier
#

1?

urban turtle
#

1 what?

heady pier
#

Oh, I thought line-length was set to 1.

urban turtle
#

Ah, no, 120 was the most popular. I could see if there's a 1, would not suprise me in the least. There's crasy stuff in the tails. A few projects don't even have parsable pyproject.tomls.

#

3 projects set it to 1000. A fourth sets it to "1000", since black supports using strings anywhere (also, I'm not bothering with line_length, which is also supported)

#

The smallest line-length I see is 30.

austere niche
#

After 120, what are the next most popular?

urban turtle
#

All the ones over 100:

120: 5755
88: 3578
79: 2876
100: 2714
80: 834
99: 811
119: 485
110: 359
90: 269
95: 169
140: 165
160: 112

(Less than that, then I probably should start including strings & the other way(s?) to specify it like line_length) Total is 19,076 pyproject.tomls that have tool.black

urban turtle
#

Here's a better breakdown for scikit-build:

tool.scikit-build.*.*:
metadata.version: 49
wheel.packages: 49
build-dir: 46
cmake.minimum-version: 42
sdist.include: 39
sdist.exclude: 37
minimum-version: 36
wheel.expand-macos-universal-tags: 35
cmake.define: 33
wheel.py-api: 29
cmake.build-type: 27
cmake.verbose: 26
cmake.args: 21
wheel.install-dir: 16
ninja.minimum-version: 15
logging.level: 13
experimental: 12
cmake.targets: 10
wheel.license-files: 8
ninja.make-fallback: 8
metadata.readme: 5
generate: 5
sdist.reproducible: 5
cmake.source-dir: 4
install.components: 3
metadata.scripts: 2
wheel.exclude: 2
strict-config: 2
metadata.optional-dependencies: 1
sdist.cmake: 1
overrides: 1
install.strip: 1

By the way, it is fantastic that this is so clean. Anything without the equivalent of strict-config=True has a weird tail.

#

For example, build-system has only three allowed entries (the top three below).

build-system.*:
requires: 118470
build-backend: 114328
backend-path: 104
packages: 29
dependencies: 21
requires_python: 19
classifiers: 18
requires-python: 17
build_backend: 16
include: 12
readme: 11
license: 10
keywords: 8
sdist: 8
exclude: 7
bdist_wheel: 7
name: 6
version: 5

(Many, many lines with <5 omitted, with things like require, build-system (again), etc)

#

Similar story for top-level:

*:
build-system: 118488
tool: 85504
project: 39624
options: 250
tools: 237
metadata: 168
coverage: 131
flake8: 116
mypy: 116
requires: 114
build-backend: 75
virtualenvs: 74
pytest: 71
dependencies: 61
bdist_wheel: 34
build: 30
build_system: 29

(Again, many more omitted, like build-sytem, too, took, rool, bluid-system, tool-<stuff>, and lots of things that should have been in subsections)

cinder aurora
urban turtle
tawdry rover
#

I'm building a wheel file using skikit-build-core that uses a .pyd and some bindings with pybind11. The wheel is generated, but the structure isn't what I'm going for. Inside the wheel, a .lib is in a lib subdirectory and the .pyd is in the bin subdirectory. How can I place these both inside the proj subdirectory where the __init__.py file is?

urban turtle
#

Do you have the CMakeLists? You need to tell CMake not to use standard subdirs.

tawdry rover
#

That sounds like it might be what I'm looking for. Do you know what the name of the dir would be for the "main" project folder of the wheel (where init.py is) ? So instead of placing it in bin/ for example, it'd be "whatever dir has init.py"

urban turtle
#

That’s dependent on your project. If it’s the name of the project, then IIRC ${SKBUILD_PROJECT_NAME}

#

Installing to “.” is site-packages. You can have multiple packages so we don’t assume a package name. You can set one as wheel.install-dir though.

tawdry rover
#

Perfect, thank you so much!

lusty barn
#

Noob question, I started using scikit-build to build a PyTorch extension. I have been setting the build-dir to "build" and mostly been using the no build env editable mode. My cmake lists creates a .so: python_add_library(driss_torch SHARED ${CU_SOURCES} ${CPP_SOURCES} )

That I load at runtime based off the /build path. I am trying to figure out how this should work for pip install . commands with wheels. How do I get my library to get installed in the correct location to site package?

I have noticed that every example uses MODULE not sure if that makes a differenc

#

From looking at the above, cmake stack overflow I imagine that I need to install my lib into a set path in the top level project and not the build dir

lusty barn
#

hmmm idk looking at the site packages even though i have a package / lib/ sharedlib.so

The sitepackage seems to drop this for some reason

urban turtle
#

You need to install for scikit-build-core. You generally shouldn’t run from the build-dir. I’d try scikit-build-core’s editable modes (there are two to pick from).

#

I think python_add_library treats everything as MODULE since that’s the only thing that is importable in Python.

#

So you’d install(TARGET driss_torch LIBRARY DESTINATION package/lib) or wherever you want it to go.

lusty barn
#

I think what I am trying to do is even more naive than this;
Scikit build properly builds the shared lib on invocation both in in editable and top level pip install . its really just that this .so lib is not being added to the final wheel.

I tried doing:

[tool.scikit-build.package-data]
driss_torch = ["lib/*"]

but this is not an option. Idk if I am fundamentally not using scikit-build-core in the manner it was designed for

urban turtle
#

Do you have source somewhere to look at?

lusty barn
lusty barn
#

hmm okay after some more monkeying around I i seem to now get the lib installed into the directory and non editable installs seem to work fo rme

urban turtle
#

tool.setuptools doesn’t do anything, we aren’t setuptools and don’t use setuptools. 🙂

#

It’s the install command that fixed it.

lusty barn
#

Ahaha okay that makes much more sense, and was just AB testing that now

#

Thanks for the help!

proper ridge
#

Hello! I apologize if this is a dumb question but here we go...

Context:
In scikit-build-core the examples use a CMakeLists.txt file which seems like a superset of what a CMakeLists.txt file would be for just the C++ code itself. For the codebase I'm working on (call it X), there is already a very large (~1000 lines) CMakeLists.txt file. For the python wrapper (call it pyX) we currently have a massive and messy setup.py that uses setuptools to 1) download and install all the dependencies for X (calling their make/cmake files), 2) build X by passing in CMake arguments, 3) run swig on the interface files to generate the wrapper code, 4) build the python library.

Questions:

  • Can scikit-build or scikit-build-core replace all of these steps (including git clone, checkout, etc)?
  • Would I need to edit the CMakeLists.txt file of X or could I keep it separate in the pyX library?

In theory I really like the idea of scikit-build and see a lot of projects I subscribe to using it, but I haven't seen any large examples with swig that are doing our pyX build is doing. I'm an amateur with cmake and just starting to explore scikit-buidl so I apologize if this is naive. Thanks!

urban turtle
#

Sorry @proper ridge, I'm on parental leave so haven't been following closely (all the spam causing all channels to be "unread" didn't help.

  1. Yes, you can write a CMakeLists instead of a setup.py that does all of that. As long as you don't mind setting a pretty recent version of CMake (which scikit-build-core helps you with), it's pretty simple and clean.
  2. You can put your Python CMakeLists in a subdirectory. However, you can't include upward in CMake, so I'd recommend some minor edits to the main CMakeLists to add the python subdirectory if the user is trying to build bindings.
    Some larger examples (not sure the binding tech, though) include ITK and VTK moving to it, and rapidai. SWIG should be pretty easy from CMake, though, there's a built-in module for it, I've used it in the past.
sage lily
#

Hello,
I asked the question is another channel and I was told to come here

I have a cuda/python package with nanobind as the binder

How can I use scikit build core to compile, upload to pypi so users just pip install and get the wheel according to their OS?.

I already have a scikit build core TOML that builds when I do pip install.

I want to be able to compile myself (no cibuild) because I use NVC++ as compiler .
Upload to pypi
Then a user can use pip install to get the binary that corresponds to their OS.

Thank you

sage lily
#

can someone tell me how can I properly use a shared library in a scikit build core built project?
Using an old setup.py it worked like a charm

upper lintel
#

I'm running into a problem when installing a package in user space (pip install --user). There script generated in the bin folder relies on the platlib, but that will point to /usr/local whic is wrong since the binaries are left in .local/... . Has anyone seen that before?
Note; this is for scikit-build-core

urban turtle
# sage lily Hello, I asked the question is another channel and I was told to come here I h...

CUDA is a hard problem without a good solution yet. Shipping an SDist and letting the user build it with CUDA is an option, but wheels are hard. It's technically impossible to ship true manylinux wheels, since you have to have an external dependency (CUDA can't be redistributed, it's not open source, plus it's big). But even if you add the CUDA libs to auditwheel's allowlist, you still have a CUDA dependency; CUDA 11 vs. CUDA 12, for example. nvidia/warp handles this by shipping a CUDA 11 wheel and putting the CUDA 12 wheel as a GitHub download. cupy and several others do this by adding the cuda version to the name of the package, and just releasing lots of packages. You can look around; RapidsAI's packages use scikit-build-core, but it really doesn't matter, that just makes building easier on user machines, building the binaries face the same challenges.

At a guess, you generally would be using cibuildwheel, adding CUDA to a manylinux image, maybe saving and using the resulting image since this might be slow (?), and listing the cuda libs in the allowlist. Then solving the CUDA versioning issues mentioned above. I'll be researching this in more detail in the future.

This was a major point at the packaging summit at PyCon, by the way; there's work toward a way to improve this.

urban turtle
urban turtle
# upper lintel I'm running into a problem when installing a package in user space (`pip install...

You can't rely on platlib from scripts any any system (scikit-build-core or otherwise). There's no way to get the path to platlib from scripts. I'd love this, since if there was I could ship cmake in the scripts dir and avoid the Python wrapper generated by the console entry point, which would save the Python start time and would also avoid issues forwarding things. (That's the solution, by the way, you have to use console entry points instead of writing to scripts if you have something that needs platlib)

(The installer does know where platlib is, and writes it into the script it generates from the console entrypoint, which is why it works that way)

hardy plover
#

I forget which nvidia package it was but they now also have a build-backend that downloads wheels from their index based on more-granular platform compatibility/capability checks, and only upload an sdist to PyPI that uses that build backend.

upper lintel
urban turtle
#

executing script should be updated
Which script? Installers like pip generate the correct script shebang for an entypoint, and I'd forgotten about it, but if you have a python script in /scripts, then #!python is converted to the correct shabang path, and scikit-build-core (like setuptools) will rewrite a "normal" shabang line like #!/usr/local/env python into #!python for you so that the installer will write out the correct shabang path.

#

Installers can't rewrite random scripts though. They only process shabang lines.

upper lintel
# urban turtle > executing script should be updated Which script? Installers like pip generate ...

I see, that makes sense. I was referring to the packaging process, which somehow generates a Python script that executes a compiled binary (see original issue here https://github.com/aestream/aestream/issues/112).
I'm honestly a little confused about the entire process, so I don't know where that script file is coming from. Do you know where I can go to figure that our/learn more about it?

GitHub
Issues · aestream/aestream

Efficient streaming of sparse event data supporting files, network I/O, GPU peripherals (via Torch/Jax/Numpy) and neuromorphic protocols - Issues · aestream/aestream

urban turtle
#

I'll need to investigate, something strange is happening here. Scikit-build-core doesn't add anything like that, but it seems to be in the wheel (https://inspector.pypi.io/project/aestream/0.6.4/packages/57/e7/47d82fda143b650b9e5273a7f81d1d567c5f4cf6133b81f39c2b2e749cda/aestream-0.6.4-cp310-cp310-manylinux_2_28_x86_64.whl/), while other packages like https://inspector.pypi.io/project/clang-format/18.1.7/packages/f7/dc/099a5e4b2b9b8cccfafa6a73d089178371ad53b6735a05fd6a9d8d9aa005/clang_format-18.1.7-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/ are fine.

#

FYI, cibuildwheel doesn't use pipx anymore, you don't need to install it for ACT running.

#

If you configure cibuildwheel via pyproject.toml, it's much easier to test locally (or use other CIs, etc)

#

The core problem is the executable links to other things in platlib, which is invalid; you should swap it for a console entry point wrapping the executable, or statically link so that it doesn't depend on platlib. auditwheel is trying to fix this for you, and is applying a fix that ignores user platlib.

upper lintel
upper lintel
#

Right, so just to be clear, would you recommend both statically linking the binary and wrapping it in a console entry point?

#

My only concern with the static linking is that we're compiling against CUDA (for one flavour of the package). Won't it risk blowing up the binary?

urban turtle
#

Don't statically link CUDA. But CUDA isn't in your site-packages, is it?

#

(And the explicit wrapper method is fine, IMO it's better to do it manually rather than hope that auditwheel fixes it via this method)

#

I knew it wasn't coming from scikit-build-core (as I wrote that), and I searched your repo for "execv" without success, so I checked inspector.pypi.io and saw it in the wheel, so it had to come from the repair step and not the installer. Your repair step was not exactly default, but not different in any important way, so then I searched for "execv" in auditwheel and found it.

#

Feel free to open an issue if no one comments on the closed one I commented on a few days.

#

would you recommend both
No, either-or. 🙂

sage lily
#

Cuda with scikit build core

upper lintel
upper lintel
#

Quick question regarding CUDA; I would love an install option like package[cuda]. To my knowledge there are no ways to install cudatoolkit via pip, so I wonder whether this is possible at all (without asking the user to install the dependencies beforehand). Do you happen to know of a way to drag in cudatoolkit as a pip-based dependency?

heady pier
upper lintel
heady pier
#

Ah, then no. 🙂

upper lintel
#

Thanks for the link though 🙂 Would another option would be to pre-build the wheels? Ofc that would require cross-compiling for all CUDA versions, which sounds like a bit of a 🤯

urban turtle
#

cuda-python is a low level Python library for CUDA that high level libraries like cupy are supposed to use. You have to have drivers + the CUDA toolkit (only the runtime is requied) to use cuda-python. You can't install the cudatoolkit from pip, even if it's possible there likley would be licencing issues. It was quite hard to get it into conda apperently because of this, requried special permissions from NVIDIA.

#

Pre-built wheels would still require the CUDA redistributable, but not a compiler.

upper lintel
urban turtle
#

Depends on user setup. You can install it with your system package manager in some cases, usually from an NVIDIA repo. NVIDIA has instructions. For example, on our clusters at Princeton, you'd use module load cuda/12.4 or something like that (many HPC clusters use Lua Environment Modules, "lmod"). Conda-forge has the ability to distribute CUDA, but you'd still need a sufficient driver version.

upper lintel
#

Right, that makes sense. I'm wondering whether pip can (1) pick up the right driver version and (2) fetch the right wheel without too much hassle on the user-side. Ideally, the user would just do pip install ... and then it'd automatically find the right wheel.
As an example, I recall that you had to manually add the path for your wheels to ealier versions of PyTorch. Can that be avoided now in "vanilla" pip?

urban turtle
#

Teaching it to do that is what was being discussed at the packaging summit at PyCon this year. It's a lot of work, especially if the system is fairly general. You can come up with several other use cases, like MPI, AVX builds, etc. Building from SDist can detect what you have, but not wheels.

hardy plover
sage lily
#

I would like to install the libraries in my CMake to a specifique path

Using setuptools this was trivial and I did it like so

cmake_args = [ f"-DCMAKE_LIBRARY_OUTPUT_DIRECTORY={extdir}{os.sep}", f"-DPYTHON_EXECUTABLE={sys.executable}", f"-DCMAKE_BUILD_TYPE={cfg}", # not used on MSVC, but no harm ]

Where extdir is a relative path that I gave (relative to site-packages)

Using scikit-build-core this is much more difficult

this

`[tool.scikit-build]
minimum-version = "0.8"
cmake.version = ">=3.19"
build-dir = "build/{wheel_tag}"
wheel.py-api = "py3"
cmake.build-type = "Release"

Add any additional configurations for scikit-build if necessary

cmake.args = [
"-DCMAKE_INSTALL_PREFIX:PATH=my-package/_src/",
"-DCMAKE_LIBRARY_OUTPUT_DIRECTORY:PATH=my-package/_src/",
]`

is setting the path to be absoulute paths and they are not relative to site packages

if I do this

`cmake.args = [
"-DCMAKE_INSTALL_PREFIX:PATH=${SKBUILD_PLATLIB_DIR}/my-package/_src/",
"-DCMAKE_LIBRARY_OUTPUT_DIRECTORY:PATH=${SKBUILD_PLATLIB_DIR}/my-package/_src/",
]``

The path is not expanded

I can just install my packges but DCMAKE_LIBRARY_OUTPUT_DIRECTORY was convinient because I also use shared libraries

Anyone know a solution?

#

possibly just by using a TOML without a python setup.py

urban turtle
#

Don't set CMAKE_INSTALL_PREFIX, scikit-build-core doesn't use it. Use wheel.install-dir = "my-package/_src". CMAKE_LIBRARY_OUTPUT_DIRECTORY, if you need it, then should be relative to the install dir, IIRC.

#

(Internally, scikit-build-core is passing --prefix to cmake --install)

# 
[tool.scikit-build]
minimum-version = "0.8"
cmake.version = ">=3.19"
build-dir = "build/{wheel_tag}"
wheel.py-api = "py3"
wheel.install-dir = "my-package/_src"
#

(And, only if you need it)

[tool.scikit-build.cmake.define]
CMAKE_LIBRARY_OUTPUT_DIRECTORY = ""
sage lily
#

All libraries and shared libraries went to the correct path

urban turtle
#

Yes, scikit-build-core always installs. That way CMake correctly handles RPaths, and strips the binaries, etc. You aren't supposed to run from the build directory.

sage lily
#

Okay I see

#

I think the setuptools script worked by accident

#

Where should I installed a shared library to get it to work?

I install it to my-package/_src

but without having the correct LD_LIBRARY_PATH it didn't work
I don't expect my users to update their LD_LIBRARY_PATH

urban turtle
#

Running from the build path usually works, but it's not correct and isn't going to work in some situations.

#

There's a setting for RPath, let me check

sage lily
#

Ok thank you 🙂

urban turtle
#

Did your old SDist work? I get add_subdirectory given source "third_party/cuDecomp" which is not an existing directory. when doing source -> SDist -> wheel on your original code.

sage lily
#

it is a submodule

urban turtle
#

Yes, I pulled it, it still doesn't like going through the SDist first. Going directly to wheel now.

sage lily
#

Yeah I did not package their code in the SDist

#

is this an issue?

urban turtle
#

I'm going directly to wheel now.

sage lily
#

running what command (sorry I am new to packaging)
python -m build?

#

I usually pip install and the setuptools does some magic

urban turtle
#

Yes, pipx run build technically, and that goes source -> SDist -> wheel. Using pipx run build --wheel which skips the SDist just like pip install.

#

Now just working on setting up an environment. It's quite picky: Requested CUDA version not available: 12.2 Available versions: 12.4 🙂

sage lily
#

Yeah building it will be a pain
it needs NVHPC

urban turtle
#

Yes, I'm using nvhpc/24.5 and openmpi/nvhpc-24.5/4.1.6

#

And cudatoolkit/12.2 but it doesn't seem to like that

sage lily
#

yeah It seems that on this branch I had coded it
One sec

urban turtle
#

Ah, I see that. Can modify

sage lily
#

yeah just do

option(NVHPC_CUDA_VERSION "CUDA version to build for" 12.4)

urban turtle
#

Yeah, it's much happier now. Is there something I can run to see if the wheel works?

sage lily
#

just

import jaxdecomp

#

in a script launched by mpirun or srun

urban turtle
#

option is boolean, by the way, you need set(... CACHE)

sage lily
#

Oh ok thanks for the info

urban turtle
#

ModuleNotFoundError: No module named 'jax'

sage lily
#

yeah you need to do
pip install jax[cuda12]

#

you are going through so much trouble, thank you so much

urban turtle
#

ImportError: /home/henryfs/jaxDecomp/.venv/lib/python3.12/site-packages/jaxdecomp/_src/_jaxdecomp.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZNSt10filesystem7__cxx114path14_M_split_cmptsEv

#

Is there a missing link to stdc++fs?

sage lily
#

uhm no this makes no sens

#

You compiled with nvc right?

#

not gcc

urban turtle
#

Yes, 24.5

sage lily
#

this should happend

urban turtle
#

Adding target_link_libraries(_jaxdecomp PRIVATE stdc++fs) fixed it. Okay, that's the old one working, now will try the new one.

sage lily
#

I don't use python3.12 maybe that's why (this helps me alot thank you 🙂 )

urban turtle
#

If you use std::filesystem, older stdlib's require this link. Maybe you have a newer stdlib?

sage lily
#

I use the one provided by my HPC.
But I will add the link to my CMake so it works for everyone (thank you again)

urban turtle
#

Hmm, interesting, one problem is that CMAKE_INSTALL_PREFIX is hardcoded into cuDecomp's CMakeLists. It shouldn't be, install defaults to CMAKE_INSTALL_PREFIX and hard coding it messes with using --prefix. I can probably make that work, though. As a quick fix, I've removed it manually.

sage lily
#

Yeah this is why I had to add it

#

Maybe I should ask them to make it more flexible

#

but if you remove it it works?

urban turtle
#

There's one more issue I'm working on now with the RPath.

#

set_target_properties(_jaxdecomp PROPERTIES INSTALL_RPATH "$ORIGIN/lib") did it. Was looking for a more elegant solution, but that does do it. Also removed the extra install line for cudecomp.

#

I'll see if I can make CMAKE_INSTALL_PREFIX not break things.

sage lily
#

thank you so much 🙏🙏🙏

#

ORIGIN is a SKBUILD variable?

urban turtle
#

No, it's part of the RPath spec.

sage lily
#

Ok I see.
I am gonna make an issue to ask them to remove the CMAKE_INSTALL_PREFIX
Thank you for your help 🙂 🙂

urban turtle
#

(Though they should fix their lib still)

#

Also, I highly recommend using src structure for your lib, I had to use -P (PYTHONSAFEPATH, 3.11+ only) to make it respect the install over the local path. But if you use src structure, that just works.

#

And I'd avoid calling the directory inside your package "src" if possible, that's usually reserved for src-structure.

sage lily
#

what is an src structure ?
using src and include folders ?

urban turtle
sage lily
#

okay I see

urban turtle
sage lily
#

not just the cuda code but everything in src

#

Thank you for the info

urban turtle
#

Yes, so that import jaxdecomp will never try to import the source directory instead of the installed package.

sage lily
#

ahh okay
This is way I have a circular dependance

urban turtle
# 
pipx run build --installer=uv --wheel
uv venv
uv pip install jax[cuda12]
uv pip install dist/jaxdecomp-0.1.0-py3-none-linux_x86_64.whl
# salloc --nodes=1 --ntasks=1 --mem=32G --time=00:10:00 --gres=gpu:1
.venv/bin/python -Pc "import jaxdecomp"

This is working for me with using the scikit-build-core PR linked above.

urban turtle
#

0.9.8 should support this (but please still open an issue with cuDecomp, they should not be doing this)

hardy geyser
#

Hi 👋

I'm trying to adapt the scikit-build-core example with swig to c++. I'm able to use scikit-build just fine with the c example, not sure what I did wrong with the adaptation. pip installing works fine, but on import I get an error.

>>> import example
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    import example
  File "/home//.local/lib/python3.8/site-packages/example.py", line 12, in <module>
    import _example
ImportError: dynamic module does not define module export function (PyInit__example)

square.hpp

#ifndef SQUARE_HPP
#define SQUARE_HPP
float square(float N);
#endif

square.cpp

#include "square.hpp"
float square(float N) {
    return N * N;
}

example.i

%module example
%{
#include "square.hpp"
%}
%include "square.hpp"

pyproject.toml

[build-system]
requires = ["scikit-build-core", "swig", "numpy"]
build-backend = "scikit_build_core.build"

[project]
name = "example"
version = "0.0.1"

CMakeLists.txt

cmake_minimum_required(VERSION 3.15...3.26)
project(${SKBUILD_PROJECT_NAME} LANGUAGES CXX)

find_package(Python3 REQUIRED COMPONENTS Interpreter Development.Module NumPy)
find_package(SWIG REQUIRED COMPONENTS python)
include(UseSWIG)

include_directories(${Python3_INCLUDE_DIRS} ${Python3_NumPy_INCLUDE_DIRS})

swig_add_library(
  example
  LANGUAGE python
  OUTPUT_DIR "${SKBUILD_PLATLIB_DIR}"
  SOURCES src/square.cpp src/example.i
)

if(WIN32)
  set_property(TARGET example
               PROPERTY SUFFIX ".${Python3_SOABI}.pyd")
else()
  set_property(TARGET example
               PROPERTY SUFFIX ".${Python3_SOABI}${CMAKE_SHARED_MODULE_SUFFIX}")
endif()
target_link_libraries(example PRIVATE Python3::Module)

install(TARGETS example DESTINATION .)
urban turtle
#

IIRC there’s a property to set on the file to mark it as c++ for swig. Can check on a computer later.

#

Link to Python::NumPy too and then you can drop the include directories.

# 
set_property(SOURCE example.i PROPERTY CPLUSPLUS ON)
hardy geyser
#

before swig_add_library

urban turtle
#

I’ll have to try it then. It’s not getting the init function, usually because the name is wrong.

#

You shouldn’t output directly to platlib, and use install instead.

hardy geyser
#

sorry I'm still pretty new to cmake. I mostly just copied directly from the scikit-build-core swig c example. By "use install instead" do you mean editing the final line install(TARGETS example DESTINATION .)?

urban turtle
#

I think there are some minor issues with the example, was helping someone a couple of days ago and noticed that. Will update once they report back that it's working for them.

I'm referring to the OUTPUT_DIR "${SKBUILD_PLATLIB_DIR}", that looks problematic to me.

hardy geyser
#

oops i should have clarified, my question implied deleting that line, but I wasn't sure what to do instead.

#

Okay, thank you!

urban turtle
#

Yeah, the current way adds a examplePYTHON_wrap.c into the wheel.

#

Ahh, missed it was src/example.i, did you correct that?

#

(I'm breifly on a computer now)

hardy geyser
#

oo doh

#

let me try, i missed that too

#

ah okay, that did the trick (+ adding src to include_directories). it didn't import without the PLATLIB line though so I added that back in

urban turtle
#

It must not be getting the helper file

hardy geyser
#

is not using PLATLIB specific to the fact this is c++ or is that something that is deprecated in scikit-build-core?

urban turtle
#

There’s probably a way to install that file (other than knowing it’s “example.py”)

#

I think the simple example is a little bit too simple and it doesn’t use enough of swig, it passes missing a bit that it actually needs for any more realistic example

hardy geyser
#

yeah to be fair I guess a lot of this can be found out through cmake docs

#

I'm kind of new to both so I'm pretty lost haha. thanks for the help

urban turtle
#

The general pattern is that build product should go in the build directory. Then install puts everything in platlib. But not sure how to nicely install the generated example.py.

#

(On my phone again)

hardy geyser
#

oh yeah just to clarify, I did get the import to work, using PLATLIB

urban turtle
#

In the docs, I see a couple of ways to do it. Will update here later.

hardy geyser
#

thank you!

urban turtle
#

This is the "easy" method: set OUTFILE_DIR so that the cpp file gets generated in in the binary dir instead:

cmake_minimum_required(VERSION 3.15...3.26)
project(${SKBUILD_PROJECT_NAME} LANGUAGES CXX)

find_package(Python3 REQUIRED COMPONENTS Interpreter Development.Module NumPy)
find_package(SWIG REQUIRED COMPONENTS python)
include(UseSWIG)

set_property(SOURCE src/example.i PROPERTY CPLUSPLUS ON)

swig_add_library(
  example
  LANGUAGE python
  OUTPUT_DIR "${SKBUILD_PLATLIB_DIR}"
  OUTFILE_DIR "${CMAKE_CURRENT_BINARY_DIR}"
  SOURCES src/square.cpp src/example.i
)

if(WIN32)
  set_property(TARGET example
               PROPERTY SUFFIX ".${Python3_SOABI}.pyd")
else()
  set_property(TARGET example
               PROPERTY SUFFIX ".${Python3_SOABI}${CMAKE_SHARED_MODULE_SUFFIX}")
endif()
target_link_libraries(example PRIVATE Python3::Module Python3::NumPy)
target_include_directories(example PRIVATE src)

install(TARGETS example DESTINATION .)
#

Note that you should surround filenames with variables in them with quotes, so they will work if they have a space in them. No need if there's no variable.

#

You might want SWIG_USE_TARGET_INCLUDE_DIRECTORIES too. Now there are several steps inbetween, but I'm going to try to do it the nice/modern way, avoiding writing to the platlib dir inside the build step, which is not ideal (but supported).

hardy geyser
#

Thanks @urban turtle !

#

that works too

urban turtle
# 
cmake_minimum_required(VERSION 3.15...3.26)
project(${SKBUILD_PROJECT_NAME} LANGUAGES CXX)

find_package(Python3 REQUIRED COMPONENTS Interpreter Development.Module NumPy)
find_package(SWIG REQUIRED COMPONENTS python)

set(UseSWIG_MODULE_VERSION 2)
include(UseSWIG)

set_property(SOURCE src/example.i PROPERTY CPLUSPLUS ON)

swig_add_library(
  example
  LANGUAGE python
  SOURCES src/square.cpp src/example.i
)

if(WIN32)
  set_property(TARGET example
               PROPERTY SUFFIX ".${Python3_SOABI}.pyd")
else()
  set_property(TARGET example
               PROPERTY SUFFIX ".${Python3_SOABI}${CMAKE_SHARED_MODULE_SUFFIX}")
endif()
target_link_libraries(example PRIVATE Python3::Module Python3::NumPy)
target_include_directories(example PRIVATE src)

get_property(support_dir TARGET example PROPERTY SWIG_SUPPORT_FILES_DIRECTORY)

install(TARGETS example DESTINATION .)
install(DIRECTORY "${support_dir}/" DESTINATION .)
#

I'm using pipx run build --installer=uv --wheel && unzip -l dist/*.whl to see what it makes.

hardy geyser
#

Sorry to ask about another topic - regarding passing flags, is the recommended way in scikit-build-core to use environmental variables? I am trying to modernize our old build script which has many (dozens) of flags; some of them could probably be removed but many will need to stay, for example --parallel will install MPI, mpi4py, and also pass a flag to the cmake of a dependency. I am looking at the docs and it seems like this is possible through config-settings but it is pretty verbose, e.g. to set a parallel flag that gets passed to cmake I think I'd need to do this? pip install . --config-settings=cmake.define.parallel=1 (this is using tool.scikit-build.cmake.define in the toml file, not sure if there is a better way)

hardy geyser
urban turtle
#

Ouch, I'd not use --parallel for comping MPI, that's usually building in parallel. -Ccmake.define.USE_MPI=1 would be a good way. You'd also want to add an extra that installs MPI and mpi4py (I'm not sure installing MPI is a good idea, though, clusters usually have their own MPI installs). I'd probalby have it detect MPI by default, but allow it to be opted in or out depending on the value passed to the define.

#

You just need to use it in the CMake file, you don't need it in both config-settings and pyproject.toml.

#

The extra would look like pip install .[mpi] for example.

#

I've been tempted to add a D shortcut in the past for that, -CDUSE_MPI=1 if that were added. But before doing that, I have something else I want to try, probably for 0.11 or so.

dusky junco
#

Hi! I've recently converted the build-system of python bindings for a c-library from setuptools to scikit-build-core. I have started considering how to create an sdist (so we can upload to PyPI) and I've noticed there are some weird challenges that are probably tied to our directory layout. It seems like I might not be doing things "properly"(since my scenario seems would be fairly common), and I wanted to ask if there was an established convention for doing things.

For concreteness, the c library is called Grackle (it is used in C or C++ or Fortran programs that run numerical simulations) and the python bindings are called Pygrackle. The bindings originated as a way to easily run integration tests and are now used for post-processing simulation-data. Below is a rough sketch of my project's directory layout:

grackle/
    CMakeLists.txt
    cmake/
        *.cmake
    src/
        clib/
            CMakeLists.txt
            *.c
            *.h
        python/
            CMakeLists.txt
            pyproject.toml
            tests/
            pygrackle/
                *.py
                *.pyx
#

The src/python/CMakeLists.txt file builds the extension-module portions of Pygrackle by:

  • (for testing) linking to an existing build/install of Grackle via find_package OR
  • (preferred) embedding a Grackle build via add_subdirectory(../../ my-sub-build-dir).
    Other than that (ugly) add_subdirectory call, the src/python/CMakeLists.txt file is completely isolated from the other cmake files.

When installing from an sdist, I would want to use the same sort of add_subdirectory logic. But (as I understand it), the files outside of the grackle/src/python directory. I know I can hack something together that works based on (this part)[https://scikit-build-core.readthedocs.io/en/latest/configuration.html#configuring-source-file-inclusion] of the docs, but that approach doesn't seem ideal (the fact the sdist would effectively have an inverted layout from the source-directory would probably be confusing). Is there a convention for how people normally address this sort of problem?

I could imagine moving my pyproject.toml to the root directory:

  • is there a way I can tell scikit-build-core to invoke cmake using the CMakeLists.txt file within the python subdirectory?
  • Or would it need to use the CMakeLists.txt file in the same directory? (I don't really want to mix python build logic into the root-level CMakeLists.txt file, but if that is the established "Right Thing to Do," I'll do it).
#

(Sorry about the length -- if this isn't the proper forum to ask these questions just let me know. I was just trying to be detailed)

urban turtle
#

The usual way to do this is to have pyproject.toml at the root. That's easiest for people using your repo directly (name @ git+https://.... works), and it means the repo and the SDist have the same structure. You certainly can point at a CMakeLists.txt in a subdirectory with cmake.source-dir.

dusky junco
#

oh, I didn't know about cmake.source-dir

#

That sounds like it would help. (I also hadn't thought about installing from my repo -- that's a great point)

Thank you!

urban turtle
#

Though if you need to build other parts of your repo, you might want the main CMakeLists.txt and just have checks for if you are building (SKBUILD is defined to 2, for example, or you can set a define like BUILD_PYTHON_BINDINGS)

#

(Though you can do it the way you described too; that's basically how pybind11 does it)

#

Pybind11's actually an example of a pyroject.toml in a subdirectory, it's for building the tests so it's in the tests subdirectory. So you can do it that way, it's just not usually the best way.

#

(Might still be in a pybind11 PR, it's the one adding pyodide WebAssembly tests, I think we haven't merged since I was discussing adding some flags)

dusky junco
#

Though if you need to build other parts of your repo, you might want the main CMakeLists.txt and just have checks for if you are building (SKBUILD is defined to 2, for example, or you can set a define like BUILD_PYTHON_BINDINGS)
Yeah, I thought a little about that. Our root-level CMakeLists.txt file is already a little more "noisy" than I would like and I worry that doing this would add more noise that would complicate/confuse people (it's probably worth revisiting down the road after I get time to refactor).

I took a look at that pybind11 PR -- It was insightful. With that said, I'm definitely going to move the pyproject.toml to the root of the repository. It definitely sounds better for our scenario.

hardy geyser
#

Hi again 👋

Is it possible to use scikit-build-core with a pre-built c++ library (or one that has been built earlier in the cmake file)? The examples all have the c++ source code in the same directory as the toml file. Where should I be building or copying a pre-built c++ library to? I see in ~/.local/lib/python3/site-packages/ there is: (_example.cpython...so), example-0.0.1.dist-info/, and example.py. My best guess is somewhere in dist-info? Thank you

#

oh, forgot to mention this is using swig

urban turtle
#

Yes, I'd assume "earlier in the file" - and you'd just install it to where you want it to go? If you built it separately, that's a lot tricker, since you need to make sure it gets built (pip install . on the repo woudl likley fail, for example, as would building from SDist). But same idea, just "install" the file you want. That does in the wheel.

#

The best place to put files is in site-packages/yourpackage, but you can use the other available wheel directories if you want. SKBUILD_SCRIPTS_DIR goes to /bin (or /Scripts on Windows), SKBUILD_DATA_DIR is the root of the install tree (this coudl be / or /usr/local on some machines without venvs!), SKBUILD_HEADERS_DIR is for headers (no one uses this), and SKBUILD_METADATA_DIR is the dist-info dir, it's very rarely used, mostly for adding licenses or special plugin-like things.

hardy geyser
#

it might be separate because people have their own custom versions of the library they want to use. in the example there are the lines

get_property(support_dir TARGET example PROPERTY SWIG_SUPPORT_FILES_DIRECTORY)
install(DIRECTORY "${support_dir}/" DESTINATION .)

should I be using support_dir for this?

hardy geyser
#

also to find the site-packages/yourpackage directory should I be using SKBUILD_PLATLIB_DIR (or Python3_SITEARCH)?

urban turtle
#

You are asking where to install to, not from, right? I assume you know where the custom version of the library someone supplied is, so that's where you get it from. Just like support_dir above is where SWIG put the support files, and you have to install them to the wheel.

"." is wheel.install-dir, which by default should be thought of as site-packages (and is SKBUILD_PLATLIB_DIR). If you set that to something (usually the name of the package), then it's ${SKBUILD_PLATLIB_DIR}/whatever-you-set-it-to.

#

So install(FILE x.so DESTINATION .) will install directly to site-packages unless you've set wheel.install-dir. install(FILE x.so DESTINATION yourpackage) or wheel.install-dir="yourpackage" , your pick (just don't do both at the same time!)

hardy geyser
#

sorry I'm not being very clear. My main question is where to build the c++ library/dependencies that swig will need to bind to. In a previous answer you mentioned that building in the build dir and transferring everything to the platlib is the recommended way, even though installing into platlib directly is possible. In that case I think I'd want to build the c++ library in SKBUILD_DATA_DIR?

urban turtle
#

Building in the build dir and then installing is best. IIRC previously for SWIG we were handling the extra support files it generates.

#

Building into the platlib directly is not recommended. CMake fixes up rpaths and strips the binaries when installing.

proper ridge
#

I'm trying to pull & build an external library, but when I pip install with scikit-build-core I get an error about not being able to find Ninja.

Here is my CMakeLists.txt. I can run it fine using cmake/make by itself, the error is only when trying to pip install. Any ideas on whats going on? Thank you!

cmake_minimum_required(VERSION 3.15...3.26)
# project(${SKBUILD_PROJECT_NAME} LANGUAGES CXX)
project(pymfem_dev LANGUAGES CXX)
include(ExternalProject)

find_package(Python3 REQUIRED COMPONENTS Interpreter Development.Module NumPy)

message(STATUS "SKBUILD_DATA_DIR : ${SKBUILD_DATA_DIR}")

# (Step 1) Add mfem
ExternalProject_Add(
    mfem
    GIT_REPOSITORY https://github.com/mfem/mfem.git
    CMAKE_ARGS
        -DBUILD_SHARED_LIBS=1
        -DMFEM_ENABLE_EXAMPLES=1
        -DMFEM_ENABLE_MINIAPPS=1
        -DCMAKE_SHARED_LINKER_FLAGS=
        -DMFEM_USE_ZLIB=1
        -DCMAKE_CXX_FLAGS=-std=c++11
        -DCMAKE_CXX_COMPILER=c++
        -DMFEM_USE_EXCEPTIONS=1
)
image.png
urban turtle
#

Ah, sorry, missed this. In a week-long event, and forgot about this. This is an issue in the inner project not finding ninja (the outer one is working fine). Quick thing to check: Can you just pass through -DCMAKE_MAKE_PROGRAM="${CMAKE_MAKE_PROGRAM}"?

charred merlin
#

how do you export debug info? setting build-type to Debug or RelWithDebInfo isn't doing it

sage lily
#

Hello
I am wondering if there a clean way that we can use to send "options" or arguments with the pip install to define some variables in the CMake building process
Other than env variables ofcourse

urban turtle
#

-Ccmake.define.FOO=BAR?

sage lily
#

Exactly what I want.
Thanks again @urban turtle

sage lily
#

Hello,
I think I found an issue with skbuild when trying to call python scripts from CMake

I am following this tutorial here
https://jax.readthedocs.io/en/latest/ffi.html#foreign-function-interface-ffi
using this CMake

Where we use the python interpeter to find include dirs

Like so

find_package(Python 3.8
  REQUIRED COMPONENTS Interpreter Development.Module
  OPTIONAL_COMPONENTS Development.SABIModule)

message(STATUS "Python_EXECUTABLE: ${Python_EXECUTABLE}")
execute_process(
  COMMAND "${Python_EXECUTABLE}"
          "-c" "from jax.extend import ffi; print(ffi.include_dir())"
  OUTPUT_STRIP_TRAILING_WHITESPACE OUTPUT_VARIABLE XLA_DIR)
message(STATUS "XLA include directory: ${XLA_DIR}")

But it says that I don't have JAX, but I definitly do

When I printed the PYTHONPATH I got

-- PYTHONPATH: /tmp/pip-build-env-hkmybt1g/site

Which explains why I found no modules or packages

Anyone knows a workaround

heady pier
#

Did you add JAX to your build dependencies?

sage lily
#

oh in the TOML?

heady pier
#

Yeah.

sage lily
#

yup that was it
Thx

minor wasp
#

Heya, I’m trying to package micromamba 2.0 for Arch Linux.

Before 2.0, one could just build all the C++ stuff together, then just package the Python module:

cd micromamba-root
cmake -D BUILD_LIBMAMBAPY=ON --preset=...
cd libmambapy
python -m build --wheel --no-isolation

but now that fails because the last line runs CMake again for just the libmambapy target instead of just packaging what’s there.
I tried using SKBUILD_CONFIGURE_OPTIONS=... with the options I used before, but it didn’t work since it runs just for the libmambapy target and tries to find the libmamba.so (which isn’t installed system-wide at that point)

Is there a way to either

  1. skip running CMake and just packaging stuff when running python -m build, or
  2. tell scikit-build to target the CMakeLists.txt in the parent directory to build everything instead of just the CMakeLists.txt in the libmambapy directory?
minor wasp
#

Is this the wrong forum to ask? If so, where can I get an answer to this?

urban turtle
#

This is probably fine, I’m just a bit behind. 2 should work, you can set the source dir to find CMake from

minor wasp
#

Thanks! Do I have to patch the package’s setup.py to set skbuild.setup(..., cmake_source_dir='..') or is there a way to override this option using environment variables?

urban turtle
#

Ah, this is scikit-build classic? With scikit-build-core, yes. I seem to remember starting to look into this.

#

What's the current state? Does it just rebuild, are you missing stuff, etc? I see several things I could improve, so I'll probably investicate and make a PR.

minor wasp
#

I’ll tell you next weekend. I was on holiday and now I’m away from my personal machine.

tiny plaza
#

hello. I am starting to use the scikit build core. would be possible to use this tool to both package CPP extensions and python code. In my current situation I need to build a cpp extension using pybind11 and I want to avoid to have two repositories one for the extension and another for the python code. thanks

urban turtle
#

Yes, most projects have both some Python and extensions inside. Scikit-build-core will auto discover src/<package_name> and a few similar patterns, or you can explicitly list packages in tool.scikit-build.wheel.packages.

tiny plaza
#

thanks I will try

tiny plaza
#

hello @urban turtle even after setting tool.scikit-build.wheel.packages the python files aren't copied after the installation any hints ?

urban turtle
#

Do you have a repo?

#

It needs to be the path to the top level of the package, including the package name.

tiny plaza
#

hello @urban turtle it worked. after I have done python -m build

#

I was using pip3 install .

minor wasp
#

cmake parent directory

thorny yarrow
#

Hey, I am trying to build a fortran90 module using scitkit-buld core - basically just using the example here (https://scikit-build-core.readthedocs.io/en/latest/getting_started.html), slightly modified.

[build-system]
requires = ["scikit-build-core", "numpy"]
build-backend = "scikit_build_core.build"

[project]
name = "example"
version = "0.0.1"
dependencies = ["numpy"]

[tool.scikit-build]
ninja.version = ">=1.10"
cmake.version = ">=3.17.2"

cmake_minimum_required(VERSION 3.17.2...3.29)
project(${SKBUILD_PROJECT_NAME} LANGUAGES C Fortran)

find_package(
  Python
  COMPONENTS Interpreter Development.Module NumPy
  REQUIRED)

# F2PY headers
execute_process(
  COMMAND "${PYTHON_EXECUTABLE}" -c
          "import numpy.f2py; print(numpy.f2py.get_include())"
  OUTPUT_VARIABLE F2PY_INCLUDE_DIR
  OUTPUT_STRIP_TRAILING_WHITESPACE)

add_library(fortranobject OBJECT "${F2PY_INCLUDE_DIR}/fortranobject.c")
target_link_libraries(fortranobject PUBLIC Python::NumPy)
target_include_directories(fortranobject PUBLIC "${F2PY_INCLUDE_DIR}")
set_property(TARGET fortranobject PROPERTY POSITION_INDEPENDENT_CODE ON)

add_custom_command(
  OUTPUT examplemodule.c example-f2pywrappers.f
  DEPENDS example.f90
  VERBATIM
  COMMAND "${Python_EXECUTABLE}" -m numpy.f2py
          "${CMAKE_CURRENT_SOURCE_DIR}/example.f90" -m example --lower)

python_add_library(
  example MODULE "${CMAKE_CURRENT_BINARY_DIR}/examplemodule.c"
  "${CMAKE_CURRENT_BINARY_DIR}/example-f2pywrappers.f"
  "${CMAKE_CURRENT_SOURCE_DIR}/example.f90" WITH_SOABI)
target_link_libraries(example PRIVATE fortranobject)

install(TARGETS example DESTINATION .)

finally, the fortran file. This works fine:

real function square(x)
  real, intent(in) :: x
  square = x * x
end function square

python3 -c "import example; print(example.square(2))"

nicely returns 4.0

However, if I convert it to a module, it does not work any longer!

# 
module math_operations
  implicit none
  
contains
  real function square(x)
    real, intent(in) :: x
    square = x * x
  end function square

end module math_operations

now I get this:

#

What I am wondering is, why does it say Fortran 90 wrappers are saved to "./example-f2pywrappers2.f90" where does the "2" come from? Now it makes sense it does not find /tmp/tmpstav11xc/build/example-f2pywrappers.f

But what do I have to change to make it work with a module? I was not able to find any documentation for that.

Just using f2py, it works

#

Discord does not allow me to paste the whole output...

urban turtle
#

Also, did you remove the output dir before rerunning as a module? Just in case that's why it changed the name.

#

I'm not very familar with Fortran. I'd probalby start by investicating why f2py adds a 2.

thorny yarrow
#

Mhm, now I get Fatal Error: Cannot open module file ‘math_operations.mod’ for reading at (1):

thorny yarrow
urban turtle
#

Ah, yes, that's a typo, thanks!

thorny yarrow
#

I got it working now with

cmake_minimum_required(VERSION 3.17.2...3.29)
project(
  ${SKBUILD_PROJECT_NAME}
  LANGUAGES C Fortran
  VERSION ${SKBUILD_PROJECT_VERSION})

set(ModuleName "_thermal_comfort")
set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -O3 -Wall -march=native -ftree-vectorize -funroll-loops")

find_package(
  Python
  COMPONENTS Interpreter Development.Module NumPy ${SKBUILD_SABI_COMPONENT}
  REQUIRED)

# F2PY headers
execute_process(
  COMMAND "${PYTHON_EXECUTABLE}" -c
          "import numpy.f2py; print(numpy.f2py.get_include())"
  OUTPUT_VARIABLE F2PY_INCLUDE_DIR
  OUTPUT_STRIP_TRAILING_WHITESPACE)

add_library(fortranobject OBJECT "${F2PY_INCLUDE_DIR}/fortranobject.c")
target_link_libraries(fortranobject PUBLIC Python::NumPy)
target_include_directories(fortranobject PUBLIC "${F2PY_INCLUDE_DIR}")
set_property(TARGET fortranobject PROPERTY POSITION_INDEPENDENT_CODE ON)

add_custom_command(
  OUTPUT "${ModuleName}module.c" "${ModuleName}-f2pywrappers2.f90"
  DEPENDS "src/${ModuleName}.f90"
  VERBATIM
  COMMAND "${Python_EXECUTABLE}" -m numpy.f2py
          "${CMAKE_CURRENT_SOURCE_DIR}/src/${ModuleName}.f90" -m "${ModuleName}" --lower)

if(NOT "${SKBUILD_SABI_COMPONENT}" STREQUAL "")
  python_add_library(
    ${ModuleName} MODULE
    "${CMAKE_CURRENT_BINARY_DIR}/${ModuleName}module.c"
    "${CMAKE_CURRENT_BINARY_DIR}/${ModuleName}-f2pywrappers2.f90"
    "${CMAKE_CURRENT_SOURCE_DIR}/src/${ModuleName}.f90"
    WITH_SOABI USE_SABI 3.7)
else()
python_add_library(
  ${ModuleName} MODULE
  "${CMAKE_CURRENT_BINARY_DIR}/${ModuleName}module.c"
  "${CMAKE_CURRENT_BINARY_DIR}/${ModuleName}-f2pywrappers2.f90"
  "${CMAKE_CURRENT_SOURCE_DIR}/src/${ModuleName}.f90"
  WITH_SOABI)
endif()

target_link_libraries("${ModuleName}" PRIVATE fortranobject)

install(TARGETS "${ModuleName}" DESTINATION "thermal_comfort")

still no idea though why the 2 is appended though...

#

...but it paid of 57x seedup for scalar values and 5x speedup for array operations 🥳

urban turtle
#

Adding F90 should fix it with f2py-cmake

river smelt
#

Hello, I'm trying to set a CMake define based on the path of the pyproject.toml, is there a way to do that?

river smelt
#

Hello, I'm trying to set a CMake define

river smelt
# river smelt Hello, I'm trying to set a CMake define based on the path of the `pyproject.toml...

Hello! I'm running into an issue where scikit-build-core doesn't produce a valid importable wheel. Here are the contents of the unzipped wheel:

$ tree -L 1
.
├── bin
├── finch_mlir-0.1.dev60+g27552e4.dist-info
├── include
├── lib
├── python_packages
├── share
└── src

And here is my scikit-build config in pyproject.toml:

[tool.scikit-build]
minimum-version = "build-system.requires"
ninja.make-fallback = false
cmake.args = ["-UNB_SUFFIX", "-UNB_SUFFIX_S"]
wheel.packages = ["python_packages/finch/mlir_finch"]
wheel.exclude = ["bin/**", "lib/**", "share/**", "src/**", "python_packages/mlir_core/**"]

...

As you can see, unnecessary directories are included whereas the necessary python_packages/finch/mlir_finch isn't really moved to site-packages, IIUC.

urban turtle
#

Hello! I'm running into an issue where

river smelt
#

Hello. I'm having issues getting scikit-build-core to play nice with ccache. I test it by repeating the following steps twice:

  1. ccache -C && rm -rf build dist (clear ccache only the first time)
  2. time python -m build --sdist --wheel (note timing)
  3. ccache -s

Funnily enough, using the setuptools plugin, ccache works like a charm, but ${SKBUILD_PLATLIB_DIR} is undefined. Is there an alternative in that case as a workaround? Also I lose the ability to control the contents of the wheel there.

The repo/branch is here: https://github.com/hameerabbasi/Finch-mlir/tree/pyproject-toml

urban turtle
#

Hello. I'm having issues getting `scikit

river smelt
strange sky
strange sky
urban turtle
#

Dev meeting in about half an hour (12 Eastern)

analog glacier
#

[scikit-build-core as backend] Hi, I want to skip the CMake build when deploying my project's documentation through an integration process like Cloudflare Pages. I don’t want the deployment process to install my project—only the dependencies specified in optional.doc. However, it seems that --no-deps only skips dependency installation, not the CMake build. I’d prefer not to use a requirements-doc.txt file. Is there a way to skip the CMake build directly in the pip command?

urban turtle
#

Ah, I seem to forget to check this - do you still need an answer? Yes, it's possible, -Cwheel.cmake=False would do it. Though it would be better to use dependnecy-groups instead of project.optional-dependencies. The next version of pip will have it, but you can simply use the pip install dependency-groups package or uv today to get a way to install them.

limber zodiac
#

Hi. I am trying to port an existing setuptools based package to scikit-build-core and have run into an issue with creation of an sdist. What is the right way to provide what files should be included in the sdist tarball? I have a [tool.scikit-build.sdist] section in my pyproject.toml where I specify the include list. However, it seems that irrespective of what I add there everything in my source tree gets added to the sdist. I have used python -c "from scikit_build_core.build import build_sdist; build_sdist('dist')" and also python -m build --sdist --no-isolation with the same result. I will be very grateful for some insight into how to get sdist build to work.

urban turtle
#

Do you have a exclude like !.*? Excludes apply on top of includes.

limber zodiac
#

Yes, I do.

    "**/*.o",
    "**/*.so",
    "**/*.dylib",
    "**/*.a",
    "**/__pycache__/**/*",
    "**/*.pyc",
    "**/.git/**/*",
    "**/.vscode/**/*",
    "**/.DS_Store"
urban turtle
#

Do you have a repo to look at? I'm looking for a negative exclude

limber zodiac
#

Sure that will be awesome. Let me push my changes to my repo quickly.

urban turtle
#

Do you have an example of a file you don't want in the SDist that's getting added?

limber zodiac
#

Yes, I am seeing all files inside the 3rdparty directory getting added instead of just the 3rdparty/cutlass/include/**/*. I also see every folder in the root directory such as benchmarks getting added. I want only the directories I specified in the include list.

urban turtle
#

Perfect, thanks, checking on it

limber zodiac
#

Thank you very much!

urban turtle
# 
[tool.scikit-build.sdist]
cmake = false
exclude = [
    "**",
]
include = [
    "3rdparty/cutlass/include/**/*",
    "3rdparty/cutlass/tools/util/include/**/*",
    "cmake/**/*",
    "flashinfer/**/*",
    "libflashinfer/**/*",
    "tvm_binding/**/*",
    "LICENSE",
    "README.md",
    "pyproject.toml",
    "!.venv/**",
]

Should do what you want. There are default excludes, so you don't need those. You do need !.venv since LICENSE and README.md will match every LICENSE and README.md, even in virtual environments - I think that's a bug, we need to shortcut checking inside directories if the directory is ignored. Also, in the future there will be a "mode" setting that should include an "explicit" mode, which will not include everything that doesn't match a rule.

#

Currently, everything is checked per file, and includes always win over excludes.

limber zodiac
#

Let me quickly try. Thank you for taking time out to look at the issue.

urban turtle
#

I'm adding INFO logging to explain why each file matches or is excluded to make understanding this easier (that's what I used here)

limber zodiac
#

It did generate a much better sdist than before, but I still see some issues. Everything in 3rdparty got added and the _skbuild build directory also was added.

(fi-devel-torch24-cuda124-py312) ~/devel/flashinfer/dist$ ls flashinfer-0.2.4.post18
3rdparty  flashinfer     LICENSE   profiler        README.md  tvm_binding
cmake     libflashinfer  PKG-INFO  pyproject.toml  _skbuild
(fi-devel-torch24-cuda124-py312) ~/devel/flashinfer/dist$ ls flashinfer-0.2.4.post18/3rdparty/
composable_kernels  cutlass  googletest  mscclpp  nvbench  spdlog
#

The profiler directory also somehow got added.

urban turtle
#

The build directory should be auto-ignored, that's worrying. Let me check. I'm fiddling with scikit-build-core's source and I've done an editable install of it, so will take a minute 🙂

limber zodiac
#

I think I have a hunch why profiler gets added. I see a README in that directory.

urban turtle
#

Yes, I'm working on fixing the recusion now. Though if a parent directory is not ignored, it's correct to include it

#

Did you intentionally not include your /licenses folder, by the way?

limber zodiac
#

No, that was an oversight. 🙂

urban turtle
#

The most common pattern is to just take the default includes (which is based on the .gitignore) and add extra ignores on top for things that shouldn't be in the SDist but should be in the git repo.

#

Ahh, I see why _skbuild is slipping in, it's due to it having some of those files, and it's added as an exclude, so the includes are taking priority.

#

The directory thing would fix that

limber zodiac
#

I should then flip the inclusion/exclusion logic. Just add excludes explicitly and not the other way round.

urban turtle
#

That's what works best, usually.

#

The directory fix will help a lot (and it's faster!), and we'll have an explicit mode for this in the future.

limber zodiac
#

is that part of 0.12?

urban turtle
#

The fix I think is fine for a patch release (which is soon).

#

Mode probably could probably be 0.12, or maybe 1.0 if it misses that.

limber zodiac
#

nice!

urban turtle
#

The check-sdist tool is nice if you'd like to check with that.

#

I'd recommend not putting ninja and cmake in your build-system.requires, scikit-build-core will intelligently add them as needed (and you should even see a warning the way it is now)

limber zodiac
#

I am trying out the excludes patterns now. I wanted to ask you another question. How can I package the _version.py file that setuptools-scm generates inside the sdist? Do I need to run cmake before sdist generation?

limber zodiac
urban turtle
#

Oh, you can remove setuptools_scm too when using the provider, it adds it for you. Doesn't really hurt anything though.

#

If you use the tool.setuptools_scm options, it works just like it does with setuptools. It adds the file in-tree and you need to add it to the explicit includes (one reason that's there and overrides the excludes, since it will be in your gitignore!). However, you can also use the generate feature which gives you three choices about where to add the file and automically handles the include part.

unreal quail
# urban turtle I'd recommend not putting ninja and cmake in your build-system.requires, scikit-...

Side-comment/observation mostly for @urban turtle because I stumbled over that sometime in the past: This auto addition will not work if you are building without build isolation since scikit-build-core auto adds CMake and Ninja to the build dependencies, which are ignored for builds without isolation. In that case, one needs to remember to explicitly install CMake and/or Ninja if a more recent version is needed than currently provided by the system.

urban turtle
#

If you are building without build isolation, you are installing everything yourself anyway, not pulling from the build-system.requires table?

unreal quail
# urban turtle If you are building without build isolation, you are installing everything yours...

Yeah. I just stumbled over this because the isolated build will automagically add CMake & Ninja as required without them being present in the build-system.requires table, while the non-isolated build will not pull these in.
I was installing all the build-system.requires dependencies manually for the non-isolation build. I suppose there is no easy way for scikit-build-core to inject the CMake and/or Ninja packages in a non-isolated build, right? Probably by definition of a non-isolated build.

vernal hazel
#

By definition.

#

I mean, technically scikit-build-core could mess with the environment, but that would be not be playing nice.

urban turtle
#

Hmmm, interesting command idea. scikit-build build-deps could spit out the list of packages needed to build. (We don't really have a working CLI yet)

vernal hazel
#

FWIW, this also goes into the heart of why a --only-build-deps pip install flag is hard, because build dependencies can be added dynamically. Most users of such a flag probably don't care and wouldn't want pip to call the backend at all, but it wouldn't be fully spec-compliant and cause issues when there are dynamically added dependencies.

#

I don't know what the current consensus on the design is, the conversation happened way before I became a pip maintainer.

urban turtle
#

I mean, you can do this .venv/bin/python -c "from scikit_build_core.build import get_requires_for_build_wheel as bw; print(bw({}))" -> ['cmake>=3.26.1', 'setuptools-scm']. Combined with tomllib you could print out what you needed.

limber zodiac
urban turtle
#

But for a general tool it would have to support -C, --wheel/--sdist

vernal hazel
#

Oh yeah, I forgot that sdist dependencies can be different. Fortunately, pip never builds a sdist and exposes it to the user so that doesn't actually matter(...?)

unreal quail
urban turtle
#

Depends on what you are doing and why you want them. 🙂 But in pip's case, doing just the build one is likely what you'd want since it can't build SDists.

vernal hazel
#

we have pip wheel as the --build flag, no?

#

Ah you meant --wheel. That makes more sense.

urban turtle
#

A pip install --build-deps would have to be two stage, first installing the static build deps, then running the build backend to get the dynamic ones

vernal hazel
#

IIRC people didn't want pip to call the backend for some reason.

#

I should read that thread in full...

limber zodiac
# limber zodiac Setting the excludes explicitly as you had suggested works so much better. The o...

I almost got it to work. With the following config, I am able to exclude all of 3rdparty/cutlass except the two sub-directories I want to add to the sdist.

[tool.scikit-build.sdist]
cmake = false
exclude = [
    "examples",
    "profiler",
    "benchmarks",
    "docker",
    "scripts",
    "tests",
    "docs",
    "3rdparty/mscclpp/**/*",
    "3rdparty/gooogletest/**/*",
    "3rdparty/nvbench/**/*",
    "3rdparty/spdlog/**/*",
    "3rdparty/composable_kernels/**/*",
    "3rdparty/cutlass/**/*",
    "ci",
    "aot_build_utils",
    "Jenkinsfile",
    "format.sh",
    "custom_backend.py",
    "version.txt",
]
include = [
    "3rdparty/cutlass/include/**/*",
    "3rdparty/cutlass/tools/**/*",
]

Although, for some reason googletest is getting added despite me excluding it.

limber zodiac
#

@urban turtle Thank you for your support today! I very much appreciate it.

urban turtle
#

The bottom of the Google page used to have goooooooogle IIRC. 🙂

river smelt
limber zodiac
#

Hi! I need some help with enabling editable installs in my repo is https://github.com/diptorupd/flashinfer/tree/feature/scikit-build. The issue may be due to something incorrect I am doing in my CMake scripts.

The project is a CUDA header-only kernel library with a C++ API and a Python API that exposes PyTorch extension modules. The headers are installed as part of the package inside ("${SKBUILD_PLATLIB_DIR}/flashinfer/include") since the Python API includes a JIT compilation functionality. The installation works as expected during Wheels builds. When doing an editable install I encounter the following behaviour:

a) the headers get installed properly the first time I run python -m pip install --no-build-isolation --config-settings=editable.rebuild=true -ve..
b) Any subsequent change to a header lead to a rebuild of the kernels and the torch extension as expected. The rebuilt extension gets installed into the ${SKBUILD_PLATLIB_DIR}/flashinfer/correctly.
c) But, the headers themselves are not reinstalled.

It will get great to get some insight. The most relevant CMake files in my repo are: https://github.com/diptorupd/flashinfer/blob/feature/scikit-build/libflashinfer/CMakeLists.txt, https://github.com/diptorupd/flashinfer/blob/feature/scikit-build/flashinfer/csrc/CMakeLists.txt.

limber zodiac
#

@urban turtle I investigated some more about the editable install configuration for my project using scikit-build-core and wanted to clarify a conceptual thing.

In my project I need to install some CUDA headers inside the package as they are used at runtime for JIT compilation. I have the following install command in my CMakeLists.txt
https://github.com/diptorupd/flashinfer/blob/49d94305fbc100c6959f41a81ddd618a5b21ad28/libflashinfer/CMakeLists.txt#L176

Then to look up the location of the headers, I have a method like NumPy's get_include:

import os
import pathlib
from sysconfig import get_path

def get_include():
    platlib = get_path("platlib")
    rootdir = os.path.join(platlib, "flashinfer")
    include_dir = os.path.join(rootdir, "include")
    return str(include_dir)

The logic works fine for regular wheel build. For editable installs I see the headers getting installed into site-packages\flashinfer\include when I first run python -m pip install --no-build-isolation --config-settings=editable.rebuild=true -ve. However, on subsequent re-installs of the headers I see them getting installed only into the tmp directory. Is this the expected behaviour?

GitHub
flashinfer/libflashinfer/CMakeLists.txt at 49d94305fbc100c6959f41a8...

FlashInfer: Kernel Library for LLM Serving. Contribute to diptorupd/flashinfer development by creating an account on GitHub.

urban turtle
#

Is this the expected behaviour?
I don't think so. The rebuild uses logic that's embedded into the hook it installs, so I bet we might be making a mistake there.

limber zodiac
#

Log excerpt from first pip install -ve.

  -- Build files have been written to: /home/diptodeb/devel/flashinfer/_skbuild
  2025-04-28 16:28:24,198 - scikit_build_core - WARNING - Unsupported CMAKE_ARGS ignored: -DCMAKE_BUILD_TYPE=Release
  *** Building project with Ninja...
  2025-04-28 16:28:24,202 - scikit_build_core - INFO - RUN: /data/diptorup/micromamba/envs/fi-devel-torch26-cude126-py312/lib/python3.12/site-packages/cmake/data/bin/cmake --build _skbuild -v
  Change Dir: '/home/diptodeb/devel/flashinfer/_skbuild'
  Run Build Command(s): /data/diptorup/micromamba/envs/fi-devel-torch26-cude126-py312/bin/ninja -v
  ninja: no work to do.
  *** Installing project into wheel...
  2025-04-28 16:28:24,278 - scikit_build_core - INFO - RUN: /data/diptorup/micromamba/envs/fi-devel-torch26-cude126-py312/lib/python3.12/site-packages/cmake/data/bin/cmake --install _skbuild --prefix /scratch/diptorup/tmp/tmp_ara78wm/wheel/platlib --strip
  -- Install configuration: "Release"
  -- Installing: /scratch/diptorup/tmp/tmp_ara78wm/wheel/platlib/flashinfer/include/flashinfer/configure.h
  -- <elided ...>
  -- Installing: /scratch/diptorup/tmp/tmp_ara78wm/wheel/platlib/flashinfer/include/flashinfer/layout.cuh
  -- <elided ...>
  *** Making editable...
  2025-04-28 16:28:24,469 - scikit_build_core - INFO - Discovered Python package at flashinfer
  *** Created flashinfer-0.2.5.post17-cp39-abi3-linux_x86_64.whl
  Building editable for flashinfer (pyproject.toml): finished with status 'done'
  Created wheel for flashinfer: filename=flashinfer-0.2.5.post17-cp39-abi3-linux_x86_64.whl size=226183 sha256=b9a0732e8c2ee91c84a2eb8192fdb5ba395b3cfd87f1562e6040582fdd0d1f2d
  Stored in directory: /scratch/diptorup/tmp/pip-ephem-wheel-cache-ilsb8g0k/wheels/3b/20/68/059c8d9639eca00a03199fb022caa95846b079d98094cddd18
Successfully built flashinfer
Installing collected packages: flashinfer
Successfully installed flashinfer-0.2.5.post17
#

The first time I build how is the include directory copied from the tmp directory into the actual site-packges directory /micromamba/envs/fi-devel-torch26-cude126-py312/lib/python3.12/site-packages/flashinfer/include?

urban turtle
#

It's copied to the temp wheel directory, then that temp wheel is installed. That's how wheel-based editable installs work. The auto rebuild feature directly installs, since it's not able to rely on your installer's temporary wheels.

limber zodiac
#

I don't think so. The rebuild uses logic that's embedded into the hook it installs, so I bet we might be making a mistake there.

auto rebuild feature directly installs,

The auto rebuild should install into the actual install location (site-packages/flashinfer/include) and not into the temp wheel. Is that a right understanding of your first statement?

urban turtle
#

Yes

plush forge
#

Bit of a long shot but is it possible to avoid full rebuilds without having to specify --no-build-isolation? Finding that just setting build-dir isn't enough. Guessing it might be something quirky with temporarily installing cmake but I wouldn't really be able to prove that.

verbal charm
#

The issue is typically with the python build dependencies. E.g. if you use pybind11 then it will reconfigure and rebuild each time

plush forge
#

Ah. I'm using nanobind but it's probably the same problem. I suppose I could set up a venv to get nanobind pre-installed but at that point may as well just pre-install everything into the venv and pass --no-build-isolation :P
Still, knowing where the problem probably was is valuable information so thank you.

verbal charm
#

You can conditionally add the build dependency for nanobind, although we don't have a built-in way for doing that based on find_package calls

plush forge
#

--no-build-isolation appears to solve multiple problems in my life. This may be a bit of a tangent.
I asked my question in the first place because I was having trouble building my package as part of a docker image or inside a docker container. Project is C++ with build dependencies on scikit-build-core and nanobind. pip install was using twice as much memory inside the docker build than it was on the host (~7GBvs~3GB). Even worse, project had dependencies with CUDA code and building that took 16-24GB. When the host is a Jetson with 8GB memory, that doesn't quite work!
For giggles, I ran pip install --no-build-isolation inside the container and my problem appear to have disappeared, the memory usage while building was the same inside the container as I would see outside.
It's a bit confusing. If it was just the bloat of temporary requirements, I could imagine higher memory usage but I'd expect it to happen immediately and not continue climbing as things were being built. And I wouldn't expect the CUDA compilation to take so much more memory either when the cost is negligible on the host.
That is to say, I don't know if this is a scikit-build-core issue, a pip issue, a PEBKAC, or something else.

violet quail
#

Perhaps the build requirements fetched in the isolated environment differ (in version, particularly) from what your container image has?
but yeah if you're using a container then build isolation is normally redundant, unless you have to build multiple things with conflicting build tools for some reason

#

(which TBF, I have no idea how common that is!)

plush forge
minor wasp
#

If you already have a Docker container for building stuff, it’s indeed the cleanest solution to install your build dependencies system-wide instead of using virtual environments. That way you have one predictable build environment (the Docker container) instead of environments inside of environments.

The same applies if you e.g. want to use another dependency manager like when you make a package for a Linux package manager that wraps a Python package. You’d like to use that package manager’s build dependency system here instead of having pip or so download stuff. The disadvantage is that you need to translate the dependencies (e.g. on Arch Linux, some system packages contain multiple Python packages so you don’t have a 1:1 package name mapping).

rotund hearth
quaint river
#

in uv, we have uv init --build-backend scikit that tries to kinda invert the problem steps by setting you up with a package and then letting the user fill in their C or C++, though i'm not sure if that fulfills the distribution aspect

formal pulsar
#

Having something like maturin new for the other backends would be extremely helpful. The fact that maturin new gives you a working, importable python project out of the box made it easy to experiment. I could immediately try small changes, like switching sum_as_string(u32, u32) to sum_as_string(String, String), and generally get a feel for how everything fits together. In contrast, if I just want to quickly try an equivalent C extension that implements something as simple as sum_as_string(int, int) -> str, it currently takes a lot more setup and effort.

urban turtle
#

You can use cookiecutter gh:scientific-python/cookie, that supports scikit-build-core, meson-python, and maturin (and several other pure python backends). uv init also supports scikit-build-core and maturin, and is a bit lighter weight. We could probably add something like this pretty easily to scikit-build-core itself, too. (The light-weight one)

#

Hmm, interesting about distribution. That is going to be tied to CI. The scientific-python cookiecutter assumes GHA (or GitLab CI if you put a gitlab URL for the project, a little less featured though)

coarse ore
#

Ralf Gommers was just talking to me about a project to test all these combinations of tools quickly to aid development of the new stable ABI for python 3.15. Seems to be a lot of overlap. I’ll bring it up with him that there seems to be a lot of overlap in this space and it’s also a documentation issue

coarse ore
coarse ore
#

shower thought: make a website called pyproject.new that asks a bunch of questions about the kind of project (app/library, pure python or extensions, which language the extensions are in, maybe other axes) and then generates a URL a user can clone with git to get going.

dusty moth
coarse ore
#

anything more constructive to offer than a meme?

digital sinew
#

Also note that the .new TLD is one of those "brand name" TLDs that's absurdly expensive for no good reason
(Price in USD)

image.png
coarse ore
#

yikes, different url hack needed then

verbal charm
#

!xkcd 927? 😛

#

If there is an interface you like among the well-known tools, we could probably help make some templates for those rather than making yet anothe thing

dusty moth
urban turtle
#

I've played with the idea of running something like cookiecutter in WebAssembly. I also worked with a student in the past to make a Textual cookiecutter/copier interface, the project is still around, still a CLI though

plush forge
#

Crazy coffee thoughts.
When using scikit-build-core as a backend for a C++ project using Conan, would it be possible to have the conan dependencies installed by the build system instead of as a separate step? This gets a bit tricky because I don't know how dependencies are actually resolved for a package, could be impossible if it really depends on pulling from an index or something python package-like

verbal charm
#

Doesn't fixing build-dir do that?

plush forge
#

I'm not following, sorry. I should also say that this isn't a specific trouble I'm encountering, just a thought (though of course it's relevant to work I'm doing)

verbal charm
#

Coming from limited to no knowledge of how conan works, but if the build cache (wherever it builds/installs the dependencies) is under the CMake build directory or somewhere similarly predictable, then if you make that persistent (e.g. setting tool.scikit-build.build-dir) then the dependencies would persist across builds

#

On re-read, are you more looking about running conan/vcpkg etc. commands within pip install, right before the scikit-build-core's cmake build? It may depend what do you need as arguments for those commands, but it could be possible to expose a hook for that. Would everyone have to use conan to build your package though?

plush forge
# verbal charm On re-read, are you more looking about running conan/vcpkg etc. commands within ...

yes this is the scenario I'm imagining. And I can understand why it wouldn't be a feature in general because of that hard requirement on conan (though conan is available on pypi!) as well as conan's build cache being separate from everything else but in my case, it's all in-house building.
The real value add, and I suppose this might be the harder thing to do depending on what hooks are available, would be better layer management in a dockerfile. With uv, you can do uv sync --locked --no-install-project to install the dependencies of a project without installing the project itself. How it does that, I don't know. That's where I'm wondering if it's possible at all.

verbal charm
#

About adding it I've seen projects that wrap around scikit-build-core overwriting some functions to inject some calls, you can of course do that. Putting it in scikit-build-core proper would not be impossible, it's more of a question of what interface to provide under (can go into more workshoping in an issue if you want to open it)

#

For something like dockerfile, wouldn't you want to run conan separately for the caching of layers?

#

On the uv note, that would not be relevant unless you build with no-build-isolation

plush forge
#

While there is a general component to this, I feel like my specific goals might be too niche to justify opening an issue. And there's too many ways to utilize conan in a project that it would be hard to handle them all.

As for the dockerfile, "yes". But there's a few ways to go about that. You could of course run conan separately. And technically that is the best way to do it since those C++ dependencies probably aren't changing as often as Python dependencies. But say the project is in a workspace and every other project in that workspace just requires the uv sync to handle dependencies. Now this other package requires special treatment. And if the image is configured to not install the package, you're still paying the cost of installing its dependencies because there is no conditional layer.
I suppose there's no perfect solution to the problem, just a lot of thoughts effectively tangential to scikit-build-core as a project. maybe I'm rambling too much 😅

verbal charm
#

On the nichness, I don't think it would be that niche. Basically would not support conan specifically, just a way to hook into various points of the build process

#

Hooking into the uv sync, that would not be possible by us, and I don't think uv would want to expose that either. Maybe a different question, why not break the dockerfile into smaller components (and join them with multiple FROM at the workspace containerfile) that can be templated and extended as needed?

plush forge
verbal charm
#

There are peps for supporting system/distro packages, but discussion and progress on those seem to stall or get sidetracked

plush forge
#

If anyone involved in those discussions are prone to impulse thoughts like me, can definitely understand why that would happen :P

verbal charm
#

It's more like too many conflicting opinions coming from different ecosystems ultimately discovering why distro packaging is a thing and the way it is now

#

Cargo ecosystem has a better approach to this imo. It's swiss cheese when it comes to implementation, but the concept is sane

coarse ore
#

this is the problem that conda/pixi solves, FWIW

#

mlx is basically, "pytorch, but on apple silicon GPUs"

verbal charm
#

I thought you would be showing a custom setup.py monstrosity. That one is quite straightforward

frigid ginkgo
plush forge
# frigid ginkgo What build dependencies do you need?

sorry for taking so long to get back to you, I had to remember why I was asking for this in the first place even though I rambled about the specifics later on.
build dependencies would just be conan I think. But the tricky bit is that conan has its separate list of dependencies that it wants to install when invoked and those dependencies are from a different, non-python index (conan center).