ENH: Vendor meson/meson-python for multi-target build support #24365
Conversation
The sources included are only those for the build backend itself, plus the necessary license files and a .gitignore, not any other files (tests, docs, ci, etc.). The files included are from meson-python commit 2e99c7b5bce53403a9, as of 26 July 2023. This is the most recent commit, and are a development version (0.14.0-dev). No changes have been made to any included files, except for removing some entries from .gitignore.
This vendors the current state of mesonbuild/meson#11307, which is the "feature module" for the multi-target build support that NumPy needs to build its SIMD support (by Sayed Adel). That PR is, as vendored, based on top of the state of the Meson master branch in commit 9d88d0d5c, which is a dev commit in between the 1.2.0 and 1.2.1 tags from 18 July 2023. No modifications have been made to any of the vendored files.
Note that this is a bit messy. I tried to vendor spin's `meson.py` separately, but it's not possible to do so cleanly as far as I can tell. The import machinery is unhappy bouncing around between `bin/spin`, `.spin/cmds.py and `.spin/meson.py`. So it was either folding the `spin/cmds/meson.py` content all into cmds.py, or vendor all of the spin package. This seems to work.
These licenses are for: - Meson - meson-python - spin Importantly, Meson is Apache 2.0 licensed, which is incompatible with NumPy's BSD-3 license. However, the Meson files are not installed, but only used as a build tool. All installed files are still BSD-3 or compatible with BSD-3.
|
|
||
| # Try joining file paths that contain spaces | ||
|
|
||
| reg_start = re.compile(r'^([A-Za-z]:)?/(.*/)*[^./]+$') |
Check failure
Code scanning / CodeQL
Inefficient regular expression
I think this is quite understandable in order to move forward with numpy, it's a very friendly fork indeed. However, it seems a bit unfortunate to vendor meson-python as well, just to set the meson_cli path (I think?). I wonder if it would be possible to instead add your own tiny in-tree build-backend that:
And stick a wrapper in that directory which invokes the meson.py script properly. It may also make sense to include meson as a submodule -- this would probably be friendlier for |
+1 |
Good idea, thanks. I'll give that a try this morning.
I also edited the meson-python dependencies to remove
I think the main issue is keeping that working permanently. The PR branch is likely to be rebased multiple times, so we can't rely on it. So we'd have to create a repo at https://github.com/numpy/meson with branches or tags to point to, and keep it there forever. I hope this is a short-lived vendoring, and it can go away towards the end of the year (please correct me if you think that that is unlikely to be the case). We can go this way if other maintainers prefer this, but I think I personally have a preference for not having another repo around permanently. |
I think it will be easier to reason about upstream changes if we manage a fork. Meson seems to still be under active development (over 12 PRs merged in 3 days, and while we may not need many of those changes, managing that churn will be easier if it is a separate repo. Once our changes are merged upstream, we can archive the repo. |
|
Okay, thanks Matti. I'll create a new |
|
Alternative PR up at gh-24379. |
|
The approach in gh-24379 works and seems preferable, so closing this PR. |
The purpose of this PR is to enable merging the SIMD support for the Meson builds in gh-23096. That PR relies on mesonbuild/meson#11307, which is a significant new feature for Meson. That is multi-target support, allowing us to recompile the same sources multiple times, with compile flags set for different sets of SIMD instructions. Given that that PR is not going to get merged into Meson in time, we'll want to vendor it in order to be able to use it now.
This PR vendors the following:
mesonas of commit 9d88d0d5c (1.2.1-dev, last change 18 July 2023), plus thefeaturemodule from modules: The 'features' module mesonbuild/meson#11307meson-pythonas of commit 2e99c7b5bc (current HEAD, 0.14.0-dev0, last change 26 July 2023)spin/cmds/meson.pyAnd then does the required plumbing in order to use the vendored code for building.
I have tested that the combination of this PR and gh-23096 works together, and that the sdist/wheels produced look the way they should. The plan is to merge this PR first, and then rebase gh-23096 on top (Cc @seiko2plus).
I chose vendoring everything over packaging up a separate package (something like
meson-numpy), because that'd be a real pain for distributions, they'd have to package that in all of their distros as a new package in other to be able to build numpy 1.26.0.Licensing-wise: Meson is Apache 2.0 licensed, but because we don't install it as part of a built numpy package, I don't think that's a problem.
@eli-schwartz in case you have any insights or concerns here regarding the approach, or things you'd like to see in terms of validation for mesonbuild/meson#11307, please let us know.