ENH: BLD: enable building SciPy with Meson #14847
Conversation
|
Nice work! |
|
Exciting times! Thank you for this vital infra work @rgommers |
|
Thanks Ralf! |
|
Does the new build system provide a "smart" build+install yet, where I can edit, say |
Yes, ninja does that intelligently and rebuilds only the subset of files+dependencies required |
|
If I'm reading the output correctly, it looks like the change is detected, but the |
|
While you can pass |
|
I suspect what I am seeing is mesonbuild/meson#9049 |
|
In that case, the change is NOT detected? |
|
Perhaps I misunderstood that issue; I'm also not up-to-date on any workarounds that are in the meson.build files, or on meson itself. Currently, I'm just a user who hasn't RTFM, hoping that https://github.com/scipy/scipy/blob/master/doc/source/dev/contributor/meson.rst is enough to get me going. Here's what I did. First, I removed the (That's a new test that I'm adding--it is not in the master branch.) Then I changed When I ran the command again, with no additional changes, the output that was I've checked the timestamps on the |
|
Thanks everyone! @WarrenWeckesser yes indeed, that's mesonbuild/meson#9049. Dependency tracking of We touch |
|
I think in total we have 3 known issues:
Those are highest priority to work on. After that, we can deal with supporting other BLAS/LAPACK libraries, cross-compiling, and then switching over sdist/wheel generation. |
|
With pxd/pxi mess, I'd mention a workaround (suggested by Ralf a while ago) which works today: use a two-step incantation in meson.build and set https://github.com/ev-br/mc_lib/blob/master/mc_lib/meson.build#L23 |
Is it currently possible to make a wheel with meson? |
|
This is currently handled by https://pypi.org/project/mesonpep517/ which handles the pep 517 build backend and creates the dist-info metadata. Meson could grow direct support, and we want to add it, but it's not perfectly clear how to. Part of the problem is that we probably need a toml parser, but the python steering council seems to be hesitant to add one to the stdlib despite it being a core requirement of the metadata specs, required for basically any part of the packaging ecosystem, a stable spec for a while now (which was the original explanation for why a toml module was not added years ago -- "we will, but first we want to release a toml 1.0 spec") and currently being vendored into pip, setuptools, flit, etc because it is a PEP violation (bootstrap cycle) to require flit to build tomli, and have flit depend on tomli, while pip and setuptools meanwhile have a "vendor everything because we cannot have dependencies" policy. Meson has a "only stdlib, we do not vendor and we do not add dependencies either" policy. This is needed because we are being used as a low level tool for bootstrapping operating systems, and it needs to be easy to run meson with nothing but a python interpreter. So, mesonpep517 is where people are testing out "wheel creation based on meson". It can require dependencies like a toml library, and once the ecosystem coalesces we may be able to merge mesonpep517 directly into meson, or use the experience and lessons of that project to design another API. tl;dr Let's see where the future takes us. |
|
Thanks for the response. Ive been thinking about experimenting with cibuildwheel for how we build scipy wheels, but it seems like it's too early at this point? |
|
Notice that there is already consensus on including tomli in the stdlib, just a PEP is missing https://discuss.python.org/t/adopting-recommending-a-toml-parser/4068/84?u=astrojuanlu |
Yes it is, however (a) you need a patch, and (b) it's not tested yet so you may run into various sharp edges. Building a wheel will not be too difficult in the future, with either mesonpep517 or meson-python. So I'd say it's fine to start playing with Here's the easiest way to get a wheel to build that you can install locally (note that it won't be manylinux compliant etc., so not suitable for redistribution - but fine for use in the env in which you built it), if you want to try:
diff --git a/pyproject.toml b/pyproject.toml
index d3a01ecfe..546c965d1 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -8,8 +8,10 @@
# "pybind11>=2.4.3,<2.5.0",
[build-system]
+build-backend = "mesonpep517.buildapi"
requires = [
"wheel",
+ "mesonpep517",
"setuptools<60.0", # do not increase, 60.0 enables vendored distutils
"Cython>=0.29.18",
"pybind11>=2.4.3",
@@ -45,7 +47,7 @@ requires = [
[project]
name = "SciPy"
license = {file = "LICENSE.txt"}
-
+version = "1.9.0.dev0"
maintainers = [
{name = "SciPy Developers", email = "scipy-dev@python.org"},
]
@@ -76,7 +78,6 @@ classifiers = [
"Operating System :: Unix",
"Operating System :: MacOS",
]
-dynamic = ['version', 'description']
[project.optional-dependencies]
test = [
@@ -109,3 +110,8 @@ documentation = "https://docs.scipy.org/doc/scipy/"
source = "https://github.com/scipy/scipy"
download = "https://github.com/scipy/scipy/releases"
tracker = "https://github.com/scipy/scipy/issues"
+
+[tool.mesonpep517.metadata]
+name = 'scipy'
+version = '1.9.0.dev0'
+I gave this a quick spin on Linux, and that does work. It should work cross-platform, and in conda envs too. As you can see from the commands, there's some things to fix UX-wise.
I t
8000
hink it should require a separate package in the end, but ideally under the Meson org. A simple "zip up the result of |
"Consensus that it's okay to propose it in a PEP" != "consensus to add it", and the larger issue here is that the topic has been carefully bikeshedded on multiple fronts for multiple years for something that should not be permitted to be missing. :) I hope it will happen, but I won't believe it until I see it.
I didn't think that meson should handle that. setuptools via setup.py doesn't handle it either, if I understand correctly? Aren't these both command-line tools run after you have a wheel and want to convert "a wheel" to "a manylinuxed wheel" (or a macOS I-dont-know-what-to-call-it wheel)? It's definitely not something you want to run for any reason other than "I'm uploading a PyPI release for a project I maintain". (It would be trivial to add a |
correct, 2x. but, see below
I agree. Unfortunately, the Python packaging crowd needs some educating/convincing there. Some of the design of |
|
That sounds like a truly concerning case, even more concerning than doing Why is the PyPI wheel relevant at all, if you never use it? And if you do use it, then building from source is irrelevant. And even if you did care about making them the same, auditwheel repair won't help!!! It explicitly doesn't handle glibc versioned symbol differences, and it in general doesn't help at all for https://reproducible-builds.org |
|
Yeah, I know it doesn't make too much sense (also, why cache local builds at all, they should be unique). I'm working on a set of docs for conceptual issues like these. I've also had discussions around UX of build tools, and even "what is a build system" that were truly mind-boggling. Setuptools for example has declared "it doesn't have a supported CLI, and no one should use a build system CLI anyway" - why would one ever need more than a single PEP 517 hook:) |
|
caching is much better done via a proper build system that does incremental builds + ccache :D For local installs, roundtripping everything through a zip file "for consistency" is just a layer of bureaucracy... Meson doesn't consider it a burden to implement a CLI, that's our bread and butter -- and so is installing. That's why I want the future to hold a core meson feature for doing pep 517 builds / pep 621 dist-info. Meson supports nearly everything that "a python package" needs and is only missing
So it seems completely reasonable to support that, but... implementing brand new support for something and deliberately choosing a method that doesn't match current official recommendations for interoperability (pep 621) seems really, really bad, so for this reason alone we really, really want toml support. And in order to provide the correct experience demanded by the rest of meson -- functionality should not randomly fail to work because of python dependency issues that cannot be bootstrapped -- our choices are either ignore pep 621 entirely, or leave dist-info and wheel creation to other layers like mesonpep517. Otherwise we will end up with people developing base system libraries at the heart of an OS bootstrap, which include python bindings that either cannot be built due to errors or silently provide a broken install that is missing dist-info and cause heisenbugs when pip / build / importlib.metadata report that dependencies are missing and return errors. I had at one time had high hopes that python 3.10 was going to include a toml library in fulfillment of the promise "we are just waiting for the format spec to stabilize as 1.0". :( :( |
|
Currently |
|
Responding to my own comment:
Yes, @rgommers noted this over in #13615 (comment):
|
|
There's no MKL support yet, and yes it will follow (and Netlib, BLIS, and generic BLAS). For order of next steps, see #14847 (comment). The issue for generic and configurable BLAS/LAPACK support in Meson is mesonbuild/meson#2835. |
|
FYI, re: TOML in the stdlib,
PEP 680 was accepted and a PR to CPython was opened: python/cpython#31498 |
This PR adds:
-Werror.As I explained in #13615 (comment), having the
-Werrorbuild merged into SciPy master will be valuable, because new build warnings will keep leaking in without CI (and it took a lot of effort to get to zero warnings).I personally also want to develop on SciPy with Meson - it's far faster and more pleasant to use than the distutils-based build. I imagine some other maintainers may want to do this too.
There is enough left to do after this PR. In order of importance:
Windows builds(EDIT: works now), non-OpenBLAS BLAS/LAPACK support,a more polished(EDIT: could be improved, but now has parity withdev.pyinterface (similar toruntests.py)runtests.pyat least), and sdist/wheel generation (see the tracking issue).I have completely rewritten the history of the
mesonintegration branch in my fork, to have one commit per submodule or other tool/topic. I have carefully added other contributors as authors or co-authors (if I forgot anyone, please comment or ping me!), namely: @czgdp1807, @Smit-create, @HarshCasper, @matthew-brett and @dhruv9vats.Right now the Meson version needed is
0.60.0rc1, ormaster0.60.xreleases and master work now). This is pinned inenvironment_meson.yml. I have 8000 not includedpyproject.tomlor other changes, so this should not interfere with the regularsetup.pybased build or release process.Once we merge this, I volunteer to resolve issues people may have with adding support, in case they are making changes in a
setup.pyfile and then find that the Meson CI job doesn't pass. This should be very little work.There are also improvements in here, like the CI caching and enabling the use of
act, which we should sync to the other CI jobs because they're generally useful. But that's left for later, to not have any cross-talk with other PRs - this PR is Meson-only as much as possible.For more context see: