ENH: meson: implement BLAS/LAPACK auto-detection and many CI jobs #24893
Conversation
…eel build]
This reimplements the auto-detection and many of the switches that
`numpy.distutils` offered. Beyond that, it implements several new
features:
- Auto-detect the symbol suffix for ILP64 OpenBLAS (can be none or
`64_`)
- MKL ILP64 support, threading control, and use of the layered library
model for MKL >=2023.0
- FlexiBLAS support (LP64 and ILP64)
- Support for the upcoming standard in Reference LAPACK for `_64` ILP64
symbol suffix convention.
- A test suite for BLAS/LAPACK libraries, covering:
- OpenBLAS: LP64, ILP64 detected via pkg-config and with a "system
dependency" (i.e., custom code inside Meson)
- MKL: LP64, ILP64 (layered) and LP64 (SDL)
- Accelerate: LP64, ILP64 on macOS >=13.3
- FlexiBLAS: LP64, ILP64 on Fedora
- ATLAS (LP64, via pkg-config only)
- BLIS (LP64, via pkg-config only)
- plain libblas/liblapack (Netlib, LP64 only)
The list of libraries that is tried with the default 'auto' setting
excludes a couple of libraries, because they're either no longer
developed (ATLAS), not mature (libflame), or can't be tested and
may be re-added later (ArmPL, ssl2). Those libraries can still be
quite easily used via pkg-config.
The new CI jobs are running by default right now. Once things settle
down, the plan is to disable them by default and allow triggering
them via a `[blas ci]` command in the commit message (just like for
wheel builds).
Docs will be included in a separate PR with the pending rewrite of
all the build/install docs. For now, the CI jobs and the
`meson_options.txt` file serve as guidance for how to use this.
Note that the test suite contains a few hacks, because of packaging
bugs for MKL on PyPI (broken .pc files) and BLIS (missing .pc file in
Debian).
0616e86 to
34afbb9
Compare
[skip actions] [skip cirrus] [skip circle]
|
This is now all green. I'd say it's ready for review - but at the same time I should flag that such a large PR probably isn't fully reviewable. The proof is in the pudding I'd say, so I suggest reviewing the added CI jobs mainly, and then quickly merging this, backporting it to the I on purpose made this a single large commit for ease of backporting (please squash-merge, there's a tiny second commit). There's one or two follow-ups that will be for 2.0 only, which I'll hold off on for a bit until we got some more real-world testing done with 1.26.1:
|
I was planning on it :) Agree that the proof is in the pudding, best to get it out to the customers. |
|
Thanks Ralf. Note that 1.26 is not currently building wheels with Accelerate. |
|
Thanks for the quick review & merge!
Yes I know, I think we should keep it that way. We need an extra set of wheels for macOS >=14.0 in the 2.0 release; I want to touch as little as possible in the 1.26.x wheel build setup. |
|
These backports make me a bit nervous because of the changes in |
|
In case it doesn't pass straight away, I can investigate tomorrow morning. There's only a few tweaks in the end aside from the new jobs. |
|
Aside: how ready do you think this is for merging into meson upstream? It is still a draft PR but I see this past week you've been updating it (and here too obviously). |
|
It's very close to ready I think - the Meson API with What I'd like to do though is wait for at least a month after the NumPy 1.26.1 release before proposing to merge it into Meson, to ensure we get this tested on a wide variety of platforms and user configs, and know that things are pretty robust. |
|
Excellent! Looking forward to it. |
This reimplements the auto-detection and many of the switches that
numpy.distutilsoffered. Beyond that, it implements several new features:Auto-detect the symbol suffix for ILP64 OpenBLAS (can be none or
64_)MKL ILP64 support, threading control, and use of the layered library model for MKL >=2023.0
FlexiBLAS support (LP64 and ILP64)
Support for the upcoming standard in Reference LAPACK for
_64ILP64 symbol suffix convention.A test suite for BLAS/LAPACK libraries, covering:
The list of libraries that is tried with the default 'auto' setting excludes a couple of libraries, because they're either no longer developed (ATLAS), not mature (libflame), or can't easily be tested and may be re-added later (ArmPL, ssl2). Those libraries can still be quite easily used via pkg-config.
The new CI jobs are running by default right now. Once things settle down, the plan is to disable them by default and allow triggering them via a
[blas ci]command in the commit message (just like for wheel builds).Docs will be included in a separate PR with the pending rewrite of all the build/install docs. For now, the CI jobs and the
meson_options.txtfile serve as guidance for how to use this.Note that the test suite contains a few hacks, because of packaging bugs for MKL on PyPI (broken .pc files) and BLIS (missing .pc file in Debian).
Closes gh-24808
Closes gh-24846
Addresses the most important remaining task from the introduction of Meson (see gh-23981).