8000 ENH: meson: implement BLAS/LAPACK auto-detection and many CI jobs [wh… · numpy/numpy@ad4f480 · GitHub
[go: up one dir, main page]
Skip to content

Commit ad4f480

Browse files
rgommerscharris
rgommers
authored and
charris
committed
ENH: meson: implement BLAS/LAPACK auto-detection and many CI jobs [wheel build]
This reimplements the auto-detection and many of the switches that `numpy.distutils` offered. Beyond that, it implements several new features: - Auto-detect the symbol suffix for ILP64 OpenBLAS (can be none or `64_`) - MKL ILP64 support, threading control, and use of the layered library model for MKL >=2023.0 - FlexiBLAS support (LP64 and ILP64) - Support for the upcoming standard in Reference LAPACK for `_64` ILP64 symbol suffix convention. - A test suite for BLAS/LAPACK libraries, covering: - OpenBLAS: LP64, ILP64 detected via pkg-config and with a "system dependency" (i.e., custom code inside Meson) - MKL: LP64, ILP64 (layered) and LP64 (SDL) - Accelerate: LP64, ILP64 on macOS >=13.3 - FlexiBLAS: LP64, ILP64 on Fedora - ATLAS (LP64, via pkg-config only) - BLIS (LP64, via pkg-config only) - plain libblas/liblapack (Netlib, LP64 only) The list of libraries that is tried with the default 'auto' setting excludes a couple of libraries, because they're either no longer developed (ATLAS), not mature (libflame), or can't be tested and may be re-added later (ArmPL, ssl2). Those libraries can still be quite easily used via pkg-config. The new CI jobs are running by default right now. Once things settle down, the plan is to disable them by default and allow triggering them via a `[blas ci]` command in the commit message (just like for wheel builds). Docs will be included in a separate PR with the pending rewrite of all the build/install docs. For now, the CI jobs and the `meson_options.txt` file serve as guidance for how to use this. Note that the test suite contains a few hacks, because of packaging bugs for MKL on PyPI (broken .pc files) and BLIS (missing .pc file in Debian).
1 parent c7e073c commit ad4f480

File tree

11 files changed

+447
-176
lines changed

11 files changed

+447
-176
lines changed

‎.github/workflows/linux_meson.yml

Lines changed: 283 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,36 @@
1-
name: Test Meson build (Linux)
1+
name: BLAS tests (Linux)
2+
3+
# This file is meant for testing different BLAS/LAPACK flavors and build
4+
# options on Linux. All other yml files for Linux will only test without BLAS
5+
# (mostly because that's easier and faster to build) or with the same 64-bit
6+
# OpenBLAS build that is used in the wheel jobs.
7+
#
8+
# Jobs and their purpose:
9+
#
10+
# - openblas32_stable_nightly:
11+
# Uses the 32-bit OpenBLAS builds, both the latest stable release
12+
# and a nightly build.
13+
# - openblas_no_pkgconfig_fedora:
14+
# Test OpenBLAS on Fedora. Fedora doesn't ship .pc files for OpenBLAS,
15+
# hence this exercises the "system dependency" detection method.
16+
# - flexiblas_fedora:
17+
# Tests FlexiBLAS (the default on Fedora for its own packages), via
18+
# pkg-config. FlexiBLAS allows runtime switching of BLAS/LAPACK
19+
# libraries, which is a useful capability (not tested in this job).
20+
# - openblas_cmake:
21+
# Tests whether OpenBLAS LP64 is detected correctly when only CMake
22+
# and not pkg-config is installed.
23+
# - netlib:
24+
# Installs vanilla blas/lapack, which is the last option tried in
25+
# auto-detection.
26+
# - mkl:
27+
# Tests MKL installed from PyPI (because easiest/fastest, if broken) in
28+
# 3 ways: both LP64 and ILP64 via pkg-config, and then using the
29+
# Single Dynamic Library (SDL, or `libmkl_rt`).
30+
# - blis:
31+
# Simple test for LP64 via pkg-config
32+
# - atlas:
33+
# Simple test for LP64 via pkg-config
234

335
on:
436
pull_request:
@@ -71,3 +103,253 @@ jobs:
71103
export NPY_RUN_MYPY_IN_TESTSUITE=1
72104
pip install pytest pytest-xdist hypothesis typing_extensions
73105
spin test -j auto
106+
107+
108+
openblas_no_pkgconfig_fedora:
109+
if: "github.repository == 'numpy/numpy'"
110+
runs-on: ubuntu-latest
111+
container: fedora:39
112+
name: "OpenBLAS (Fedora, no pkg-config, LP64/ILP64)"
113+
steps:
114+
- name: Install system dependencies
115+
run: |
116+
dnf install git gcc-gfortran g++ python3-devel openblas-devel -y
117+
118+
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4.1.0
119+
with:
120+
submodules: recursive
121+
fetch-depth: 0
122+
123+
- name: Install dependencies
124+
run: |
125+
pip install -r build_requirements.txt
126+
pip install pytest hypothesis typing_extensions
127+
128+
- name: Build (LP64)
129+
run: spin build -- -Dblas=openblas -Dlapack=openblas -Ddisable-optimization=true
130+
131+
- name: Test
132+
run: spin test -- numpy/linalg
133+
134+
- name: Build (ILP64)
135+
run: |
136+
rm -rf build
137+
spin build -- -Duse-ilp64=true -Ddisable-optimization=true
138+
139+
- name: Test
140+
run: spin test -- numpy/linalg
141+
142+
143+
flexiblas_fedora:
144+
if: "github.repository == 'numpy/numpy'"
145+
runs-on: ubuntu-latest
146+
container: fedora:39
147+
name: "FlexiBLAS (LP64, ILP64 on Fedora)"
148+
steps:
149+
- name: Install system dependencies
150+
run: |
151+
dnf install git gcc-gfortran g++ python3-devel flexiblas-devel -y
152+
153+
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4.1.0
154+
with:
155+
submodules: recursive
156+
fetch-depth: 0
157+
158+
- name: Install dependencies
159+
run: |
160+
pip install -r build_requirements.txt
161+
pip install pytest hypothesis typing_extensions
162+
163+
- name: Build
164+
run: spin build -- -Ddisable-optimization=true
165+
166+
- name: Test
167+
run: spin test -- numpy/linalg
168+
169+
- name: Build (ILP64)
170+
run: |
171+
rm -rf build
172+
spin build -- -Ddisable-optimization=true -Duse-ilp64=true
173+
174+
- name: Test (ILP64)
175+
run: spin test -- numpy/linalg
176+
177+
178+
openblas_cmake:
179+
if: "github.repository == 'numpy/numpy'"
180+
runs-on: ubuntu-latest
181+
name: "OpenBLAS with CMake"
182+
steps:
183+
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4.1.0
184+
with:
185+
submodules: recursive
186+
fetch-depth: 0
187+
- uses: actions/setup-python@61a6322f88396a6271a6ee3565807d608ecaddd1 # v4.7.0
188+
with:
189+
python-version: '3.11'
190+
191+
- name: Install dependencies
192+
run: |
193+
pip install -r build_requirements.txt
194+
pip install pytest pytest-xdist hypothesis typing_extensions
195+
sudo apt-get install libopenblas-dev cmake
196+
sudo apt-get remove pkg-config
197+
198+
- name: Build
199+
run: spin build -- -Ddisable-optimization=true
200+
201+
- name: Test
202+
run: spin test -j auto -- numpy/linalg
203+
204+
205+
netlib:
206+
if: "github.repository == 'numpy/numpy'"
207+
runs-on: ubuntu-latest
208+
name: "Netlib BLAS/LAPACK"
209+
steps:
210+
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4.1.0
211+
with:
212+
submodules: recursive
213+
fetch-depth: 0
214+
- uses: actions/setup-python@61a6322f88396a6271a6ee3565807d608ecaddd1 # v4.7.0
215+
with:
216+
python-version: '3.11'
217+
218+
- name: Install dependencies
219+
run: |
220+
pip install -r build_requirements.txt
221+
sudo apt-get install liblapack-dev pkg-config
222+
223+
- name: Build
224+
run: |
225+
spin build -- -Ddisable-optimization=true
226+
227+
- name: Test
228+
run: |
229+
pip install pytest pytest-xdist hypothesis typing_extensions
230+
spin test -j auto -- numpy/linalg
231+
232+
233+
mkl:
234+
if: "github.repository == 'numpy/numpy'"
235+
runs-on: ubuntu-latest
236+
name: "MKL (LP64, ILP64, SDL)"
237+
steps:
238+
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4.1.0
239+
with:
240+
submodules: recursive
241+
fetch-depth: 0
242+
- uses: actions/setup-python@61a6322f88396a6271a6ee3565807d608ecaddd1 # v4.7.0
243+
with:
244+
python-version: '3.11'
245+
246+
- name: Install dependencies
247+
run: |
248+
pip install -r build_requirements.txt
249+
pip install pytest pytest-xdist hypothesis typing_extensions
250+
pip install mkl mkl-devel
251+
252+
- name: Repair MKL pkg-config files and symlinks
253+
run: |
254+
# MKL 2023.2 works when installed from conda-forge (except for `-iomp`
255+
# and `-tbb` pkg-config files), Spack, or with the standalone Intel
256+
# installer. The standalone installer is the worst option, since it's
257+
# large and clumsy to install and requires running a setvars.sh script
258+
# before things work. The PyPI MKL packages are broken and need the
259+
# fixes in this step. For details, see
260+
# https://github.com/conda-forge/intel_repack-feedstock/issues/34
261+
cd $Python3_ROOT_DIR/lib/pkgconfig
262+
sed -i 's/\/intel64//g' mkl*.pc
263+
# add the expected .so -> .so.2 symlinks to fix linking
264+
cd ..
265+
for i in $( ls libmkl*.so.2 ); do ln -s $i ${i%.*}; done
266+
267+
- name: Build with defaults (LP64)
268+
run: |
269+
pkg-config --libs mkl-dynamic-lp64-seq # check link flags
270+
spin build -- -Ddisable-optimization=true
271+
272+
- name: Test
273+
run: spin test -- numpy/linalg
274+
275+
- name: Build with ILP64
276+
run: |
277+
git clean -xdf > /dev/null
278+
pkg-config --libs mkl-dynamic-ilp64-seq
279+
spin build -- -Duse-ilp64=true -Ddisable-optimization=true
280+
281+
- name: Test
282+
run: spin test -- numpy/linalg
283+
284+
- name: Build without pkg-config (default options, SDL)
285+
run: |
286+
git clean -xdf > /dev/null
287+
pushd $Python3_ROOT_DIR/lib/pkgconfig
288+
rm mkl*.pc
289+
popd
290+
export MKLROOT=$Python3_ROOT_DIR
291+
spin build -- -Ddisable-optimization=true
292+
293+
- name: Test
294+
run: spin test -- numpy/linalg
295+
296+
blis:
297+
if: "github.repository == 'numpy/numpy'"
298+
runs-on: ubuntu-latest
299+
name: "BLIS"
300+
steps:
301+
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4.1.0
302+
with:
303+
submodules: recursive
304+
fetch-depth: 0
305+
- uses: actions/setup-python@61a6322f88396a6271a6ee3565807d608ecaddd1 # v4.7.0
306+
with:
307+
python-version: '3.11'
308+
309+
- name: Install dependencies
310+
run: |
311+
pip install -r build_requirements.txt
312+
pip install pytest pytest-xdist hypothesis typing_extensions
313+
sudo apt-get install libblis-dev libopenblas-dev pkg-config
314+
315+
- name: Add BLIS pkg-config file
316+
run: |
317+
# Needed because blis.pc missing in Debian:
318+
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989076
319+
# The alternative here would be to use another distro or Miniforge
320+
sudo cp tools/ci/_blis_debian.pc /usr/lib/x86_64-linux-gnu/pkgconfig/blis.pc
321+
# Check if the patch works:
322+
pkg-config --libs blis
323+
pkg-config --cflags blis
324+
325+
- name: Build
326+
run: spin build -- -Dblas=blis -Ddisable-optimization=true
327+
328+
- name: Test
329+
run: spin test -- numpy/linalg
330+
331+
atlas:
332+
if: "github.repository == 'numpy/numpy'"
333+
runs-on: ubuntu-latest
334+
name: "ATLAS"
335+
steps:
336+
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4.1.0
337+
with:
338+
submodules: recursive
339+
fetch-depth: 0
340+
- uses: actions/setup-python@61a6322f88396a6271a6ee3565807d608ecaddd1 # v4.7.0
341+
with:
342+
python-version: '3.11'
343+
344+
- name: Install dependencies
345+
run: |
346+
pip install -r build_requirements.txt
347+
pip install pytest pytest-xdist hypothesis typing_extensions
348+
sudo apt-get install libatlas-base-dev pkg-config
349+
350+
- name: Build
351+
run: spin build -- -Dblas=blas-atlas -Dlapack=lapack-atlas -Ddisable-optimization=true
352+
353+
- name: Test
354+
run: spin test -- numpy/linalg
355+

‎.github/workflows/linux_musl.yml

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -33,12 +33,12 @@ jobs:
3333
# using git commands to clone because versioneer doesn't work when
3434
# actions/checkout is used for the clone step in a container
3535
36-
git config --global --add safe.directory $PWD
37-
36+
git config --global --add safe.directory $PWD
37+
3838
if [ $GITHUB_EVENT_NAME != pull_request ]; then
3939
git clone --recursive --branch=$GITHUB_REF_NAME https://github.com/${GITHUB_REPOSITORY}.git $GITHUB_WORKSPACE
4040
git reset --hard $GITHUB_SHA
41-
else
41+
else
4242
git clone --recursive https://github.com/${GITHUB_REPOSITORY}.git $GITHUB_WORKSPACE
4343
git fetch origin $GITHUB_REF:my_ref_name
4444
git checkout $GITHUB_BASE_REF
@@ -54,14 +54,14 @@ jobs:
5454
source test_env/bin/activate
5555
5656
# required for figuring out the system tags in openblas_support
57-
pip install packaging
58-
57+
pip install packaging
58+
5959
# install openblas by co-opting the CIBW setup script
6060
RUNNER_OS=Linux sh tools/wheels/cibw_before_build.sh .
6161
6262
pip install -r build_requirements.txt
6363
pip install pytest pytest-xdist hypothesis typing_extensions
6464
65-
# use meson to build and test
66-
spin build
65+
# use meson to build and test
66+
spin build --with-scipy-openblas=64 -- -Duse-ilp64=true
6767
spin test -j auto

‎.github/workflows/macos.yml

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ jobs:
100100
ccache -s
101101
102102
accelerate:
103-
name: Accelerate ILP64
103+
name: Accelerate (LP64, ILP64)
104104
if: "github.repository == 'numpy/numpy'"
105105
runs-on: macos-13
106106
steps:
@@ -122,14 +122,15 @@ jobs:
122122
pip install -r build_requirements.txt
123123
pip install pytest pytest-xdist hypothesis
124124
125-
- name: Build NumPy against Accelerate (ILP64)
126-
run: |
127-
spin build -- -Dblas=accelerate -Dlapack=accelerate -Duse-ilp64=true
125+
- name: Build against Accelerate (LP64)
126+
run: spin build -- -Ddisable-optimization=true
128127

129-
- name: Show meson-log.txt
130-
if: always()
131-
run: 'cat build/meson-logs/meson-log.txt'
128+
- name: Test (linalg only)
129+
run: spin test -j2 -- numpy/linalg
132130

133-
- name: Test
131+
- name: Build NumPy against Accelerate (ILP64)
134132
run: |
135-
spin test -j2
133+
spin build -- -Duse-ilp64=true
134+
135+
- name: Test (fast tests)
136+
run: spin test -j2

‎azure-steps-windows.yml

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,10 @@ steps:
4242
python -m pip install . -v -Csetup-args="--vsenv" -Csetup-args="-Dblas=none" -Csetup-args="-Dlapack=none" -Csetup-args="-Dallow-noblas=true"
4343
}
4444
elseif ( Test-Path env:NPY_USE_BLAS_ILP64 ) {
45-
python -m pip install . -v -Csetup-args="--vsenv" -Csetup-args="-Duse-ilp64=true" -Csetup-args="-Dblas-symbol-suffix=64_"
45+
python -m pip install scipy-openblas64 spin
46+
spin config-openblas --with-scipy-openblas=64
47+
$env:PKG_CONFIG_PATH="$pwd/.openblas"
48+
python -m pip install . -v -Csetup-args="--vsenv" -Csetup-args="-Duse-ilp64=true"
4649
} else {
4750
python -m pip install . -v -Csetup-args="--vsenv"
4851
}

‎meson.build

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,6 @@ project(
1212
'b_ndebug=if-release',
1313
'c_std=c99',
1414
'cpp_std=c++17',
15-
'blas=openblas',
16-
'lapack=openblas',
1715
'pkgconfig.relocatable=true',
1816
],
1917
)

0 commit comments

Comments
 (0)
0