From 7977994dd6f90eb7b93d9a7dbb3f4ea5523c0442 Mon Sep 17 00:00:00 2001 From: Jerry Morrison <1fish2@users.noreply.github.com> Date: Fri, 9 Jul 2021 22:21:11 -0700 Subject: [PATCH 1/7] Document BLAS/LAPACK linking rules ... per recent numpy-discussion@python.org discussion. My aim was to explain the rules clearly including background that developers might need to learn. I made some guesses to fill in details. For sure let me know whatever's wrong or unclear here. Should the website installation page https://numpy.org/install/ link to this doc so developers can learn how to install NumPy for their situation? --- numpy/distutils/misc_util.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/numpy/distutils/misc_util.py b/numpy/distutils/misc_util.py index 60696438f346..cf65717b8d11 100644 --- a/numpy/distutils/misc_util.py +++ b/numpy/distutils/misc_util.py @@ -2362,6 +2362,30 @@ def show(): * ``not found``: dispatched features that are not supported in the system + NumPy BLAS/LAPACK Installation Notes + ------------------------------------ + Installing a numpy wheel (e.g. ``pip install numpy``) includes + an embedded OpenBLAS library on every platform that has a wheel + available. (OpenBLAS implements the BLAS and LAPACK linear + algebra APIs.) In this case, ``show_config()`` reports + ``library_dirs = ['/usr/local/lib']`` per the build time + configuration even if there's no libblas in that directory at + run time. The embedded OpenBLAS is a static link library + compiled with gcc/gfortran. + + Installing numpy from source (e.g. + ``pip install numpy --no-binary numpy)`` looks for BLAS and + LAPACK dynamic link libraries at build time. This search is + influenced by the environment variables + NPY_BLAS_ORDER/NPY_LAPACK_ORDER, if set, and the file + ``~/.numpy-site.cfg``, if present. + On macOS, 'accelerate' (Apple's Accelerate BLAS library) is in + the default search order (but not in numpy 1.20) after + 'openblas'. + Loading a numpy library that's linked to Accelerate on + macOS < 11.3 will detect an Accelerate bug and raise a + RuntimeError. + Examples -------- >>> import numpy as np From 83f2b22411936dafe8a085d2e42c28b6e39e7cbb Mon Sep 17 00:00:00 2001 From: Jerry Morrison <1fish2@users.noreply.github.com> Date: Fri, 9 Jul 2021 22:54:48 -0700 Subject: [PATCH 2/7] .gitignore virtualenv's .python-version --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index a2a1f2b68725..d856762492df 100644 --- a/.gitignore +++ b/.gitignore @@ -76,6 +76,7 @@ doc/cdoc/build MANIFEST .cache pip-wheel-metadata +.python-version # Paver generated files # ######################### From 05d6e615095e04c5f3bc6e9256882bcdd2950f12 Mon Sep 17 00:00:00 2001 From: Jerry Morrison <1fish2@users.noreply.github.com> Date: Sat, 10 Jul 2021 14:16:58 -0700 Subject: [PATCH 3/7] fix ")" position --- numpy/distutils/misc_util.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/numpy/distutils/misc_util.py b/numpy/distutils/misc_util.py index cf65717b8d11..68d5bdfe950c 100644 --- a/numpy/distutils/misc_util.py +++ b/numpy/distutils/misc_util.py @@ -2374,7 +2374,7 @@ def show(): compiled with gcc/gfortran. Installing numpy from source (e.g. - ``pip install numpy --no-binary numpy)`` looks for BLAS and + ``pip install numpy --no-binary numpy``) looks for BLAS and LAPACK dynamic link libraries at build time. This search is influenced by the environment variables NPY_BLAS_ORDER/NPY_LAPACK_ORDER, if set, and the file From 445292716b0d43b97a6bf7646c424c8b0a2ec435 Mon Sep 17 00:00:00 2001 From: Jerry Morrison <1fish2@users.noreply.github.com> Date: Mon, 12 Jul 2021 00:45:55 -0700 Subject: [PATCH 4/7] apply suggestions to the wheel case --- numpy/distutils/misc_util.py | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/numpy/distutils/misc_util.py b/numpy/distutils/misc_util.py index 68d5bdfe950c..53b2ba56bdab 100644 --- a/numpy/distutils/misc_util.py +++ b/numpy/distutils/misc_util.py @@ -2365,13 +2365,12 @@ def show(): NumPy BLAS/LAPACK Installation Notes ------------------------------------ Installing a numpy wheel (e.g. ``pip install numpy``) includes - an embedded OpenBLAS library on every platform that has a wheel - available. (OpenBLAS implements the BLAS and LAPACK linear - algebra APIs.) In this case, ``show_config()`` reports - ``library_dirs = ['/usr/local/lib']`` per the build time - configuration even if there's no libblas in that directory at - run time. The embedded OpenBLAS is a static link library - compiled with gcc/gfortran. + an OpenBLAS implementation of the BLAS and LAPACK linear algebra + APIs. In this case, ``library_dirs`` reports the build time + configuration; the OpenBLAS library is actually in + ``site-packages/numpy.libs/`` or + ``site-packages/numpy/.dylibs/`` and it's compiled with + gcc/gfortran. Installing numpy from source (e.g. ``pip install numpy --no-binary numpy``) looks for BLAS and From 8a67d77924c8303c75a8f3261139342e0a7544c2 Mon Sep 17 00:00:00 2001 From: Jerry Morrison <1fish2@users.noreply.github.com> Date: Mon, 12 Jul 2021 16:51:11 -0700 Subject: [PATCH 5/7] code review iteration * Give the OS-specific site-packages directories for NumPy's OpenBLAS. * Include the `pip install numpy --only-binary numpy` variation, but not `:numpy:` since pip install doesn't document that. `pip install --only-binary numpy numpy` also works but that's confusing to parse. * Drop the sentence on the Accelerate bug on macOS < 11.3 since this isn't the place for it. * Also drop the search-order part about Accelerate? This applies to Numpy 1.19 as well as 1.21+. * Clarify the part about finding external BLAS/LAPACK libs. Some doc should explain what people need to know about installation. If they need portable, reproducible results, they should probably use the same BLAS/LAPACK libs compiled with the same compiler everywhere. It's unfortunate to install NumPy + SciPy linked to Accelerate, along with more pips, only to discover that NumPy won't load on their Mac and have to start over. Or will SciPy reject the older LAPACK API and have problems using different BLAS/LAPACK libs? Installing pips in Docker takes over an hour (vs. 15 minutes) if linking NumPy and SciPy to an external BLAS/LAPACK (and another pip conditionally compiles from source if NumPy did). Worse to have to debug CI failures because their CI on Linux computes different output than Accelerate. Should this also explain how to tell which release of OpenBLAS is included in a NumPy installation? (What got us into installing NumPy from source was needing an OpenBLAS bug fix release.) --- numpy/distutils/misc_util.py | 37 ++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/numpy/distutils/misc_util.py b/numpy/distutils/misc_util.py index 53b2ba56bdab..59c337848ca1 100644 --- a/numpy/distutils/misc_util.py +++ b/numpy/distutils/misc_util.py @@ -2364,26 +2364,27 @@ def show(): NumPy BLAS/LAPACK Installation Notes ------------------------------------ - Installing a numpy wheel (e.g. ``pip install numpy``) includes + Installing a numpy wheel (``pip install numpy`` or force it + via ``pip install numpy --only-binary numpy``) includes an OpenBLAS implementation of the BLAS and LAPACK linear algebra - APIs. In this case, ``library_dirs`` reports the build time - configuration; the OpenBLAS library is actually in - ``site-packages/numpy.libs/`` or - ``site-packages/numpy/.dylibs/`` and it's compiled with - gcc/gfortran. - - Installing numpy from source (e.g. - ``pip install numpy --no-binary numpy``) looks for BLAS and - LAPACK dynamic link libraries at build time. This search is - influenced by the environment variables - NPY_BLAS_ORDER/NPY_LAPACK_ORDER, if set, and the file - ``~/.numpy-site.cfg``, if present. + APIs. In this case, ``library_dirs`` reports the original build + time configuration as compiled with gcc/gfortran; at run time + the OpenBLAS library is in + ``site-packages/numpy.libs/`` (linux), or + ``site-packages/numpy/.dylibs/`` (macOS), or + ``site-packages/numpy/.libs/`` (windows). + + Installing numpy from source + (``pip install numpy --no-binary numpy``) searches for BLAS and + LAPACK dynamic link libraries at build time as influenced by + environment variables NPY_BLAS_LIBS, NPY_CBLAS_LIBS, and + NPY_LAPACK_LIBS; or NPY_BLAS_ORDER and NPY_LAPACK_ORDER; + or the optional file ``~/.numpy-site.cfg``. + NumPy remembers those locations and expects to load the same + libraries at run-time. On macOS, 'accelerate' (Apple's Accelerate BLAS library) is in - the default search order (but not in numpy 1.20) after - 'openblas'. - Loading a numpy library that's linked to Accelerate on - macOS < 11.3 will detect an Accelerate bug and raise a - RuntimeError. + the default build-time search order -- except in numpy 1.20 -- + after 'openblas'. Examples -------- From e883802124a88f21cb85e6baee7b64864a5961c8 Mon Sep 17 00:00:00 2001 From: Jerry Morrison <1fish2@users.noreply.github.com> Date: Mon, 12 Jul 2021 23:07:46 -0700 Subject: [PATCH 6/7] Update numpy/distutils/misc_util.py `:numpy:` Thanks, Matti! Can you illuminate what the colons do in pip install? The doc webpage and CLI help says: > Accepts either ":all:" to disable all binary packages, ":none:" to empty the set (notice the colons), or one or more package names with commas between them **(no colons).** Co-authored-by: Matti Picus --- numpy/distutils/misc_util.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/numpy/distutils/misc_util.py b/numpy/distutils/misc_util.py index 59c337848ca1..84f2f616db35 100644 --- a/numpy/distutils/misc_util.py +++ b/numpy/distutils/misc_util.py @@ -2365,7 +2365,7 @@ def show(): NumPy BLAS/LAPACK Installation Notes ------------------------------------ Installing a numpy wheel (``pip install numpy`` or force it - via ``pip install numpy --only-binary numpy``) includes + via ``pip install numpy --only-binary :numpy: numpy``) includes an OpenBLAS implementation of the BLAS and LAPACK linear algebra APIs. In this case, ``library_dirs`` reports the original build time configuration as compiled with gcc/gfortran; at run time From 599a72fb9dd22e27dd30ee6a3d5d28822c518007 Mon Sep 17 00:00:00 2001 From: Jerry Morrison <1fish2@users.noreply.github.com> Date: Mon, 12 Jul 2021 23:27:08 -0700 Subject: [PATCH 7/7] review feedback * Accelerate in "NumPy 1.21+". I thought the NumPy 1.20.0 release note decommissioned "the Accelerate library as a candidate" but it doesn't mention BLAS. https://github.com/numpy/numpy/pull/15759/files#diff-6fde01624b4d27874d419c0f8aeae3743c4f5e7e1c9f2b039eb453a714d0cbb1R396 seems to affect both but --- numpy/distutils/misc_util.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/numpy/distutils/misc_util.py b/numpy/distutils/misc_util.py index 84f2f616db35..20cbc855aa0a 100644 --- a/numpy/distutils/misc_util.py +++ b/numpy/distutils/misc_util.py @@ -2382,9 +2382,9 @@ def show(): or the optional file ``~/.numpy-site.cfg``. NumPy remembers those locations and expects to load the same libraries at run-time. - On macOS, 'accelerate' (Apple's Accelerate BLAS library) is in - the default build-time search order -- except in numpy 1.20 -- - after 'openblas'. + In NumPy 1.21+ on macOS, 'accelerate' (Apple's Accelerate BLAS + library) is in the default build-time search order after + 'openblas'. Examples --------