DOC: BLAS/LAPACK linking rules #19447
Conversation
... per recent numpy-discussion@python.org discussion. My aim was to explain the rules clearly including background that developers might need to learn. I made some guesses to fill in details. For sure let me know whatever's wrong or unclear here. Should the website installation page https://numpy.org/install/ link to this doc so developers can learn how to install NumPy for their situation?
numpy/distutils/misc_util.py
Outdated
| Installing a numpy wheel (e.g. ``pip install numpy``) includes | ||
| an embedded OpenBLAS library on every platform that has a wheel | ||
| available. (OpenBLAS implements the BLAS and LAPACK linear | ||
| algebra APIs.) In this case, ``show_config()`` reports | ||
| ``library_dirs = ['/usr/local/lib']`` per the build time | ||
| configuration even if there's no libblas in that directory at | ||
| run time. The embedded OpenBLAS is a static link library | ||
| compiled with gcc/gfortran. |
There was a problem hiding this comment.
| Installing a numpy wheel (e.g. ``pip install numpy``) includes | |
| an embedded OpenBLAS library on every platform that has a wheel | |
| available. (OpenBLAS implements the BLAS and LAPACK linear | |
| algebra APIs.) In this case, ``show_config()`` reports | |
| ``library_dirs = ['/usr/local/lib']`` per the build time | |
| configuration even if there's no libblas in that directory at | |
| run time. The embedded OpenBLAS is a static link library | |
| compiled with gcc/gfortran. | |
| Installing a numpy wheel (e.g. ``pip install numpy``) includes | |
| an OpenBLAS implementation of the BLAS and LAPACK linear | |
| algebra APIs. In this case, ``library_dirs`` reports the build time | |
| configuration, the actual OpenBLAS shared object is located in | |
| ``site-packages/numpy.libs``. |
There was a problem hiding this comment.
This looks good except I'm not finding numpy.libs anywhere within my pyenv root,
$ fd 'numpy\.libs' $(pyenv root)
$ find $(pyenv root) -name "numpy.libs"
Searching site-packages/numpy in one of the virtualenvs that contains numpy with embedded OpenBLAS,
$ fd '\.so$'
core/_multiarray_tests.cpython-38-darwin.so
core/_multiarray_umath.cpython-38-darwin.so
core/_operand_flag_tests.cpython-38-darwin.so
core/_rational_tests.cpython-38-darwin.so
core/_struct_ufunc_tests.cpython-38-darwin.so
core/_umath_tests.cpython-38-darwin.so
fft/_pocketfft_internal.cpython-38-darwin.so
linalg/_umath_linalg.cpython-38-darwin.so
linalg/lapack_lite.cpython-38-darwin.so
random/_bounded_integers.cpython-38-darwin.so
random/_common.cpython-38-darwin.so
random/_generator.cpython-38-darwin.so
random/_mt19937.cpython-38-darwin.so
random/_pcg64.cpython-38-darwin.so
random/_philox.cpython-38-darwin.so
random/_sfc64.cpython-38-darwin.so
random/bit_generator.cpython-38-darwin.so
random/mtrand.cpython-38-darwin.so
Is the embedded OpenBLAS one of the two shared objects in site-packages/numpy/linalg?
There was a problem hiding this comment.
Is this a numpy installed by pip install numpy (not one built from source)? What version of numpy is it? You should have a directory one level higher, directly under site-packages:
$ ls /tmp/venv/lib/python3.8/site-packages/numpy.libs
libgfortran-2e0d59d6.so.5.0.0 libquadmath-2d0c479f.so.0.0.0
libopenblasp-r0-5bebc122.3.13.dev.so libz-eb09ad1d.so.1.2.3
There was a problem hiding this comment.
The file listing below is for numpy 1.19.5 installed from a wheel via pip install numpy. (I think it's the same virtualenv listed above.) Looking from one directory higher:
$ cd /usr/local/var/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/
$ ls numpy* numpy/*/*.so
numpy/core/_multiarray_tests.cpython-38-darwin.so* numpy/random/_bounded_integers.cpython-38-darwin.so*
numpy/core/_multiarray_umath.cpython-38-darwin.so* numpy/random/_common.cpython-38-darwin.so*
numpy/core/_operand_flag_tests.cpython-38-darwin.so* numpy/random/_generator.cpython-38-darwin.so*
numpy/core/_rational_tests.cpython-38-darwin.so* numpy/random/_mt19937.cpython-38-darwin.so*
numpy/core/_struct_ufunc_tests.cpython-38-darwin.so* numpy/random/_pcg64.cpython-38-darwin.so*
numpy/core/_umath_tests.cpython-38-darwin.so* numpy/random/_philox.cpython-38-darwin.so*
numpy/fft/_pocketfft_internal.cpython-38-darwin.so* numpy/random/_sfc64.cpython-38-darwin.so*
numpy/linalg/_umath_linalg.cpython-38-darwin.so* numpy/random/bit_generator.cpython-38-darwin.so*
numpy/linalg/lapack_lite.cpython-38-darwin.so* numpy/random/mtrand.cpython-38-darwin.so*
numpy:
LICENSE.txt _distributor_init.py ctypeslib.py lib/ random/
__config__.py _globals.py distutils/ linalg/ setup.py
__init__.cython-30.pxd _pytesttester.py doc/ ma/ testing/
__init__.pxd compat/ dual.py matlib.py tests/
__init__.py conftest.py f2py/ matrixlib/ version.py
__pycache__/ core/ fft/ polynomial/
numpy-1.19.5.dist-info:
INSTALLER LICENSES_bundled.txt RECORD WHEEL top_level.txt
LICENSE.txt METADATA REQUESTED entry_points.txt
(The files for numpy 1.21.0 installed from a wheel look very similar, with one more .so file.)
libopenblasp was a good clue! It's in a hidden directory with a similar filename:
$ find * -name "*libopenblas*"
numpy/.dylibs/libopenblas.0.dylib
scipy/.dylibs/libopenblas.0.dylib
$ fd -H libopenblas
numpy/.dylibs/libopenblas.0.dylib
scipy/.dylibs/libopenblas.0.dylib
This is on MacOS 10.14.6.
There was a problem hiding this comment.
The wheels can be examined from the pypi downloads, after all they are renamed zip files. So for macOS the directory is numpy/.dylibs, for linux it is numpy.libs, and for windows numpy/.libs [0]. I wonder why you cannot find it in your installation? Perhaps pip is building from source and not downloading a binary. What happens when you force binary mode (which in my opinion should be the default) via pip install -- only-binary :all: numpy
[0] aside: the linux directory is probably meant to be numpy/.lib but oh well, not gonna change that now
There was a problem hiding this comment.
:-) I found it after forcing fd and ls to look in hidden directories (why would anyone make those directories hidden?) and going back to looking for libblas* instead of *.so.
Judging by the console messages and speed, pip install numpy does install a wheel while pip install numpy --no-binary numpy does build a wheel and install it.
|
I am not sure about the wording of the "build from source" paragraph. It seems to have too much information for an end user who simply wants to know what BLAS/LAPACK is being used, but I don't have a better idea. Maybe the whole idea of showing build-time data in this function is outdated? A link to the email discussion starts here, and @rgommer 's comment that led to this PR starts here. I am not sure what is up with nabble, the thread is not there? |
|
In my team's experience (and Stack Overflow Q&A), the main aim is to figure out how to control which BLAS/LAPACK gets used. Yes, showing build-time data is not ideal for verifying which libraries get used at runtime. |
* Give the OS-specific site-packages directories for NumPy's OpenBLAS. * Include the `pip install numpy --only-binary numpy` variation, but not `:numpy:` since pip install doesn't document that. `pip install --only-binary numpy numpy` also works but that's confusing to parse. * Drop the sentence on the Accelerate bug on macOS < 11.3 since this isn't the place for it. * Also drop the search-order part about Accelerate? This applies to Numpy 1.19 as well as 1.21+. * Clarify the part about finding external BLAS/LAPACK libs. Some doc should explain what people need to know about installation. If they need portable, reproducible results, they should probably use the same BLAS/LAPACK libs compiled with the same compiler everywhere. It's unfortunate to install NumPy + SciPy linked to Accelerate, along with more pips, only to discover that NumPy won't load on their Mac and have to start over. Or will SciPy reject the older LAPACK API and have problems using different BLAS/LAPACK libs? Installing pips in Docker takes over an hour (vs. 15 minutes) if linking NumPy and SciPy to an external BLAS/LAPACK (and another pip conditionally compiles from source if NumPy did). Worse to have to debug CI failures because their CI on Linux computes different output than Accelerate. Should this also explain how to tell which release of OpenBLAS is included in a NumPy installation? (What got us into installing NumPy from source was needing an OpenBLAS bug fix release.)
|
Getting closer: just a few small tweaks, thanks! |
`:numpy:` Thanks, Matti! Can you illuminate what the colons do in pip install? The doc webpage and CLI help says: > Accepts either ":all:" to disable all binary packages, ":none:" to empty the set (notice the colons), or one or more package names with commas between them **(no colons).** Co-authored-by: Matti Picus <matti.picus@gmail.com>
* Accelerate in "NumPy 1.21+". I thought the NumPy 1.20.0 release note decommissioned "the Accelerate library as a candidate" but it doesn't mention BLAS. https://github.com/numpy/numpy/pull/15759/files#diff-6fde01624b4d27874d419c0f8aeae3743c4f5e7e1c9f2b039eb453a714d0cbb1R396 seems to affect both but
|
Thanks, Matti! |
|
?? on macOS 10.14.6 (without |
|
Sorry, I am confused. This PR started as "what in the world is If you build 1.21+ from source on some newer version of xcode (I am not sure which one), you will get an Accelerate backend without that bug. I am failing to find the issue/PR from Apple people to back up that claim, @rgommers do you remember where the discussion was? |
|
It's fine. (I'm confused, too.) My PR description was muddled. The aim of the email thread is to learn the rules to install NumPy with the desired BLAS/LAPACK (including Mac complications like how to control whether it uses Accelerate, how to ensure it doesn't, when/how to use the NPY_ environment vars, how to tell if it's using an embedded OpenBLAS or Accelerate and where is that embedded OpenBLAS, ...). There's some documentation for using You proposed adding to permanent documentation, which is a great idea. There's a case to add to the https://numpy.org/install/ installation page as well as the I tried installing 1.19.5 to check if Accelerate is in the default build-time search order "except in numpy 1.20" vs. "in 1.21+". Searching for "Accelerate" on the release notes page finds
|
|
The current version looks good to me. |
|
Thanks @1fish2. Any further clarifications can be done in follow on PRs. |
|
Nice! I learned some things here, esp. in finding the hidden dynamic library. Do you want to document some of these lessons for people installing NumPy, say on the "Installing" page with links to the "Building from source" page? I'm totally fine with a "no," and surely everyone has higher priorities. |
|
Thanks for this @1fish2! I think the Installing page on numpy.org is not a good place, this is way too detailed for that high-level overview. This would be the right page for it: https://numpy.org/devdocs/user/building.html#accelerated-blas-lapack-libraries |
Document BLAS/LAPACK linking rules per recent numpy-discussion@python.org discussion.
My aim was to explain the rules clearly including some background that developers might need to learn. I hope this is clear and helpful.
I made some guesses to fill in details. For sure let me know whatever's inaccurate or unclear here.
Should the website link to this info so developers can learn how to install NumPy for their situation?
The Contributing guide says to follow PEP 8 but PyCharm and flake8 list lots of PEP 8 warnings and other warnings in this file. I could make the risk-free fixes if y'all want -- maybe in a separate PR?