8000 DOC: BLAS/LAPACK linking rules by 1fish2 · Pull Request #19447 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

DOC: BLAS/LAPACK linking rules #19447

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jul 13, 2021
Merged

DOC: BLAS/LAPACK linking rules #19447

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
7977994
Document BLAS/LAPACK linking rules
1fish2 Jul 10, 2021
83f2b22
.gitignore virtualenv's .python-version
1fish2 Jul 10, 2021
05d6e61
fix ")" position
1fish2 Jul 10, 2021
4452927
apply suggestions to the wheel case
1fish2 Jul 12, 2021
8a67d77
code review iteration
1fish2 Jul 12, 2021
e883802
Update numpy/distutils/misc_util.py
1fish2 Jul 13, 2021
599a72f
review feedback
1fish2 Jul 13, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ doc/cdoc/build
MANIFEST
.cache
pip-wheel-metadata
.python-version

# Paver generated files #
#########################
Expand Down
24 changes: 24 additions & 0 deletions numpy/distutils/misc_util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2362,6 +2362,30 @@ def show():
* ``not found``: dispatched features that are not supported
in the system

NumPy BLAS/LAPACK Installation Notes
------------------------------------
Installing a numpy wheel (e.g. ``pip install numpy``) includes
an embedded OpenBLAS library on every platform that has a wheel
available. (OpenBLAS implements the BLAS and LAPACK linear
algebra APIs.) In this case, ``show_config()`` reports
``library_dirs = ['/usr/local/lib']`` per the build time
configuration even if there's no libblas in that directory at
run time. The embedded OpenBLAS is a static link library
compiled with gcc/gfortran.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Installing a numpy wheel (e.g. ``pip install numpy``) includes
an embedded OpenBLAS library on every platform that has a wheel
available. (OpenBLAS implements the BLAS and LAPACK linear
algebra APIs.) In this case, ``show_config()`` reports
``library_dirs = ['/usr/local/lib']`` per the build time
configuration even if there's no libblas in that directory at
run time. The embedded OpenBLAS is a static link library
compiled with gcc/gfortran.
Installing a numpy wheel (e.g. ``pip install numpy``) includes
an OpenBLAS implementation of the BLAS and LAPACK linear
algebra APIs. In this case, ``library_dirs`` reports the build time
configuration, the actual OpenBLAS shared object is located in
``site-packages/numpy.libs``.

Copy link
Contributor Author
@1fish2 1fish2 Jul 10, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good except I'm not finding numpy.libs anywhere within my pyenv root,

$ fd 'numpy\.libs' $(pyenv root)
$ find $(pyenv root) -name "numpy.libs"

Searching site-packages/numpy in one of the virtualenvs that contains numpy with embedded OpenBLAS,

$ fd '\.so$'
core/_multiarray_tests.cpython-38-darwin.so
core/_multiarray_umath.cpython-38-darwin.so
core/_operand_flag_tests.cpython-38-darwin.so
core/_rational_tests.cpython-38-darwin.so
core/_struct_ufunc_tests.cpython-38-darwin.so
core/_umath_tests.cpython-38-darwin.so
fft/_pocketfft_internal.cpython-38-darwin.so
linalg/_umath_linalg.cpython-38-darwin.so
linalg/lapack_lite.cpython-38-darwin.so
random/_bounded_integers.cpython-38-darwin.so
random/_common.cpython-38-darwin.so
random/_generator.cpython-38-darwin.so
random/_mt19937.cpython-38-darwin.so
random/_pcg64.cpython-38-darwin.so
random/_philox.cpython-38-darwin.so
random/_sfc64.cpython-38-darwin.so
random/bit_generator.cpython-38-darwin.so
random/mtrand.cpython-38-darwin.so

Is the embedded OpenBLAS one of the two shared objects in site-packages/numpy/linalg?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a numpy installed by pip install numpy (not one built from source)? What version of numpy is it? You should have a directory one level higher, directly under site-packages:

$ ls /tmp/venv/lib/python3.8/site-packages/numpy.libs
libgfortran-2e0d59d6.so.5.0.0         libquadmath-2d0c479f.so.0.0.0
libopenblasp-r0-5bebc122.3.13.dev.so  libz-eb09ad1d.so.1.2.3

Copy link
Contributor Author
@1fish2 1fish2 Jul 12, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file listing below is for numpy 1.19.5 installed from a wheel via pip install numpy. (I think it's the same virtualenv listed above.) Looking from one directory higher:

$ cd /usr/local/var/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/
$ ls numpy* numpy/*/*.so
numpy/core/_multiarray_tests.cpython-38-darwin.so*   numpy/random/_bounded_integers.cpython-38-darwin.so*
numpy/core/_multiarray_umath.cpython-38-darwin.so*   numpy/random/_common.cpython-38-darwin.so*
numpy/core/_operand_flag_tests.cpython-38-darwin.so* numpy/random/_generator.cpython-38-darwin.so*
numpy/core/_rational_tests.cpython-38-darwin.so*     numpy/random/_mt19937.cpython-38-darwin.so*
numpy/core/_struct_ufunc_tests.cpython-38-darwin.so* numpy/random/_pcg64.cpython-38-darwin.so*
numpy/core/_umath_tests.cpython-38-darwin.so*        numpy/random/_philox.cpython-38-darwin.so*
numpy/fft/_pocketfft_internal.cpython-38-darwin.so*  numpy/random/_sfc64.cpython-38-darwin.so*
numpy/linalg/_umath_linalg.cpython-38-darwin.so*     numpy/random/bit_generator.cpython-38-darwin.so*
numpy/linalg/lapack_lite.cpython-38-darwin.so*       numpy/random/mtrand.cpython-38-darwin.so*

numpy:
LICENSE.txt             _distributor_init.py    ctypeslib.py            lib/                    random/
__config__.py           _globals.py             distutils/              linalg/                 setup.py
__init__.cython-30.pxd  _pytesttester.py        doc/                    ma/                     testing/
__init__.pxd            compat/                 dual.py                 matlib.py               tests/
__init__.py             conftest.py             f2py/                   matrixlib/              version.py
__pycache__/            core/                   fft/                    polynomial/

numpy-1.19.5.dist-info:
INSTALLER             LICENSES_bundled.txt  RECORD                WHEEL                 top_level.txt
LICENSE.txt           METADATA              REQUESTED             entry_points.txt

(The files for numpy 1.21.0 installed from a wheel look very similar, with one more .so file.)

libopenblasp was a good clue! It's in a hidden directory with a similar filename:

$ find * -name "*libopenblas*"
numpy/.dylibs/libopenblas.0.dylib
scipy/.dylibs/libopenblas.0.dylib
$ fd -H libopenblas
numpy/.dylibs/libopenblas.0.dylib
scipy/.dylibs/libopenblas.0.dylib

This is on MacOS 10.14.6.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wheels can be examined from the pypi downloads, after all they are renamed zip files. So for macOS the directory is numpy/.dylibs, for linux it is numpy.libs, and for windows numpy/.libs [0]. I wonder why you cannot find it in your installation? Perhaps pip is building from source and not downloading a binary. What happens when you force binary mode (which in my opinion should be the default) via pip install -- only-binary :all: numpy

[0] aside: the linux directory is probably meant to be numpy/.lib but oh well, not gonna change that now

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:-) I found it after forcing fd and ls to look in hidden directories (why would anyone make those directories hidden?) and going back to looking for libblas* instead of *.so.

Judging by the console messages and speed, pip install numpy does install a wheel while pip install numpy --no-binary numpy does build a wheel and install it.


Installing numpy from source (e.g.
``pip install numpy --no-binary numpy``) looks for BLAS and
LAPACK dynamic link libraries at build time. This search is
influenced by the environment variables
NPY_BLAS_ORDER/NPY_LAPACK_ORDER, if set, and the file
``~/.numpy-site.cfg``, if present.
On macOS, 'accelerate' (Apple's Accelerate BLAS library) is in
the default search order (but not in numpy 1.20) after
'openblas'.
Loading a numpy library that's linked to Accelerate on
macOS < 11.3 will detect an Accelerate bug and raise a
RuntimeError.

Examples
--------
>>> import numpy as np
Expand Down
0