8000 ENH: building wheels for musl libc · Issue #20780 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

ENH: building wheels for musl libc #20780

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dimpase opened this issue Jan 10, 2022 · 11 comments
Closed

ENH: building wheels for musl libc #20780

dimpase opened this issue Jan 10, 2022 · 11 comments
Labels
32 - Installation Problems installing or compiling NumPy

Comments

@dimpase
Copy link
Contributor
dimpase commented Jan 10, 2022 •
edited
Loading

Proposed new feature or change:

One notices that the only Linux wheels provided by numpy on PyPI are manylinux* wheels (glibc-based).

Are there plans/problems regarding musl libc-based wheels? Note that a musl-libc Linux distribution, Alpine,
does provide numpy 1.22, see https://pkgs.alpinelinux.org/packages?name=*numpy*&branch=edge - but
as far as I understand, they build numpy calling python setup.py build, see https://git.alpinelinux.org/aports/tree/community/py3-numpy
for their build script (they use openblas and gfortran) and a small patch they apply.

A naive attempt to build wheels for numpy we tried didn't work with a rather obscure error, see
https://trac.sagemath.org/ticket/33083#comment:46, or
below: #20780 (comment)
@dimpase dimpase mentioned this issue Jan 10, 2022
Closed
@rgommers rgommers added the 32 - Installation Problems installing or compiling NumPy label Jan 10, 2022
@rgommers
Copy link
Member

Are there plans/problems regarding musl libc-based wheels?

To be considered after the ongoing migration to cibuildwheel is complete. If it's very easy to do so with cibuildwheel, then probably yes we'll support it.

See also https://mail.python.org/archives/list/numpy-discussion@python.org/message/46HT2SYDHBNLOC6N5RTXI7CN32YWIJWR/

@dimpase
Copy link
Contributor Author
dimpase commented Jan 10, 2022

I recently added building musl libc wheels for gmpy2 PyPI package, here: aleaxit/gmpy#318, using cibuildwheel - it was quite easy indeed, subject to few quirks.

@dimpase
Copy link
Contributor Author
dimpase commented Jan 11, 2022

What really bothers me here, is an apparent bug in setting up the CLIB, and I don't understand what file can't be found, and why.

...
INFO: building extension "numpy.random._bounded_integers" sources
  INFO: building extension "numpy.random.mtrand" sources
  INFO: building data_files sources
  INFO: build_src: building npy-pkg config files
  running build_py
  copying build/src.linux-x86_64-3.10/numpy/__config__.py -> build/lib.linux-x86_64-3.10/numpy
  copying build/src.linux-x86_64-3.10/numpy/distutils/__config__.py -> build/lib.linux-x86_64-3.10/numpy/distutils
  UPDATING build/lib.linux-x86_64-3.10/numpy/_version.py
  set build/lib.linux-x86_64-3.10/numpy/_version.py to '1.23.0.dev0+400.g2d7497233'
  running build_clib
  INFO: customize UnixCCompiler
  INFO: customize UnixCCompiler using new_build_clib
  INFO: CCompilerOpt.__init__[798] : load cache from file -> /home/dima/numpy/build/temp.linux-x86_64-3.10/ccompiler_opt_cache_clib.py
  INFO: CCompilerOpt.__init__[809] : hit the file cache
  INFO: building 'npymath' library
  INFO: compiling C sources
  INFO: C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fomit-frame-pointer -g -fomit-frame-pointer -g -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC
  
  INFO: compile options: '-Ibuild/src.linux-x86_64-3.10/numpy/core/src/npymath -Inumpy/core/include -Ibuild/src.linux-x86_64-3.10/numpy/core/include/numpy -Ibuild/src.linux-x86_64-3.10/numpy/distutils/include -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/src/_simd -I/usr/include/python3.10 -Ibuild/src.linux-x86_64-3.10/numpy/core/src/common -Ibuild/src.linux-x86_64-3.10/numpy/core/src/npymath -c'
  extra options: '-msse -msse2 -msse3'
  error: [Errno 2] No such file or directory
  INFO:
  ########### CLIB COMPILER OPTIMIZATION ###########
  INFO: Platform      :
    Architecture: x64
    Compiler    : gcc
  
  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none
  
  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
    Generated   : none
  INFO: CCompilerOpt.cache_flush[842] : write cache to path -> /home/dima/numpy/build/temp.linux-x86_64-3.10/ccompiler_opt_cache_clib.py
  ----------------------------------------
  ERROR: Failed building wheel for numpy
Failed to build numpy
ERROR: Failed to build one or more wheels

This was the result of pip wheel -e . on Alpine Linux edge (musl libc), on the current numpy master.

@rgommers
Copy link
Member

That's a really odd build command, you are mixing wheel building with an editable build. Do you get the same for python setup.py bdist_wheel?

@dimpase
Copy link
Contributor Author
dimpase commented Jan 11, 2022

Do you get the same for python setup.py bdist_wheel?

yes, same error.

@rgommers
Copy link
Member

I don't know then. It certainly works on Alpine. Just use https://alpine-wheels.github.io/index/?

@dimpase
Copy link
Contributor Author
dimpase commented Jan 11, 2022

These wheels are built without an external BLAS/LAPACK.
I also fail to see what particular versions of Apline Linux is used in the wheel builder you are referring to.

Looking back at my error log, I see three warnings:
WARN: Could not locate executable armflang - weird - I even tried

--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,7 +1,7 @@
 [build-system]
 # Minimum requirements for the build system to execute.
 requires = [
-    "packaging==20.5; platform_machine=='arm64'",  # macos M1
+    "packaging==20.5",  # macos M1
     "setuptools==59.2.0",
     "wheel==0.37.0",
     "Cython==0.29.24"  # pin until PyPy releases v7.3.8

but it's still there. It's apparently harmless though.

Also, an import and a distutils warning, below:

  INFO:
  Warning: attempted relative import with no known parent package
  /usr/lib/python3.10/distutils/dist.py:274: UserWarning: Unknown distribution option: define_macros'
    warnings.warn(msg)
  running bdist_wheel
  running build
  running config_cc
  INFO: unifing config_cc, config, build_clib, build_ext, build commands --compiler options
  running config_fc
  INFO: unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
  running build_src
  INFO: build_src
  INFO: building py_modules sources
  INFO: building library "npymath" sources
  WARN: Could not locate executable armflang

@rgommers
Copy link
Member

These wheels are built without an external BLAS/LAPACK.

Ah, great. Well, I guess you're using Alpine for small image size, not optimal performance ....

Also, an import and a distutils warning, below:

Those should be harmless.

@dimpase
Copy link
Contributor Author
dimpase commented Jan 11, 2022

We're using Apline merely as a platform to build musl libc wheels, not for actual computing work.
Musl libc is popular with some crowds (besides, in 2022 the size of Numpy with libs can't be called big).

@mattip
Copy link
Member
mattip commented Jan 25, 2022

Duplicate of #20089 which also references a mailing list thread

@h-vetinari h-vetinari mentioned this issue Feb 10, 2022
Merged
@mattip
Copy link
Member
mattip commented Feb 14, 2022

Closing in favor of #20089.

@mattip mattip closed this as completed Feb 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
32 - Installation Problems installing or compiling NumPy
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rgommers @mattip @dimpase
0