BLD: fail to build on Apple M1 #17807
Labels
Comments
|
I am experiencing the same issue when pip installing numpy on M1. |
|
You should use openblas with numpy, not Accelerate. As noted in this comment, you should be able to do |
|
Hi, @mattip thanks for the response.
Yields the same |
|
It seems you are still using the buggy Accelerate backend, which thankfully clang refuses to do. I don't have a macOSx so I can't really help you. Maybe someone else can. A few things to check in the mean time:
|
Thanks for looking into this so far. Hopefully someone with an M1 device can look into this further. |
|
Hey everyone, I had the same issue. To get it working I installed homebrew while specifying the x86 arch You can then use Hopefully this helps :) |
|
Build failure: |
6D40
|
@picn0113 follow the steps of my comment above to install the x86-64 version. |
|
In fact, numpy 1.19.4 has been installed, but |
|
Ultimately, installing the x86-64 version is a workaround that isn't sustainable, particularly given the large number of downstream packages that rely on numpy. An ARM build is the only real solution going forward, particularly given that Apple seems committed to this transition in the next two years. I realize that some of the issues are out of everyone's control (e.g. an open FORTRAN compiler), but hopefully we can collectively make some substantive progress toward better support. I did a quick test building OpenBLAS on the M1 (works fine), but numpy still doesn't work when linked against that library. It produces roughly the same errors as @picn0113's build above (which also includes OpenBLAS). Are there other explicit packages that need to be built and referenced for numpy to build properly? |
|
Thanks for trying. OpenBLAS (and system provided libraries like libm, glibc or their equivalent) are the only external libraries for NumPy. To provided a unversal2 binary wheel we would need a CI provider to make that machine available in order to run tests, but getting a local build working is a big first step. This is the thing you should focus on next. It picks up an improperly compiled OpenBLAS in |
|
So I didn't realize that his used X86_64. Mine is native arm64. I do get a similar linker warning:
From my OpenBLAS compile:
I followed this with a "make PREFIX=/opt/OpenBLAS/ install", and uncommented the appropriate lines in numpy's site.cfg. Complete output (badly formatted) below: Thanks for your help - I'll try to keep poking at this, but since I don't have a ton of experience with the guts of numpy, any hints on where to look next would speed the process up tremendously. |
|
There is still a clang error early in the process: which is preventing you from using OpenBLAS. I suggest you reach out in the OpenBLAS repo, perhaps there is a problem compiling for M1. |
|
Hello @jwischka. How do you build OpenBLAS? I get a
|
|
@picn0113 I just downloaded and did a simple make - mine is (correctly) skipping the fortran part:
|
|
@mattip Maybe I'm reading this wrong, but it looks like numpy/pip/python/somebody is sticking in an x86_64 in addition to the arm64, which is causing some of the errors (see below). Is there a way to explicitly limit the arch that is passed to clang (via pip, or similar)?
|
|
Then I had a thought... why am I trying to build 1.19.4 when I should be trying to build off the current source? So this is a problem that will, ultimately, work itself out. Any chance we could get an expedited release of 1.20? |
|
The problem is I have installed numpy 1.19.4, why pandas tries to build numpy itself? |
|
@picn0113 I'm trying to build purely on Apple Silicon, only using arm64 packages (no x86_64). I suspect mixing those is going to be a recipe for a lot of hurt down the road. The solution for me on the arm64 side was to clone the numpy tree, then run the pip3 install you see above. After that, you should be able to install pandas in similar fashion - or at least it worked for me (pip3 install pandas --no-binary :all: --no-use-pep517). |
|
@mattip @charris Are you guys aware of anything specific that would have changed from the 1.19.4 stable release to now that would have specifically addressed this issue? I primarily ask because I'm getting similar build errors on the SciPy side, and I'm hopeful that the answer may be over here in the commit history. That said, obviously there are a huge amount of commits recently, and I suspect it will be easy to miss. Any thoughts you might have would be appreciated. |
|
Okay. I succeed too. |
|
My guess would be some of the changes in |
|
I can confirm that this #17807 (comment) works on the M1 chip. |
|
For information, as a short-term stop gap for numpy users on the new Apple machines, it's possible to get the full stack (Python, numpy, scipy, openblas, gfortran compiler, pandas, scikit-learn...) working natively on Apple Silicon M1 using the miniforge installer from the conda-forge community project: choose the It would be good to get native arm64 macos wheels on PyPI but that requires upgrading the full wheel building infrastructure to setup cross-compiling on the intel based macos workers on Azure Pipelines and co (this is how conda-forge did it, using https://pypi.org/project/crossenv/). For those interested, I also patched numpy master to build against Big Sur's Accelerate, just to see the speed difference with OpenBLAS on this hardware. SGEMM is 1.7x faster with Accelerate but LAPACK operations (e.g. |
Did Apple fix the bugs in Accelerate that caused NumPy and SciPy to blacklist its use? |
No: the polyfit test case still fails with Big Sur's Accelerate on M1 with in arm64 mode. I also tried to use netlib's LAPACK in combination with Accelerate's BLAS by using a hack from @isuruf: while the polyfit test pass, many scikit-learn tests fail (while they pass when numpy and scipy are linked only to netlib or OpenBLAS BLAS+LAPACK). So I would definitely not recommend linking against Accelerate even if SGEMM is faster than with the OpenBLAS version. Accelerate is still too unreliable. |
|
@ogrisel, that hack is not enough as I realized that Accelerate uses g77 ABI. Needs more hacking. |
This was referenced Dec 10, 2020
|
It's wheel problem. |
|
(Sorry to spam, but this issue seems to have the most info about this problem that can crop up while pip installing pandas, numpy, or scipy on Apple M1's) In my case: pip install scipywas hitting the error: However, this worked: OPENBLAS="$(brew --prefix openblas)" pip install scipyI already had |
OMG you save my life!!! thanks! |
|
I think I have the solution to this issue.
for people who bought the Macbook air with the M1, we probably ended up installing newer versions of Python like 3.9/3.10 I ended up using Python 3.7.9, and inside that, I had no issues pip installing scipy. Solution (for me) : Use Python 3.7 |
|
Since numpy now comes with pre-built wheels for People who really want to build from source can have a look at the build configuration from https://github.com/MacPython/numpy-wheels to see how to fetch an OpenBLAS binary for I believe we can close this issue. |
|
Agreed, thanks @ogrisel & everyone who pitched in here! |
|
Thanks to all for fixing this issue! |
|
Finally! Thanks for the great work! 🎉🎉🎉 |
|
BTW, could you also build numpy for ARM Windows? |
No, there is no CI support for Windows on arm64 (which we require to even consider this) nor enough user demand at the moment. So: not any time soon. Please open a new issue for unrelated questions rather than commenting on an already-merged PR. |
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reproducing code example:
pip install pandas
Error message:
< 8000 svg aria-hidden="true" height="16" viewBox="0 0 16 16" version="1.1" width="16" data-view-component="true" class="octicon octicon-info">