Please provide universal2 wheels on macOS #18143
Comments
|
Thanks @ronaldoussoren, we hope to do so when it's feasible (in addition to thin We also still have some issues, in particular this build issue: gh-17807. And this performance issue: gh-17989. I'll link pypa/cibuildwheel#473 here as probably the most relevant issue for |
|
I ran into gh-17807 myself, my crude workaround was to remove the code adding "-faltivec" from setup.py. That worked for me, but is not a proper solution. I'm a very light user of numpy through pandas at best, I'm basically only using pandas to create graphs from a couple of CSV files with some light calculations. That said, I can't promise quick responses but feel free to let me know if there's something I can do or test. BTW. What's needed to build NumPy in a way that matches the official wheels? Looking at the azure-pipelines.yml in the repo I'd say gfortran and OpenBlas as the only non-system dependencies. Is that correct? |
|
Accelerate should be avoided for NumPy as it has bugs. NumPy doesn't use gfortran, SciPy does. But there currently no arm64-darwin OpenBLAS build on https://anaconda.org/multibuild-wheels-staging/openblas-libs/files. I am not sure how we would build that without support for arm64-darwin from a CI system. |
|
I opened MacPython/openblas-libs#49 about building OpenBLAS for |
For building you only need a CI system that provides macOS 11 hosts, or with some effort macOS 10.15 hosts with Xcode 12.2. For testing you actually need M1 systems in the CI system, which will likely be a blocker for you and could take some time (I have no idea how constrained the supply of M1 systems is, but I expect that a CI provider like Azure Pipelines will require a lot of M1 systems). |
Have you (the numpy project) files bugs about this in Apple's tracker? Or are these bugs in numpy's use of Accelerate? |
Yes, multiple. Also other scientific projects - Apple basically doesn't care and hasn't been serious about maintaining Accelerate in a long time. Not enough value in it for them I guess. SciPy has more linear algebra functionality than NumPy, and dropped Accelerate earlier than NumPy. We have two good options for BLAS/LAPACK - OpenBLAS (our default for wheels) and Intel MKL. So we'd rather spend our effort on OpenBLAS rather than work around Apple's poor support. |
Yes, I'd be very nervous about releasing something that we can't test. Our stance for other platforms (e.g. Linux on ARM64) is that CI services need to be available; in the absence of that we merge fixes but don't release wheels.
Thanks!
In addition to what @mattip said, here is the actual wheel build machinery: https://github.com/MacPython/numpy-wheels |
|
I feel your pain, I'm hearing similar experiences from app developers and most issues I've filed with them have gone completely unanswered :-( |
|
@ronaldoussoren will the universal2 python.org installer pick up thin arm64 wheels if they are available and Python is started in native mode? |
|
That's more a pip question than python version, but yes pip will use arm64 wheels when running in natively on M1 Macs. |
|
@ronaldoussoren - sorry - just checking because I was surprised. You're saying that if I do: on an M1 machine, using the Python.org |
|
That's correct. That's something that surprised me too. Even worse (IMHO) pip will prefer a native wheel over a universal2 one. Most users won't care, except for folks like myself that wish to redistribute binaries to other systems (in my case mostly using py2app). I've filed an feature request about that, but the pip team is not convinced yet that preferring universal2 wheels for a universal2 build is a good idea. I'll probably create a PR to show the impact this change would have on their code base. Note that it is also possible to install an x86_64 wheel on an M1 system by running python in emulated mode. That is, This will prefer an arm64 wheel (also without the invocation of the This will prefer an x86_64 wheel: P.S. I've not looked into the python3 included in Apple's compiler tools, that version might not support the |
|
@ronaldoussoren - ouch! - and thanks for working that through, that's very helpful. |
|
Interesting. I'm not sure that's a bad thing. There are scientific packages that already exceed the PyPI version limit, so a doubling of wheel size for what (for scientific libraries) is a very niche use case doesn't seem very sustainable. I didn't have the energy to jump in that packaging discussion, but I suspect scientific libs may prefer thin wheels. We work hard on keeping binary sizes reasonable. |
|
Yeah, I noticed that some some packages are very large. Those can always choose to not provide universal2 wheels at all, that will work for most users. At a risk of going completely off-topic, I'd like to have more control on what gets installed than pip is currently giving. When I'm doing something on my local machine only I might prefer a thin wheel (smaller, faster install), but when I intend to redistribute I might prefer a universal wheel (and possibly even one that's synthesised from two thin ones on PyPI). |
|
More control would be nice indeed, if I'd actually prefer if it didn't stay, because also for redistribution there isn't much reason to do it as |
|
Just superficially, wouldn't it be more simple for Python.org to provide separate M1 and x86_64 installers - as it does for Windows 32 and 64 bit? It's hard to imagine many people using the And, as Ralf says, I bet that will usually break, when you get to install packages. |
|
Separate installers requires users to understand what kind of machine they have. With a universal2 installer things just work in that regard. The only problem is when projects do no provide wheels with native code for arm64 (either universal wheels or thin arm64 wheels), but that should correct itself in due time (especially once M1 systems are available in cloud CI systems). I expect that it will be years before we can ignore intel Macs, even after Apple transitions all their hardware to arm64. Universal support is more or less required for folks distributing application bundles, having separate application downloads for intel and arm is just not what Mac users expect. Universal support is not about being able to run x86_64 on arm, but about having a single binary that just works everywhere. |
|
I don't think it's much to ask of a user that they know they have an M1 Mac, honestly. And for now, it would be reasonable to make the x86_64 installer the default, so they have to specifically ask for the M1 build. I presume an x86_64 build will also work on M1 via Rosetta? What happens for: Does this look for an |
|
I don't understand what problem universal2 pip packages solve. If the package is pure-python, pip will download that. If the package has c-extension modules, pip should choose the proper hardware model. |
|
I think the problem they solve is where the user sometimes wants to run Python in M1 mode, and sometimes in x86_64 mode. They can choose their mode with the
The other problem is the one Ronald mentioned - a universal binary for Python itself means it will work on either hardware. I agree though, that the first use-case doesn't seem all that important, and the second seems a rather minor advantage compared to the cost in terms of packaging confusion. |
That's specifically a general "Mac end user" problem though, and is unrelated to PyPI and wheels. Having thin wheels plus the right |
Agreed, at least 6-7 years I'd think. |
|
|
|
I think multibuild has full support too - thanks to @isuruf. |
|
Those tests are disabled on windows where |
|
I have a hard time understanding the meaning of those tests but here is what I get on an ARM64 macos numpy: >>> from numpy.f2py.tests.test_array_from_pyobj import Type
>>> Type("LONGDOUBLE").elsize
8.0
>>> Type("LONGDOUBLE").dtype
dtype('float64')
>>> from pprint import pprint
>>> for t in Type("LONGDOUBLE").cast_types():
... print(t, t.elsize, t.dtype, t.type_num)
...
<numpy.f2py.tests.test_array_from_pyobj.Type object at 0x148bedfd0> 1.0 int8 1
<numpy.f2py.tests.test_array_from_pyobj.Type object at 0x148bff040> 1.0 uint8 2
<numpy.f2py.tests.test_array_from_pyobj.Type object at 0x148bff070> 2.0 int16 3
<numpy.f2py.tests.test_array_from_pyobj.Type object at 0x148bff0a0> 2.0 uint16 4
<numpy.f2py.tests.test_array_from_pyobj.Type object at 0x148bff0d0> 4.0 int32 5
<numpy.f2py.tests.test_array_from_pyobj.Type object at 0x148bff130> 8.0 int64 7
<numpy.f2py.tests.test_array_from_pyobj.Type object at 0x148bff160> 8.0 uint64 8
<numpy.f2py.tests.test_array_from_pyobj.Type object at 0x148bff1f0> 4.0 float32 11
<numpy.f2py.tests.test_array_from_pyobj.Type object at 0x148bff220> 8.0 float64 12
<numpy.f2py.tests.test_array_from_pyobj.Type object at 0x148bff280> 8.0 float64 13 |
|
Right. In I think we should skip this test as it is skipped in windows where there's no |
|
Ok so the problem is in those lines then: https://github.com/numpy/numpy/blob/main/numpy/f2py/tests/test_array_from_pyobj.py#L117-L128 but still I do not understand what the test is supposed to test so I am not confident such a PR. |
|
For reference: >>> import sys
>>> sys.platform
'darwin'
>>> import platform
>>> platform.platform()
'macOS-11.2.3-arm64-arm-64bit'
>>> platform.architecture()
('64bit', '')
>>> platform.processor()
'arm'
>>> platform.system()
'Darwin'So adding a condition for I will open a PR. |
Is that true? I thought AArch64 and Power9 both supported quad precision floats in hardware. It may depend on the compiler.
|
|
Apple silicon ABI is slightly different. See https://developer.apple.com/documentation/xcode/writing-arm64-code-for-apple-platforms#//apple_ref/doc/uid/TP40013702-SW1
|
Cheapskates :) Thanks for the information, it is like MSVC as far as long doubles go. |
|
Going to close this, NumPy now has both thin and universal2 wheels, although we cannot test on M1. |
|
Sorry if this is a trivial question, but is there a way to install the universal wheel explicitly with "pip install"? I would like to have an environment that supports both Intel and ARM, just like Python itself does. I am aware that I can download and install the wheel manually, I am looking specifically with an automated way. |
|
There is no way to do that AFAIK. |
|
@charris I do not see universal wheels labeled at PyPI, where can I find the builds? https://pypi.org/project/numpy/#files Thanks! |
|
There aren't any, |
Feature
The binary wheels for macOS on PyPI are currently for x86_64. Please also provide a universal2 wheel (x86_64 and arm64) for use with the (currently experimental) universal2 Python 3.9.1 installer on Python.org.
The text was updated successfully, but these errors were encountered: