8000 BLD: use github to build macos-arm64 wheels with OpenBLAS and update to 0.3.30 by mattip · Pull Request #29069 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

BLD: use github to build macos-arm64 wheels with OpenBLAS and update to 0.3.30 #29069

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

BLD: use github to build macos-arm64 wheels with OpenBLAS and update to 0.3.30 #29069

wants to merge 3 commits into from
+5 −145

Conversation

mattip
Copy link
Member
@mattip mattip commented May 27, 2025

Move the macos-arm64 wheel builds with OpenBLAS (targeting MACOSX_DEPLOYMENT_TARGET=11) on github runners instead of cirrus. If this works, maybe it will impact the build failure in #29039. It should simplify CI, and will be less expensive, but does make us more dependent on github.

@github-actions github-actions bot added the 36 - Build Build related PR label May 27, 2025
andyfaff
andyfaff reviewed May 27, 2025
View reviewed changes
@mattip
Copy link
Member Author
mattip commented May 27, 2025

A single parameterized test is failing for the csingle and cdouble cases:

     @pytest.mark.parametrize('dtype', [single, double, csingle, cdouble])
      def test_types(self, dtype):
          x = np.array([[1, 0.5], [0.5, 1]], dtype=dtype)
  >       assert_equal(np.linalg.det(x).dtype, dtype)
...
RuntimeWarning: divide by zero encountered in det

@mattip mattip mentioned this pull request May 27, 2025
Merged
@mattip
Copy link
Member Author
mattip commented May 29, 2025

Changing to draft to indicate that this is stuck until we can figure out what is going wrong

@mattip mattip marked this pull request as draft May 29, 2025 12:34
@mattip
Copy link
Member Author
mattip commented Jun 5, 2025

In case the logs go away before this gets revisited: the (failing) meson build here reports

clang 15.0.0 "Apple clang version 15.0.0 (clang-1500.3.9.4)"

while the (successful) cirrus CI one reports clang 16.0.0 "Apple clang version 16.0.0 (clang-1600.0.26.4)"

@mattip mattip mentioned this pull request Jun 27, 2025
Closed
@mattip mattip force-pushed the github-macos-arm64-wheels branch from fb3bf2e to 17d0308 Compare June 29, 2025 16:47
@mattip
Copy link
Member Author
mattip commented Jun 29, 2025

I hacked the call to zgetrf to reset the FPE registers on macos-arm64, since I couldn't actually fix the problem. In my defense most people with macos-arm64 machines will be using the Accelerate LAPACK wheel and not the OpenBLAS one, since they will have macos-14+ (released Sep 2023).

@mattip
Copy link
Member Author
mattip commented Jun 29, 2025 •
edited
Loading

I wonder why scipy does not see this

Edit: scipy, if I am not mistaken, does not check FPE registers?

@mattip
Copy link
Member Author
mattip commented Jun 29, 2025

The goal of this PR is to build macos-arm64 wheels on github, and with the hacky fix it is passing. The pypy failure is unrelated.

@martin-frbg
Copy link

If you can tell me what to look out for, I may find time eventually, but please don't expect me to wade through numpy logs. I take it you are seeing fpe exception registers getting set without an exception actually being raised, i.e. something like an underflow or a quiet NaN that depends on xcode/AppleClang version (so might be an Apple compiler bug) ?

@charris
Copy link
Member
charris commented Jun 29, 2025

depends on xcode/AppleClang

IIRC, it is using SME that has that problem, apparently it is part of the spec. See #29223.

@mattip
Copy link
Member Author
mattip commented Jun 30, 2025

@martin-frbg sorry for the noise, I will try to whittle it down to a minimal reproducer in C. The general idea is that this sets FPE error registers in the call to zgetrf when I use a NumPy built locally or with github CI on macOS-arm64, but does not when build on cirrusCI.

x = np.array([[1, 0.5], [0.5, 1]], dtype=np.complex128)
d = np.linalg.det(x)

@mattip
Copy link
Member Author
mattip commented Jul 2, 2025

I don't know if I should open an issue on OpenBLAS, I am not sure they commit to correct handling of FPE registers. Here is the minimum reproducer I could create, starting in the OpenBLAS repo HEAD on macos-arm64

  • Compile OpenBLAS with

    CFLAGS=' -arch arm64 -fvisibility=protected -Wno-uninitialized' \
make BUFFERSIZE=20 DYNAMIC_ARCH=1 QUIET_MAKE=1 USE_OPENMP=0 NUM_THREADS=64 BINARY=64 INTERFACE64=1 TARGET=VORTEX

  • Compile this test.c with clang test.c -L. -lopenblas -lm

    test.c 
    #include <stdio.h>
#include <fenv.h>
#include <stdint.h>

typedef struct { double r, i; } f2c_doublecomplex;
typedef int64_t fortran_int;

fortran_int
zgetrf_(fortran_int *m, fortran_int *n,
            f2c_doublecomplex a[], fortran_int *lda,
            fortran_int ipiv[],
            fortran_int *info);

int main(int argv, const char *argc[]) {

  f2c_doublecomplex a[] = {{1, 0}, {0.5, 0}, {0.5, 0}, {1, 0}};
  fortran_int m = 2, n=2, lda=2, info=1, ipiv[] = {0, 0};
  fortran_int ret = zgetrf_(&m, &n, a, &lda, ipiv, &info);
  int fpstatus = fetestexcept(FE_DIVBYZERO | FE_OVERFLOW | FE_UNDERFLOW | FE_INVALID);
  printf("zgetrf returned %lld, fpstatus=%d, info=%lld", ret, fpstatus, info);
  return 0;
}

  • Run the test with OPENBLAS_VERBOSE=2 ./a.out. On my macbook m2 it prints

    Core: neoversen1
zgetrf returned 0, fpstatus=3, info=0

    where on x86_64 it prints fpstatus=0. I did try OPENBLAS_CORETYPE=ARMV8 but still got fpstatus=3.

@mattip
Copy link
Member Author
mattip commented Jul 2, 2025

@seiko2plus pointed out we need to add -ftrapping-math when using clang. Indeed, the reproducer above does not emit FP errors when using that flag 👍

@mattip mattip force-pushed the github-macos-arm64-wheels branch from 17d0308 to 88e865d Compare July 3, 2025 04:30
@mattip mattip changed the title BLD: use github to build macos-arm64 wheels with OpenBLAS BLD: use github to build macos-arm64 wheels with OpenBLAS and update to 0.3.30 Jul 3, 2025
@mattip
Copy link
Member Author
mattip commented Jul 3, 2025

rebased and updated to use OpenBLAS 0.3.30 compiled with -ftrapping-math

@mattip mattip marked this pull request as ready for review July 3, 2025 05:29
@mattip
Copy link
Member Author
mattip commented Jul 3, 2025

wheel building on macos is passing, now that OpenBLAS is built with -ftrapping-math. Apparently this is known behaviour, see #19479, #24060.

The PyPy failures are fixed on PyPy-HEAD, the wheel building uses the last release.

@seberg
Copy link
Member
seberg commented Jul 3, 2025 •
edited
Loading

Would be nice if this ended up fixing the mac-os matmul warning heisenbug :).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rgommers rgommers rgommers left review comments

@andyfaff andyfaff andyfaff left review comments

Assignees
No one assigned
Labels
36 - Build Build related PR
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@mattip @martin-frbg @charris @seberg @rgommers @andyfaff
0