8000 BUG: SciPy `linalg.logm` function signature mismatch with NumPy >=2.1.0rc1 due to hidden visibility for API tables · Issue #27865 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

BUG: SciPy linalg.logm function signature mismatch with NumPy >=2.1.0rc1 due to hidden visibility for API tables #27865

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
agriyakhetarpal opened this issue Nov 26, 2024 · 4 comments
Open

BUG: SciPy linalg.logm function signature mismatch with NumPy >=2.1.0rc1 due to hidden visibility for API tables #27865

agriyakhetarpal opened this issue Nov 26, 2024 · 4 comments
Labels
00 - Bug

Comments

@agriyakhetarpal
Copy link
Contributor
agriyakhetarpal commented Nov 26, 2024 •
edited
Loading

Describe the issue:

Hi there, we've been facing an issue wherein we have a mismatch in the function signature for the scipy.linalg.logm function (internally, this is coming from the Schur decomposition and, in turn, from the dgees LAPACK routine) for WASM/Pyodide in pyodide/pyodide#4925 when NumPy is compiled with the Emscripten toolchain.

It is to be noted that Meson sets -fvisibility=hidden by default, thus overriding what emcc sets by default (which is -fvisibility=default, confirmed using emcc --flags) – so this is the same issue as pyodide/pyodide#4310 that has been previously reported: the _npy_f2py_ARRAY_API symbol is not exposed anymore (which it shouldn't be in a normal scenario, though).

Compiling against NumPy >=1.26.4,<=2.0.2 works fine; @rgommers has pointed out that this has been coming from the changes introduced via gh-26103 or gh-26286 that first appeared in NumPy versions 2.1.0rc1 and later—which I agree are good in principle—but we have been overriding the symbol visibility anyway post the introduction of pyodide/pyodide#4313, which is something that we ought to fix, hence while this issue may inspect on a problem with numpy.f2py, it's also centred on guidance how to proceed with compiling SciPy to work correctly going forward against changes in NumPy and whether we should retain a patch that undoes such changes.

More specifically, in the WASM code, logm is defined in the relevant memory address call_indirect (param i32) (result i32) – but the value in the WASM table returns null. For versions of NumPy that work, this symbol shows up with the name ndarray_from_pyobj.

cc: @rgommers, @ryanking13, @hoodmane

For a reproducer, it is currently required to compile SciPy in-tree, since we don't build NumPy v2 and dependent packages with it yet (hence the PR to update it). Here are all the prerequisites (some system-level packages might still be required via apt/yum/dnf/Homebrew):

# Set up Emscripten
git clone https://github.com/emscripten-core/emsdk.git ~/emsdk # or any other location
pushd ~/emsdk
./emsdk install 3.1.58
./emsdk activate 3.1.58
popd
source ./emsdk_env.sh

# Set up a development environment
git clone https://github.com/agriyakhetarpal/pyodide.git
git checkout update/numpy-v2

# Set up a debug build for Pyodide, libraries, and packages
python3.12 -m virtualenv .venv
source .venv/bin/activate
python -m pip install pyodide-build
PYODIDE_DEBUG=1 PYODIDE_JOBS=$(nproc) PYODIDE_PACKAGES="scipy" make

Once OpenBLAS and other libraries + Emscripten wheels for NumPy and SciPy get compiled, we can use them:

# Create and activate Pyodide venv
pyodide venv .venv-pyodide
source .venv-pyodide/bin/activate
python -m pip install dist/*.whl  # install SciPy and all other wheels

In this activated Python/Pyodide interpreter, please run the following code to

Reproduce the code example:

import numpy as np
from scipy.linalg import logm

np.random.seed(1234)

n = 2
scale = 1e-4
A = (np.eye(n) + np.random.rand(n, n) * scale).astype(np.float64)

logm(A)

Error message:

Pyodide has suffered a fatal error. Please report this to the Pyodide maintainers.
The cause of the fatal error was:
RuntimeError: null function or function signature mismatch
    at wasm://wasm/0003f1c2:wasm-function[99]:0xb4ef
    at invoke_iiiiiiii (/src/dist/pyodide.asm.js:116336:40)
    at wasm://wasm/0060573e:wasm-function[225]:0x3d319
    at wasm://wasm/0003f1c2:wasm-function[101]:0xc3b6
    at _PyEM_TrampolineCall_JS (/src/dist/pyodide.asm.js:10307:33)
    at pyodide.asm.wasm._PyObject_MakeTpCall (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[1155]:0x1cc781)
    at pyodide.asm.wasm.PyObject_Vectorcall (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[1164]:0x1ccf92)
    at pyodide.asm.wasm._PyEval_EvalFrameDefault (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[3432]:0x2aaefa)
    at pyodide.asm.wasm.PyEval_EvalCode (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[3430]:0x2a0968)
    at pyodide.asm.wasm.run_mod (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[4235]:0x313fc7) {
  pyodide_fatal_error: true
}
Stack (most recent call first):
  File "/src/.venv-pyodide/lib/python3.12/site-packages/scipy/linalg/_decomp_schur.py", line 149 in schur
  File "/src/.venv-pyodide/lib/python3.12/site-packages/scipy/linalg/_matfuncs_inv_ssq.py", line 874 in _logm
  File "/src/.venv-pyodide/lib/python3.12/site-packages/scipy/linalg/_matfuncs.py", line 200 in logm
  File "<stdin>", line 1 in <module>
RuntimeError: null function or function signature mismatch
    at wasm://wasm/0003f1c2:wasm-function[99]:0xb4ef
    at invoke_iiiiiiii (/src/dist/pyodide.asm.js:116336:40)
    at wasm://wasm/0060573e:wasm-function[225]:0x3d319
    at wasm://wasm/0003f1c2:wasm-function[101]:0xc3b6
    at _PyEM_TrampolineCall_JS (/src/dist/pyodide.asm.js:10307:33)
    at pyodide.asm.wasm._PyObject_MakeTpCall (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[1155]:0x1cc781)
    at pyodide.asm.wasm.PyObject_Vectorcall (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[1164]:0x1ccf92)
    at pyodide.asm.wasm._PyEval_EvalFrameDefault (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[3432]:0x2aaefa)
    at pyodide.asm.wasm.PyEval_EvalCode (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[3430]:0x2a0968)
    at pyodide.asm.wasm.run_mod (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[4235]:0x313fc7) {
  pyodide_fatal_error: true
}

Python and NumPy Versions:

2.1.3
3.12.7 (main, Nov 26 2024, 19:46:55) [Clang 19.0.0git (https:/github.com/llvm/llvm-project 0a8cd1ed1f4f35905df318015b

Runtime Environment:

[{'numpy_version': '2.1.3',
  'python': '3.12.7 (main, Nov 26 2024, 19:46:55) [Clang 19.0.0git '
            '(https:/github.com/llvm/llvm-project 0a8cd1ed1f4f35905df318015b',
  'uname': uname_result(system='Emscripten', node='emscripten', release='3.1.58', version='#1', machine='wasm32')},
 {'simd_extensions': {'baseline': [], 'found': [], 'not_found': []}}]

Context for the issue:

This affects pyodide/pyodide#4925, where we are trying to upgrade NumPy to version >=2, along with all its dependents.

We have some notes on debugging a function signature mismatch in Pyodide, which is where this bug report has originated from: https://pyodide.org/en/stable/development/debugging.html#debugging-runtimeerror-function-signature-mismatch

There are probably some useful insights from here onwards, too: pyodide/pyodide#4925 (comment)
@agriyakhetarpal

This comment was marked as resolved.

Sign in to view
@agriyakhetarpal
Copy link
Contributor Author

@ryanking13, could you please re-recheck if setting
-DNPY_API_SYMBOL_ATTRIBUTE=__attribute__((visibility("default"))) was working for you? I'm unable to get logm passing with that flag. The task of simply reverting gh-26103 and gh-26286 does not fully fix the issue either.

@ryanking13
Copy link
Contributor

@ryanking13, could you please re-recheck if setting
-DNPY_API_SYMBOL_ATTRIBUTE=attribute((visibility("default"))) was working for you? I'm unable to get logm passing with that flag. The task of simply reverting #26103 and #26286 does not fully fix the issue either.

Sure, I'll recheck it.

Meanwhile, if NumPy >=1.26.4,<=2.0.2 works fine, how about upgrading to Numpy 2.0.2 first and upgrading to a newer version in a separate PR (in Pyodide)? I think it could narrow down the points we should focus on.

@rgommers
Copy link
Member

it's also centred on guidance how to proceed with compiling SciPy to work correctly going forward against changes in NumPy and whether we should retain a patch that undoes such changes.

Keeping a patch for this seems perfect 8666 ly fine to me, whatever works there. You can also define NPY_API_SYMBOL_ATTRIBUTE (see docs in gh-26103), and perhaps it helps to add and cc.get_id() != 'emscripten' to this check (?):

https://github.com/rgommers/numpy/blob/84f283a5ab2564a50c4e7a295d36280fc579930e/numpy/_core/meson.build#L505

For versions of NumPy that work, this symbol shows up with the name ndarray_from_pyobj.

This is an f2py utility defined in numpy/f2py/src/fortranobject.c. That file is built by the project using f2py, not by NumPy itself. For SciPy, it's here:

https://github.com/scipy/scipy/blob/a58579ed5f57e6fddc7b7e2f3ad2e53890a61d59/scipy/meson.build#L128-L134

@agriyakhetarpal agriyakhetarpal mentioned this issue Nov 27, 2024
Merged
2 tasks
@agriyakhetarpal agriyakhetarpal mentioned this issue Jan 28, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rgommers @ryanking13 @agriyakhetarpal
0