10000 BUG: Undefined minimum with numpy compiled using intel compilers · Issue #27840 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

BUG: Undefined minimum with numpy compiled using intel compilers #27840

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
DavidHuber-NOAA opened this issue Nov 25, 2024 · 13 comments
Closed

BUG: Undefined minimum with numpy compiled using intel compilers #27840

DavidHuber-NOAA opened this issue Nov 25, 2024 · 13 comments
Labels
00 - Bug

Comments

@DavidHuber-NOAA
Copy link

Describe the issue:

When attempting to use an ndarray method (like min()) with an invalid subscripted array, the returning value is not repeatable when Numpy is compiled with GNU and Intel Classic compilers (note that Intel LLVM compilers behave like GNU). With Intel, the returning value is NaN, with GNU, it is the expected result.

Reproduce the code example:

import numpy as np

rows = np.asarray([0, 0])
val = np.asarray([0.11])
print(val[rows].min())  # Returns 0.11 with GNU and LLVM, NaN with Intel Classic

Error message:

N/A in both cases

Python and NumPy Versions:

Numpy: 1.25.2
Python: 3.11.7 (main, Aug 29 2024, 15:22:52) [GCC Intel(R) C++ gcc 9.2 mode]

Note that this is repeatable on other systems with different versions of gcc

Runtime Environment:

Compiled via spack.
[{'numpy_version': '1.25.2',
'python': '3.11.7 (main, Aug 29 2024, 15:22:52) [GCC Intel(R) C++ gcc 9.2 '
'mode]',
'uname': uname_result(system='Linux', node='hfe02', release='4.18.0-553.22.1.el8_10.x86_64', version='#1 SMP Wed Sep 25 09:20:43 UTC 2024', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2',
'AVX512F',
'AVX512CD',
'AVX512_SKX'],
'not_found': ['AVX512_KNL',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL']}},
{'architecture': 'Haswell',
'filepath': '/scratch1/NCEPDEV/global/David.Huber/SPACK/ss_matplotlib/envs/matplotlib_intel/install/intel/2021.5.0/openblas-0.3.24-tmyu6qy/lib/libopenblas-r0.3.24.so',
'internal_api': 'openblas',
'num_threads': 1,
'prefix': 'libopenblas',
'threading_layer': 'disabled',
'user_api': 'blas',
'version': '0.3.24'}]

Context for the issue:

Found while debugging matplotlib/matplotlib#28762
This was referenced Nov 25, 2024
Closed
Closed
@seberg
Copy link
Member
seberg commented Nov 25, 2024

with an invalid subscripted array

This is a fully valid subscripted array and just repeats the two values returning a contiguous copy.
Can you please narrow down by checking the result of the subscript and see if that is good, also trying to just use np.array([0.11, 0.11]).min() directly?

@DavidHuber-NOAA
Copy link
Author

Interesting:

>>> rows = np.array([0,0])
>>> val = np.array([0.11])
>>> val[rows]
array([0.11, 0.11])
>>> np.array([0.11, 0.11]).min()
nan

@DavidHuber-NOAA
Copy link
Author

Also interesting that it matters if they are floats or ints:

>>> np.array([0, 0]).min()
0
>>> np.array([0.1, 0.1]).min()
nan
>>> np.array([-1, -1]).min()
-1
>>> np.array([-1.1, -1.1]).min()
nan

@seberg
Copy link
Member
seberg commented Nov 25, 2024

Well, such a compiler sensitive thing was likely to be SIMD related. Length 1 is not very interesting because there is no reduction.
Length 2 or 3 makes sense for example, because the compiler may use SIMD instructions. For minimum, though, we would still call the reduction with a single length 1

Since this is apparently compiler dependent, there is a decent chance it is a compiler bug (incorrect optimization).

You may be able to make it work if you set NPY_DISABLE_CPU_FEATURES=AVX, but not sure (see np.show_runtime() for more tries).
Since there is a single value, this may kick in even in the always present SSE version.

CC @r-devulap and @seiko2plus. I am also not sure if this may just be fixed in newer versions, since 1.25 is pretty old. (also there were a lot of discussions about intel compiler problems recently I haven't quite kept up with).

@mhvk mhvk changed the title BUG: Unusual behavior when indexing 1-D ndarrays with length 2+ indexing ndarrays BUG: Undefined minimum with numpy compiled using intel compilers Nov 25, 2024
@DavidHuber-NOAA
8000 Copy link
Author

The issue is present with numpy up to version 1.26.4. I haven't tried 1.26.5 as it doesn't have a spack recipe yet. Similarly, the version of spack-stack that I am working with does not have any recipes for numpy v2.x. I may try upgrading to a newer version of spack and seeing if the problem still exists for v2.1.2.

That said, disabling one of two CPU features was successful:

> export NPY_DISABLE_CPU_FEATURES=AVX512F && python -c "import numpy as np; print(np.array([-1.1, -1.1]).min())"
-1.1
> export NPY_DISABLE_CPU_FEATURES=AVX512CD && python -c "import numpy as np; print(np.array([-1.1, -1.1]).min())"
-1.1

Disabling any other combination of AVX or SSE features returned nan, so it seems to be an AVX512 feature that causes the problem.

@DavidHuber-NOAA DavidHuber-NOAA mentioned this issue Nov 26, 2024
Closed
@DavidHuber-NOAA
Copy link
Author

After setting NPY_DISABLE_CPU_FEATURES=AVX512F, numpy, when compiled with Intel classic compilers, calculates the minimum correctly. I have opened Spack PR JCSDA/spack#486 to set this flag on any system that has AVX512 instructions. Once merged, I think we can close this issue.

@seberg
Copy link
Member
seberg commented Nov 26, 2024

You can disable it at build time rather than runtime. Although, of course the real issue here is probably a compiler bug. But depending on how old the compiler is, I am not sure that is worth thinking about...

(But it may be a shame to drag this hack around indefinitely.)

@DavidHuber-NOAA
Copy link
Author

@seberg That's a good point RE building. The 'classic' Intel compilers are officially retired, though NOAA is still in the process of migrating to the newer LLVM compilers (which do not have this issue) and I suspect many groups will be using the classic compilers for at least a few years to come. I believe the last update to the classic compilers came at the beginning of this year.

@r-devulap
Copy link
Member
r-devulap commented Nov 27, 2024 •
edited
Loading

@seberg I wonder if this PR #26281 fixed the bug reported here

@r-devulap
Copy link
Member

@DavidHuber-NOAA is it possible to patch #26281 and see if your bug goes away?

@seberg
Copy link
Member
seberg commented Nov 27, 2024

Hmmm, that bug does seem likely relate and probably can be included as a patch in the spack build.

@DavidHuber-NOAA
Copy link
Author

@r-devulap Thanks for the suggestion. Unfortunately, after including that patch, I am still getting nan. That said, I think I will still include the patch in JCSDA/spack#486.

@DavidHuber-NOAA DavidHuber-NOAA mentioned this issue Dec 5, 2024
Merged
@DavidHuber-NOAA
Copy link
Author

spack/spack#47941 was merged. Closing this issue.

@DavidHuber-NOAA DavidHuber-NOAA mentioned this issue Jan 8, 2025
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@seberg @r-devulap @DavidHuber-NOAA
0