8000 BUG: Undefined math symbols when compiling from source against intel MKL · Issue #27111 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content

BUG: Undefined math symbols when compiling from source against intel MKL #27111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AgilentGCMS opened this issue Aug 4, 2024 · 32 comments
Closed
Labels

Comments

@AgilentGCMS
Copy link

Describe the issue:

I have compiled python 3.12.4 from source on a computing cluster, and I am trying to compile numpy. The goal is to install it in the same tree as the locally compiled python binary. We have intel compilers on the cluster, so I am trying to get numpy to use intel MKL libraries. I compiled numpy with

CC=icx CXX=icpx python -m pip install --no-build-isolation -v . -Cbuild-dir=build -Csetup-args=-Dallow-noblas=false -Csetup-args=-Dblas-order=mkl -Csetup-args=-Dlapack-order=mkl -Csetup-args=-Dblas=mkl-static-lp64-iomp -Csetup-args=-Dlapack=mkl-static-lp64-iomp

This installed and compiled without complaints. However, when I try to import numpy from within python, I get

$ python
Python 3.12.4 (main, Jul 29 2024, 10:14:44) [Clang 18.0.0 (icx 2024.1.0.20240308)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/overrides.py", line 8, in <module>
    from numpy._core._multiarray_umath import (
ImportError: /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so: undefined symbol: pow8_h

An invocation of ldd shows the following libraries linked to the above mentioned shared object file:

$ ldd /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so
	linux-vdso.so.1 (0x00007ffef53dd000)
	libimf.so => /apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2024.1.0-76aou5yu2oqfcz72vq7lp77bp3rqcoht/compiler/2024.1/lib/libimf.so (0x000014a0f6287000)
	libiomp5.so => /apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2024.1.0-76aou5yu2oqfcz72vq7lp77bp3rqcoht/compiler/2024.1/lib/libiomp5.so (0x000014a0f5ea1000)
	libsvml.so => /apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2024.1.0-76aou5yu2oqfcz72vq7lp77bp3rqcoht/compiler/2024.1/lib/libsvml.so (0x000014a0f484d000)
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x000014a0f4626000)
	libintlc.so.5 => /apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2024.1.0-76aou5yu2oqfcz72vq7lp77bp3rqcoht/compiler/2024.1/lib/libintlc.so.5 (0x000014a0f45c4000)
	libc.so.6 => /usr/lib64/libc.so.6 (0x000014a0f43bb000)
	librt.so.1 => /usr/lib64/librt.so.1 (0x000014a0f43b6000)
	libdl.so.2 => /usr/lib64/libdl.so.2 (0x000014a0f43b1000)
	libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x000014a0f43aa000)
	/lib64/ld-linux-x86-64.so.2 (0x000014a0f8526000)
	libm.so.6 => /usr/lib64/libm.so.6 (0x000014a0f42cf000)
	libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x000014a0f42b4000)

So what is missing here?

Reproduce the code example:

import numpy

Error message:

Traceback (most recent call last):
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/overrides.py", line 8, in <module>
    from numpy._core._multiarray_umath import (
ImportError: /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-x86_64-linux-gnu.so: undefined symbol: pow8_h

Python and NumPy Versions:

Python 3.12.4
Numpy 2.0.1

Runtime Environment:

Cannot show this because importing numpy fails

Context for the issue:

No response

@AgilentGCMS
Copy link
Author

@rgommers Apologies for bothering you, but in the past you've helped solve issues at the confluence of numpy and MKL, so perhaps this problem has an obvious solution to you?

@mattip
Copy link
Member
mattip commented Aug 5, 2024

The missing symbol is pow8_h. What cpu architecture are you targeting? Google does not seem to be able to find that symbol at all.

@rgommers
Copy link
Member
rgommers commented Aug 5, 2024

I can't find any reports about pow8_h in this issue tracker, but I'm sure I've seen it before - it's a known issue with SVML IIRC.

@AgilentGCMS
Copy link
Author

The missing symbol is pow8_h. What cpu architecture are you targeting? Google does not seem to be able to find that symbol at all.

Intel Ice Lake, or, from /proc/cpuinfo,

vendor_id	: GenuineIntel
cpu family	: 6
model		: 106
model name	: Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz
stepping	: 6
microcode	: 0xd0003d1
cpu MHz		: 2300.000
cache size	: 61440 KB
physical id	: 1
siblings	: 80
core id		: 39
cpu cores	: 40
apicid		: 207
initial apicid	: 207
fpu		: yes
fpu_exception	: yes
cpuid level	: 27
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities
vmx flags	: vnmi preemption_timer posted_intr invvpid ept_x_only ept_ad ept_1gb flexpriority apicv tsc_offset vtpr mtf vapic ept vpid unrestricted_guest vapic_reg vid ple shadow_vmcs pml ept_mode_based_exec tsc_scaling
bugs		: spectre_v1 spectre_v2 spec_store_bypass swapgs mmio_stale_data eibrs_pbrsb
bogomips	: 4605.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 57 bits virtual

@r-devulap
Copy link
Member

I have seen this before, let me try and find what the mitigation was

@AgilentGCMS
Copy link
Author

Not that this helps, when I compile against the dynamic library (i.e., mkl-dynamic-lp64-iomp instead of mkl-static-lp64-iomp), I get an error about undefined symbol: tan8_h. It's probably the first undefined symbol found, i.e., I don't believe that tan8_h was defined in the static library but is undefined in the dynamic library.

@r-devulap
Copy link
Member

Could you add this to CFLAGS and rebuild? -fveclib=none. That should fix the error.

@r-devulap
Copy link
Member

I had filed this issue a while back and forgot about it :) #26257. This flag should be added into our meson build for icx/icpx compiler. I will submit a PR to do that.

@AgilentGCMS
Copy link
Author

Could you add this to CFLAGS and rebuild? -fveclib=none. That should fix the error.

You mean just CFLAGS='-fveclib=none' on the same line where I'm defining CC and CXX? I am not setting any other CFLAGS anywhere.

@r-devulap
Copy link
Member

Could you add this to CFLAGS and rebuild? -fveclib=none. That should fix the error.

You mean just CFLAGS='-fveclib=none' on the same line where I'm defining CC and CXX? I am not setting any other CFLAGS anywhere.

yup

@AgilentGCMS
Copy link
Author

@r-devulap That indeed fixed the issue. Thanks!

@AgilentGCMS
Copy link
Author

I think I need to reopen this issue, unless what I'm seeing is unrelated. With that addition to CFLAGS, numpy built successfully. However, it fails import numpy; numpy.test() with the following:

Fatal Python error: Segmentation fault

Thread 0x000014baf042a640 (most recent call first):
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/threading.py", line 355 in wait
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/threading.py", line 655 in wait
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/IPython/core/history.py", line 903 in run
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/IPython/core/history.py", line 61 in only_when_enabled
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/decorator.py", line 232 in fun
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/threading.py", line 1030 in _bootstrap

Current thread 0x000014baf3d8a740 (most recent call first):
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/fromnumeric.py", line 57 in _wrapfunc
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/fromnumeric.py", line 1200 in argsort
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/tests/test_multiarray.py", line 10106 in test_argsort_float
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/_pytest/python.py", line 159 in pytest_pyfunc_call
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/_pytest/python.py", line 1627 in runtest
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/_pytest/runner.py", line 174 in pytest_runtest_call
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/_pytest/runner.py", line 242 in <lambda>
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/_pytest/runner.py", line 241 in call_and_report
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/_pytest/runner.py", line 132 in runtestprotocol
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/_pytest/main.py", line 362 in pytest_runtestloop
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/_pytest/main.py", line 337 in _main
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/_pytest/main.py", line 283 in wrap_session
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_pytesttester.py", line 195 in __call__
  File "<ipython-input-2-dab0198860c9>", line 1 in <module>
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3577 in run_code
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3517 in run_ast_nodes
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3334 in run_cell_async
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/IPython/core/async_helpers.py", line 128 in _pseudo_sync_runner
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3130 in _run_cell
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3075 in run_cell
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/IPython/terminal/interactiveshell.py", line 910 in interact
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/IPython/terminal/interactiveshell.py", line 917 in mainloop
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/IPython/terminal/ipapp.py", line 317 in start
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/traitlets/config/application.py", line 1075 in launch_instance
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/IPython/__init__.py", line 130 in start_ipython
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/bin/ipython", line 8 in <module>

@AgilentGCMS AgilentGCMS reopened this Aug 8, 2024
@seberg
Copy link
Member
seberg commented Aug 8, 2024

I tried debugging with gdb --args python -c 'import numpy; numpy.test()', here's the part of the output with errors

The warnings are maybe just Intel compiler enabling fast-math by default and not related to the segfault? (I thought our build process disables it but dunno.)

It may be easier to run the test explicitly on its own. (I suspect the above started a new process maybe?)
(Although, I don't really understand why the gdb verson would have problems in the collecton if the normal test run doesn't, does it already fail during collection?)

import numpy as np
from numpy.testing import assert_equal

def assert_arg_sorted(arr, arg):
    # resulting array should be sorted and arg values should be unique
    assert_equal(arr[arg], np.sort(arr))
    assert_equal(np.sort(arg), np.arange(len(arg)))

dtype = np.float64  # might also be float32
N = 512  # might actually be any value between 1 and 512

for dtype in (np.float32, np.float64):
    for N in np.arange(1, 512):
        rnd = np.random.RandomState(116112)
        # (1) Regular data with a few nan: doesn't use vectorized sort
        arr = -0.5 + rnd.random(N).astype(dtype)
        arr[rnd.choice(arr.shape[0], 3)] = np.nan
        assert_arg_sorted(arr, np.argsort(arr, kind='quick'))
        # (2) Random data with inf at the end of array
        # See: https://github.com/intel/x86-simd-sort/pull/39
        arr = -0.5 + rnd.rand(N).astype(dtype)
        arr[N-1] = np.inf
        assert_arg_sorted(arr, np.argsort(arr, kind='quick'))

This goes into highway, I wonder if it might make sense to pull from main, but I think the fixes in highway sorting were just for MacOS.

@AgilentGCMS
Copy link
Author

@seberg Thanks for that test code! It fails with the following:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
    [... skipping hidden 1 frame]

File /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/arrayprint.py:762, in array2string(a, max_line_width, precision, suppress_small, separator, prefix, style, formatter, threshold, edgeitems, sign, floatmode, suffix, legacy)
    760     return "[]"
--> 762 return _array2string(a, options, separator, prefix)

File /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/arrayprint.py:534, in _recursive_guard.<locals>.decorating_function.<locals>.wrapper(self, *args, **kwargs)
    533 try:
--> 534     return f(self, *args, **kwargs)
    535 finally:

File /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/arrayprint.py:560, in _array2string(a, options, separator, prefix)
    559 # find the right formatting function for the array
--> 560 format_function = _get_format_function(data, **options)
    562 # skip over "["

File /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/arrayprint.py:493, in _get_format_function(data, **options)
    492     else:
--> 493         return formatdict['float']()
    494 elif issubclass(dtypeobj, _nt.complexfloating):

File /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/arrayprint.py:432, in _get_formatdict.<locals>.<lambda>()
    423 def _get_formatdict(data, *, precision, floatmode, suppress, sign, legacy,
    424                     formatter, **kwargs):
    425     # note: extra arguments in kwargs are ignored
    426
    427     # wrapped in lambdas to avoid taking a code path
    428     # with the wrong type of data
    429     formatdict = {
    430         'bool': lambda: BoolFormat(data),
    431         'int': lambda: IntegerFormat(data, sign),
--> 432         'float': lambda: FloatingFormat(
    433             data, precision, floatmode, suppress, sign, legacy=legacy),
    434         'longfloat': lambda: FloatingFormat(
    435             data, precision, floatmode, suppress, sign, legacy=legacy),
    436         'complexfloat': lambda: ComplexFloatingFormat(
    437             data, precision, floatmode, suppress, sign, legacy=legacy),
    438         'longcomplexfloat': lambda: ComplexFloatingFormat(
    439             data, precision, floatmode, suppress, sign, legacy=legacy),
    440         'datetime': lambda: DatetimeFormat(data, legacy=legacy),
    441         'timedelta': lambda: TimedeltaFormat(data),
    442         'object': lambda: _object_format,
    443         'void': lambda: str_format,
    444         'numpystr': lambda: repr_format}
    446     # we need to wrap values in `formatter` in a lambda, so that the interface
    447     # is the same as the above values.

File /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/arrayprint.py:964, in FloatingFormat.__init__(self, data, precision, floatmode, suppress_small, sign, legacy)
    962 self.large_exponent = False
--> 964 self.fillFormat(data)

File /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/arrayprint.py:996, in FloatingFormat.fillFormat(self, data)
    995 frac_strs, _, exp_strs = zip(*(s.partition('e') for s in strs))
--> 996 int_part, frac_part = zip(*(s.split('.') for s in frac_strs))
    997 self.exp_size = max(len(s) for s in exp_strs) - 1

ValueError: not enough values to unpack (expected 2, got 1)

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[3], line 7
      5 arr = -0.5 + rnd.random(N).astype(dtype)
      6 arr[rnd.choice(arr.shape[0], 3)] = np.nan
----> 7 assert_arg_sorted(arr, np.argsort(arr, kind='quick'))
      8 # (2) Random data with inf at the end of array
      9 # See: https://github.com/intel/x86-simd-sort/pull/39
     10 arr = -0.5 + rnd.rand(N).astype(dtype)

Cell In[1], line 6, in assert_arg_sorted(arr, arg)
      4 def assert_arg_sorted(arr, arg):
      5     # resulting array should be sorted and arg values should be unique
----> 6     assert_equal(arr[arg], np.sort(arr))
      7     assert_equal(np.sort(arg), np.arange(len(arg)))

    [... skipping hidden 1 frame]

File /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_utils/__init__.py:85, in _rename_parameter.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
     83             raise TypeError(msg)
     84         kwargs[new_name] = kwargs.pop(old_name)
---> 85 return fun(*args, **kwargs)

    [... skipping hidden 1 frame]

File /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/contextlib.py:81, in ContextDecorator.__call__.<locals>.inner(*args, **kwds)
     78 @wraps(func)
     79 def inner(*args, **kwds):
     80     with self._recreate_cm():
---> 81         return func(*args, **kwds)

File /work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/testing/_private/utils.py:893, in assert_array_compare(comparison, x, y, err_msg, verbose, header, precision, equal_nan, equal_inf, strict, names)
    889 header = f'error during assertion:\n\n{efmt}\n\n{header}'
    891 msg = build_err_msg([x, y], err_msg, verbose=verbose, header=header,
    892                     names=names, precision=precision)
--> 893 raise ValueError(msg)

ValueError:
error during assertion:

Traceback (most recent call last):
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 854, in assert_array_compare
    + array2string(max_abs_error))
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/arrayprint.py", line 762, in array2string
    return _array2string(a, options, separator, prefix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/arrayprint.py", line 534, in wrapper
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/arrayprint.py", line 560, in _array2string
    format_function = _get_format_function(data, **options)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/arrayprint.py", line 493, in _get_format_function
    return formatdict['float']()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/arrayprint.py", line 432, in <lambda>
    'float': lambda: FloatingFormat(
                     ^^^^^^^^^^^^^^^
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/arrayprint.py", line 964, in __init__
    self.fillFormat(data)
  File "/work2/noaa/co2/sbasu/packages/python/3.12.4/lib/python3.12/site-packages/numpy/_core/arrayprint.py", line 996, in fillFormat
    int_part, frac_part = zip(*(s.split('.') for s in frac_strs))
    ^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)


Arrays are not equal
 ACTUAL: [repr failed for <ndarray>: not enough values to unpack (expected 2, got 1)]
 DESIRED: [repr failed for <ndarray>: not enough values to unpack (expected 2, got 1)]

The bit about disabling fast-math rings a bell, I will try recompiling numpy with -fp-model=strict.

@seberg
Copy link
Member
seberg commented Aug 8, 2024

Not clear whats going on there, I was hoping for segfault from gdb not some python errors anyway...

@AgilentGCMS
Copy link
Author
AgilentGCMS commented Aug 8, 2024

Not clear whats going on there, I was hoping for segfault from gdb not some python errors anyway...

The errors I posted from gdb might not have been accurate, so I deleted that comment and am trying to debug just based on the python output. Apologies for that confusion.

Basically, on our HPC systems we have "login" nodes where one can run single (or few) threaded applications, and "compute" nodes where one gets up to 80 cores. I realized that the error depended on the number of cores available, not gdb vs non-gdb.

@AgilentGCMS
Copy link
Author

I recompiled numpy with

CC=icx CXX=icpx CFLAGS='-fveclib=none -fp-model=strict' FFLAGS='-fp-model=strict' CXXFLAGS='-fp-model=strict' python -m pip install --no-build-isolation -v . -Cbuild-dir=build -Csetup-args=-Dallow-noblas=false -Csetup-args=-Dblas-order=mkl -Csetup-args=-Dlapack-order=mkl -Csetup-args=-Dblas=mkl-dynamic-lp64-iomp -Csetup-args=-Dlapack=mkl-dynamic-lp64-iomp

and now python -c 'import numpy; numpy.test() does not lead to a segfault, yay! However, it does fail 9 tests as below. I'm not sure if I should be concerned or whether I can expect a few tests to fail.

===================================================================================== FAILURES =====================================================================================
_________________________________________________________________ TestFloatExceptions.test_floating_exceptions[G] __________________________________________________________________

self = <numpy._core.tests.test_numeric.TestFloatExceptions object at 0x1507e415a630>, typecode = 'G'

    @pytest.mark.skipif(IS_WASM, reason="no wasm fp exception support")
    @pytest.mark.parametrize("typecode", np.typecodes["AllFloat"])
    def test_floating_exceptions(self, typecode):
        if 'bsd' in sys.platform and typecode in 'gG':
            pytest.skip(reason="Fallback impl for (c)longdouble may not raise "
                               "FPE errors as expected on BSD OSes, "
                               "see gh-24876, gh-23379")

        # Test basic arithmetic function errors
        with np.errstate(all='raise'):
            ftype = obj2sctype(typecode)
            if np.dtype(ftype).kind == 'f':
                # Get some extreme values for the type
                fi = np.finfo(ftype)
                ft_tiny = fi._machar.tiny
                ft_max = fi.max
                ft_eps = fi.eps
                underflow = 'underflow'
                divbyzero = 'divide by zero'
            else:
                # 'c', complex, corresponding real dtype
                rtype = type(ftype(0).real)
                fi = np.finfo(rtype)
                ft_tiny = ftype(fi._machar.tiny)
                ft_max = ftype(fi.max)
                ft_eps = ftype(fi.eps)
                # The complex types raise different exceptions
                underflow = ''
                divbyzero = ''
            overflow = 'overflow'
            invalid = 'invalid'

            # The value of tiny for double double is NaN, so we need to
            # pass the assert
            if not np.isnan(ft_tiny):
                self.assert_raises_fpe(underflow,
                                    lambda a, b: a/b, ft_tiny, ft_max)
                self.assert_raises_fpe(underflow,
                                    lambda a, b: a*b, ft_tiny, ft_tiny)
            self.assert_raises_fpe(overflow,
                                   lambda a, b: a*b, ft_max, ftype(2))
            self.assert_raises_fpe(overflow,
                                   lambda a, b: a/b, ft_max, ftype(0.5))
            self.assert_raises_fpe(overflow,
                                   lambda a, b: a+b, ft_max, ft_max*ft_eps)
            self.assert_raises_fpe(overflow,
                                   lambda a, b: a-b, -ft_max, ft_max*ft_eps)
>           self.assert_raises_fpe(overflow,
                                   np.power, ftype(2), ftype(2**fi.nexp))

divbyzero  = ''
fi         = finfo(resolution=1e-18, min=-1.189731495357231765e+4932, max=1.189731495357231765e+4932, dtype=float128)
ft_eps     = np.clongdouble('1.084202172485504434e-19+0j')
ft_max     = np.clongdouble('1.189731495357231765e+4932+0j')
ft_tiny    = np.clongdouble('3.3621031431120935063e-4932+0j')
ftype      = <class 'numpy.clongdouble'>
invalid    = 'invalid'
overflow   = 'overflow'
rtype      = <class 'numpy.longdouble'>
self       = <numpy._core.tests.test_numeric.TestFloatExceptions object at 0x1507e415a630>
typecode   = 'G'
underflow  = ''

../python/3.12.4/lib/python3.12/site-packages/numpy/_core/tests/test_numeric.py:965:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <numpy._core.tests.test_numeric.TestFloatExceptions object at 0x1507e415a630>, fpeerr = 'overflow', flop = <ufunc 'power'>, x = np.clongdouble('2+0j')
y = np.clongdouble('32768+0j')

    def assert_raises_fpe(self, fpeerr, flop, x, y):
        ftype = type(x)
        try:
            flop(x, y)
>           assert_(False,
                    "Type %s did not raise fpe error '%s'." % (ftype, fpeerr))
E                   AssertionError: Type <class 'numpy.clongdouble'> did not raise fpe error 'overflow'.

flop       = <ufunc 'power'>
fpeerr     = 'overflow'
ftype      = <class 'numpy.clongdouble'>
self       = <numpy._core.tests.test_numeric.TestFloatExceptions object at 0x1507e415a630>
x          = np.clongdouble('2+0j')
y          = np.clongdouble('32768+0j')

../python/3.12.4/lib/python3.12/site-packages/numpy/_core/tests/test_numeric.py:899: AssertionError
____________________________________________________________________ TestRemainder.test_float_divmod_errors[g] _____________________________________________________________________

self = <numpy._core.tests.test_umath.TestRemainder object at 0x1507e11a13d0>, dtype = 'g'

    @pytest.mark.skipif(IS_WASM, reason="fp errors don't work in wasm")
    @pytest.mark.xfail(sys.platform.startswith("darwin"),
            reason="MacOS seems to not give the correct 'invalid' warning for "
                   "`fmod`.  Hopefully, others always do.")
    @pytest.mark.parametrize('dtype', np.typecodes['Float'])
    def test_float_divmod_errors(self, dtype):
        # Check valid errors raised for divmod and remainder
        fzero = np.array(0.0, dtype=dtype)
        fone = np.array(1.0, dtype=dtype)
        finf = np.array(np.inf, dtype=dtype)
        fnan = np.array(np.nan, dtype=dtype)
        # since divmod is combination of both remainder and divide
        # ops it will set both dividebyzero and invalid flags
        with np.errstate(divide='raise', invalid='ignore'):
            assert_raises(FloatingPointError, np.divmod, fone, fzero)
        with np.errstate(divide='ignore', invalid='raise'):
>           assert_raises(FloatingPointError, np.divmod, fone, fzero)

dtype      = 'g'
finf       = array(inf, dtype=float128)
fnan       = array(nan, dtype=float128)
fone       = array(1., dtype=float128)
fzero      = array(0., dtype=float128)
self       = <numpy._core.tests.test_umath.TestRemainder object at 0x1507e11a13d0>

../python/3.12.4/lib/python3.12/site-packages/numpy/_core/tests/test_umath.py:796:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../python/3.12.4/lib/python3.12/unittest/case.py:778: in assertRaises
    return context.handle('assertRaises', args, kwargs)
        args       = (<ufunc 'divmod'>, array(1., dtype=float128), array(0., dtype=float128))
        context    = None
        expected_exception = <class 'FloatingPointError'>
        kwargs     = {}
        self       = <numpy.testing._private.utils._Dummy testMethod=nop>
../python/3.12.4/lib/python3.12/unittest/case.py:237: in handle
    with self:
        args       = [array(1., dtype=float128), array(0., dtype=float128)]
        callable_obj = <ufunc 'divmod'>
        kwargs     = {}
        name       = 'assertRaises'
        self       = None
../python/3.12.4/lib/python3.12/unittest/case.py:260: in __exit__
    self._raiseFailure("{} not raised by {}".format(exc_name,
        exc_name   = 'FloatingPointError'
        exc_type   = None
        exc_value  = None
        self       = <unittest.case._AssertRaisesContext object at 0x1507b5c59a60>
        tb         = None
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <unittest.case._AssertRaisesContext object at 0x1507b5c59a60>, standardMsg = 'FloatingPointError not raised by divmod'

    def _raiseFailure(self, standardMsg):
        msg = self.test_case._formatMessage(self.msg, standardMsg)
>       raise self.test_case.failureException(msg)
E       AssertionError: FloatingPointError not raised by divmod

msg        = 'FloatingPointError not raised by divmod'
self       = <unittest.case._AssertRaisesContext object at 0x1507b5c59a60>
standardMsg = 'FloatingPointError not raised by divmod'

../python/3.12.4/lib/python3.12/unittest/case.py:200: AssertionError
________________________________________________________________ TestRemainder.test_float_remainder_errors[fmod-g] _________________________________________________________________

self = <numpy._core.tests.test_umath.TestRemainder object at 0x1507e11a31a0>, dtype = 'g', fn = <ufunc 'fmod'>

    @pytest.mark.skipif(hasattr(np.__config__, "blas_ssl2_info"),
            reason="gh-22982")
    @pytest.mark.skipif(IS_WASM, reason="fp errors don't work in wasm")
    @pytest.mark.xfail(sys.platform.startswith("darwin"),
           reason="MacOS seems to not give the correct 'invalid' warning for "
                  "`fmod`.  Hopefully, others always do.")
    @pytest.mark.parametrize('dtype', np.typecodes['Float'])
    @pytest.mark.parametrize('fn', [np.fmod, np.remainder])
    def test_float_remainder_errors(self, dtype, fn):
        fzero = np.array(0.0, dtype=dtype)
        fone = np.array(1.0, dtype=dtype)
        finf = np.array(np.inf, dtype=dtype)
        fnan = np.array(np.nan, dtype=dtype)

        # The following already contain a NaN and should not warn.
        with np.errstate(all='raise'):
>           with pytest.raises(FloatingPointError,
                    match="invalid value"):
E                   Failed: DID NOT RAISE <class 'FloatingPointError'>

dtype      = 'g'
finf       = array(inf, dtype=float128)
fn         = <ufunc 'fmod'>
fnan       = array(nan, dtype=float128)
fone       = array(1., dtype=float128)
fzero      = array(0., dtype=float128)
self       = <numpy._core.tests.test_umath.TestRemainder object at 0x1507e11a31a0>

../python/3.12.4/lib/python3.12/site-packages/numpy/_core/tests/test_umath.py:823: Failed
______________________________________________________________ TestRemainder.test_float_remainder_errors[remainder-g] ______________________________________________________________

self = <numpy._core.tests.test_umath.TestRemainder object at 0x1507e11a3560>, dtype = 'g', fn = <ufunc 'remainder'>

    @pytest.mark.skipif(hasattr(np.__config__, "blas_ssl2_info"),
            reason="gh-22982")
    @pytest.mark.skipif(IS_WASM, reason="fp errors don't work in wasm")
    @pytest.mark.xfail(sys.platform.startswith("darwin"),
           reason="MacOS seems to not give the correct 'invalid' warning for "
                  "`fmod`.  Hopefully, others always do.")
    @pytest.mark.parametrize('dtype', np.typecodes['Float'])
    @pytest.mark.parametrize('fn', [np.fmod, np.remainder])
    def test_float_remainder_errors(self, dtype, fn):
        fzero = np.array(0.0, dtype=dtype)
        fone = np.array(1.0, dtype=dtype)
        finf = np.array(np.inf, dtype=dtype)
        fnan = np.array(np.nan, dtype=dtype)

        # The following already contain a NaN and should not warn.
        with np.errstate(all='raise'):
>           with pytest.raises(FloatingPointError,
                    match="invalid value"):
E                   Failed: DID NOT RAISE <class 'FloatingPointError'>

dtype      = 'g'
finf       = array(inf, dtype=float128)
fn         = <ufunc 'remainder'>
fnan       = array(nan, dtype=float128)
fone       = array(1., dtype=float128)
fzero      = array(0., dtype=float128)
self       = <numpy._core.tests.test_umath.TestRemainder object at 0x1507e11a3560>

../python/3.12.4/lib/python3.12/site-packages/numpy/_core/tests/test_umath.py:823: Failed
________________________________________________________________________ TestSpecialFloats.test_log_values _________________________________________________________________________

self = <numpy._core.tests.test_umath.TestSpecialFloats object at 0x1507e1153620>

    @pytest.mark.skipif(IS_WASM, reason="fp errors don't work in wasm")
    def test_log_values(self):
        with np.errstate(all='ignore'):
            x = [np.nan, np.nan, np.inf, np.nan, -np.inf, np.nan]
            y = [np.nan, -np.nan, np.inf, -np.inf, 0.0, -1.0]
            y1p = [np.nan, -np.nan, np.inf, -np.inf, -1.0, -2.0]
            for dt in ['e', 'f', 'd', 'g']:
                xf = np.array(x, dtype=dt)
                yf = np.array(y, dtype=dt)
                yf1p = np.array(y1p, dtype=dt)
                assert_equal(np.log(yf), xf)
                assert_equal(np.log2(yf), xf)
                assert_equal(np.log10(yf), xf)
                assert_equal(np.log1p(yf1p), xf)

        with np.errstate(divide='raise'):
            for dt in ['e', 'f', 'd']:
                assert_raises(FloatingPointError, np.log,
                              np.array(0.0, dtype=dt))
>               assert_raises(FloatingPointError, np.log2,
                              np.array(0.0, dtype=dt))

dt         = 'e'
self       = <numpy._core.tests.test_umath.TestSpecialFloats object at 0x1507e1153620>
x          = [nan, nan, inf, nan, -inf, nan]
xf         = array([ nan,  nan,  inf,  nan, -inf,  nan], dtype=float128)
y          = [nan, nan, inf, -inf, 0.0, -1.0]
y1p        = [nan, nan, inf, -inf, -1.0, -2.0]
yf         = array([ nan,  nan,  inf, -inf,   0.,  -1.], dtype=float128)
yf1p       = array([ nan,  nan,  inf, -inf,  -1.,  -2.], dtype=float128)

../python/3.12.4/lib/python3.12/site-packages/numpy/_core/tests/test_umath.py:1463:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../python/3.12.4/lib/python3.12/unittest/case.py:778: in assertRaises
    return context.handle('assertRaises', args, kwargs)
        args       = (<ufunc 'log2'>, array(0., dtype=float16))
        context    = None
        expected_exception = <class 'FloatingPointError'>
        kwargs     = {}
        self       = <numpy.testing._private.utils._Dummy testMethod=nop>
../python/3.12.4/lib/python3.12/unittest/case.py:237: in handle
    with self:
        args       = [array(0., dtype=float16)]
        callable_obj = <ufunc 'log2'>
        kwargs     = {}
        name       = 'assertRaises'
        self       = None
../python/3.12.4/lib/python3.12/unittest/case.py:260: in __exit__
    self._raiseFailure("{} not raised by {}".format(exc_name,
        exc_name   = 'FloatingPointError'
        exc_type   = None
        exc_value  = None
        self       = <unittest.case._AssertRaisesContext object at 0x1507b581e870>
        tb         = None
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <unittest.case._AssertRaisesContext object at 0x1507b581e870>, standardMsg = 'FloatingPointError not raised by log2'

    def _raiseFailure(self, standardMsg):
        msg = self.test_case._formatMessage(self.msg, standardMsg)
>       raise self.test_case.failureException(msg)
E       AssertionError: FloatingPointError not raised by log2

msg        = 'FloatingPointError not raised by log2'
self       = <unittest.case._AssertRaisesContext object at 0x1507b581e870>
standardMsg = 'FloatingPointError not raised by log2'

../python/3.12.4/lib/python3.12/unittest/case.py:200: AssertionError
_________________________________________________________________________ TestSpecialFloats.test_arcsincos _________________________________________________________________________

self = <numpy._core.tests.test_umath.TestSpecialFloats object at 0x1507e1177530>

    @pytest.mark.skipif(IS_WASM, reason="fp errors don't work in wasm")
    def test_arcsincos(self):
        with np.errstate(all='ignore'):
            in_ = [np.nan, -np.nan, np.inf, -np.inf]
            out = [np.nan, np.nan, np.nan, np.nan]
            for dt in ['e', 'f', 'd']:
                in_arr = np.array(in_, dtype=dt)
                out_arr = np.array(out, dtype=dt)
                assert_equal(np.arcsin(in_arr), out_arr)
                assert_equal(np.arccos(in_arr), out_arr)

        for callable in [np.arcsin, np.arccos]:
            for value in [np.inf, -np.inf, 2.0, -2.0]:
                for dt in ['e', 'f', 'd']:
                    with np.errstate(invalid='raise'):
>                       assert_raises(FloatingPointError, callable,
                                      np.array(value, dtype=dt))

callable   = <ufunc 'arcsin'>
dt         = 'e'
in_        = [nan, nan, inf, -inf]
in_arr     = array([ nan,  nan,  inf, -inf])
out        = [nan, nan, nan, nan]
out_arr    = array([nan, nan, nan, nan])
self       = <numpy._core.tests.test_umath.TestSpecialFloats object at 0x1507e1177530>
value      = inf

../python/3.12.4/lib/python3.12/site-packages/numpy/_core/tests/test_umath.py:1627:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../python/3.12.4/lib/python3.12/unittest/case.py:778: in assertRaises
    return context.handle('assertRaises', args, kwargs)
        args       = (<ufunc 'arcsin'>, array(inf, dtype=float16))
        context    = None
        expected_exception = <class 'FloatingPointError'>
        kwargs     = {}
        self       = <numpy.testing._private.utils._Dummy testMethod=nop>
../python/3.12.4/lib/python3.12/unittest/case.py:237: in handle
    with self:
        args       = [array(inf, dtype=float16)]
        callable_obj = <ufunc 'arcsin'>
        kwargs     = {}
        name       = 'assertRaises'
        self       = None
../python/3.12.4/lib/python3.12/unittest/case.py:260: in __exit__
    self._raiseFailure("{} not raised by {}".format(exc_name,
        exc_name   = 'FloatingPointError'
        exc_type   = None
        exc_value  = None
        self       = <unittest.case._AssertRaisesContext object at 0x1507b571f830>
        tb         = None
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <unittest.case._AssertRaisesContext object at 0x1507b571f830>, standardMsg = 'FloatingPointError not raised by arcsin'

    def _raiseFailure(self, standardMsg):
        msg = self.test_case._formatMessage(self.msg, standardMsg)
>       raise self.test_case.failureException(msg)
E       AssertionError: FloatingPointError not raised by arcsin

msg        = 'FloatingPointError not raised by arcsin'
self       = <unittest.case._AssertRaisesContext object at 0x1507b571f830>
standardMsg = 'FloatingPointError not raised by arcsin'

../python/3.12.4/lib/python3.12/unittest/case.py:200: AssertionError
___________________________________________________________________________ TestClog.test_special_values ___________________________________________________________________________

self = <numpy._core.tests.test_umath_complex.TestClog object at 0x1507e0fa1eb0>

    @platform_skip
    @pytest.mark.skipif(platform.machine() == "armv5tel", reason="See gh-413.")
    def test_special_values(self):
        xl = []
        yl = []

        # From C99 std (Sec 6.3.2)
        # XXX: check exceptions raised
        # --- raise for invalid fails.

        # clog(-0 + i0) returns -inf + i pi and raises the 'divide-by-zero'
        # floating-point exception.
        with np.errstate(divide='raise'):
            x = np.array([ncu.NZERO], dtype=complex)
            y = complex(-np.inf, np.pi)
>           assert_raises(FloatingPointError, np.log, x)

self       = <numpy._core.tests.test_umath_complex.TestClog object at 0x1507e0fa1eb0>
x          = array([-0.+0.j])
xl         = []
y          = (-inf+3.141592653589793j)
yl         = []

../python/3.12.4/lib/python3.12/site-packages/numpy/_core/tests/test_umath_complex.py:154:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../python/3.12.4/lib/python3.12/unittest/case.py:778: in assertRaises
    return context.handle('assertRaises', args, kwargs)
        args       = (<ufunc 'log'>, array([-0.+0.j]))
        context    = None
        expected_exception = <class 'FloatingPointError'>
        kwargs     = {}
        self       = <numpy.testing._private.utils._Dummy testMethod=nop>
../python/3.12.4/lib/python3.12/unittest/case.py:237: in handle
    with self:
        args       = [array([-0.+0.j])]
        callable_obj = <ufunc 'log'>
        kwargs     = {}
        name       = 'assertRaises'
        self       = None
../python/3.12.4/lib/python3.12/unittest/case.py:260: in __exit__
    self._raiseFailure("{} not raised by {}".format(exc_name,
        exc_name   = 'FloatingPointError'
        exc_type   = None
        exc_value  = None
        self       = <unittest.case._AssertRaisesContext object at 0x1507b26478c0>
        tb         = None
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <unittest.case._AssertRaisesContext object at 0x1507b26478c0>, standardMsg = 'FloatingPointError not raised by log'

    def _raiseFailure(self, standardMsg):
        msg = self.test_case._formatMessage(self.msg, standardMsg)
>       raise self.test_case.failureException(msg)
E       AssertionError: FloatingPointError not raised by log

msg        = 'FloatingPointError not raised by log'
self       = <unittest.case._AssertRaisesContext object at 0x1507b26478c0>
standardMsg = 'FloatingPointError not raised by log'

../python/3.12.4/lib/python3.12/unittest/case.py:200: AssertionError
________________________________________________________________________________ TestKind.test_int _________________________________________________________________________________

self = <numpy.f2py.tests.test_kind.TestKind object at 0x1507df6894c0>

    @pytest.mark.skipif(sys.maxsize < 2 ** 31 + 1,
                        reason="Fails for 32 bit machines")
    def test_int(self):
        """Test `int` kind_func for integers up to 10**40."""
        selectedintkind = self.module.selectedintkind

        for i in range(40):
>           assert selectedintkind(i) == selected_int_kind(
                i
            ), f"selectedintkind({i}): expected {selected_int_kind(i)!r} but got {selectedintkind(i)!r}"
E           AssertionError: selectedintkind(19): expected 16 but got -1
E           assert -1 == 16
E            +  where -1 = <fortran function selectedintkind>(19)
E            +  and   16 = selected_int_kind(19)

i          = 19
selectedintkind = <fortran function selectedintkind>
self       = <numpy.f2py.tests.test_kind.TestKind object at 0x1507df6894c0>

../python/3.12.4/lib/python3.12/site-packages/numpy/f2py/tests/test_kind.py:23: AssertionError
________________________________________________________________________________ TestKind.test_real ________________________________________________________________________________

self = <numpy.f2py.tests.test_kind.TestKind object at 0x1507df689d60>

    def test_real(self):
        """
        Test (processor-dependent) `real` kind_func for real numbers
        of up to 31 digits precision (extended/quadruple).
        """
        selectedrealkind = self.module.selectedrealkind

        for i in range(32):
>           assert selectedrealkind(i) == selected_real_kind(
                i
            ), f"selectedrealkind({i}): expected {selected_real_kind(i)!r} but got {selectedrealkind(i)!r}"
E           AssertionError: selectedrealkind(16): expected 10 but got 16
E           assert 16 == 10
E            +  where 16 = <fortran function selectedrealkind>(16)
E            +  and   10 = selected_real_kind(16)

i          = 16
selectedrealkind = <fortran function selectedrealkind>
self       = <numpy.f2py.tests.test_kind.TestKind object at 0x1507df689d60>

../python/3.12.4/lib/python3.12/site-packages/numpy/f2py/tests/test_kind.py:35: AssertionError
============================================================================= short test summary info ==============================================================================
FAILED numpy/_core/tests/test_numeric.py::TestFloatExceptions::test_floating_exceptions[G] - AssertionError: Type <class 'numpy.clongdouble'> did not raise fpe error 'overflow'.
FAILED numpy/_core/tests/test_umath.py::TestRemainder::test_float_divmod_errors[g] - AssertionError: FloatingPointError not raised by divmod
FAILED numpy/_core/tests/test_umath.py::TestRemainder::test_float_remainder_errors[fmod-g] - Failed: DID NOT RAISE <class 'FloatingPointError'>
FAILED numpy/_core/tests/test_umath.py::TestRemainder::test_float_remainder_errors[remainder-g] - Failed: DID NOT RAISE <class 'FloatingPointError'>
FAILED numpy/_core/tests/test_umath.py::TestSpecialFloats::test_log_values - AssertionError: FloatingPointError not raised by log2
FAILED numpy/_core/tests/test_umath.py::TestSpecialFloats::test_arcsincos - AssertionError: FloatingPointError not raised by arcsin
FAILED numpy/_core/tests/test_umath_complex.py::TestClog::test_special_values - AssertionError: FloatingPointError not raised by log
FAILED numpy/f2py/tests/test_kind.py::TestKind::test_int - AssertionError: selectedintkind(19): expected 16 but got -1
FAILED numpy/f2py/tests/test_kind.py::TestKind::test_real - AssertionError: selectedrealkind(16): expected 10 but got 16
9 failed, 48061 passed, 312 skipped, 2789 deselected, 58 xfailed, 5 xpassed in 256.52s (0:04:16)
628C

@r-devulap
Copy link
Member

@AgilentGCMS It is still possible that icx compiler doesn't raise all the appropriate floating point exceptions. Could you compile this with icpx and tell me output?

#include <stdio.h>
#include <math.h>
#include <immintrin.h>

int main () {
    printf("before: 0x%x\n", _mm_getcsr());
    volatile float temp = log(-10.0);
    printf("after: 0x%x\n", _mm_getcsr());
    return 0;
}

Output when using gcc, the last bit is set to 1 which means it raised an invalid FPE:

before: 0x1f80
after: 0x1f81

@r-devulap
Copy link
Member

compile with the -fp-model=strict flag enabled, obviously :)

@AgilentGCMS
Copy link
Author

@r-devulap I compiled your code with icx -fp-model=strict test_rdevulap.c and ran the executable:

$ ./a.out
before: 0x1f80
after: 0x1f81

@seberg
Copy link
Member
seberg commented Aug 9, 2024

The errors look pretty harmless overall. I would guess the math libraries are still using fast-math, so anything that calls out of NumPy into math libs is still sloppy about FPEs.
(Of course they might still require users to disable checking FPEs.)

EDIT: Ah, not sure about the longdouble versions being calls out into math libs, I suppose they may also be related to longdouble itself. I don't think the f2py thing is a problem.

@AgilentGCMS
Copy link
Author

@seberg Any reason why math libraries would still use fast-math even after I've specified -fp-model=strict?

@seberg
Copy link
Member
seberg commented Aug 9, 2024

They are dynamically linked and there is only one version of them, no? So either strict, or not.

@r-devulap
Copy link
Member

@seberg I agree, these look specific to ICPX. Most the test failures seem harmless, except I am not sure about these two:

FAILED numpy/f2py/tests/test_kind.py::TestKind::test_int - AssertionError: selectedintkind(19): expected 16 but got -1
FAILED numpy/f2py/tests/test_kind.py::TestKind::test_real - AssertionError: selectedrealkind(16): expected 10 but got 16

Do you know what these failures are?

@AgilentGCMS
Copy link
Author

They are dynamically linked and there is only one version of them, no? So either strict, or not.

So these will disappear if I link with the static library? I'll try.

@seberg
Copy link
Member
seberg commented Aug 9, 2024

Do you know what these failures are?

I forgot the exact cause but IIRC they are some wrong dtype mappings in f2py (and with that I think just the tests) and probably harmless as well, i.e.the test setup is brittle w.r.t. to new platforms.

@r-devulap
Copy link
Member

static linking or dynamic link shouldn't fix it, I think. The libraries are precompiled and shipped, changing the way you link to them won't affect their behavior.

@r-devulap
Copy link
Member

From the docs, -fp-model=strict should raise all appropriate FP exceptions.

-fp-model=strict or /fp:strict | Tells the compiler to strictly adhere to value-safe optimizations when implementing floating-point calculations and enables floating-point exception semantics.

@AgilentGCMS
Copy link
Author
AgilentGCMS commented Aug 9, 2024

static linking or dynamic link shouldn't fix it, I think. The libraries are precompiled and shipped, changing the way you link to them won't affect their behavior.

I can confirm that linking to the static library did not fix/change anything. I get the same failures, including the two you were concerned about.

FAILED numpy/_core/tests/test_numeric.py::TestFloatExceptions::test_floating_exceptions[G] - AssertionError: Type <class 'numpy.clongdouble'> did not raise fpe error 'overflow'.
FAILED numpy/_core/tests/test_umath.py::TestRemainder::test_float_divmod_errors[g] - AssertionError: FloatingPointError not raised by divmod
FAILED numpy/_core/tests/test_umath.py::TestRemainder::test_float_remainder_errors[fmod-g] - Failed: DID NOT RAISE <class 'FloatingPointError'>
FAILED numpy/_core/tests/test_umath.py::TestRemainder::test_float_remainder_errors[remainder-g] - Failed: DID NOT RAISE <class 'FloatingPointError'>
FAILED numpy/_core/tests/test_umath.py::TestSpecialFloats::test_log_values - AssertionError: FloatingPointError not raised by log2
FAILED numpy/_core/tests/test_umath.py::TestSpecialFloats::test_arcsincos - AssertionError: FloatingPointError not raised by arcsin
FAILED numpy/_core/tests/test_umath_complex.py::TestClog::test_special_values - AssertionError: FloatingPointError not raised by log
FAILED numpy/f2py/tests/test_kind.py::TestKind::test_int - AssertionError: selectedintkind(19): expected 16 but got -1
FAILED numpy/f2py/tests/test_kind.py::TestKind::test_real - AssertionError: selectedrealkind(16): expected 10 but got 16

@mattip
Copy link
Member
mattip commented Aug 11, 2024

IIRC they are some wrong dtype mappings in f2py

I seem to remember this as well.

@seberg
Copy link
Member
seberg commented Aug 11, 2024

@AgilentGCMS I am not sure about those exact test failures. However:

  • Compiler and/or math library bugs around correct (strict) FPE handling are not super uncommon. So there is also some chance that it is impossible to remove the warnings.
  • Under normal runtime, these will be warnings not errors. Also the first of these are in longdouble handling which very few users use. (Leaving those in log2 and arcsin) So you can probably get away with this by telling your users that these are harmless warnings and that they can suppress them with np.errstate. (Not suppressing warnings does come with a non-negligible overhead for small arrays.)
  • For the kind errors, see e.g. BUG: loongarch doesn't use REAL(10) #24904 or search a bit more. It might be nice to see what we have to change to make the test pass simply. (Or maybe we should finally give up on this brittle test...)

@AgilentGCMS
Copy link
Author

OK, I'm going to close this for now. Thanks for all your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants
0