8000 SystemError on invalid ndarray / Path operation · Issue #7360 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

SystemError on invalid ndarray / Path operation #7360

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
anntzer opened this issue Feb 28, 2016 · 12 comments
Closed

SystemError on invalid ndarray / Path operation #7360

anntzer opened this issue Feb 28, 2016 · 12 comments
Labels
00 - Bug component: numpy._core
Milestone
1.21.0 release

Comments

@anntzer
Copy link
Contributor
anntzer commented Feb 28, 2016

Running

from pathlib import Path
import numpy as np
np.arange(300000) / Path("foo")

raises

TypeError: argument should be a path or str object, not <class 'int'>

During handling of the above exception, another exception occurred:

SystemError: <built-in method __new__ of type object at 0x7fb4df227fc0> returned a result with an error set

<elided: many repeats of the same>

During handling of the above exception, another exception occurred:

SystemError: <built-in method __new__ of type object at 0x7fb4df227fc0> returned a result with an error set

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/foo.py", line 3, in <module>
    np.arange(300000) / Path("foo")
  File "/usr/lib/python3.5/pathlib.py", line 879, in __rtruediv__
    return self._from_parts([key] + self._parts)
  File "/usr/lib/python3.5/pathlib.py", line 637, in _from_parts
    self = object.__new__(cls)
SystemError: <built-in method __new__ of type object at 0x7fb4df227fc0> returned a result with an error set

Note that this does NOT appear for small arrays; I haven't determined the threshold.
Crossposted as http://bugs.python.org/issue26453.
Python 3.5.1, numpy 1.10.4.
@rgommers
Copy link
Member

Looks like a pathlib issue, although the many repeating errors in a loop are odd. For small arrays it's more sensible:

>>> foo = Path('foo')
>>> x = np.arange(3)
>>> x / foo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/rgommers/Code/pythons/lib/python3.5/pathlib.py", line 879, in __rtruediv__
    return self._from_parts([key] + self._parts)
  File "/home/rgommers/Code/pythons/lib/python3.5/pathlib.py", line 638, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/home/rgommers/Code/pythons/lib/python3.5/pathlib.py", line 630, in _parse_args
    % type(a))
TypeError: argument should be a path or str object, not <class 'int'>

Compared with what happens when dividing with a string:

>>> x / 'foo'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: ufunc 'true_divide' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

So Path implements __rtruediv__ which seems to take precedence over any ndarray methods here. And then inside __rtruediv__ something goes wrong when the left-hand side of the division is a large array.

Python bug, hence close this issue?

@serhiy-storchaka
Copy link
Contributor

TypeError is not a bug. SystemError is a bug. And this looks as NumPy bug.

I can't reproduce the bug on 32-bit Python 3.4.3 with NumPy 1.8.2.

@vstinner
Copy link
Contributor

Traceback:

python
>>> from pathlib import Path; import numpy as np; np.arange(300000) / Path("foo")
python: Objects/abstract.c:2158: PyObject_Call: Assertion `!PyErr_Occurred()' failed.
(...)
(gdb) where
...
#3  0x00007ffff71222d2 in __assert_fail () from /lib64/libc.so.6
#4  0x000000000046ad09 in PyObject_Call (func=<method at remote 0x7ffff7f2fde8>, arg=(8192,), kw=0x0) at Objects/abstract.c:2158

#5  0x00000000004f2ed0 in call_maybe (o=<PosixPath at remote 0x7fffea9814b8>, nameid=0x961820 <rop_id>, format=0x6b010d "(O)") at Objects/typeobject.c:1483
#6  0x00000000005042c3 in slot_nb_true_divide (self=8192, other=<PosixPath at remote 0x7fffea9814b8>) at Objects/typeobject.c:5973
#7  0x0000000000466d09 in binary_op1 (v=8192, w=<PosixPath at remote 0x7fffea9814b8>, op_slot=240) at Objects/abstract.c:765
#8  0x0000000000466e37 in binary_op (v=8192, w=<PosixPath at remote 0x7fffea9814b8>, op_slot=240, op_name=0x698c1b "/") at Objects/abstract.c:788
#9  0x00000000004677b7 in PyNumber_TrueDivide (v=8192, w=<PosixPath at remote 0x7fffea9814b8>) at Objects/abstract.c:956
#10 0x00007fffef053ed1 in PyUFunc_OO_O (args=0xb41658, dimensions=0xb41618, steps=0xb41640, func=0x46778d <PyNumber_TrueDivide>) at numpy/core/src/umath/loops.c.src:536

#11 0x00007fffef0c324f in iterator_loop (ufunc=0xbcb0a0, op=0x7fffffffbb30, dtype=0x7fffffffb700, order=NPY_KEEPORDER, buffersize=8192, arr_prep=0x7fffffffb5f0, arr_prep_args=0x0, 
    innerloop=0x7fffef053dd7 <PyUFunc_OO_O>, innerloopdata=0x46778d <PyNumber_TrueDivide>) at numpy/core/src/umath/ufunc_object.c:1505
#12 0x00007fffef0c3f7e in execute_legacy_ufunc_loop (ufunc=0xbcb0a0, trivial_loop_ok=0, op=0x7fffffffbb30, dtypes=0x7fffffffb700, order=NPY_KEEPORDER, buffersize=8192, 
    arr_prep=0x7fffffffb5f0, arr_prep_args=0x0) at numpy/core/src/umath/ufunc_object.c:1660
#13 0x00007fffef0c6887 in PyUFunc_GenericFunction (ufunc=0xbcb0a0, args=(<numpy.ndarray at remote 0x7ffff06d3940>, <PosixPath at remote 0x7fffea9814b8>), kwds=0x0, op=0x7fffffffbb30)
    at numpy/core/src/umath/ufunc_object.c:2627
#14 0x00007fffef0ccbd2 in ufunc_generic_call (ufunc=0xbcb0a0, args=(<numpy.ndarray at remote 0x7ffff06d3940>, <PosixPath at remote 0x7fffea9814b8>), kwds=0x0)
    at numpy/core/src/umath/ufunc_object.c:4253

#15 0x000000000046adac in PyObject_Call (func=<numpy.ufunc at remote 0xbcb0a0>, arg=(<numpy.ndarray at remote 0x7ffff06d3940>, <PosixPath at remote 0x7fffea9814b8>), kw=0x0)
    at Objects/abstract.c:2170
#16 0x000000000046bed6 in PyObject_CallFunctionObjArgs (callable=<numpy.ufunc at remote 0xbcb0a0>) at Objects/abstract.c:2450
#17 0x00007fffef79f32f in PyArray_GenericBinaryFunction (m1=0x7ffff06d3940, m2=<PosixPath at remote 0x7fffea9814b8>, op=<numpy.ufunc at remote 0xbcb0a0>)
    at numpy/core/src/multiarray/number.c:321
#18 0x00007fffef7a0de0 in array_true_divide (m1=0x7ffff06d3940, m2=<PosixPath at remote 0x7fffea9814b8>) at numpy/core/src/multiarray/number.c:737

#19 0x0000000000466c67 in binary_op1 (v=<numpy.ndarray at remote 0x7ffff06d3940>, w=<PosixPath at remote 0x7fffea9814b8>, op_slot=240) at Objects/abstract.c:759
#20 0x0000000000466e37 in binary_op (v=<numpy.ndarray at remote 0x7ffff06d3940>, w=<PosixPath at remote 0x7fffea9814b8>, op_slot=240, op_name=0x698c1b "/") at Objects/abstract.c:788
#21 0x00000000004677b7 in PyNumber_TrueDivide (v=<numpy.ndarray at remote 0x7ffff06d3940>, w=<PosixPath at remote 0x7fffea9814b8>) at Objects/abstract.c:956
#22 0x00000000005b9a68 in PyEval_EvalFrameEx (f=Frame 0x7ffff06d6238, for file <stdin>, line 1, in <module> (), throwflag=0) at Python/ceval.c:1524

@vstinner
Copy link
Contributor

iterator_loop() calls PyUFunc_OO_O() which calls Path.rtruediv() which raises an error, but it doesn't handle immediatly the error: it continue to call PyUFunc_OO_O() with an exception set.

Python 3.5 helps to detect bugs in C code, it is now more strict. It's not more possible to call arbitrary Python function with an exception set. You have two main choices:

  • completly ignore the exception (PyErr_Clear()) -- usually a bad choice
  • detect that PyUFunc_OO_O() failed and stop the loop -- probably the better choice

It's up to you to decide how to handle this case.

I modified Python 3.5 to make it more strict to avoid loosing exceptions by mistake. For example, calling hasattr() can clear the current exception. So if you call hasattr() while you handle an exception, you may simply loose the current exception. It may work or not. Python 3.5 now requires to be more careful.

@seberg
Copy link
Member
seberg commented Feb 28, 2016

The threshold is going to be the iterator buffer size (8196 elements I think). Though it surprises me that it should be different if one loop or more loops are called. Maybe we don't check the error state after every loop but only at the very end. We will have to check.

@vstinner vstinner mentioned this issue Mar 9, 2016
Closed
@eric-wieser
Copy link
Member

It looks to me like this got fixed. On windows 10 with a recent master, I get only the TypeError.

@anntzer
Copy link
Contributor Author
anntzer commented Oct 10, 2020

I still get the error with Py3.8.6/numpy 1.19.2...

@eric-wieser
Copy link
Member

This sounds like the type of thing that @seberg may have fixed after 1.19 was released.

@eric-wieser eric-wieser added this to the 1.20.0 release milestone Oct 10, 2020
@eric-wieser eric-wieser reopened this Oct 10, 2020
@eric-wieser
Copy link
Member

Milestoning and reopening so we remember to test this.

@seberg
Copy link
Member
seberg commented Oct 11, 2020

Probably fixed (or at least mostly fixed) by gh-17029. To be fair, that does include a few new tests which may already cover this one (I am not certain).

@charris
Copy link
Member
charris commented Nov 23, 2020

I cannot reproduce on master with Python 3.9.

In [1]: from pathlib import Path 
   ...: import numpy as np 
   ...: np.arange(300000) / Path("foo")                                         
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-ea09b45ac9e8> in <module>
      1 from pathlib import Path
      2 import numpy as np
----> 3 np.arange(300000) / Path("foo")

TypeError: unsupported operand type(s) for /: 'int' and 'PosixPath'

@anntzer
Copy link
Contributor Author
anntzer commented Nov 23, 2020

Indeed, seems fixed on master.

@anntzer anntzer closed this as completed Nov 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug component: numpy._core
Projects
None yet
Milestone
1.21.0 release
Development

No branches or pull requests
7 participants
@seberg @charris @rgommers @vstinner @eric-wieser @anntzer @serhiy-storchaka
0