8000 segfault with extension dtype not defining nb_int · Issue #10693 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

segfault with extension dtype not defining nb_int #10693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
alexhsamuel opened this issue Mar 5, 2018 · 14 comments
Open

segfault with extension dtype not defining nb_int #10693

alexhsamuel opened this issue Mar 5, 2018 · 14 comments
Labels
00 - Bug component: numpy.dtype

Comments

@alexhsamuel
Copy link
alexhsamuel commented Mar 5, 2018 •
edited
Loading

It appears that if you create an extension type that derives PyGenericArrType and doesn't define nb_int, NumPy crashes when calling int() on an instance of your type.

This is straightforward to demonstrate with test_rational.c: replace pyrational_int with 0 on line 631 when defining nb_int. Then,

from numpy.core.test_rational import rational
r = rational(2, 3)
int(r)

This results in an infinite recursion. On OS/X, this is the bottom of the stack.

    frame #262073: 0x000000010000dfeb python`PyNumber_Long + 123
    frame #262074: 0x0000000103154c3e multiarray.cpython-36m-darwin.so`array_int + 78
    frame #262075: 0x0000000103163eba multiarray.cpython-36m-darwin.so`gentype_int + 42
    frame #262076: 0x000000010006ccd4 python`_PyLong_FromNbInt + 52
    frame #262077: 0x000000010000dfeb python`PyNumber_Long + 123
    frame #262078: 0x0000000103154c3e multiarray.cpython-36m-darwin.so`array_int + 78
    frame #262079: 0x0000000103163eba multiarray.cpython-36m-darwin.so`gentype_int + 42
    frame #262080: 0x000000010006ccd4 python`_PyLong_FromNbInt + 52
    frame #262081: 0x000000010000dfeb python`PyNumber_Long + 123
    frame #262082: 0x000000010006059b python`long_new + 139
    frame #262083: 0x00000001000b74ef python`type_call + 47
    frame #262084: 0x0000000100009df1 python`_PyObject_FastCallDict + 177
    frame #262085: 0x0000000100163348 python`call_function + 392
    frame #262086: 0x0000000100160f4c python`_PyEval_EvalFrameDefault + 47100
    frame #262087: 0x0000000100154589 python`_PyEval_EvalCodeWithName + 425
    frame #262088: 0x00000001001ac63c python`PyRun_FileExFlags + 252
    frame #262089: 0x00000001001abdee python`PyRun_SimpleFileExFlags + 366
    frame #262090: 0x00000001001d1dd6 python`Py_Main + 3718
    frame #262091: 0x0000000100001e7d python`main + 509
    frame #262092: 0x00007fff4f91c145 libdyld.dylib`start + 1
    frame #262093: 0x00007fff4f91c145 libdyld.dylib`start + 1

If the type doesn't subclass PyGenericArrType, this does not appear to occur.

Similar failure and similar stack trace on Linux/GCC/gdb.

Tested on 1.14, 1.13, 1.12 (the latter has a different stack trace) with Python 3.
@eric-wieser
Copy link
Member

Yep, looks like generic.__int__ defers to ndarray.__int__, and array.__int__ defers to generic.__int__...

This is like the __str__ problem all over again (cc @ahaldane)

@eric-wieser
Copy link
Member
eric-wieser commented Mar 5, 2018 •
edited
Loading

Did this happen in 1.13? It's possible I introduced this in #10042, or in the commit described here: #10042 (comment)

@alexhsamuel
Copy link
Author

Just tried 1.13.3; same problem. (I used a hacked up test_rational.c.src from head, however; has this changed recently?)

@eric-wieser
Copy link
Member
eric-wieser commented Mar 5, 2018 •
edited
Loading

1.12? The Pr above was 1.14. but the comment within it references a 1.13 change

@alexhsamuel
Copy link
Author

Also crashes with 1.12.1. Again, same extension module source though.

@alexhsamuel
Copy link
Author

Different stack trace on 1.12 though: only gentype_int and array_int. In later versions it bounces between these and PyNumber_Long, _PyLong_FromNb_Int.

@eric-wieser
Copy link
Member

Python 2 or 3?

@alexhsamuel
Copy link
Author

Python 3.

@alexhsamuel
Copy link
Author

Also likewise for nb_float / float().

A simple workaround is to point these two at functions that unconditionally raise TypeError.

@eric-wieser
Copy link
Member

Perhaps gentype_int should not exist at all?

@eric-wieser
Copy link
Member
eric-wieser commented Mar 19, 2018 •
edited
Loading

Ok, so here's the actual path:

g = np.generic(...); g.__int__ -> arr = np.ndarray(g); arr.__int__ -> p = arr.item(); p.__int__

Normally, p is a python scalar, but for rational it returns the numpy scalar type, which is just a copy of g.

I'm of the opinion that every internal use of .item() is an error within numpy, and that eliminating those uses is the way to fix these bugs.

@eric-wieser
Copy link
Member

Attempting to fix this opens the can of worms that is #9972

@mattip
Copy link
Member
mattip commented Aug 11, 2018

we should revisit this class of bugs once PR #10915 lands and we can refactor the code that set slots after calling PyType_Ready

@mattip
Copy link
Member
mattip commented Feb 20, 2019

Setting PyRational_Type.tp_as_number->nb_int = 0; just before PyType_Ready(&PyRational_Type) in `_rational_tests.c.src still crashes the interpreter.

from numpy.core._rational_tests import rational
r = rational(2, 3)
int(r)

results in the code path in Eric's comment, nothing changed after #10915

@moble moble mentioned this issue Jun 26, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug component: numpy.dtype
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@eric-wieser @mattip @alexhsamuel
0