10000 numpy.ndarray.clip segfaults on big-endian dtype (Trac #1588) · Issue #2184 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

numpy.ndarray.clip segfaults on big-endian dtype (Trac #1588) #2184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
numpy-gitbot opened this issue Oct 19, 2012 · 12 comments
Closed

numpy.ndarray.clip segfaults on big-endian dtype (Trac #1588) #2184

numpy-gitbot opened this issue Oct 19, 2012 · 12 comments
Labels
00 - Bug component: numpy._core
Milestone
NumPy 1.7

Comments

@numpy-gitbot
Copy link

Original ticket http://projects.scipy.org/numpy/ticket/1588 on 2010-08-20 by trac user bdkern, assigned to unknown.

import numpy
numpy.zeros((1,), dtype='>i2').clip(min=0)

segfaults on three systems tried: Ubuntu / Python 2.5.2 / Numpy 1.1.1, Red Hat / py 2.6.4 / np 1.3.0, WinXP / py 2.5.4 / np 1.3.0. This same setup also segfaults for dtypes '>f4', '>f8', etc.
@numpy-gitbot
Copy link
Author

@fengy-research wrote on 2012-05-08

I am confirming this is still and issue with 1.6.1 with IPython, on python 2.7.3

won't seg fault:
.clip(min=0, max=1), with ipython
.clip(min=0), with python (I believe there may still be corrupted data)
will seg fault:
.clip(min=0), with ipython.

I made a gdb attaching to the ipython process and obtained a backtrace:

0x00002aaab026d5df in _update_descr_and_dimensions (subtype=0x2aaab050d040, descr=0x64, nd=2, dims=0x7fffea80d410, strides=0x0, data=0x12c98ea0,
flags=1287, obj=0x12c9d2e0) at numpy/core/src/multiarray/ctors.c:265
265 Py_INCREF(*des);
(gdb) bt
#0 0x00002aaab026d5df in _update_descr_and_dimensions (subtype=0x2aaab050d040, descr=0x64, nd=2, dims=0x7fffea80d410, strides=0x0, data=0x12c98ea0,
flags=1287, obj=0x12c9d2e0) at numpy/core/src/multiarray/ctors.c:265
#1 PyArray_NewFromDescr (subtype=0x2aaab050d040, descr=0x64, nd=2, dims=0x7fffea80d410, strides=0x0, data=0x12c98ea0, flags=1287, obj=0x12c9d2e0)
at numpy/core/src/multiarray/ctors.c:961
#2 0x00002aaab02a999d in PyArray_Newshape (self=0x12c9d2e0, newdims=, order=NPY_CORDER) at numpy/core/src/multiarray/shape.c:301
#3 0x00002aaab02af2d4 in PyArray_Ravel (a=0x0, order=NPY_CORDER) at numpy/core/src/multiarray/shape.c:845
#4 0x00002aaab02af4b9 in array_ravel (self=0x12c9d2e0, args=, kwds=) at numpy/core/src/multiarray/methods.c:2030
#5 0x000000000049b574 in call_function (f=0x12e47d50, throwflag=) at Python/ceval.c:4012
#6 PyEval_EvalFrameEx (f=0x12e47d50, throwflag=) at Python/ceval.c:2665
#7 0x000000000049d3bb in PyEval_EvalCodeEx (co=0x11f957b0, globals=, locals=, args=0x2, argcount=1,
kws=0x12e47cd8, kwcount=0, defs=0x11f93728, defcount=1, closure=0x0) at Python/ceval.c:3252
#8 0x000000000049aeb5 in call_function (f=0x12e47ae0, throwflag=) at Python/ceval.c:4108
#9 PyEval_EvalFrameEx (f=0x12e47ae0, throwflag=) at Python/ceval.c:2665
#10 0x000000000049d3bb in PyEval_EvalCodeEx (co=0x11f8f2b0, globals=, locals=, args=0x5, argcount=6,
kws=0x12e47aa8, kwcount=0, defs=0x11f9e260, defcount=2, closure=0x0) at Python/ceval.c:3252
#11 0x000000000049aeb5 in call_function (f=0x12e478b0, throwflag=) at Python/ceval.c:4108
#12 PyEval_EvalFrameEx (f=0x12e478b0, throwflag=) at Python/ceval.c:2665
#13 0x000000000049d3bb in PyEval_EvalCodeEx (co=0x11f8f430, globals=, locals=, args=0x7, argcount=6,
kws=0x12e47878, kwcount=0, defs=0x11f98548, defcount=6, closure=0x0) at Python/ceval.c:3252
#14 0x000000000049aeb5 in call_function (f=0x12e47680, throwflag=) at Python/ceval.c:4108
#15 PyEval_EvalFrameEx (f=0x12e47680, throwflag=) at Python/ceval.c:2665
#16 0x000000000049d3bb in PyEval_EvalCodeEx (co=0x11eeddb0, globals=, locals=, args=0x4, argcount=1, kws=0x0,
kwcount=0, defs=0x2b9bb36971a8, defcount=3, closure=0x0) at Python/ceval.c:3252
#17 0x00000000004fb1bf in function_call (func=0x11f8e938, arg=0x123239d0, kw=0x0) at Objects/funcobject.c:526
#18 0x00000000004186ed in PyObject_Call (func=0x11f8e938, arg=0x123239d0, kw=0x0) at Objects/abstract.c:2522
#19 0x00000000004945a6 in PyEval_CallObjectWithKeywords (func=0x11f8e938, arg=0x123239d0, kw=0x0) at Python/ceval.c:3881
#20 0x00002aaab025cedd in array_repr (self=) at numpy/core/src/multiarray/arrayobject.c:435
#21 0x000000000044d57a in PyObject_Repr (v=0x12c9d2e0) at Objects/object.c:381
#22 0x000000000049b736 in call_function (f=0x12e47460, throwflag=) at Python/ceval.c:4000
#23 PyEval_EvalFrameEx (f=0x12e47460, throwflag=) at Python/ceval.c:2665
#24 0x000000000049d3bb in PyEval_EvalCodeEx (co=0x1197e930, globals=, locals=, args=0x12e46e08, argcount=3,
kws=0x12e46e20, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252
#25 0x000000000049aeb5 in call_function (f=0x12e46c50, throwflag=) at Python/ceval.c:4108
#26 PyEval_EvalFrameEx (f=0x12e46c50, throwflag=) at Python/ceval.c:2665
8000

@numpy-gitbot
Copy link
Author

@fengy-research wrote on 2012-05-08

Another investigation running gdb with a breakpoint on PyArray_Clip shows the function runs properly.

And there is no way for PyArray_Clip to fall into the PyArray_Ravel path. Who is calling PyArray_Ravel?

@numpy-gitbot
Copy link
Author

@fengy-research wrote on 2012-05-08

More strangely,

with ipython -c 'import numpy; a=numpy.array([11214], dtype=">f4"); a.clip(min=0);'

The code always crashes.

But with the interactive ipython -pylab, the code sometimes does crash sometimes does not.

@numpy-gitbot
Copy link
Author

@charris wrote on 2012-05-11

Still present in 1.7.0.dev-bb162a2.

>>> a = numpy.zeros((2,), dtype='>i2').clip(min=0)
>>> a + a
Segmentation fault (core dumped)

I'm guessing a reference counting bug.

@numpy-gitbot
Copy link
Author

Milestone changed to NumPy 1.7 by @charris on 2012-05-11

@numpy-gitbot
Copy link
Author

trac user ocertik wrote on 2012-08-31

Here is the stacktrace as of master 5c944b9:

https://gist.github.com/3548633

for the following code:

import numpy
a = numpy.zeros((2,), dtype='>i2').clip(min=0)
print "OK 1"
x = a + a
print "OK 2"
y = str(x)
print "OK 3"
print y
print "OK 4"

It fails on the line "y = str(x)".

@numpy-gitbot
Copy link
Author

trac user ocertik wrote on 2012-08-31

When compiled using:

CFLAGS="-g -ggdb -O0" python setup.py install

the full stacktrace is:

https://gist.github.com/3548988

@numpy-gitbot
Copy link
Author

trac user ocertik wrote on 2012-08-31

It segfaults on the line:

        return descr->f->getitem(data, base);

Where descr->f is ok, but descr->f->getitem points to a memory (not NULL) that cannot be accessed. A full output from valgrind: https://gist.github.com/3549063

@numpy-gitbot
Copy link
Author

trac user ocertik wrote on 2012-08-31

The following patch seems to fix it (now there are leaks):

diff --git a/numpy/core/src/multiarray/calculation.c b/numpy/core/src/multiarray/calculation.c
index 618a871..1dd4a18 100644
--- a/numpy/core/src/multiarray/calculation.c
+++ b/numpy/core/src/multiarray/calculation.c
@@ -931,7 +931,7 @@ PyArray_Clip(PyArrayObject *self, PyObject *min, PyObject *max, PyArrayObject *o
     }
     if (max != NULL) {
         newdescr = PyArray_DescrFromObject(max, indescr);
-        Py_XDECREF(indescr);
+        //Py_XDECREF(indescr);
         if (newdescr == NULL) {
             return NULL;
         }
@@ -959,12 +959,12 @@ PyArray_Clip(PyArrayObject *self, PyObject *min, PyObject *max, PyArrayObject *o
         indescr = PyArray_DESCR(self);
         Py_INCREF(indescr);
     }
-    Py_DECREF(newdescr);
+    //Py_DECREF(newdescr);

     if (!PyDataType_ISNOTSWAPPED(indescr)) {
         PyArray_Descr *descr2;
         descr2 = PyArray_DescrNewByteorder(indescr, '=');
-        Py_DECREF(indescr);
+        //Py_DECREF(indescr);
         if (descr2 == NULL) {
             goto fail;
         }
@@ -981,7 +981,7 @@ PyArray_Clip(PyArrayObject *self, PyObject *min, PyObject *max, PyArrayObject *o
     }
     else {
         /* Side-effect of PyArray_FromAny */
-        Py_DECREF(indescr);
+        //Py_DECREF(indescr);
     }

     /*
@@ -998,14 +998,14 @@ PyArray_Clip(PyArrayObject *self, PyObject *min, PyObject *max, PyArrayObject *o
             zero = PyInt_FromLong(0);
             cmp = PyObject_RichCompareBool(min, zero, Py_LT);
             if (cmp == -1) {
-                Py_DECREF(zero);
+                //Py_DECREF(zero);
                 goto fail;
             }
             if (cmp == 1) {
                 min = zero;
             }
             else {
-                Py_DECREF(zero);
+                //Py_DECREF(zero);
                 Py_INCREF(min);
             }
         }
@@ -1017,7 +1017,7 @@ PyArray_Clip(PyArrayObject *self, PyObject *min, PyObject *max, PyArrayObject *o
         Py_INCREF(indescr);
         mina = (PyArrayObject *)PyArray_FromAny(min, indescr, 0, 0,
                                  NPY_ARRAY_DEFAULT, NULL);
-        Py_DECREF(min);
+        //Py_DECREF(min);
         if (mina == NULL) {
             goto fail;
         }
@@ -1147,11 +1147,11 @@ PyArray_Clip(PyArrayObject *self, PyObject *min, PyObject *max, PyArrayObject *o
     func(PyArray_DATA(newin), PyArray_SIZE(newin), min_data, max_data, PyArray_DATA(newout));

     /* Clean up temporary variables */
-    Py_XDECREF(mina);
-    Py_XDECREF(maxa);
-    Py_DECREF(newin);
+    //Py_XDECREF(mina);
+    //Py_XDECREF(maxa);
+    //Py_DECREF(newin);
     /* Copy back into out if out was not already a nice array. */
-    Py_DECREF(newout);
+    //Py_DECREF(newout);
     return (PyObject *)out;

  fail:

So the problem is right here somewhere.

@numpy-gitbot
Copy link
Author

trac user ocertik wrote on 2012-08-31

I am going to post the rest of the issues here:

#398

Due to trac unreliability.

@numpy-gitbot
Copy link
Author

trac user ocertik wrote on 2012-08-31

This is fixed by: #405

@numpy-gitbot
Copy link
Author

trac user ocertik wrote on 2012-08-31

PR is merged.

@khaeru khaeru mentioned this issue Apr 3, 2013
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug component: numpy._core
Projects
None yet
Milestone
NumPy 1.7
Development

No branches or pull requests
1 participant
@numpy-gitbot
0