8000 Segmentation fault when reversing an array enough times (Trac #466) · Issue #1064 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

Segmentation fault when reversing an array enough times (Trac #466) #1064

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
numpy-gitbot opened this issue Oct 19, 2012 · 16 comments
Closed

Segmentation fault when reversing an array enough times (Trac #466) #1064

numpy-gitbot opened this issue Oct 19, 2012 · 16 comments
Labels
00 - Bug component: numpy._core
Milestone
NumPy 1.7

Comments

@numpy-gitbot
Copy link

Original ticket http://projects.scipy.org/numpy/ticket/466 on 2007-03-08 by @FrancescAlted, assigned to @dmcooke.

Hi,

I'm getting a consistent seg fault by running this code:

import numpy

print "numpy version-->", numpy.__version__
N = 1000*1000
a = numpy.array([1,2])
for i in xrange(N):
    a = a[::-1]

and the output for my machine:

numpy version--> 1.0.2.dev3546
Violació de segment

I need 1 million of iterations for getting the segfault, but your mileage may vary.
@numpy-gitbot
Copy link
Author

@alberts wrote on 2007-03-10

Confirmed with 1.0.2.dev3572. I ran the provided code through Valgrind. After a while Valgrind crashes with:

==3912== Stack overflow in thread 1: can't grow stack to 0xBE416FEC
==3912== 
==3912== Process terminating with default action of signal 11 (SIGSEGV)
==3912==  Access not within mapped region at address 0xBE416FEC
==3912==    at 0x444CC9B: array_dealloc (arrayobject.c:1907)

Running under gdb, I get the following backtrace when it crashes:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208523072 (LWP 3998)]
0x00262c9b in array_dealloc (self=0xa8c42a0) at numpy/core/src/arrayobject.c:1907
1907    array_dealloc(PyArrayObject *self) {
(gdb) bt
#0  0x00262c9b in array_dealloc (self=0xa8c42a0) at numpy/core/src/arrayobject.c:1907
#1  0x00262d32 in array_dealloc (self=0xa8c42e0) at numpy/core/src/arrayobject.c:1933
#2  0x00262d32 in array_dealloc (self=0xa8c4320) at numpy/core/src/arrayobject.c:1933
#3  0x00262d32 in array_dealloc (self=0xa8c4360) at numpy/core/src/arrayobject.c:1933
... many more of the same ...

@numpy-gitbot
Copy link
Author

@pv wrote on 2007-04-01

Not only does [::-1] crash, but also the following loops cause a crash:

for i in xrange(N):
    a = a.T




for i in xrange(N):
    a = a[:]

I notice that on each iteration, some memory is reserved, and the size of the python process grows continuously while the loop is running.

The following does not exhibit this behavior:

for i in xrange(N):
    a = numpy.array(a[::-1], copy=True)

I think that the code makes numpy to generate a chain of views, unlimited in length, each referring to its parent. The bottom views are never freed before the topmost one, which causes memory hogging and everything to be freed only when the program ends. When freeing, DECREFing self->base calls array_dealloc recursively until stack overflows.

If this hypothesis is correct, could the problem be fixed by walking self->base pointers upwards always when creating new views?

@numpy-gitbot
Copy link
Author

@teoliphant wrote on 2007-05-11

I think the hypothesis is correct. This definitely looks like a stack-overflow. I am hesitant to walk self-base pointers upwards as there may be situations where this creates other problems.

There are other ways in Python to create stack over-flows like this, so I'm not sure a lot of effort should be spent on fixing this.

@numpy-gitbot
Copy link
Author

Milestone changed to 1.0.3 Release by @dmcooke on 2007-05-11

@numpy-gitbot
Copy link
Author

@sebhaase wrote on 2007-05-14

Hi,
Is there really no way of converting this stack-over-flow into an exception ?
I thought that in (non-numpy / standard) python there was absolutely no way of creating a segmentation fault ?

-Sebastian

@numpy-gitbot
Copy link
Author

@dmcooke wrote on 2007-05-14

I don't agree with the wontfix. I don't think walking the self-base pointers upwards will cause other problems: you'll get a very wide tree with the base as the root, instead of a long skinny one. Using the original base would make views always views of the original array, instead of the previous.

Reopening, will assign this to me. I don't think it'll be fixed for 1.0.3, though.

@numpy-gitbot
Copy link
Author

Milestone changed to 1.1 by @dmcooke on 2007-05-14

@numpy-gitbot
Copy link
Author

Milestone changed to 1.3.0 by @cournape on 2008-08-13

@numpy-gitbot
Copy link
Author

Milestone changed to Unscheduled by @cournape on 2009-03-02

@numpy-gitbot
Copy link
Author

@mwiebe wrote on 2011-03-23

There should be a clean API mechanism for creating views, since currently fixing this requires hand-modifying many places.

One fix will be to do a separation between "arrays" and "buffers," where a buffer holds elements, and arrays are always a view into a buffer. A view into another view would end up looking into the buffer, so no chain of references could exist.

@numpy-gitbot
Copy link
Author

@teoliphant wrote on 2011-03-23

I am not sure what is meant by a "clean API mechanism". I'm pretty sure that all that needs to be done is to push the reference to the original object exposing the memory instead of leaving it to the "most recent" array exposing the view.

This could have been changed and it is not that difficult of a change.

I dont' know what you mean by a "separation" of an array and a "buffer". An array is an example of a buffer. There are many examples of "buffers" and an array is a kind of "buffer"

@numpy-gitbot
Copy link
Author

@mwiebe wrote on 2011-07-19

I've made a new function PyArray_SetBase(arr, obj), which encapsulates setting the base. This is what I meant by a clean API mechanism. I've changed the many places this was being done to use PyArray_SetBase.

The NPY_ARRAY_UPDATEIFCOPY flag needed special handling, and the memmap ndarray subclass had bad assumptions about how views worked, and hopefully the way I've dealt with those is good. Anything that's not should be caught with a long testing period just like during the 1.6 release.

The changes are here:

#116

@numpy-gitbot
Copy link
Author

@charris wrote on 2011-08-13

Can this be closed?

@numpy-gitbot
Copy link
Author

@mwiebe wrote on 2011-08-13

The examples in the original report don't crash for me, I'll go ahead and close it.

@numpy-gitbot
Copy link
Author

Milestone changed to NumPy 2.0 by @mwiebe on 2011-08-13

@numpy-gitbot
Copy link
Author

Milestone changed to NumPy 1.7 by @charris on 2012-04-26

This was referenced Oct 23, 2012
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug component: numpy._core
Projects
None yet
Milestone
NumPy 1.7
Development

No branches or pull requests
1 participant
@numpy-gitbot
0