8000 Deadlock happening when using numpy/scipy with python subinterpreters (Trac #2213) · Issue #665 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

Deadlock happening when using numpy/scipy with python subinterpreters (Trac #2213) #665

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
numpy-gitbot opened this issue Oct 19, 2012 · 6 comments
Closed

Deadlock happening when using numpy/scipy with python subinterpreters (Trac #2213) #665

numpy-gitbot opened this issue Oct 19, 2012 · 6 comments
Labels
01 - Enhancement component: numpy._core Embedded Issues regarding embedded python interpreters

Comments

@numpy-gitbot
Copy link

Original ticket http://projects.scipy.org/numpy/ticket/2213 on 2012-09-12 by trac user raphael.de.feraudy, assigned to unknown.

Hello,

I've encountered a deadlock while trying to import a scipy subpackage using Python C API in my application.

I've managed to build a minimal example showing this, which I join to this ticket.

According to what I've found on the internet, I guess this is caused by the (documented) unsupported use of both PyGilState_* functions in numpy code and Subinterpreters in my code (As you will see in my example, the simple use of a new subinterpreter instead of the main default one is enough to produce the deadlock).

I use an ubuntu 12.04 Precise Pangolin 64bits distribution with standard python-numpy (1.6.1) and python-scipy (0.9.0) package , but the deadlock still happens with last revisions of git repositories.

I tried to fix this by myself, only using PyEval_* functions in a working copy, but didn't succeed.

My questions are :

  • Are you aware of this issue ?
  • Do you plan to address it in next milestone ?
  • Do you have any workaround to suggest ?
@numpy-gitbot
Copy link
Author

Attachment added by trac user raphael.de.feraudy on 2012-09-12: deadlock.tar.gz

@numpy-gitbot
Copy link
Author

@rgommers wrote on 2012-09-15

Don't think that has been discussed recently. It isn't adressed by the upcoming 1.7.0 release. You should probably bring this up on the mailing list, perhaps someone there can suggest a workaround.

@numpy-gitbot
Copy link
Author

trac user raphael.de.feraudy wrote on 2012-09-24

Replying to [comment:1 rgommers]:

Don't think that has been discussed recently. It isn't adressed by the upcoming 1.7.0 release. You should probably bring this up on the mailing list, perhaps someone there can suggest a workaround.

Thanks Ralf for your answer and advice.
I actually found a discussion related to my problem (as far as i understand), but no workaround was definitely validated there : http://comments.gmane.org/gmane.comp.python.numeric.general/46220.
I shall post a comment.

@petli1m
Copy link
petli1m commented Jan 21, 2013

I would like to confirm this problem.

We've encountered it with 'multiple interpreters mode' in uWSGI (which uses subinterpreters) with two applications reusing the same processes.

We've noticed hanging of uWSGI processes in which the numpy was imported in a subinterpreter, after the other application (also using numpy) was initialised first.

It resulted in a timeout on uWSGI and the following backtrace:

Mon Jan 21 10:45:12 2013 - *** backtrace of 31696 ***
Mon Jan 21 10:45:12 2013 - /usr/local/bin/uwsgi(uwsgi_backtrace+0x29) [0x807ff89]
Mon Jan 21 10:45:12 2013 - /usr/local/bin/uwsgi(what_i_am_doing+0x21) [0x80804e1]
Mon Jan 21 10:45:12 2013 - [0xc5c500]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_RestoreThread+0x48) [0x4c1628]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyGILState_Ensure+0x70) [0x4e51c0]
Mon Jan 21 10:45:12 2013 - /env/local/lib/python2.7/site-packages/numpy/core/umath.so(+0x1a66f) [0x5dd766f]
Mon Jan 21 10:45:12 2013 - /env/local/lib/python2.7/site-packages/numpy/core/umath.so(+0x1aa58) [0x5dd7a58]
Mon Jan 21 10:45:12 2013 - /env/local/lib/python2.7/site-packages/numpy/core/umath.so(+0x1ab38) [0x5dd7b38]
Mon Jan 21 10:45:12 2013 - /env/local/lib/python2.7/site-packages/numpy/core/umath.so(+0x2077b) [0x5ddd77b]
Mon Jan 21 10:45:12 2013 - /env/local/lib/python2.7/site-packages/numpy/core/umath.so(+0x20a1f) [0x5ddda1f]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x5c) [0x416bfc]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(_PyObject_CallFunction_SizeT+0x65) [0x419e05]
Mon Jan 21 10:45:12 2013 - /env/local/lib/python2.7/site-packages/numpy/core/multiarray.so(+0x5cb73) [0x4390b73]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0x2f713) [0x417713]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyNumber_Multiply+0x30) [0x4186b0]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2c07) [0x4c44b7]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5c87) [0x4c7537]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7f0) [0x4c7fe0]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0x5f217) [0x447217]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x5c) [0x416bfc]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0x40bd4) [0x428bd4]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x5c) [0x416bfc]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0x9c98d) [0x48498d]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0x92855) [0x47a855]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x5c) [0x416bfc]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4132) [0x4c59e2]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7f0) [0x4c7fe0]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x47bd) [0x4c606d]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7f0) [0x4c7fe0]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0x5f217) [0x447217]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x5c) [0x416bfc]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0x92b63) [0x47ab63]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0x9280e) [0x47a80e]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x5c) [0x416bfc]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4132) [0x4c59e2]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7f0) [0x4c7fe0]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalCode+0x63) [0x4c8123]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0xad) [0x4d8d4d]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0xf3864) [0x4db864]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0xf486f) [0x4dc86f]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0xf4b6c) [0x4dcb6c]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0xf5184) [0x4dd184]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x3b) [0x4dd6eb]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0xd8306) [0x4c0306]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyCFunction_Call+0x128) [0x460ea8]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x5c) [0x416bfc]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x54) [0x4c08d4]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x19fa) [0x4c32aa]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7f0) [0x4c7fe0]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalCode+0x63) [0x4c8123]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0xad) [0x4d8d4d]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0xf3864) [0x4db864]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0xf4190) [0x4dc190]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0xf486f) [0x4dc86f]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0xf4e20) [0x4dce20]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0xf5453) [0x4dd453]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x3b) [0x4dd6eb]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(+0xd8306) [0x4c0306]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyCFunction_Call+0x128) [0x460ea8]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x5c) [0x416bfc]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x54) [0x4c08d4]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x19fa) [0x4c32aa]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7f0) [0x4c7fe0]
Mon Jan 21 10:45:12 2013 - /usr/lib/libpython2.7.so.1.0(PyEval_EvalCode+0x63) [0x4c8123]
Mon Jan 21 10:45:12 2013 - *** end of backtrace ***

@pv
Copy link
Member
pv commented Jan 21, 2013

Numpy does not support working with multiple Python sub-interpreters in the same process at the moment:
http://mail.scipy.org/pipermail/numpy-discussion/2009-July/044046.html

@charris
Copy link
Member
charris commented Feb 18, 2014

Closing. This sounds like a major rewrite and there may be other solutions for computational problems of this sort. I suspect we would need support at the interpreter level.

@charris charris closed this as completed Feb 18, 2014
@ylikx ylikx mentioned this issue Jun 10, 2018
Open
@paulmueller paulmueller mentioned this issue Jun 15, 2018
Closed
@mattip mattip mentioned this issue Dec 8, 2019
Closed
@m-f-h m-f-h mentioned this issue Mar 12, 2020
Closed
@rgommers rgommers added the Embedded Issues regarding embedded python interpreters label Feb 1, 2021
@ngoldbaum ngoldbaum mentioned this issue Aug 12, 2024
Closed
@AA-Turner AA-Turner mentioned this issue May 21, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
01 - Enhancement component: numpy._core Embedded Issues regarding embedded python interpreters
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@pv @charris @rgommers @numpy-gitbot @petli1m
0