8000 Crash calling import_array() in C++-Api when python runs in separate thread · Issue #21588 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

Crash calling import_array() in C++-Api when python runs in separate thread #21588

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agr 8000 ee to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Gockel92 opened this issue May 24, 2022 · 13 comments
Closed

Crash calling import_array() in C++-Api when python runs in separate thread #21588

Gockel92 opened this issue May 24, 2022 · 13 comments
Labels
00 - Bug

Comments

@Gockel92
Copy link
Gockel92 commented May 24, 2022 •
edited
Loading

Describe the issue:

I get an EXC_BAD_ACCESS Error when calling import_array() in an c++ program if Python runs in an different thread.

In detail I have written a module that uses Numpy. If I initialize Python in my main thread and import my module everything works fine. But if I initialize Python in an different thread and import the module in this thread, numpy crashes when calling import_array(). More precise the crash happens wenn PyImport_ImportModule("numpy.core._multiarray_umath") is called in _import_array() (_multiarray_api.h).

I'm using macOS Monterey 12.3.1 on an Apple Silicon (M1) processor. My Python version is 3.9.6 and the NumPy I'm using is 1.21.4. Python and Numpy are for ARM architecture. As compiler I'm using Clang 12.0.5.
The issue can also be reproduced on Mac machines with intel processors

On Windows no problem occurs.

Reproduce the code example:

#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "numpy/arrayobject.h"
#include <iostream>





static PyObject *
spam_func(PyObject *self, PyObject *args)
{
    PyObject* rhsNpArray = NULL;
    if (!PyArg_ParseTuple(args, "O!", &PyArray_Type, &rhsNpArray))
        return NULL;
    int number = PyArray_NDIM(rhsNpArray);
    return PyLong_FromSize_t(number);
}
static PyMethodDef SpamMethods[] = {
    {"func",  spam_func, METH_VARARGS,
     "Execute spam_func."},
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

static struct PyModuleDef spammodule = {
    PyModuleDef_HEAD_INIT,
    "spam",
    NULL,
    -1,       
    SpamMethods
};

PyMODINIT_FUNC
PyInit_spam(void)
{
    import_array();// here the crash happens
    
    if (PyErr_Occurred()) {
        std::cerr << "Failed to import numpy Python module(s)." << std::endl;
        return NULL;
    }
    assert(! PyErr_Occurred());
    return PyModule_Create(&spammodule);
}
void doPythonThing()
{

    if (PyImport_AppendInittab("spam", PyInit_spam) == -1) {
        fprintf(stderr, "Error: could not extend in-built modules table\n");
         exit(1);
     }

     Py_Initialize();

     PyObject *pmodule = PyImport_ImportModule("spam");
}





int
main(int argc, char *argv[])
{ 
    //calling doPythonThing without a new thread works fine
    std::thread thread(doPythonThing);
    thread.join();
    return 0; 
}

Error message:

Process 5308 launched: '/Users/robin/Desktop/build/pythonTest/spam' (arm64)
Process 5308 stopped
* thread #4, stop reason = EXC_BAD_ACCESS (code=2, address=0x16ff1bff8)
    frame #0: 0x000000018ec250cc libsystem_pthread.dylib`___chkstk_darwin + 60
libsystem_pthread.dylib`___chkstk_darwin:
->  0x18ec250cc <+60>: ldur   x11, [x11, #-0x8]
    0x18ec250d0 <+64>: mov    x10, sp
    0x18ec250d4 <+68>: cmp    x9, #0x1, lsl #12         ; =0x1000 
    0x18ec250d8 <+72>: b.lo   0x18ec250f0               ; <+96>
Target 0: (spam) stopped.

NumPy/Python version information:

1.21.4 3.9.6 (v3.9.6:db3ff76da1, Jun 28 2021, 11:14:58)
[Clang 12.0.5 (clang-1205.0.22.9)]
@seberg
Copy link
Member
seberg commented May 24, 2022

I am not sure about the subtleties that may exist if a Python interpreter is initialized in a thread itself. But I doubt that this has anything to do with NumPy here.

Maybe check: https://docs.python.org/3/c-api/init.html#non-python-created-threads I suspect you must add that thread management, but that probably it is OK to keep the Py_Initialize() where it is (but it should likely be guarded to make sure you don't call it again from another thread)

@Gockel92
Copy link
Author

Thank you for your answer. Yes, I suspected that with the GIL, but without success. I called PyGILState_Ensure() right after Py_Initialize(), but the crash is unavoidable.

What confuses me a lot about this is that the problem cannot be reproduced on Windows, Ubuntu, Fedora and Raspbian....

@seberg
Copy link
Member
seberg commented May 24, 2022

Maybe try using a newer NumPy version? I guess the problem may just be the M1 here then? Are you running in the compatibility mode?

@Gockel92
Copy link
Author

Okay this might be worth trying :)
No I'm not using the compatibility mode right now...
But I could also try to get a x64 Python and NumPy and check if this helps to run on Rosetta

I will report when I have tried

@Gockel92
Copy link
Author

Updating NumPy to 1.22.4 didn't fix the issue...

@tpham3783
Copy link
tpham3783 commented Jun 3, 2022 •
edited
Loading

I am having a similar crash, exactly at import_array().

But it is related to how python loaded hashlib (openssl) library.
The way I got around it was running the app with this command:

/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --preload /usr/lib/x86_64-linux-gnu/libssl.so ./upsNativeInterfence

stacktrace hint:

importing np
--Type <RET> for more, q to quit, c to continue without paging--

Thread 1 "upsNativeInterf" received signal SIGSEGV, Segmentation fault.
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65
65	../sysdeps/x86_64/multiarch/strlen-avx2.S: No such file or directory.
(gdb) bt
#0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65
#1  0x00007ffff7c6fb7d in PyUnicode_FromString () at /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0
#2  0x00007fff3000c31e in  () at /usr/lib/python3.8/lib-dynload/_hashlib.cpython-38-x86_64-linux-gnu.so
#3  0x00007fffd7b86207 in OPENSSL_LH_doall_arg () at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#4  0x00007fffd7b9238b in OBJ_NAME_do_all () at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#5  0x00007fffd7b7c869 in EVP_MD_do_all () at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#6  0x00007fff3000ddd4 in PyInit__hashlib () at /usr/lib/python3.8/lib-dynload/_hashlib.cpython-38-x86_64-linux-gnu.so
#7  0x00007ffff7be7006 in _PyImport_LoadDynamicModuleWithSpec () at /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0
#8  0x00007ffff7be9075 in  () at /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0
#9  0x00007ffff7c9f537 in  () at /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0

@Gockel92
Copy link
Author

I will check if this solves my issue…

@Gockel92 Gockel92 mentioned this issue Sep 7, 2022
Closed
@ronaldoussoren
Copy link

I've done some researching on this on the CPython end and this seems to be related to OpenBlas' threading support. If I run the program with OPENBLAS_NUM_THREADS set to 1 in the shell environment it does not crash, without the environment variable or with a higher number of threads I get a bus error as described in this issue.

And while writing this: The issue might be related to the stack size given the top of the stack trace. The default stack size for threads on macOS is fairly small. In CPython we explicitly set the stack size for threads we create in the threading module to a much larger value.

  * frame #0: 0x00000001aa81ce6c libsystem_pthread.dylib`___chkstk_darwin + 60

@Gockel92
Copy link
Author

After some googling I found that std::thread does not provide an option to set the stack size. QThreads have such an option. Increasing the thread size seems to fix the problem for now :). I raised the thread size to 8MB and the error was gone. Still, it's not really satisfying that this is not possible with std::threads. Does anyone know another possibility?

@krsvojte
Copy link

Can confirm that the following fixes the problem:

  • setting OPENBLAS_NUM_THREADS=1 OR
  • setting stack size at or above 540KB

After some googling I found that std::thread does not provide an option to set the stack size. QThreads have such 1an option. Increasing the thread size seems to fix the problem for now :). I raised the thread size to 8MB and the error was gone. Still, it's not really satisfying that this is not possible with std::threads. Does anyone know another possibility?

The only alternative I found was to launch create the thread via pthread_create. Another alternative is to use boost::thread. It seems that due to the design of std::thread there is no way to set stack size before creating the thread (and getting the native_handle from std::thread is too late). There are ways to set stack size via compiler or setrlimit(RLIMIT_STACK, ...), however those seem to only affect the main thread.

@TroikaTronix
Copy link
TroikaTronix commented Jan 11, 2023 •
edited
Loading

I also can confirm success on macOS Ventura 13.1 (ARM/M1) and Python 3.9.13. I created a lightweight replacement for std::thread that uses pthreads, set the stack size to 768K (more than recommended above, just to be safe) and the crash is solved. Many thanks to all above who worked through this issue and found a solution. ;-)

@albertz albertz mentioned this issue Jun 2, 2023
Open
3 tasks
@albertz
Copy link
albertz commented Jun 2, 2023

I just want to mention that I got a similar (or exactly the same?) crash involving libcrypto, Python, TensorFlow and Numpy.

It helped a lot to use LD_DEBUG=bindings to debug the issue. (I also wrote this down here.)

You also find a similar crash in pybind/pybind11#3543.

With linking TF, I got outputs like this:

2306643:     binding file /u/zeyer/.linuxbrew/Cellar/python@3.11/3.11.2_1/lib/python3.11/lib-dynload/_hashlib.cpython-311-x86_64-linux-gnu.so [0] to /u/zeyer/.local/lib/python3.11/site-packages/tensorflow/libtensorflow_framework.so.2 [0]: normal symbol `EVP_MD_size' [OPENSSL_1_1_0]

Just running python -c "import hashlib" gives me outputs like this:

   2307740:     binding file /u/zeyer/.linuxbrew/Cellar/python@3.11/3.11.2_1/lib/python3.11/lib-dynload/_hashlib.cpython-311-x86_64-linux-gnu.so [0] to /work/tools/users/zeyer/linuxbrew/lib/libcrypto.so.1.1 [0]: normal symbol `EVP_MD_size' [OPENSSL_1_1_0]

As you see, in the case with TF, it finds those OpenSSL related symbols in libtensorflow_framework.so.2, in the other case, it finds them in libcrypto.so.1.1 (as it should be).

Our solution here was to add -L/work/tools/users/zeyer/linuxbrew/lib -Wl,-rpath,/work/tools/users/zeyer/linuxbrew/lib -Wl,-no-as-needed -lcrypto to our linker flags, before we link the TF library. So we enforce that libcrypto.so.1.1 is loaded first. That solves the problem.

@seberg
Copy link
Member
seberg commented Jun 21, 2023

Sounds like all the issues mentioned here were fixed in some form and not related to NumPy directly. So closing, please open a new issue if I missed something.

@seberg seberg closed this as completed Jun 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@albertz @seberg @ronaldoussoren @krsvojte @tpham3783 @TroikaTronix @Gockel92
0