DOC: site.cfg: warn against OpenBLAS and multiprocessing #4194
Conversation
|
I wouldn't say its a flaw in python, its even very useful for non portable code. |
|
I don't know what workaround ATLAS uses, and I'm not looking for an argument about the merits of platform-specific hacks. But I've encountered so many people who came across this exact problem that I figured it should be documented in plenty of places and this seems like one of the right places. |
|
right, but maybe add a note that ATLAS does not have this problem for users of older pythons. |
Fixes numpy#654 by not fixing it; I don't think NumPy *can* actually fix the problem as it's a design flaw in Python's multiprocessing. Listed various alternatives (Python 3.4 forkserver, single-threaded OpenBLAS, Python threading).
|
Added a note about ATLAS. |
|
thanks |
DOC: site.cfg: warn against OpenBLAS and multiprocessing
|
Sounds like a good first step. In the future we might want to try to detect if numpy is linked against OpenBLAS or OSX Accelerate and automatically switch to either the multiprocessing |
|
My last comment was about the scikit-learn / joblib projects that use multiprocessing. For numpy I don't think we can do more than just documenting the limitation for threading libraries like OpenBLAS with unsafe forking programs that use the |
|
I'm not convinced openblas can't be fixed so I'm not in favor of implementing a bunch of complicated workarounds. But I did not find time to try fixing it yet. |
|
accelerate is a bigger problem as that can't be fixed ... |
|
Here is a summary of my understanding of the problem: AFAIK OpenBLAS has two modes: one without OpenMP (to be buildable with clang / llvm) and the other one that uses OpenMP and hence the GOMP implementation when built with GCC. By trying to understand how to "fix" the problem by contributing a hackish patch to libgomp and getting it rejected with constructive comments it is now my understanding that this is not possible: the POSIX OpenMP standard and the POSIX fork / thread standards are making it impossible to have safe fork without exec in an program that links against an OpenMP runtime. Hence the need to use Python 3.4's mutiprocessing spawn or forkerserver (without numpy / OpenBLAS preloading unfortunately). One could try to convince the OpenBLAS maintainer to stick to manual thread pool handling and use For reference @juliantaylor @larsmans and I also discussed the problem on this OpenBLAS issue: OpenMathLib/OpenBLAS#294 |
Fixes #654 by not fixing it; I don't think NumPy can actually fix the problem as it's a design flaw in Python's multiprocessing. Listed various alternatives (Python 3.4 forkserver, single-threaded OpenBLAS, Python threading).
Maybe @ogrisel wants to chip in on this one as well?