8000 numpy.dot crash with numpy.float32 input · Issue #4007 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content

numpy.dot crash with numpy.float32 input #4007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
piskvorky opened this issue Nov 4, 2013 · 155 comments
Closed

numpy.dot crash with numpy.float32 input #4007

piskvorky opened this issue Nov 4, 2013 · 155 comments

Comments

@piskvorky
Copy link

A user of gensim @fbkarsdorp reported crash (segfault) with NumPy: piskvorky/gensim#131

The crash seems to have nothing to do with gensim, so I'm transferring the issue here. It happens in dot of matrix*vector in single precision, on his OS X Maverick.

@juliantaylor
Copy link
Contributor

most likely it is a crash in the used blas library.
which one is used?
the information can be obtained with

np.__config__.show()

if its openblas, please make sure it is the latest version from git, anything older tends to crash a lot.

@fbkarsdorp
Copy link

It is vecLib that comes with OS X 10.9.

@pv
8000
Copy link
Member
pv commented Nov 4, 2013

@fbkarsdorp: and your Numpy version is?

@fbkarsdorp
Copy link

1.9.0.dev-a3e8c12

@juliantaylor
Copy link
Contributor

does it also crash with numpy 1.7 and 1.8 (make sure to also use veclib)?
can you try a different blas library like atlas, netlib blas or openblas?

please also provide a backtrace with gdb:

gdb --args python yourtestcase.py
run
set logging on logfile.txt
backtrace full
disassemble
info all-registers

@charris
Copy link
Member
charris commented Nov 4, 2013

OS X 10.9 was released October 22 and it could very well have library/xcode 5.0.1 problems. @fbkarsdorp What compiler was used for the numpy installation? Was xcode updated also?

@fbkarsdorp
Copy link

Yes, I had xcode 5.0.1 installed and I belief I used the following compiler:

Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
Target: x86_64-apple-darwin13.0.0
Thread model: posix

@fbkarsdorp
Copy link

I ran the following testcase:

import numpy
mean = numpy.random.rand(200).astype(numpy.float32)
syn0norm = numpy.random.rand(10000, 200).astype(numpy.float32)
numpy.dot(syn0norm, mean)

And this is the log:

[@pv/EDIT long traceback moved to a gist: https://gist.github.com/pv/7307192 ]

@juliantaylor
Copy link
Contributor

mac gdb stops at the beginning of each run, you need to type continue until you get a SIGSEV

@w4nderlust
Copy link

Just want to confirm the issue. I am in the exact same situation of fbkarsdorp, with the same configuration and versions (xcode, compiler, python and numpy) and i have the exactly same error.

@pv
Copy link
Member
pv commented Nov 25, 2013

Please try to obtain a gdb traceback of the SIGSEGV.

@w4nderlust
Copy link

On mavericks there's no gdb, there's lldb.
I downloaded gdb with homebrew, but i get this:
gdb --args python test.py
GNU gdb (GDB) 7.6.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin13.0.1".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
"/Library/Frameworks/Python.framework/Versions/2.7/bin/python": not in executable format: File format not recognized

@andytwigg
Copy link

I'm having the same issue. Anyone know if there a workaround / fix ?

os x 10.9
Python 2.7.6 |Anaconda 1.7.0 (x86_64)| (default, Nov 11 2013, 10:49:09)
numpy 1.7.1

@seberg
Copy link
Member
seberg commented Dec 11, 2013

@andytwigg, can you try to run gdb (juliantaylor did some explenation on how further up)? I doubt this is an issue in numpy, but maybe a debugger run can tell us more.

Not sure about a workaround, the example seems to be matrix-vector product, numpy probably calls gemv for that, maybe if you make the second a matrix, too, and play around with transposing (playing around with transposing can only work in numpy >=1.8.)

@charris
Copy link
Member
charris commented Feb 15, 2014

Can folks having this problem check if it persists with the latest xcode? I think xcode is at 5.0.2.

@w4nderlust
Copy link

Will try it tomorrow and let you know. I'm sorry i can't give you a debugger dump, but it's all llvm on 10.9 and i don't really know how does the debugger here works.
Piero

Il giorno 16/feb/2014, alle ore 00:01, Charles Harris notifications@github.com ha scritto:

Can folks having this problem check if it persists with the latest xcode? I think xcode is at 5.0.2.


Reply to this email directly or view it on GitHub.

@w4nderlust
Copy link

I tried with the same setting that was giving me the error (using gensim word2vec model, with the numpy.dot() call on line 435 here https://github.com/piskvorky/gensim/blob/develop/gensim/models/word2vec.py ).
i still get the Segmentation Fault 11 error:

Python 2.7.6 (v2.7.6:3a1db0d2747e, Nov 10 2013, 00:42:54)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import numpy
numpy.version
'1.9.0.dev-Unknown'
from gensim import corpora, models, similarities
from gensim.models import Word2Vec
import logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
model = Word2Vec.load_word2vec_format('/Users/w4nderlust/Sviluppo/word2vec/vectors.bin', binary=True)
2014-02-16 18:31:46,026 : INFO : loading projection weights from /Users/w4nderlust/Sviluppo/word2vec/vectors.bin
2014-02-16 18:31:46,867 : INFO : loaded (71291, 200) matrix from /Users/w4nderlust/Sviluppo/word2vec/vectors.bin
2014-02-16 18:31:46,867 : INFO : precomputing L2-norms of word weight vectors
model.most_similar(positive=['apple'], negative=['computer'])
Segmentation fault: 11

if you need it i can give you the vectors.bin file to reproduce this, but you can find it on word2vec website (or you can find a short description of how to obtain it from some data).
Piero

Il giorno 16/feb/2014, alle ore 00:43, w4nderlust w4nderlust@gmail.com ha scritto:

Will try it tomorrow and let you know. I'm sorry i can't give you a debugger dump, but it's all llvm on 10.9 and i don't really know how does the debugger here works.
Piero

Il giorno 16/feb/2014, alle ore 00:01, Charles Harris notifications@github.com ha scritto:

Can folks having this problem check if it persists with the latest xcode? I think xcode is at 5.0.2.


Reply to this email directly or view it on GitHub.

@omar-florez
Copy link

Having also the same problem with:
numpy: 1.6.2
OS: OS X 10.9
python: 2.7.5
BLAS: veclib
gcc: GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)
cython: /Library/Python/2.7/site-packages/Cython-0.20.1-py2.7-macosx-10.9-intel.egg/cython.py

Error:
Crashed Thread: 0 Dispatch queue: com.apple.root.default-priority

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: EXC_I386_GPFLT

Thread 0 Crashed:: Dispatch queue: com.apple.root.default-priority
0 libBLAS.dylib 0x00007fff8b508262 APL_sgemvT_unitStride_AxAligned_AVX256 + 467
1 libdispatch.dylib 0x00007fff865d38f8 _dispatch_client_callout2 + 8
2 libdispatch.dylib 0x00007fff865d3849 _dispatch_apply_invoke + 87
3 libdispatch.dylib 0x00007fff865d377a dispatch_apply_f + 603
4 libBLAS.dylib 0x00007fff8b50a324 APL_sgemvTrans_AVX + 1026
5 libBLAS.dylib 0x00007fff8b3f4332 cblas_sgemv + 590
6 _dotblas.so 0x0000000107897508 0x107895000 + 9480
7 org.python.python 0x000000010715f14d PyEval_EvalFrameEx + 8080
8 org.python.python 0x000000010715d093 PyEval_EvalCodeEx + 1641
9 org.python.python 0x00000001071638c8 0x1070dd000 + 551112
10 org.python.python 0x00000001071604d4 PyEval_EvalFrameEx + 13079
11 org.python.python 0x0000000107163864 0x1070dd000 + 551012

Binary images:
Binary Images:
0x1018f0000 - 0x1018f0fff org.python.python (2.7.5 - 2.7.5) <4E975338-99DE-3927-870D-8E0651E406D1> /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
0x1018fa000 - 0x1019e4fff org.python.python (2.7.5 - 2.7.5) <53AC42BF-A913-3995-9518-3FC2F21C7119> /System/Library/Frameworks/Python.framework/Versions/2.7/Python
0x101be8000 - 0x101be9fff termios.so (76) /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/termios.so
...
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/fft/fftpack_lite.so
0x10222e000 - 0x102231fff strop.so (76) <935E433D-305A-303F-A01A-80DEE79CD72C> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/strop.so
0x102337000 - 0x102361ff7 mtrand.so (35) /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/random/mtrand.so
0x10239c000 - 0x1023aafff _ctypes.so (76) <421053C0-2110-3BCA-AE7C-25DF92BEA34A> /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ctypes.so
0x1023b6000 - 0x10246eff7 _csr.so (35) <18CC9ADF-2827-3F17-91BD-FDE1671FE212> /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/sparse/sparsetools/_csr.so

@ogrisel
Copy link
Contributor
ogrisel commented Apr 4, 2014

I can reproduce on OSX 10.9 with numpy 1.8.1 (the wheel package from pypi) on Python 3.4 from python.org.

>>> np.show_config()
blas_opt_info:
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3)]
lapack_mkl_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
lapack_opt_info:
    extra_compile_args = ['-msse3']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3)]
atlas_blas_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
atlas_blas_threads_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE
atlas_threads_info:
  NOT AVAILABLE
atlas_info:
  NOT AVAILABLE

Here is the backtrace from the OSX crash tool:

Process:         Python [29493]
Path:            /Library/Frameworks/Python.framework/Versions/3.4/Resources/Python.app/Contents/MacOS/Python
Identifier:      Python
Version:         3.4.0 (3.4.0)
Code Type:       X86-64 (Native)
Parent Process:  bash [29378]
Responsible:     iTerm [1616]
User ID:         501

Date/Time:       2014-04-04 16:22:28.935 +0200
OS Version:      Mac OS X 10.9.2 (13C64)
Report Version:  11
Anonymous UUID:  9892C1E7-EDC2-5103-F1BA-21BC3BC1DDEC

Sleep/Wake UUID: AE944E6E-F5A3-43E1-8E32-56A202072CD6

Crashed Thread:  0  Dispatch queue: com.apple.root.default-priority

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: EXC_I386_GPFLT

Thread 0 Crashed:: Dispatch queue: com.apple.root.default-priority
0   libBLAS.dylib                   0x00007fff8ba20262 APL_sgemvT_unitStride_AxAligned_AVX256 + 467
1   libdispatch.dylib               0x00007fff88d1c8f8 _dispatch_client_callout2 + 8
2   libdispatch.dylib               0x00007fff88d1c849 _dispatch_apply_invoke + 87
3   libdispatch.dylib               0x00007fff88d1c77a dispatch_apply_f + 603
4   libBLAS.dylib                   0x00007fff8ba22324 APL_sgemvTrans_AVX + 1026
5   libBLAS.dylib                   0x00007fff8b90c332 cblas_sgemv + 590
6   _dotblas.so                     0x00000001031f42fd dotblas_matrixproduct + 5789
7   org.python.python               0x00000001000e50eb PyEval_EvalFrameEx + 31291
8   org.python.python               0x00000001000e631d PyEval_EvalCodeEx + 2349
9   org.python.python               0x00000001000e63df PyEval_EvalCode + 63
10  org.python.python               0x00000001000daafe builtin_exec + 206
11  org.python.python               0x00000001000e50eb PyEval_EvalFrameEx + 31291
12  org.python.python               0x00000001000e5051 PyEval_EvalFrameEx + 31137
13  org.python.python               0x00000001000e631d PyEval_EvalCodeEx + 2349
14  org.python.python               0x00000001000e4eb7 PyEval_EvalFrameEx + 30727
15  org.python.python               0x00000001000e631d PyEval_EvalCodeEx + 2349
16  org.python.python               0x00000001000e4eb7 PyEval_EvalFrameEx + 30727
17  org.python.python               0x00000001000e631d PyEval_EvalCodeEx + 2349
18  org.python.python               0x00000001000e4eb7 PyEval_EvalFrameEx + 30727
19  org.python.python               0x00000001000e631d PyEval_EvalCodeEx + 2349
20  org.python.python               0x00000001000e4eb7 PyEval_EvalFrameEx + 30727
21  org.python.python               0x00000001000e5051 PyEval_EvalFrameEx + 31137
22  org.python.python               0x00000001000e631d PyEval_EvalCodeEx + 2349
23  org.python.python               0x000000010003f4ca function_call + 186
24  org.python.python               0x000000010000d848 PyObject_Call + 104
25  org.python.python               0x00000001000e0cc3 PyEval_EvalFrameEx + 13843
26  org.python.python               0x00000001000e631d PyEval_EvalCodeEx + 2349
27  org.python.python               0x00000001000e4eb7 PyEval_EvalFrameEx + 30727
28  org.python.python               0x00000001000e631d PyEval_EvalCodeEx + 2349
29  org.python.python               0x00000001000e63df PyEval_EvalCode + 63
30  org.python.python               0x00000001001106be PyRun_FileExFlags + 206
31  org.python.python               0x0000000100110a6d PyRun_SimpleFileExFlags + 717
32  org.python.python               0x0000000100127f6e Py_Main + 3262
33  org.python.python               0x0000000100000e32 0x100000000 + 3634
34  org.python.python               0x0000000100000c84 0x100000000 + 3204

Thread 1:
0   libsystem_kernel.dylib          0x00007fff9377e716 __psynch_cvwait + 10
1   libsystem_pthread.dylib         0x00007fff8971bc3b _pthread_cond_wait + 727
2   org.python.python               0x0000000100124f6b PyThread_acquire_lock_timed + 475
3   org.python.python               0x000000010012add8 acquire_timed + 168
4   org.python.python               0x000000010012b119 lock_PyThread_acquire_lock + 201
5   org.python.python               0x00000001000e50eb PyEval_EvalFrameEx + 31291
6   org.python.python               0x00000001000e631d PyEval_EvalCodeEx + 2349
7   org.python.python               0x00000001000e4eb7 PyEval_EvalFrameEx + 30727
8   org.python.python               0x00000001000e631d PyEval_EvalCodeEx + 2349
9   org.python.python               0x00000001000e4eb7 PyEval_EvalFrameEx + 30727
10  org.python.python               0x00000001000e631d PyEval_EvalCodeEx + 2349
11  org.python.python               0x000000010003f4ca function_call + 186
12  org.python.python               0x000000010000d848 PyObject_Call + 104
13  org.python.python               0x00000001000e0cc3 PyEval_EvalFrameEx + 13843
14  org.python.python               0x00000001000e631d PyEval_EvalCodeEx + 2349
15  org.python.python               0x00000001000e4eb7 PyEval_EvalFrameEx + 30727
16  org.python.python   
8000
            0x00000001000e5051 PyEval_EvalFrameEx + 31137
17  org.python.python               0x00000001000e5051 PyEval_EvalFrameEx + 31137
18  org.python.python               0x00000001000e631d PyEval_EvalCodeEx + 2349
19  org.python.python               0x000000010003f4ca function_call + 186
20  org.python.python               0x000000010000d848 PyObject_Call + 104
21  org.python.python               0x000000010002922c method_call + 140
22  org.python.python               0x000000010000d848 PyObject_Call + 104
23  org.python.python               0x00000001000dba57 PyEval_CallObjectWithKeywords + 87
24  org.python.python               0x000000010012aa67 t_bootstrap + 71
25  libsystem_pthread.dylib         0x00007fff89719899 _pthread_body + 138
26  libsystem_pthread.dylib         0x00007fff8971972a _pthread_start + 137
27  libsystem_pthread.dylib         0x00007fff8971dfc9 thread_start + 13

Thread 2:: Dispatch queue: com.apple.root.default-priority
0   libBLAS.dylib                   0x00007fff8ba20262 APL_sgemvT_unitStride_AxAligned_AVX256 + 467
1   libdispatch.dylib               0x00007fff88d1c8f8 _dispatch_client_callout2 + 8
2   libdispatch.dylib               0x00007fff88d1c849 _dispatch_apply_invoke + 87
3   libdispatch.dylib               0x00007fff88d152ad _dispatch_client_callout + 8
4   libdispatch.dylib               0x00007fff88d1709e _dispatch_root_queue_drain + 326
5   libdispatch.dylib               0x00007fff88d18193 _dispatch_worker_thread2 + 40
6   libsystem_pthread.dylib         0x00007fff8971aef8 _pthread_wqthread + 314
7   libsystem_pthread.dylib         0x00007fff8971dfb9 start_wqthread + 13

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000105343320  rbx: 0x0000000105343640  rcx: 0x0000000000000000  rdx: 0x0000000105343960
  rdi: 0x0000000105343000  rsi: 0x0000000105343c80  rbp: 0x00007fff5bffcbe0  rsp: 0x00007fff5bffcb20
   r8: 0x0000000102957a00   r9: 0x00000000000000c1  r10: 0x0000000000000000  r11: 0x0000000000000320
  r12: 0x0000000103b27f70  r13: 0x0000000105343fa0  r14: 0x00000000000000c8  r15: 0x00000000000000c8
  rip: 0x00007fff8ba20262  rfl: 0x0000000000010246  cr2: 0x0000000106001000

Logical CPU:     2
Error Code:      0x00000000
Trap Number:     13


Binary Images:
       0x100000000 -        0x100000ff7 +org.python.python (3.4.0 - 3.4.0) <F02CACEE-4574-9E75-28AB-1B2C6B8882DD> /Library/Frameworks/Python.framework/Versions/3.4/Resources/Python.app/Contents/MacOS/Python
       0x100003000 -        0x1001f1fe7 +org.python.python (3.4.0, [c] 2004-2014 Python Software Foundation. - 3.4.0) <B60B5CD1-2D77-F350-08E5-26E9EE858908> /Library/Frameworks/Python.framework/Versions/3.4/Python
       0x1003f4000 -        0x1003f6fff +time.so (???) <1B89A816-9AEF-26CF-41DE-A1C567058F58> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/time.so
       0x1020f1000 -        0x1020f2ff7 +_heapq.so (???) <1E22A4D4-34DC-532F-A1CE-973C8AF04C75> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_heapq.so
       0x1020f6000 -        0x1020f6fff +grp.so (???) <A62BF19B-F78A-E42E-9FF3-D4F9A9C26F6D> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/grp.so
       0x1020f9000 -        0x1020faff7 +_bz2.so (???) <10B5541A-8E90-03D4-EF2C-91054445D1B6> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_bz2.so
       0x102321000 -        0x102326ff7 +_struct.so (???) <83D2D343-5067-35E4-8654-1FA93FF9F80F> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_struct.so
       0x10236e000 -        0x102371fe7 +binascii.so (???) <0F66D6E3-8EEE-CBCC-8DDC-EEE2F5FEC86D> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/binascii.so
       0x102375000 -        0x102379fff +zlib.so (???) <B892B7E5-7F85-AEFF-79AD-EDBC996EC719> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/zlib.so
       0x10237e000 -        0x1023aafff +_lzma.so (???) <2B636EE7-0053-99E2-1518-0541B2AD5A0F> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_lzma.so
       0x1023b4000 -        0x1023b5ff7 +_posixsubprocess.so (???) <60C59188-3D87-DA3E-E310-5BF786E9729C> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_posixsubprocess.so
       0x1023b8000 -        0x1023bbfff +select.so (???) <C0EFBC14-D946-CCBB-C22F-D9D7B9AB7147> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/select.so
       0x1023c1000 -        0x1023c6ff7 +math.so (???) <5F7EBB80-BE89-E5C4-4204-A7BAE0C90008> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/math.so
       0x10240d000 -        0x102419ff7 +parser.so (???) <0BF9789F-6407-5427-19C2-917F9660CA6E> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/parser.so
       0x10245f000 -        0x10246cfef +_datetime.so (???) <608C2C4F-C80C-F497-47DD-BEB54F288BFE> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_datetime.so
       0x102477000 -        0x1024acff7 +pyexpat.so (???) <89B416C3-0E33-36DA-FBFB-DEBC82C11FA0> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/pyexpat.so
       0x10257e000 -        0x10257efff +_opcode.so (???) <4FE41F83-0701-4FDD-E326-E12222491709> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_opcode.so
       0x102601000 -        0x102603ff7 +_hashlib.so (???) <4ED655B0-15D9-354C-B9C8-3B3EDD5D7DF1> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_hashlib.so
       0x102607000 -        0x102608ff7 +_random.so (???) <C50789F9-A2CE-78E7-9BDB-9C36D532DE1A> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_random.so
       0x10264b000 -        0x10264cfff +resource.so (???) <E647AA4D-F291-46E3-8EAA-902A76261D49> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/resource.so
       0x10264f000 -        0x102650fff +termios.so (???) <DA36EC00-BAF3-EE4C-40CF-58F8DEA172A6> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/termios.so
       0x102655000 -        0x102656fff +fcntl.so (???) <B8651930-C564-F52D-2128-41071DC6EF3C> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/fcntl.so
       0x1027db000 -        0x1027e5ff7 +_socket.so (???) <A5EC9DEB-ED80-600B-20D4-88A4BBA81D40> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_socket.so
       0x1027f1000 -        0x1027f7ff7 +_json.so (???) <623E499F-4833-4024-224F-54D597A1A658> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_json.so
       0x103100000 -        0x1031aefef +_sqlite3.so (???) <D072CE40-3F4D-9F80-68A4-CCD318A44725> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_sqlite3.so
       0x1031cc000 -        0x1031deff7 +_pickle.so (???) <6D8B47A4-D0DB-AFB2-B770-78134AFEEBAA> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_pickle.so
       0x1031ea000 -        0x1031ecff7 +_lsprof.so (???) <09BC9DF5-2083-2067-D30E-7777E0C6D7B8> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_lsprof.so
       0x1031f0000 -        0x1031f4fff +_dotblas.so (???) <196850EC-5025-38E7-8691-6C916BAE2DFA> /Users/USER/*/_dotblas.so
       0x1031f8000 -        0x1031fcff7 +_compiled_base.so (???) <5CC017E9-99A7-3AE8-8509-991B8702B1B7> /Users/USER/*/_compiled_base.so
       0x1034c0000 -        0x1034d5ff7 +_ctypes.so (???) <ECB49FED-18EA-3F04-1D8C-3A3A2DD6D226> /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib-dynload/_ctypes.so
       0x103a69000 -        0x103aa2fff +readline.so (???) <CFA73F54-2D9B-312D-AB06-39765DD34F62> /Users/USER/*/readline.so
       0x103c5e000 -        0x103da6fef +multiarray.so (???) <5E516009-5DBE-3508-AEAD-A52283676828> /Users/USER/*/multiarray.so
       0x103e8c000 -        0x103ee4fef +umath.so (???) <ECE6DDFD-B1B6-3813-A001-09960A6893E2> /Users/USER/*/umath.so
       0x103f50000 -        0x103f73ff7 +scalarmath.so (???) <A14BCE89-7D60-3168-AF3E-D10588EB5106> /Users/USER/*/scalarmath.so
       0x104043000 -        0x104047ff7 +lapack_lite.so (???) <6301DFDF-357F-3C64-B19C-257B5EC4242F> /Users/USER/*/lapack_lite.so
       0x10404b000 -        0x104060ff7 +_umath_linalg.so (???) <11A7436F-E877-365E-83F3-C27C14A408DE> /Users/USER/*/_umath_linalg.so
       0x1040ec000 -        0x1040f4ff7 +fftpack_lite.so (???) <8DF40BB0-A9B2-3F76-B9CD-636E9D435013> /Users/USER/*/fftpack_lite.so
       0x1041f8000 -        0x104243ff7 +mtrand.so (???) <126CE027-6BAA-3168-BFA0-4CE56BCF5123> /Users/USER/*/mtrand.so
    0x7fff6d38b000 -     0x7fff6d3be817  dyld (239.4) <2B17750C-ED1B-3060-B64E-21897D08B28B> /usr/lib/dyld
    0x7fff877b3000 -     0x7fff877bbfff  libsystem_dnssd.dylib (522.90.2) <A0B7CF19-D9F2-33D4-8107-A62184C9066E> /usr/lib/system/libsystem_dnssd.dylib
    0x7fff877bc000 -     0x7fff877cdff7  libz.1.dylib (53) <42E0C8C6-CA38-3CA4-8619-D24ED5DD492E> /usr/lib/libz.1.dylib
    0x7fff878b1000 -     0x7fff878baff3  libsystem_notify.dylib (121) <52571EC3-6894-37E4-946E-064B021ED44E> /usr/lib/system/libsystem_notify.dylib
    0x7fff878bb000 -     0x7fff8790dfff  libc++.1.dylib (120) <4F68DFC5-2077-39A8-A449-CAC5FDEE7BDE> /usr/lib/libc++.1.dylib
    0x7fff87ea0000 -     0x7fff87ec9ff7  libc++abi.dylib (49.1) <21A807D3-6732-3455-B77F-743E9F916DF0> /usr/lib/libc++abi.dylib
    0x7fff88d11000 -     0x7fff88d13ff7  libquarantine.dylib (71) <7A1A2BCB-C03D-3A25-BFA4-3E569B2D2C38> /usr/lib/system/libquarantine.dylib
    0x7fff88d14000 -     0x7fff88d2efff  libdispatch.dylib (339.90.1) <F3CBFE1B-FCE8-3F33-A53D-9092AB382DBB> /usr/lib/system/libdispatch.dylib
    0x7fff88e60000 -     0x7fff88e67fff  libcompiler_rt.dylib (35) <4CD916B2-1B17-362A-B403-EF24A1DAC141> /usr/lib/system/libcompiler_rt.dylib
    0x7fff895d9000 -     0x7fff895dafff  com.apple.TrustEvaluationAgent (2.0 - 25) <334A82F4-4AE4-3719-A511-86D0B0723E2B> /System/Library/PrivateFrameworks/TrustEvaluationAgent.framework/Versions/A/TrustEvaluationAgent
    0x7fff89718000 -     0x7fff8971fff7  libsystem_pthread.dylib (53.1.4) <AB498556-B555-310E-9041-F67EC9E00E2C> /usr/lib/system/libsystem_pthread.dylib
    0x7fff89729000 -     0x7fff89758fd2  libsystem_m.dylib (3047.16) <B7F0E2E4-2777-33FC-A787-D6430B630D54> /usr/lib/system/libsystem_m.dylib
    0x7fff8adb2000 -     0x7fff8adb3ff7  libDiagnosticMessagesClient.dylib (100) <4CDB0F7B-C0AF-3424-BC39-495696F0DB1E> /usr/lib/libDiagnosticMessagesClient.dylib
    0x7fff8b099000 -     0x7fff8b47affe  libLAPACK.dylib (1094.5) <7E7A9B8D-1638-3914-BAE0-663B69865986> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
    0x7fff8b47b000 -     0x7fff8b482ff3  libcopyfile.dylib (103) <5A881779-D0D6-3029-B371-E3021C2DDA5E> /usr/lib/system/libcopyfile.dylib
    0x7fff8b483000 -     0x7fff8b483fff  com.apple.Accelerate (1.9 - Accelerate 1.9) <509BB27A-AE62-366D-86D8-0B06D217CF56> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
    0x7fff8b811000 -     0x7fff8b811fff  com.apple.Accelerate.vecLib (3.9 - vecLib 3.9) <F8D0CC77-98AC-3B58-9FE6-0C25421827B6> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
    0x7fff8b8f6000 -     0x7fff8ba64ff7  libBLAS.dylib (1094.5) <DE93A590-5FA5-32A2-A16C-5D7D7361769F> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
    0x7fff8bc02000 -     0x7fff8bc03ff7  libsystem_sandbox.dylib (278.11) <5E5A6E09-33A9-391A-AB34-E57D93BB1551> /usr/lib/system/libsystem_sandbox.dylib
    0x7fff8bc04000 -     0x7fff8bed8fc7  com.apple.vImage (7.0 - 7.0) <D241DBFA-AC49-31E2-893D-EAAC31890C90> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage
    0x7fff8c39b000 -     0x7fff8c39dff3  libsystem_configuration.dylib (596.13) <B51C8C22-C455-36AC-952D-A319B6545884> /usr/lib/system/libsystem_configuration.dylib
    0x7fff8c869000 -     0x7fff8c89dfff  libssl.0.9.8.dylib (50) <B15F967C-B002-36C2-9621-3456D8509F50> /usr/lib/libssl.0.9.8.dylib
    0x7fff8cadb000 -     0x7fff8cb0bfff  libncurses.5.4.dylib (42) <BF763D62-9149-37CB-B1D2-F66A2510E6DD> /usr/lib/libncurses.5.4.dylib
    0x7fff8cc49000 -     0x7fff8cc4eff7  libunwind.dylib (35.3) <78DCC358-2FC1-302E-B395-0155B47CB547> /usr/lib/system/libunwind.dylib
    0x7fff8f05c000 -     0x7fff8f127fff  libvDSP.dylib (423.32) <3BF732BE-DDE0-38EB-8C54-E4E3C64F77A7> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib
    0x7fff8fd8c000 -     0x7fff8ff39f27  libobjc.A.dylib (551.1) <AD7FD984-271E-30F4-A361-6B20319EC73B> /usr/lib/libobjc.A.dylib
    0x7fff8ff3a000 -     0x7fff8ff47ff0  libbz2.1.0.dylib (29) <0B98AC35-B138-349C-8063-2B987A75D24C> /usr/lib/libbz2.1.0.dylib
    0x7fff8ff53000 -     0x7fff8ff64ff7  libsystem_asl.dylib (217.1.4) <655FB343-52CF-3E2F-B14D-BEBF5AAEF94D> /usr/lib/system/libsystem_asl.dylib
    0x7fff904df000 -     0x7fff90697ff3  libicucore.A.dylib (511.31) <167DDD0A-A935-31AF-B5B9-940268EC3A3C> /usr/lib/libicucore.A.dylib
    0x7fff90698000 -     0x7fff906daff7  libauto.dylib (185.5) <F45C36E8-B606-3886-B5B1-B6745E757CA8> /usr/lib/libauto.dylib
    0x7fff9075e000 -     0x7fff90768fff  libcommonCrypto.dylib (60049) <8C4F0CA0-389C-3EDC-B155-E62DD2187E1D> /usr/lib/system/libcommonCrypto.dylib
    0x7fff90779000 -     0x7fff9077afff  libunc.dylib (28) <62682455-1862-36FE-8A04-7A6B91256438> /usr/lib/system/libunc.dylib
    0x7fff91c9a000 -     0x7fff91c9bffb  libremovefile.dylib (33) <3543F917-928E-3DB2-A2F4-7AB73B4970EF> /usr/lib/system/libremovefile.dylib
    0x7fff91e92000 -     0x7fff91f71fff  libcrypto.0.9.8.dylib (50) <B95B9DBA-39D3-3EEF-AF43-44608B28894E> /usr/lib/libcrypto.0.9.8.dylib
    0x7fff9241a000 -     0x7fff9241dff7  libdyld.dylib (239.4) <CF03004F-58E4-3BB6-B3FD-BE4E05F128A0> /usr/lib/system/libdyld.dylib
    0x7fff92447000 -     0x7fff92495fff  libcorecrypto.dylib (161.1) <F3973C28-14B6-3006-BB2B-00DD7F09ABC7> /usr/lib/system/libcorecrypto.dylib
    0x7fff9249c000 -     0x7fff9249dff7  libsystem_blocks.dylib (63) <FB856CD1-2AEA-3907-8E9B-1E54B6827F82> /usr/lib/system/libsystem_blocks.dylib
    0x7fff92509000 -     0x7fff92592ff7  libsystem_c.dylib (997.90.3) <6FD3A400-4BB2-3B95-B90C-BE6E9D0D78FA> /usr/lib/system/libsystem_c.dylib
    0x7fff92a7b000 -     0x7fff92b2bff7  libvMisc.dylib (423.32) <049C0735-1808-39B9-943F-76CB8021744F> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib
    0x7fff92b2c000 -     0x7fff92b47ff7  libsystem_malloc.dylib (23.10.1) <A695B4E4-38E9-332E-A772-29D31E3F1385> /usr/lib/system/libsystem_malloc.dylib
    0x7fff92bb4000 -     0x7fff92bbaff7  libsystem_platform.dylib (24.90.1) <3C3D3DA8-32B9-3243-98EC-D89B9A1670B3> /usr/lib/system/libsystem_platform.dylib
    0x7fff93139000 -     0x7fff9313dff7  libsystem_stats.dylib (93.90.3) <1A55AF8A-B6C4-3163-B557-3AD25DA643A8> /usr/lib/system/libsystem_stats.dylib
    0x7fff9365d000 -     0x7fff9365eff7  libSystem.B.dylib (1197.1.1) <BFC0DC97-46C6-3BE0-9983-54A98734897A> /usr/lib/libSystem.B.dylib
    0x7fff93682000 -     0x7fff936a9ff7  libsystem_network.dylib (241.3) <8B1E1F1D-A5CC-3BAE-8B1E-ABC84337A364> /usr/lib/system/libsystem_network.dylib
    0x7fff936aa000 -     0x7fff936aeff7  libcache.dylib (62) <BDC1E65B-72A1-3DA3-A57C-B23159CAAD0B> /usr/lib/system/libcache.dylib
    0x7fff93769000 -     0x7fff93785ff7  libsystem_kernel.dylib (2422.90.20) <20E00C54-9222-359F-BD98-CB79ABED769A> /usr/lib/system/libsystem_kernel.dylib
    0x7fff93a87000 -     0x7fff93a8eff8  liblaunch.dylib (842.90.1) <38D1AB2C-A476-385F-8EA8-7AB604CA1F89> /usr/lib/system/liblaunch.dylib
    0x7fff9407b000 -     0x7fff94080fff  libmacho.dylib (845) <1D2910DF-C036-3A82-A3FD-44FF73B5FF9B> /usr/lib/system/libmacho.dylib
    0x7fff94255000 -     0x7fff94279fff  libxpc.dylib (300.90.2) <AB40CD57-F454-3FD4-B415-63B3C0D5C624> /usr/lib/system/libxpc.dylib
    0x7fff94a92000 -     0x7fff94c77fff  com.apple.CoreFoundation (6.9 - 855.14) <617B8A7B-FAB2-3271-A09B-C542E351C532> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
    0x7fff94fdc000 -     0x7fff95003ffb  libsystem_info.dylib (449.1.3) <7D41A156-D285-3849-A2C3-C04ADE797D98> /usr/lib/system/libsystem_info.dylib
    0x7fff95004000 -     0x7fff95004ff7  libkeymgr.dylib (28) <3AA8D85D-CF00-3BD3-A5A0-E28E1A32A6D8> /usr/lib/system/libkeymgr.dylib

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 222746
    thread_create: 4
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=93.2M resident=18.4M(20%) swapped_out_or_unallocated=74.8M(80%)
Writable regions: Total=106.8M written=44.0M(41%) resident=48.1M(45%) swapped_out=0K(0%) unallocated=58.8M(55%)

REGION TYPE                      VIRTUAL
===========                      =======
Dispatch continuations             8192K
Kernel Alloc Once                     4K
MALLOC                             76.4M
MALLOC (admin)                       16K
MALLOC_LARGE (reserved)             256K        reserved VM address space (unallocated)
STACK GUARD                           8K
Stack                              5648K
VM_ALLOCATE                          16K
__DATA                             2376K
__LINKEDIT                         67.5M
__TEXT                             25.7M
__UNICODE                           544K
__UNIXSTACK                        16.0M
shared memory                         4K
===========                      =======
TOTAL                             202.3M
TOTAL, minus reserved VM space    202.0M


Model: MacBookAir5,2, BootROM MBA51.00EF.B02, 2 processors, Intel Core i7, 2 GHz, 8 GB, SMC 2.5f9
Graphics: Intel HD Graphics 4000, Intel HD Graphics 4000, Built-In, 1024 MB
Memory Module: BANK 0/DIMM0, 4 GB, DDR3, 1600 MHz, 0x02FE, 0x000000000000000000000000000000000000
Memory Module: BANK 1/DIMM0, 4 GB, DDR3, 1600 MHz, 0x02FE, 0x000000000000000000000000000000000000
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0xE9), Broadcom BCM43xx 1.0 (5.106.98.100.22)
Bluetooth: Version 4.2.3f10 13477, 3 services, 15 devices, 1 incoming serial ports
Network Service: Wi-Fi, AirPort, en0
Serial ATA Device: APPLE SSD SM256E, 251 GB
USB Device: Hub
USB Device: Hub
USB Device: BRCM20702 Hub
USB Device: Bluetooth USB Host Controller
USB Device: Apple Internal Keyboard / Trackpad
USB Device: Internal Memory Card Reader
USB Device: Hub
USB Device: FaceTime HD Camera (Built-in)
Thunderbolt Bus: MacBook Air, Apple Inc., 23.4

Note that I had to run the reproduction code 3 times before getting the crash (it is not deterministic) and that it does not happen when I replace 200 by 100 or less.

@juliantaylor
Copy link
Contributor

unless someone can reproduce the issue with a different blas than accelerate this seems to be an apple issue and should be reported to apple (or whoever develops accelerate)

@ogrisel
Copy link
Contributor
ogrisel commented Apr 4, 2014

I think so too. I cannot reproduce the issue with numpy master built with OpenBLAS.

Also on numpy built with Accelerate: I fixed the seed with rng = np.random.RandomState(42) and the segfault still happens half of the time so. The randomness of the segfault is not data dependent.

@smeylan
Copy link
smeylan commented Jun 1, 2014

Ran into the same problem on Mavericks; compiling numpy with openblas (after getting openblas with homebrew) fixed the problem.

@matthew-brett
Copy link
Contributor

Should we provide a numpy binary wheel with openblas instead of linking to Accelerate?

I believe most OSX users are on Mavericks already: https://www.adium.im/sparkle/#osVersion

@rgommers
Copy link
Member
rgommers commented Jun 2, 2014

@matthew-brett this was discussed a few times on the mailing list, and there were several saying "please don't do that to us". OpenBLAS will come with its own issues, so I think fixing this issue should be preferred. We managed to do that in scipy, we should manage here I'd think.

@juliantaylor
Copy link
Contributor

if you have the choice of accelerate or openblas. I'd choose openblas. accelerate also has fork issues (which are fixed in pthread openblas now) and bugs apparently don't get fixed at all nor can they be fixed by the community because its closed source (or is it open?)

@mcg1969
Copy link
mcg1969 commented Oct 11, 2014

As for the macro magic, I'm only doing that because I have to replace so many functions. Again, all you do is declare a static variable within which you will store a pointer to the original cblas_sgemv. Then, whenever you need to make a call to that original function, you use that pointer. Of course, each time you need to test if the variable is zero, and if it is, load the pointer with a loadsym call.

@mcg1969
Copy link
mcg1969 commented Oct 11, 2014

And if your code never creates misaligned data, you'll never even call loadsym.

@mcg1969
Copy link
mcg1969 commented Oct 11, 2014

Scratch that. Of course, you'll call loadsym most of the time.

@sturlamolden
Copy link
Contributor

The failmode is

if ( (((intptr_t)A)%32) || (((intptr_t)X)%32) || (((intptr_t)Y)%32) ) 

independent of transpositions.

@sturlamolden
Copy link
Contributor

I am not sure if calling loadsym most of the time is better than calling cblas_sgemm all of the time.

@mcg1969
Copy link
mcg1969 commented Oct 11, 2014

You only call loadsym once. You just have to test the pointer every time.

On Oct 11, 2014, at 3:52 PM, Sturla Molden <notifications@github.commailto:notifications@github.com> wrote:

I am not sure if calling loadsym most of the time is better than calling cblas_sgemm all of the time.


Reply to this email directly or view it on GitHubhttps://github.com//issues/4007#issuecomment-58764139.

@sturlamolden
Copy link
Contributor

Thanks, that makes sence.

@sturlamolden
Copy link
Contributor

Ok, now I understand your code, except the strange VOIDS(n) macro.

@sturlamolden
Copy link
Contributor

Apple is stalling, so here is a work-around based on an improved version of my previous fix, slightly inspired by libVecFort. Also we cannot rely on Apple users to actually have Apple's bugfix installed, whenever it might show up.

@piskvorky
Copy link
Author

I tried pinging Apple devs ("community evangelists") about that ticket a few days ago, but no response.

Apparently, Apple devs give priority to issues/tickets based on how many people report the same problem. If that's true, a single bug report doesn't stand much chance...

@matthew-brett
Copy link
Contributor

I got a reply to my bug report a couple of days ago claiming that this was fixed in Yosemite, and asking me to check, but I haven't got Yosemite. If anyone does have Yosemite, please do check with the Julian's code in #4007 (comment) (compiled with clang thecode.c -framework Accelerate) and let me know.

In any case, there are going to be a large number of people with this bug for a a long time, so I think we would still need to work around it.

@mcg1969
Copy link
mcg1969 commented Oct 18, 2014

Just ran it on Yosemite: "0 16 0".

@juliantaylor
Copy link
Contributor

thats very good news, I tried my old testcases on yosemite and they indeed do not crash anymore

@sturlamolden
Copy link
Contributor

@mcg1969 Here is a full patch for all fail modes. You might take a look at it for libVecFort:
https://github.com/sturlamolden/numpy/blob/maintenance/1.9.x/numpy/core/blasdot/apple_sgemv_patch.c

@juliantaylor
Copy link
Contributor

should be worked around in the 1.9 branch now

@mcg1969
Copy link
mcg1969 commented Oct 26, 2014

It has been a pleasure to see y'all working so well on this. Nicely done everyone.

@matthew-brett
Copy link
Contributor

Sturla - a particular shout-out to you, thanks very much for your patience and hard work on this one.

@sturlamolden
Copy link
Contributor

Here is more of the same, and I still have NumPy master (1.10.x) to fix.
scipy/scipy#4105
scipy/scipy#4104

@charris
Copy link
Member
charris commented Oct 27, 2014

@sturlamolden I'm inclined to wait on Apple to get their act together before applying a fix to numpy master. A PR while the work is fresh in your mind would be good, but it will sit for a while. Don't mean to discourage you, though ;)

@sturlamolden
Copy link
Contributor

I think I should do this while I have it in my head, then you can sit on it as long as you want :-)

A fix for NumPy master would be more like the ones for SciPy though, i.e. putting it in numpy/build_utils, which is a bit cleaner.

I cannot imagine why Apple would not want to fix a segfault in SGEMV, given its importance in computer graphics and multimedia.

@sturlamolden
Copy link
Contributor

This is for @charris to sit on:
#5237

@matthew-brett
Copy link
Contributor

I got a reply from the Apple engineers on my bug report:

There are no plans to backport the fix to 10.9.

We are closing this bug report.

@sturlamolden
Copy link
Contributor

Bummer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

0