8000 numpy.dot cannot handle arrays with >2**31 elements without support in the underlying BLAS · Issue #5533 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content
numpy.dot cannot handle arrays with >2**31 elements without support in the underlying BLAS #5533
Closed
@dschallis

Description

@dschallis

[The bug here is that some BLAS libraries will crash if you pass in > 16 GiB arrays, because they use 32-bit indices internally. The most obvious solution is to break the dot call into multiple calls to dgemm, though there are also other possibilities. Original report follows:]


On python 2.7.8, numpy 1.9.1, on Mac OS X:

import numpy
numpy.random.seed(1)
X = numpy.random.random((50000,100))
numpy.dot(X, X.T)

Results in:

Segmentation fault: 11

Segfault doesn't occur on smaller arrays (e.g. 30000x100 is fine). In case it's useful, some linkage info:

>>> numpy.__config__.show()
atlas_threads_info:
  NOT AVAILABLE
blas_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-DAPPLE_ACCELERATE_SGEMV_PATCH', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    define_macros = [('NO_ATLAS_INFO', 3)]
atlas_blas_threads_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
lapack_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-DAPPLE_ACCELERATE_SGEMV_PATCH']
    define_macros = [('NO_ATLAS_INFO', 3)]
openblas_lapack_info:
  NOT AVAILABLE
atlas_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      0