8000 numpy.load fails loading large arrays on windows due to integer overflow · Issue #8469 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content
numpy.load fails loading large arrays on windows due to integer overflow #8469
Closed
@tolga-b

Description

@tolga-b

Hi,
The problem is as follows:
I have an array of size (1281024, 4096) (5247074304 elements) of type float32.
I can save and load this array using numpy.save and numpy.load functions in Ubuntu 14.04 64bit, however when I try to load the same array in Windows 64bit I get an error:
"Total size of the array must be unchanged" in numpy/lib/format.py.

I found out that the read count is incorrectly set in numpy.multiple.reduce at line 4E49 626 in format.py due to int32 overflow for any array with larger than 2,147,483,647 elements (default C int is 32bit in 64bit Windows unlike Linux). Which in turn causes only a part of the file to be read and therefore setting the shape fails. Just fixing the read count caused the read array to contain a lot of "nan" values, so there must be other parts that are broken as well.

I think this should be included in documentation if there is a file limit for numpy.save.

I am using numpy version 1.11.1 with latest Anaconda distribution.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      0