8000 Saving large array fails with savez_compressed() but works with save() · Issue #5336 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content 8000
Saving large array fails with savez_compressed() but works with save() #5336
Closed
@minhlab

Description

@minhlab

I got this strange problem on a CentOS box, Python 2.6.6, Numpy 1.9.1:

[minhle@node069 ~]$ python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.version.version
'1.9.1'
>>> a = np.ones((775890380,))
>>> np.savez_compressed('/home/minhle/scratch/test.npz', a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/npyio.py", line 560, in savez_compressed
    _savez(file, args, kwds, True)
  File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/npyio.py", line 597, in _savez
    format.write_array(fid, np.asanyarray(val))
  File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/format.py", line 562, in write_array
    array.tofile(fp)
IOError: 775890380 requested and 233691638 written
>>> np.save('/home/minhle/scratch/test.npy', a)
>>> b = np.load('/home/minhle/scratch/test.npy')
>>> b[:10]
array([ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.])
>>> quit()
[minhle@node069 ~]$ cat /etc/*-release
CentOS release 6.5 (Final)
Cluster Manager v5.2
slave
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      0