np.load() "invalid shape in fixed-type tuple" in NumPy 1.16.0 #13431

jzwinck · 2019-04-30T05:41:03Z

In NumPy 1.16.0, np.load() fails to load some files created by np.save(), saying "ValueError: invalid shape in fixed-type tuple."

The problem seems to be that #12358 changed np.lib.format._read_array_header(). It used to pass the descr to np.dtype(), but now it uses a new function descr_to_dtype() which fails for some valid inputs.

We have significant amounts of data stored on disk with formats which NumPy 1.16.0 cannot load, so this is a showstopper for upgrading beyond 1.15.

Reproducing code example:

This fails in NumPy 1.16.0, but works in all prior versions:

import numpy as np
a = np.zeros(1, np.dtype([('c', ('<f8', (5,)), (2,))]))
np.save('a.npy', a)
np.load('a.npy')

Error message:

Traceback (most recent call last):
  File "numpy/lib/npyio.py", line 440, in load
    pickle_kwargs=pickle_kwargs)
  File "numpy/lib/format.py", line 678, in read_array
    shape, fortran_order, dtype = _read_array_header(fp, version)
  File "numpy/lib/format.py", line 563, in _read_array_header
    dtype = descr_to_dtype(d['descr'])
  File "numpy/lib/format.py", line 282, in descr_to_dtype
    dt = numpy.dtype((descr_to_dtype(descr_str), shape))
  File "numpy/lib/format.py", line 282, in descr_to_dtype
    dt = numpy.dtype((descr_to_dtype(descr_str), shape))
ValueError: invalid shape in fixed-type tuple.

Numpy/Python version information:

NumPy 1.16.0
Python 3.6.6

The text was updated successfully, but these errors were encountered:

mattip · 2019-04-30T10:57:53Z

It seems these two dtypes are equivalent, we only handled the second in the refactoring

np.zeros(1, dtype=[('c', ('<f8', (5,)), (2,))])
np.zeros(1, dtype=[('c', '<f8', (2, 5))])

mattip mentioned this issue Apr 30, 2019

BUG: Handle subarrays in descr_to_dtype #13433

Merged

charris added this to the 1.16.4 release milestone Apr 30, 2019

charris added the 06 - Regression label Apr 30, 2019

seberg closed this as completed in #13433 May 12, 2019

charris mentioned this issue May 14, 2019

BUG: Handle subarrays in descr_to_dtype #13561

Merged

kousaa mentioned this issue Nov 28, 2019

Reading an h5ad file not working anymore after I run the rank_genes_groups function (used to work fine before python module update) scverse/scanpy#937

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

np.load() "invalid shape in fixed-type tuple" in NumPy 1.16.0 #13431

np.load() "invalid shape in fixed-type tuple" in NumPy 1.16.0 #13431

Uh oh!