8000 Backport 7463, BUG: fix array too big error for wide dtypes. by charris · Pull Request #7816 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

Backport 7463, BUG: fix array too big error for wide dtypes. #7816

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 10, 2016
Merged

Backport 7463, BUG: fix array too big error for wide dtypes. #7816

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
af0beed
BUG: fix array too big error for wide dtypes.
seberg Mar 26, 2016
66be3ab
MAINT: Rename size and sd to nbytes in NewFromDescr_int
seberg Mar 26, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
BUG: fix array too big error for wide dtypes. 
The error was not being raised when arr.size * arr.dtype.itemsize
was too large, but only when arr.size was too large alone.

Closes gh-7451
  • Loading branch information
@seberg @charris
seberg authored and charris committed Jul 10, 2016
commit af0beedb9962f8cf50a3161699a2cc90fe0eb977
10 changes: 6 additions & 4 deletions numpy/core/src/multiarray/ctors.c
View file
Open in desktop
< 8000 td class="blob-code blob-code-context js-file-line"> }
Original file line number Diff line number Diff line change
Expand Up @@ -929,8 +929,7 @@ PyArray_NewFromDescr_int(PyTypeObject *subtype, PyArray_Descr *descr, int nd,
return NULL;
}

/* Check dimensions */
size = 1;
/* Check datatype element size */
sd = (size_t) descr->elsize;
if (sd == 0) {
if (!PyDataType_ISSTRING(descr)) {
Expand All @@ -950,6 +949,8 @@ PyArray_NewFromDescr_int(PyTypeObject *subtype, PyArray_Descr *descr, int nd,
}

/* Check dimensions */
size = sd;
for (i = 0; i < nd; i++) {
npy_intp dim = dims[i];

Expand All @@ -976,7 +977,8 @@ PyArray_NewFromDescr_int(PyTypeObject *subtype, PyArray_Descr *descr, int nd,
*/
if (npy_mul_with_overflow_intp(&size, size, dim)) {
PyErr_SetString(PyExc_ValueError,
"array is too big.");
"array is too big; `arr.size * arr.dtype.itemsize` "
"is larger than the maximum possible size.");
Py_DECREF(descr);
return NULL;
}
Expand Down Expand Up @@ -1025,7 +1027,7 @@ PyArray_NewFromDescr_int(PyTypeObject *subtype, PyArray_Descr *descr, int nd,
* the memory, but be careful with this...
*/
memcpy(fa->strides, strides, sizeof(npy_intp)*nd);
sd *= size;
sd = size;
}
}
else {
Expand Down
15 changes: 15 additions & 0 deletions numpy/core/tests/test_multiarray.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -730,6 +730,21 @@ def __len__(self):
d = A([1,2,3])
assert_equal(len(np.array(d)), 3)

def test_array_too_big(self):
# Test that array creation succeeds for arrays addressable by intp
# on the byte level and fails for too large arrays.
buf = np.zeros(100)

max_bytes = np.iinfo(np.intp).max
for dtype in ["intp", "S20", "b"]:
dtype = np.dtype(dtype)
itemsize = dtype.itemsize

np.ndarray(buffer=buf, strides=(0,),
shape=(max_bytes//itemsize,), dtype=dtype)
assert_raises(ValueError, np.ndarray, buffer=buf, strides=(0,),
shape=(max_bytes//itemsize + 1,), dtype=dtype)


class TestStructured(TestCase):
def test_subarray_field_access(self):
Expand Down
0