8000 API: bump MAXDIMS/MAXARGS to 64 introduce NPY_AXIS_RAVEL by seberg · Pull Request #25149 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

API: bump MAXDIMS/MAXARGS to 64 introduce NPY_AXIS_RAVEL #25149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Nov 28, 2023
Merged

API: bump MAXDIMS/MAXARGS to 64 introduce NPY_AXIS_RAVEL #25149

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
fc1bb6b
API: NumPy NPY_MAXDIMS and NPY_MAXARGS to 64 introduce NPY_AXIS_RAVEL
seberg Nov 2, 2023
af55c6b
TST: Adapt test to larger max-dims and max-args
seberg Nov 14, 2023
bbc4083
TST: Add a few tests for paths that are for now limited to 32 dims
seberg Nov 14, 2023
6a878e3
API: Add 1.x support for ``PyArray_RUNTIME_VERSION`` to the compat he…
seberg Nov 15, 2023
3270231
DOC: Add documentation for maxdim increase (and related) changes
seberg Nov 15, 2023
d6840f5
TST: "fix" failing memoryview with too many dimensions test
seberg Nov 15, 2023
a5a250f
TST: Make linter happy, it was right.
seberg Nov 15, 2023
8dddc54
MAINT: Cython restore NPY_MAXDIMS and add NPY_RAVEL_AXIS
seberg Nov 15, 2023
56b792c
Apply suggestions from code review
seberg Nov 24, 2023
0498d8c
Address review comments
seberg Nov 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
API: NumPy NPY_MAXDIMS and NPY_MAXARGS to 64 introduce NPY_AXIS_RAVEL 
Bumping these two has three main caveats:
1. We cannot break ABI for the iterator macros, which means the
   iterators now need to use 32/their own macro.
2. We used axis=MAXDIMS to mean axis=None, introduce NPY_AXIS_RAVEL
   to replace this, it will be run-time outside NumPy.
3. The old style iterators cannot deal with high dimensions, meaning
   that some functions just won't work (because they use them).
  • Loading branch information
@seberg
seberg committed Nov 17, 2023
commit fc1bb6b5b751dc20d7a4fe85ec4fd4287968a0ab
5 changes: 4 additions & 1 deletion doc/source/reference/c-api/array.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3411,7 +3411,10 @@ Other constants

.. c:macro:: NPY_MAXDIMS

The maximum number of dimensions allowed in arrays.
.. note::
NumPy used to have a maximum number of dimensions set to 32 and now 64.
This was/is mostly useful for stack allocations. We now ask you to
define (and test) such a limit in your own project if needed.

.. c:macro:: NPY_MAXARGS

Expand Down
39 changes: 21 additions & 18 deletions doc/source/reference/c-api/types-and-structures.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,8 +119,11 @@ PyArray_Type and PyArrayObject
array. When nd is 0, the array is sometimes called a rank-0
array. Such arrays have undefined dimensions and strides and
cannot be accessed. Macro :c:data:`PyArray_NDIM` defined in
``ndarraytypes.h`` points to this data member. :c:data:`NPY_MAXDIMS`
is the largest number of dimensions for any array.
``ndarraytypes.h`` points to this data member.
Although most operations may be limited in dimensionality, we do not
advertise a maximum dimension. Anyone explicitly relying on one
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a remnant of an earlier version that completely removed NPY_MAXDIMS from the public API? Maybe rephrase this to say NPY_MAXDIMS is there but not to rely on it because it might change or be removed in the future.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't a remnant, but rephrased a bit.

must check for it. Before NumPy 2.0, NumPy used 32 dimensions at most
after it, the limit is currently 64.

.. c:member:: npy_intp *dimensions

Expand Down Expand Up @@ -986,11 +989,11 @@ PyArrayIter_Type and PyArrayIterObject
int nd_m1;
npy_intp index;
npy_intp size;
npy_intp coordinates[NPY_MAXDIMS];
npy_intp dims_m1[NPY_MAXDIMS];
npy_intp strides[NPY_MAXDIMS];
npy_intp backstrides[NPY_MAXDIMS];
npy_intp factors[NPY_MAXDIMS];
npy_intp coordinates[NPY_MAXDIMS_LEGACY_ITERS];
npy_intp dims_m1[NPY_MAXDIMS_LEGACY_ITERS];
npy_intp strides[NPY_MAXDIMS_LEGACY_ITERS];
npy_intp backstrides[NPY_MAXDIMS_LEGACY_ITERS];
npy_intp factors[NPY_MAXDIMS_LEGACY_ITERS];
PyArrayObject *ao;
char *dataptr;
npy_bool contiguous;
Expand Down Expand Up @@ -1086,8 +1089,8 @@ PyArrayMultiIter_Type and PyArrayMultiIterObject
npy_intp size;
npy_intp index;
int nd;
npy_intp dimensions[NPY_MAXDIMS];
PyArrayIterObject *iters[NPY_MAXDIMS];
npy_intp dimensions[NPY_MAXDIMS_LEGACY_ITERS];
PyArrayIterObject *iters[NPY_MAXDIMS_LEGACY_ITERS];
} PyArrayMultiIterObject;

.. c:macro: PyObject_HEAD
Expand Down Expand Up @@ -1141,20 +1144,20 @@ PyArrayNeighborhoodIter_Type and PyArrayNeighborhoodIterObject
PyObject_HEAD
int nd_m1;
npy_intp index, size;
npy_intp coordinates[NPY_MAXDIMS]
npy_intp dims_m1[NPY_MAXDIMS];
npy_intp strides[NPY_MAXDIMS];
npy_intp backstrides[NPY_MAXDIMS];
npy_intp factors[NPY_MAXDIMS];
npy_intp coordinates[NPY_MAXDIMS_LEGACY_ITERS]
npy_intp dims_m1[NPY_MAXDIMS_LEGACY_ITERS];
npy_intp strides[NPY_MAXDIMS_LEGACY_ITERS];
npy_intp backstrides[NPY_MAXDIMS_LEGACY_ITERS];
npy_intp factors[NPY_MAXDIMS_LEGACY_ITERS];
PyArrayObject *ao;
char *dataptr;
npy_bool contiguous;
npy_intp bounds[NPY_MAXDIMS][2];
npy_intp limits[NPY_MAXDIMS][2];
npy_intp limits_sizes[NPY_MAXDIMS];
npy_intp bounds[NPY_MAXDIMS_LEGACY_ITERS][2];
npy_intp limits[NPY_MAXDIMS_LEGACY_ITERS][2];
npy_intp limits_sizes[NPY_MAXDIMS_LEGACY_ITERS];
npy_iter_get_dataptr_t translate;
npy_intp nd;
npy_intp dimensions[NPY_MAXDIMS];
npy_intp dimensions[NPY_MAXDIMS_LEGACY_ITERS];
PyArrayIterObject* _internal_iter;
char* constant;
int mode;
Expand Down
5 changes: 0 additions & 5 deletions numpy/__init__.cython-30.pxd
View file
< 67ED /span> Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,11 +186,6 @@ cdef extern from "numpy/arrayobject.h":

NPY_ARRAY_UPDATE_ALL

cdef enum:
NPY_MAXDIMS

npy_intp NPY_MAX_ELSIZE

ctypedef void (*PyArray_VectorUnaryFunc)(void *, void *, npy_intp, void *, void *)

ctypedef struct PyArray_ArrayDescr:
Expand Down
5 changes: 0 additions & 5 deletions numpy/__init__.pxd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,11 +181,6 @@ cdef extern from "numpy/arrayobject.h":

NPY_ARRAY_UPDATE_ALL

cdef enum:
NPY_MAXDIMS

npy_intp NPY_MAX_ELSIZE
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a bit random, but NPY_MAX_ELSIZE doesn't exist (it sure did at some point).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this removal need a release note? Just to say that if someone was using and relying on it, it wasn't doing what they thought it did.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤷 I doubt anyone will notice, but sure, added.


ctypedef void (*PyArray_VectorUnaryFunc)(void *, void *, npy_intp, void *, void *)

ctypedef struct PyArray_ArrayDescr:
Expand Down
1 change: 0 additions & 1 deletion numpy/_core/include/numpy/ndarrayobject.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,6 @@ extern "C" {

#define PyArray_FILLWBYTE(obj, val) memset(PyArray_DATA(obj), val, \
PyArray_NBYTES(obj))
#define NPY_MAX_ELSIZE (2 * NPY_SIZEOF_LONGDOUBLE)

#define PyArray_ContiguousFromAny(op, type, min_depth, max_depth) \
PyArray_FromAny(op, PyArray_DescrFromType(type), min_depth, \
Expand Down
49 changes: 28 additions & 21 deletions numpy/_core/include/numpy/ndarraytypes.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,17 @@
* The array creation itself could have arbitrary dimensions but all
* the places where static allocation is used would need to be changed
* to dynamic (including inside of several structures)
*
* As of NumPy 2.0, we strongly discourage the downstream use of NPY_MAXDIMS,
* but since auditing everything seems a big ask, define it as 64.
* A future version could:
* - Increase or remove the limit and require recompilation (like 2.0 did)
* - Deprecate or remove the macro but keep the limit (at basically any time)
*/

#define NPY_MAXDIMS 32
#define NPY_MAXARGS 32
#define NPY_MAXDIMS 64
/* We cannot change this as it would break ABI: */
#define NPY_MAXDIMS_LEGACY_ITERS 32
#define NPY_MAXARGS 64

/* Used for Converter Functions "O&" code in ParseTuple */
#define NPY_FAIL 0
Expand Down Expand Up @@ -1095,18 +1102,18 @@ struct PyArrayIterObject_tag {
PyObject_HEAD
int nd_m1; /* number of dimensions - 1 */
npy_intp index, size;
npy_intp coordinates[NPY_MAXDIMS];/* N-dimensional loop */
npy_intp dims_m1[NPY_MAXDIMS]; /* ao->dimensions - 1 */
npy_intp strides[NPY_MAXDIMS]; /* ao->strides or fake */
npy_intp backstrides[NPY_MAXDIMS];/* how far to jump back */
npy_intp factors[NPY_MAXDIMS]; /* shape factors */
npy_intp coordinates[NPY_MAXDIMS_LEGACY_ITERS];/* N-dimensional loop */
npy_intp dims_m1[NPY_MAXDIMS_LEGACY_ITERS]; /* ao->dimensions - 1 */
npy_intp strides[NPY_MAXDIMS_LEGACY_ITERS]; /* ao->strides or fake */
npy_intp backstrides[NPY_MAXDIMS_LEGACY_ITERS];/* how far to jump back */
npy_intp factors[NPY_MAXDIMS_LEGACY_ITERS]; /* shape factors */
PyArrayObject *ao;
char *dataptr; /* pointer to current item*/
npy_bool contiguous;

npy_intp bounds[NPY_MAXDIMS][2];
npy_intp limits[NPY_MAXDIMS][2];
npy_intp limits_sizes[NPY_MAXDIMS];
npy_intp bounds[NPY_MAXDIMS_LEGACY_ITERS][2];
npy_intp limits[NPY_MAXDIMS_LEGACY_ITERS][2];
npy_intp limits_sizes[NPY_MAXDIMS_LEGACY_ITERS];
npy_iter_get_dataptr_t translate;
} ;

Expand Down Expand Up @@ -1230,7 +1237,7 @@ typedef struct {
npy_intp size; /* broadcasted size */
npy_intp index; /* current index */
int nd; /* number of dims */
npy_intp dimensions[NPY_MAXDIMS]; /* dimensions */
npy_intp dimensions[NPY_MAXDIMS_LEGACY_ITERS]; /* dimensions */
PyArrayIterObject *iters[NPY_MAXARGS]; /* iterators */
} PyArrayMultiIterObject;

Expand Down Expand Up @@ -1335,18 +1342,18 @@ typedef struct {
*/
int nd_m1; /* number of dimensions - 1 */
npy_intp index, size;
npy_intp coordinates[NPY_MAXDIMS];/* N-dimensional loop */
npy_intp dims_m1[NPY_MAXDIMS]; /* ao->dimensions - 1 */
npy_intp strides[NPY_MAXDIMS]; /* ao->strides or fake */
npy_intp backstrides[NPY_MAXDIMS];/* how far to jump back */
npy_intp factors[NPY_MAXDIMS]; /* shape factors */
npy_intp coordinates[NPY_MAXDIMS_LEGACY_ITERS];/* N-dimensional loop */
npy_intp dims_m1[NPY_MAXDIMS_LEGACY_ITERS]; /* ao->dimensions - 1 */
npy_intp strides[NPY_MAXDIMS_LEGACY_ITERS]; /* ao->strides or fake */
npy_intp backstrides[NPY_MAXDIMS_LEGACY_ITERS];/* how far to jump back */
npy_intp factors[NPY_MAXDIMS_LEGACY_ITERS]; /* shape factors */
PyArrayObject *ao;
char *dataptr; /* pointer to current item*/
npy_bool contiguous;

npy_intp bounds[NPY_MAXDIMS][2];
npy_intp limits[NPY_MAXDIMS][2];
npy_intp limits_sizes[NPY_MAXDIMS];
npy_intp bounds[NPY_MAXDIMS_LEGACY_ITERS][2];
npy_intp limits[NPY_MAXDIMS_LEGACY_ITERS][2];
npy_intp limits_sizes[NPY_MAXDIMS_LEGACY_ITERS];
npy_iter_get_dataptr_t translate;

/*
Expand All @@ -1355,7 +1362,7 @@ typedef struct {
npy_intp nd;

/* Dimensions is the dimension of the array */
npy_intp dimensions[NPY_MAXDIMS];
npy_intp dimensions[NPY_MAXDIMS_LEGACY_ITERS];

/*
* Neighborhood points coordinates are computed relatively to the
Expand Down
10 changes: 10 additions & 0 deletions numpy/_core/include/numpy/npy_2_compat.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,4 +52,14 @@
#endif


#if NPY_FEATURE_VERSION >= NPY_2_0_API_VERSION
#define NPY_RAVEL_AXIS NPY_MIN_INT
#elif NPY_ABI_VERSION < 0x02000000
#define NPY_RAVEL_AXIS 32
#else
#define NPY_RAVEL_AXIS \
(PyArray_RUNTIME_VERSION >= NPY_2_0_API_VERSION ? -1 : 32)
#endif


#endif /* NUMPY_CORE_INCLUDE_NUMPY_NPY_2_COMPAT_H_ */
14 changes: 9 additions & 5 deletions numpy/_core/src/multiarray/_multiarray_tests.c.src
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ static int copy_@name@(PyArrayIterObject *itx, PyArrayNeighborhoodIterObject *ni
{
npy_intp i, j;
@type@ *ptr;
npy_intp odims[NPY_MAXDIMS];
npy_intp odims[NPY_MAXDIMS_LEGACY_ITERS];
PyArrayObject *aout;

/*
Expand Down Expand Up @@ -125,7 +125,7 @@ static int copy_object(PyArrayIterObject *itx, PyArrayNeighborhoodIterObject *ni
PyObject **out)
{
npy_intp i, j;
npy_intp odims[NPY_MAXDIMS];
npy_intp odims[NPY_MAXDIMS_LEGACY_ITERS];
PyArrayObject *aout;
PyArray_CopySwapFunc *copyswap = PyArray_DESCR(itx->ao)->f->copyswap;
npy_int itemsize = PyArray_ITEMSIZE(itx->ao);
Expand Down Expand Up @@ -166,7 +166,7 @@ test_neighborhood_iterator(PyObject* NPY_UNUSED(self), PyObject* args)
PyArrayIterObject *itx;
int i, typenum, mode, st;
Py_ssize_t idxstart = 0;
npy_intp bounds[NPY_MAXDIMS*2];
npy_intp bounds[NPY_MAXDIMS_LEGACY_ITERS*2];
PyArrayNeighborhoodIterObject *niterx;

if (!PyArg_ParseTuple(args, "OOOi|n", &x, &b, &fill, &mode, &idxstart)) {
Expand Down Expand Up @@ -298,7 +298,7 @@ copy_double_double(PyArrayNeighborhoodIterObject *itx,
{
npy_intp i, j;
double *ptr;
npy_intp odims[NPY_MAXDIMS];
npy_intp odims[NPY_MAXDIMS_LEGACY_ITERS];
PyArrayObject *aout;

/*
Expand Down Expand Up @@ -338,7 +338,7 @@ test_neighborhood_iterator_oob(PyObject* NPY_UNUSED(self), PyObject* args)
PyArrayObject *ax;
PyArrayIterObject *itx;
int i, typenum, mode1, mode2, st;
npy_intp bounds[NPY_MAXDIMS*2];
npy_intp bounds[NPY_MAXDIMS_LEGACY_ITERS*2];
PyArrayNeighborhoodIterObject *niterx1, *niterx2;

if (!PyArg_ParseTuple(args, "OOiOi", &x, &b1, &mode1, &b2, &mode2)) {
Expand All @@ -358,6 +358,10 @@ test_neighborhood_iterator_oob(PyObject* NPY_UNUSED(self), PyObject* args)
if (ax == NULL) {
return NULL;
}
if (PyArray_NDIM(ax) > NPY_MAXDIMS_LEGACY_ITERS) {
PyErr_SetString(PyExc_TypeError, "too many dimensions.");
goto clean_ax;
}
if (PySequence_Size(b1) != 2 * PyArray_NDIM(ax)) {
PyErr_SetString(PyExc_ValueError,
"bounds sequence 1 size not compatible with x input");
Expand Down
2 changes: 1 addition & 1 deletion numpy/_core/src/multiarray/calculation.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ _PyArray_ArgMinMaxCommon(PyArrayObject *op,
}
else {
out_shape = _shape_buf;
if (axis_copy == NPY_MAXDIMS) {
if (axis_copy == NPY_RAVEL_AXIS) {
for (int i = 0; i < out_ndim; i++) {
out_shape[i] = 1;
}
Expand Down
4 changes: 2 additions & 2 deletions numpy/_core/src/multiarray/compiled_base.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1977,7 +1977,7 @@ NPY_NO_EXPORT PyObject *
io_pack(PyObject *NPY_UNUSED(self), PyObject *args, PyObject *kwds)
{
PyObject *obj;
int axis = NPY_MAXDIMS;
int axis = NPY_RAVEL_AXIS;
static char *kwlist[] = {"in", "axis", "bitorder", NULL};
char c = 'b';
const char * order_str = NULL;
Expand Down Expand Up @@ -2005,7 +2005,7 @@ NPY_NO_EXPORT PyObject *
io_unpack(PyObject *NPY_UNUSED(self), PyObject *args, PyObject *kwds)
{
PyObject *obj;
int axis = NPY_MAXDIMS;
int axis = NPY_RAVEL_AXIS;
PyObject *count = Py_None;
static char *kwlist[] = {"in", "axis", "count", "bitorder", NULL};
const char * c = NULL;
Expand Down
13 changes: 2 additions & 11 deletions numpy/_core/src/multiarray/conversion_utils.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ PyArray_IntpConverter(PyObject *obj, PyArray_Dims *seq)
if (len > NPY_MAXDIMS) {
PyErr_Format(PyExc_ValueError,
"maximum supported dimension for an ndarray "
"is %d, found %d", NPY_MAXDIMS, len);
"is currently %d, found %d", NPY_MAXDIMS, len);
Py_DECREF(seq_obj);
return NPY_FAIL;
}
Expand Down Expand Up @@ -325,23 +325,14 @@ NPY_NO_EXPORT int
PyArray_AxisConverter(PyObject *obj, int *axis)
{
if (obj == Py_None) {
*axis = NPY_MAXDIMS;
*axis = NPY_RAVEL_AXIS;
}
else {
*axis = PyArray_PyIntAsInt_ErrMsg(obj,
"an integer is required for the axis");
if (error_converting(*axis)) {
return NPY_FAIL;
}
if (*axis == NPY_MAXDIMS){
/* NumPy 1.23, 2022-05-19 */
if (DEPRECATE("Using `axis=32` (MAXDIMS) is deprecated. "
"32/MAXDIMS had the same meaning as `axis=None` which "
"should be used instead. "
"(Deprecated NumPy 1.23)") < 0) {
return NPY_FAIL;
}
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very niche, but, mentioned it in the release notes.

}
return NPY_SUCCEED;
}
Expand Down
4 changes: 2 additions & 2 deletions numpy/_core/src/multiarray/ctors.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2785,14 +2785,14 @@ PyArray_CheckAxis(PyArrayObject *arr, int *axis, int flags)
PyObject *temp1, *temp2;
int n = PyArray_NDIM(arr); D40E

if (*axis == NPY_MAXDIMS || n == 0) {
if (*axis == NPY_RAVEL_AXIS || n == 0) {
if (n != 1) {
temp1 = PyArray_Ravel(arr,0);
if (temp1 == NULL) {
*axis = 0;
return NULL;
}
if (*axis == NPY_MAXDIMS) {
if (*axis == NPY_RAVEL_AXIS) {
*axis = PyArray_NDIM((PyArrayObject *)temp1)-1;
}
}
Expand Down
8 changes: 8 additions & 0 deletions numpy/_core/src/multiarray/iterators.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,8 @@ PyArray_RawIterBaseInit(PyArrayIterObject *it, PyArrayObject *ao)
int nd, i;

nd = PyArray_NDIM(ao);
/* The legacy iterator only supports 32 dimensions */
assert(nd <= NPY_MAXDIMS_LEGACY_ITERS);
PyArray_UpdateFlags(ao, NPY_ARRAY_C_CONTIGUOUS);
if (PyArray_ISCONTIGUOUS(ao)) {
it->contiguous = 1;
Expand Down Expand Up @@ -190,6 +192,12 @@ PyArray_IterNew(PyObject *obj)
return NULL;
}
ao = (PyArrayObject *)obj;
if (PyArray_NDIM(ao) > NPY_MAXDIMS_LEGACY_ITERS) {
PyErr_Format(PyExc_RuntimeError,
"this function only supports up to 32 dimensions but "
"the array has %d.", PyArray_NDIM(ao));
return NULL;
}

it = (PyArrayIterObject *)PyArray_malloc(sizeof(PyArrayIterObject));
PyObject_Init((PyObject *)it, &PyArrayIter_Type);
Expand Down
Loading
0