8000 ENH: Add shape to *_like() array creation by pentschev · Pull Request #13046 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content

ENH: Add shape to *_like() array creation #13046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
Apr 25, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
f885c08
ENH: Added shape argument to *_like() array creation functions
pentschev Feb 26, 2019
5ad61b6
ENH: C backend adjustments for shape argument on *_like()
pentschev Feb 26, 2019
fd3e270
TST: Added test for shape argument in *_like() functions
pentschev Feb 26, 2019
b7202a7
ENH: Added PyArray_NewLikeArrayWithShape()
pentschev Feb 26, 2019
013cbce
BUG: Fix for PyArray_NewLikeArrayWithShape strides and ndim == 0
pentschev Feb 27, 2019
807f512
REL: Updates for C-API, version 1.17.x
pentschev Feb 27, 2019
95bbfd0
Revert "REL: Updates for C-API, version 1.17.x"
pentschev Mar 13, 2019
43b9828
Revert exposing PyArray_NewLikeArrayWithShape on C-API
pentschev Mar 13, 2019
fa7fd75
DOC: fix versionadded for *_like() shape argument
pentschev Mar 13, 2019
d57e6d3
STY: add missing spaces in array initializers
pentschev Mar 13, 2019
f1b3e91
ENH: empty_like raises ValueError
pentschev Mar 16, 2019
d24ac10
TST: test for exception of *_like() functions
pentschev Mar 16, 2019
ec26417
DOC: release note for shape argument in *_like() functions
pentschev Mar 16, 2019
40e7e9e
DOC: fix *_like() documentation on raises
pentschev Mar 17, 2019
928952d
BUG: *_like() raises for non-C/F-layout arrays
pentschev Mar 17, 2019
e394356
TST: change *_like() shapes to prevent NPY_RELAXED_STRIDE_DEBUG=1 fai…
pentschev Mar 17, 2019
d938fb9
Move empty_like() exception to C implementation
pentschev Mar 19, 2019
99a55e0
Merge branch 'master' into add-shape-to-like-array-creation
pentschev Mar 28, 2019
c087801
Update *_like() ValueError documentation
pentschev Apr 4, 2019
db7614b
Merge remote-tracking branch 'upstream/master' into add-shape-to-like…
pentschev Apr 5, 2019
669ea71
Rearrange stride computation for *_like() if new shape and order='K'
pentschev Apr 10, 2019
f53afbe
Change handling of order= for *_like()
pentschev Apr 19, 2019
c3ac08e
Merge branch 'master' into add-shape-to-like-array-creation
pentschev Apr 19, 2019
695b836
Fix *_like() tests
pentschev Apr 19, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
8000 Diff view
10 changes: 10 additions & 0 deletions doc/release/1.17.0-notes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,16 @@ New mode "empty" for ``np.pad``
This mode pads an array to a desired shape without initializing the new
entries.


``np.empty_like`` and related functions now accept a ``shape`` argument
-----------------------------------------------------------------------
``np.empty_like``, ``np.full_like``, ``np.ones_like`` and ``np.zeros_like`` now
accept a ``shape`` keyword argument, which can be used to create a new array
as the prototype, overriding its shape as well. This is particularly useful
when combined with the ``__array_function__`` protocol, allowing the creation
of new arbitrary-shape arrays from NumPy-like libraries when such an array
is used as the prototype.

Floating point scalars implement ``as_integer_ratio`` to match the builtin float
--------------------------------------------------------------------------------
This returns a (numerator, denominator) pair, which can be used to construct a
Expand Down
10 changes: 8 additions & 2 deletions numpy/core/multiarray.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,9 +71,9 @@


@array_function_from_c_func_and_dispatcher(_multiarray_umath.empty_like)
def empty_like(prototype, dtype=None, order=None, subok=None):
def empty_like(prototype, dtype=None, order=None, subok=None, shape=None):
"""
empty_like(prototype, dtype=None, order='K', subok=True)
empty_like(prototype, dtype=None, order='K', subok=True, shape=None)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed order and subok both default to None rather than the ('K' and True) values documentation proposes and other _like() functions have. Which one is the correct? It seems to me that the function definition should be order='K' and subok=True, am I correct?

Copy link
Member
8000

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little confusing, but the arguments for this function (the dispatcher) need to default to None. We actually have a test that verifies this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we then update the documentation to reflect the real defaults?


Return a new array with the same shape and type as a given array.

Expand All @@ -97,6 +97,12 @@ def empty_like(prototype, dtype=None, order=None, subok=None):
If True, then the newly created array will use the sub-class
type of 'a', otherwise it will be a base-class array. Defaults
to True.
shape : int or sequence of ints, optional.
Overrides the shape of the result. If order='K' and the number of
dimensions is unchanged, will try to keep order, otherwise,
order='C' is implied.

.. versionadded:: 1.17.0

Returns
-------
Expand Down
36 changes: 27 additions & 9 deletions numpy/core/numeric.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,12 +90,12 @@ class ComplexWarning(RuntimeWarning):
pass


def _zeros_like_dispatcher(a, dtype=None, order=None, subok=None):
def _zeros_like_dispatcher(a, dtype=None, order=None, subok=None, shape=None):
return (a,)


@array_function_dispatch(_zeros_like_dispatcher)
def zeros_like(a, dtype=None, order='K', subok=True):
def zeros_like(a, dtype=None, order='K', subok=True, shape=None):
"""
Return an array of zeros with the same shape and type as a given array.

Expand All @@ -119,6 +119,12 @@ def zeros_like(a, dtype=None, order='K', subok=True):
If True, then the newly created array will use the sub-class
type of 'a', otherwise it will be a base-class array. Defaults
to True.
shape : int or sequence of ints, optional.
Overrides the shape of the result. If order='K' and the number of
dimensions is unchanged, will try to keep order, otherwise,
order='C' is implied.

.. versionadded:: 1.17.0

Returns
-------
Expand Down Expand Up @@ -150,7 +156,7 @@ def zeros_like(a, dtype=None, order='K', subok=True):
array([0., 0., 0.])

"""
res = empty_like(a, dtype=dtype, order=order, subok=subok)
res = empty_like(a, dtype=dtype, order=order, subok=subok, shape=shape)
# needed instead of a 0 to get same result as zeros for for string dtypes
z = zeros(1, dtype=res.dtype)
multiarray.copyto(res, z, casting='unsafe')
Expand Down Expand Up @@ -210,12 +216,12 @@ def ones(shape, dtype=None, order='C'):
return a


def _ones_like_dispatcher(a, dtype=None, order=None, subok=None):
def _ones_like_dispatcher(a, dtype=None, order=None, subok=None, shape=None):
return (a,)


@array_function_dispatch(_ones_like_dispatcher)
def ones_like(a, dtype=None, order='K', subok=True):
67E6 def ones_like(a, dtype=None, order='K', subok=True, shape=None):
"""
Return an array of ones with the same shape and type as a given array.

Expand All @@ -239,6 +245,12 @@ def ones_like(a, dtype=None, order='K', subok=True):
If True, then the newly created array will use the sub-class
type of 'a', otherwise it will be a base-class array. Defaults
to True.
shape : int or sequence of ints, optional.
Overrides the shape of the result. If order='K' and the number of
dimensions is unchanged, will try to keep order, otherwise,
order='C' is implied.

.. versionadded:: 1.17.0

Returns
-------
Expand Down Expand Up @@ -270,7 +282,7 @@ def ones_like(a, dtype=None, order='K', subok=True):
array([1., 1., 1.])

"""
res = empty_like(a, dtype=dtype, order=order, subok=subok)
res = empty_like(a, dtype=dtype, order=order, subok=subok, shape=shape)
multiarray.copyto(res, 1, casting='unsafe')
return res

Expand Down Expand Up @@ -322,12 +334,12 @@ def full(shape, fill_value, dtype=None, order='C'):
return a


def _full_like_dispatcher(a, fill_value, dtype=None, order=None, subok=None):
def _full_like_dispatcher(a, fill_value, dtype=None, order=None, subok=None, shape=None):
return (a,)


@array_function_dispatch(_full_like_dispatcher)
def full_like(a, fill_value, dtype=None, order='K', subok=True):
def full_like(a, fill_value, dtype=None, order='K', subok=True, shape=None):
"""
Return a full array with the same shape and type as a given array.

Expand All @@ -349,6 +361,12 @@ def full_like(a, fill_value, dtype=None, order='K', subok=True):
If True, then the newly created array will use the sub-class
type of 'a', otherwise it will be a base-class array. Defaults
to True.
shape : int or sequence of ints, optional.
Overrides the shape of the result. If order='K' and the number of
dimensions is unchanged, will try to keep order, otherwise,
order='C' is implied.

.. versionadded:: 1.17.0

Returns
-------
Expand Down Expand Up @@ -379,7 +397,7 @@ def full_like(a, fill_value, dtype=None, order='K', subok=True):
array([0.1, 0.1, 0.1, 0.1, 0.1, 0.1])

"""
res = empty_like(a, dtype=dtype, order=order, subok=subok)
res = empty_like(a, dtype=dtype, order=order, subok=subok, shape=shape)
multiarray.copyto(res, fill_value, casting='unsafe')
return res

Expand Down
51 changes: 41 additions & 10 deletions numpy/core/src/multiarray/ctors.c
Original file line number Diff line number Diff line change
Expand Up @@ -1183,28 +1183,37 @@ PyArray_NewFromDescrAndBase(
flags, obj, base, 0, 0);
}

/*NUMPY_API
/*
* Creates a new array with the same shape as the provided one,
* with possible memory layout order and data type changes.
* with possible memory layout order, data type and shape changes.
*
* prototype - The array the new one should be like.
* order - NPY_CORDER - C-contiguous result.
* NPY_FORTRANORDER - Fortran-contiguous result.
* NPY_ANYORDER - Fortran if prototype is Fortran, C otherwise.
* NPY_KEEPORDER - Keeps the axis ordering of prototype.
* dtype - If not NULL, overrides the data type of the result.
* ndim - If not 0 and dims not NULL, overrides the shape of the result.
* dims - If not NULL and ndim not 0, overrides the shape of the result.
* subok - If 1, use the prototype's array subtype, otherwise
* always create a base-class array.
*
* NOTE: If dtype is not NULL, steals the dtype reference. On failure or when
* dtype->subarray is true, dtype will be decrefed.
*/
NPY_NO_EXPORT PyObject *
PyArray_NewLikeArray(PyArrayObject *prototype, NPY_ORDER order,
PyArray_Descr *dtype, int subok)
PyArray_NewLikeArrayWithShape(PyArrayObject *prototype, NPY_ORDER order,
PyArray_Descr *dtype, int ndim, npy_intp *dims, int subok)
{
PyObject *ret = NULL;
int ndim = PyArray_NDIM(prototype);

if (dims == NULL) {
ndim = PyArray_NDIM(prototype);
dims = PyArray_DIMS(prototype);
}
else if (order == NPY_KEEPORDER && (ndim != PyArray_NDIM(prototype))) {
order = NPY_CORDER;
}

/* If no override data type, use the one from the prototype */
if (dtype == NULL) {
Expand Down Expand Up @@ -1237,7 +1246,7 @@ PyArray_NewLikeArray(PyArrayObject *prototype, NPY_ORDER order,
ret = PyArray_NewFromDescr(subok ? Py_TYPE(prototype) : &PyArray_Type,
dtype,
ndim,
PyArray_DIMS(prototype),
dims,
NULL,
NULL,
order,
Expand All @@ -1246,11 +1255,10 @@ PyArray_NewLikeArray(PyArrayObject *prototype, NPY_ORDER order,
/* KEEPORDER needs some analysis of the strides */
else {
npy_intp strides[NPY_MAXDIMS], stride;
npy_intp *shape = PyArray_DIMS(prototype);
npy_stride_sort_item strideperm[NPY_MAXDIMS];
int idim;

PyArray_CreateSortedStridePerm(PyArray_NDIM(prototype),
PyArray_CreateSortedStridePerm(ndim,
PyArray_STRIDES(prototype),
strideperm);

Expand All @@ -1259,14 +1267,14 @@ PyArray_NewLikeArray(PyArrayObject *prototype, NPY_ORDER order,
for (idim = ndim-1; idim >= 0; --idim) {
npy_intp i_perm = strideperm[idim].perm;
strides[i_perm] = stride;
stride *= shape[i_perm];
stride *= dims[i_perm];
}
Copy link
Member
@eric-wieser eric-wieser Apr 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My 1.ii proposal would look like changing everything between the new line 1261 and here to:

if (PyArray_NDIM(prototype) >= ndim) {
    int leading_dims = PyArray_NDIM(prototype) - ndim;

    /* Use only the trailing strides */
    PyArray_CreateSortedStridePerm(ndim,
                                   PyArray_STRIDES(prototype) + leading_dims,
                                   strideperm);
    /* Build the new strides */
    stride = dtype->elsize;
    for (idim = ndim-1; idim >= 0; --idim) {
        npy_intp i_perm = strideperm[idim].perm;
        strides[i_perm] = stride;
        stride *= shape[i_perm];
    }
}
else {
    int leading_dims = ndim - PyArray_NDIM(prototype);
    /* Use all the strides */
    PyArray_CreateSortedStridePerm(PyArray_NDIM(prototype),
                                   PyArray_STRIDES(prototype),
                                   strideperm);

    /* Build the new trailing strides */
    stride = dtype->elsize;
    for (idim = PyArray_NDIM(prototype)-1; idim >= 0; --idim) {
        npy_intp i_perm = strideperm[idim].perm + leading_dims;
        strides[i_perm] = stride;
        stride *= shape[i_perm];
    }

    /* Create remaining leading strides as C order */
    for (idim = leading_dims; idim >= 0; --idim) {
        strides[idim] = stride;
        stride *= shape[idim];
    }
}

Copy link
Member
@eric-wieser eric-wieser Apr 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or trading branching for state:

/* not sure if this is clearer via min/max */
int leading_src_dims = 0;  // max(src.ndim - dst.ndim, 0)
int leading_dst_dims = 0;  // max(dst.ndim - src.ndim, 0)
int shared_dims;  // min(src.ndim, dst.ndim
if (PyArray_NDIM(prototype) >= ndim) {
    shared_dims = ndim;
    leading_src_dims = PyArray_NDIM(prototype) - ndim;
}
else {
    shared_dims = PyArray_NDIM(prototype);
    leading_dst_dims = ndim - PyArray_NDIM(prototype);
}

/* Use only the trailing strides from the source */
PyArray_CreateSortedStridePerm(shared_dims,
                               PyArray_STRIDES(prototype) + leading_src_dims,
                               strideperm);

/* Build the destrination trailing strides */
stride = dtype->elsize;
for (idim = ndim-1; idim >= 0; --idim) {
    npy_intp i_perm = strideperm[idim].perm + leading_dst_dims;
    strides[i_perm] = stride;
    stride *= shape[i_perm];
}

/* Create remaining leading strides as C order */
for (idim = leading_dst_dims; idim >= 0; --idim) {
    strides[idim] = stride;
    stride *= shape[idim];
}


/* Finally, allocate the array */
ret = PyArray_NewFromDescr(subok ? Py_TYPE(prototype) : &PyArray_Type,
dtype,
ndim,
shape,
dims,
strides,
NULL,
0,
Expand All @@ -1276,6 +1284,29 @@ PyArray_NewLikeArray(PyArrayObject *prototype, NPY_ORDER order,
return ret;
}

/*NUMPY_API
* Creates a new array with the same shape as the provided one,
* with possible memory layout order and data type changes.
*
* prototype - The array the new one should be like.
* order - NPY_CORDER - C-contiguous result.
* NPY_FORTRANORDER - Fortran-contiguous result.
* NPY_ANYORDER - Fortran if prototype is Fortran, C otherwise.
* NPY_KEEPORDER - Keeps the axis ordering of prototype.
* dtype - If not NULL, overrides the data type of the result.
* subok - If 1, use the prototype's array subtype, otherwise
* always create a base-class array.
*
* NOTE: If dtype is not NULL, steals the dtype reference. On failure or when
* dtype->subarray is true, dtype will be decrefed.
*/
NPY_NO_EXPORT PyObject *
PyArray_NewLikeArray(PyArrayObject *prototype, NPY_ORDER order,
PyArray_Descr *dtype, int subok)
{
return PyArray_NewLikeArrayWithShape(prototype, order, dtype, 0, NULL, subok);
}

/*NUMPY_API
* Generic new array creation routine.
*/
Expand Down
4 changes: 4 additions & 0 deletions numpy/core/src/multiarray/ctors.h
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ PyArray_NewFromDescr_int(PyTypeObject *subtype, PyArray_Descr *descr, int nd,
int flags, PyObject *obj, PyObject *base, int zeroed,
int allow_emptystring);

NPY_NO_EXPORT PyObject *
PyArray_NewLikeArrayWithShape(PyArrayObject *prototype, NPY_ORDER order,
PyArray_Descr *dtype, int ndim, npy_intp *dims, int subok);

NPY_NO_EXPORT PyObject *PyArray_New(PyTypeObject *, int nd, npy_intp *,
int, npy_intp *, void *, int, int, PyObject *);

Expand Down
23 changes: 14 additions & 9 deletions numpy/core/src/multiarray/multiarraymodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -1758,7 +1758,7 @@ static PyObject *
array_copyto(PyObject *NPY_UNUSED(ignored), PyObject *args, PyObject *kwds)
{

static char *kwlist[] = {"dst","src","casting","where",NULL};
static char *kwlist[] = {"dst", "src", "casting", "where", NULL};
PyObject *wheremask_in = NULL;
PyArrayObject *dst = NULL, *src = NULL, *wheremask = NULL;
NPY_CASTING casting = NPY_SAME_KIND_CASTING;
Expand Down Expand Up @@ -1803,7 +1803,7 @@ static PyObject *
array_empty(PyObject *NPY_UNUSED(ignored), PyObject *args, PyObject *kwds)
{

static char *kwlist[] = {"shape","dtype","order",NULL};
static char *kwlist[] = {"shape", "dtype", "order", NULL};
PyArray_Descr *typecode = NULL;
PyArray_Dims shape = {NULL, 0};
NPY_ORDER order = NPY_CORDER;
Expand Down Expand Up @@ -1846,23 +1846,28 @@ static PyObject *
array_empty_like(PyObject *NPY_UNUSED(ignored), PyObject *args, PyObject *kwds)
{

static char *kwlist[] = {"prototype","dtype","order","subok",NULL};
static char *kwlist[] = {"prototype", "dtype", "order", "subok", "shape", NULL};
PyArrayObject *prototype = NULL;
PyArray_Descr *dtype = NULL;
NPY_ORDER order = NPY_KEEPORDER;
PyArrayObject *ret = NULL;
int subok = 1;
PyArray_Dims shape = {NULL, 0};

< F440 /td> if (!PyArg_ParseTupleAndKeywords(args, kwds, "O&|O&O&i:empty_like", kwlist,
if (!PyArg_ParseTupleAndKeywords(args, kwds, "O&|O&O&iO&:empty_like", kwlist,
&PyArray_Converter, &prototype,
&PyArray_DescrConverter2, &dtype,
&PyArray_OrderConverter, &order,
&subok)) {
&subok,
&PyArray_IntpConverter, &shape)) {
goto fail;
}
/* steals the reference to dtype if it's not NULL */
ret = (PyArrayObject *)PyArray_NewLikeArray(prototype,
order, dtype, subok);
ret = (PyArrayObject *)PyArray_NewLikeArrayWithShape(prototype, order, dtype,
shape.len, shape.ptr, subok);
if (!ret) {
goto fail;
}
Py_DECREF(prototype);

return (PyObject *)ret;
Expand All @@ -1881,7 +1886,7 @@ static PyObject *
array_scalar(PyObject *NPY_UNUSED(ignored), PyObject *args, PyObject *kwds)
{

static char *kwlist[] = {"dtype","obj", NULL};
static char *kwlist[] = {"dtype", "obj", NULL};
PyArray_Descr *typecode;
PyObject *obj = NULL, *tmpobj = NULL;
int alloc = 0;
Expand Down Expand Up @@ -1957,7 +1962,7 @@ array_scalar(PyObject *NPY_UNUSED(ignored), PyObject *args, PyObject *kwds)
static PyObject *
array_zeros(PyObject *NPY_UNUSED(ignored), PyObject *args, PyObject *kwds)
{
static char *kwlist[] = {"shape","dtype","order",NULL};
static char *kwlist[] = {"shape", "dtype", "order", NULL};
PyArray_Descr *typecode = NULL;
PyArray_Dims shape = {NULL, 0};
NPY_ORDER order = NPY_CORDER;
Expand Down
29 changes: 29 additions & 0 deletions numpy/core/tests/test_numeric.py
Original file line number Diff line number Diff line change
Expand Up @@ -2157,6 +2157,7 @@ def setup(self):
(np.arange(24).reshape(2, 3, 4).swapaxes(0, 1), None),
(np.arange(24).reshape(4, 3, 2).swapaxes(0, 1), '?'),
]
self.shapes = [(5,), (5,6,), (5,6,7,)]

def compare_array_value(self, dz, value, fill_value):
if value is not None:
Expand Down Expand Up @@ -2222,6 +2223,34 @@ def check_like_function(self, like_function, value, fill_value=False):
assert_equal(dz.dtype, np.dtype(dtype))
self.compare_array_value(dz, value, fill_value)

# Test the 'shape' parameter
for s in self.shapes:
for o in 'CFA':
sz = like_function(d, dtype=dtype, shape=s, order=o,
**fill_kwarg)
assert_equal(sz.shape, s)
if dtype is None:
assert_equal(sz.dtype, d.dtype)
else:
assert_equal(sz.dtype, 727A np.dtype(dtype))
if o == 'C' or (o == 'A' and d.flags.c_contiguous):
assert_(sz.flags.c_contiguous)
elif o == 'F' or (o == 'A' and d.flags.f_contiguous):
assert_(sz.flags.f_contiguous)
self.compare_array_value(sz, value, fill_value)

if (d.ndim != len(s)):
assert_equal(np.argsort(like_function(d, dtype=dtype,
shape=s, order='K',
**fill_kwarg).strides),
np.argsort(np.empty(s, dtype=dtype,
order='C').strides))
else:
assert_equal(np.argsort(like_function(d, dtype=dtype,
shape=s, order='K',
**fill_kwarg).strides),
np.argsort(d.strides))

# Test the 'subok' parameter
class MyNDArray(np.ndarray):
pass
Expand Down
0