8000 ENH: Add shape to *_like() array creation by pentschev · Pull Request #13046 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

ENH: Add shape to *_like() array creation #13046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
Apr 25, 2019
Merged

ENH: Add shape to *_like() array creation #13046

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
f885c08
ENH: Added shape argument to *_like() array creation functions
pentschev Feb 26, 2019
5ad61b6
ENH: C backend adjustments for shape argument on *_like()
pentschev Feb 26, 2019
fd3e270
TST: Added test for shape argument in *_like() functions
pentschev Feb 26, 2019
b7202a7
ENH: Added PyArray_NewLikeArrayWithShape()
pentschev Feb 26, 2019
013cbce
BUG: Fix for PyArray_NewLikeArrayWithShape strides and ndim == 0
pentschev Feb 27, 2019
807f512
REL: Updates for C-API, version 1.17.x
pentschev Feb 27, 2019
95bbfd0
Revert "REL: Updates for C-API, version 1.17.x"
pentschev Mar 13, 2019
43b9828
Revert exposing PyArray_NewLikeArrayWithShape on C-API
pentschev Mar 13, 2019
fa7fd75
DOC: fix versionadded for *_like() shape argument
pentschev Mar 13, 2019
d57e6d3
STY: add missing spaces in array initializers
pentschev Mar 13, 2019
f1b3e91
ENH: empty_like raises ValueError
pentschev Mar 16, 2019
d24ac10
TST: test for exception of *_like() functions
pentschev Mar 16, 2019 < 8000 /div>
ec26417
DOC: release note for shape argument in *_like() functions
pentschev Mar 16, 2019
40e7e9e
DOC: fix *_like() documentation on raises
pentschev Mar 17, 2019
928952d
BUG: *_like() raises for non-C/F-layout arrays
pentschev Mar 17, 2019
e394356
TST: change *_like() shapes to prevent NPY_RELAXED_STRIDE_DEBUG=1 fai…
pentschev Mar 17, 2019
d938fb9
Move empty_like() exception to C implementation
pentschev Mar 19, 2019
99a55e0
Merge branch 'master' into add-shape-to-like-array-creation
pentschev Mar 28, 2019
c087801
Update *_like() ValueError documentation
pentschev Apr 4, 2019
db7614b
Merge remote-tracking branch 'upstream/master' into add-shape-to-like…
pentschev Apr 5, 2019
669ea71
Rearrange stride computation for *_like() if new shape and order='K'
pentschev Apr 10, 2019
f53afbe
Change handling of order= for *_like()
pentschev Apr 19, 2019
c3ac08e
Merge branch 'master' into add-shape-to-like-array-creation
pentschev Apr 19, 2019
695b836
Fix *_like() tests
pentschev Apr 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Move empty_like() exception to C implementation
  • Loading branch information
@pentschev
pentschev committed Mar 19, 2019
commit d938fb9d38e092c80fb564f8058d35671b41a8f1
5 changes: 0 additions & 5 deletions numpy/core/multiarray.py
8000
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,11 +139,6 @@ def empty_like(prototype, dtype=None, order=None, subok=None, shape=None):
[ 4.38791518e-305, -2.00000715e+000, 4.17269252e-309]])

"""
if (shape is not None and order not in 'CFA' and
(prototype.ndim != len(shape) or
not (prototype.flags.c_contiguous or prototype.flags.f_contiguous))):
raise ValueError(
"mismatching ndim or non-C/F-layout arrays can not keep order")
return (prototype,)


Expand Down
6 changes: 6 additions & 0 deletions numpy/core/src/multiarray/ctors.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1214,6 +1214,12 @@ PyArray_NewLikeArrayWithShape(PyArrayObject *prototype, NPY_ORDER order,
dims = PyArray_DIMS(prototype);
new_shape = 0;
}
else if (order == NPY_KEEPORDER && (ndim != PyArray_NDIM(prototype) ||
Copy link
Member
@charris charris Apr 4, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This breaks previous behavior which did not require the prototype to be contiguous.

In [1]: a = ones((3,4))[:,::2]

In [2]: a.flags
Out[2]: 
  C_CONTIGUOUS : False
  F_CONTIGUOUS : False
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

In [3]: ones_like(a, order='K')
Out[3]: 
array([[1., 1.],
       [1., 1.],
       [1., 1.]])

I'm not sure why an error should be raised, why not just override the shape and go with C order when a new shape is specified and order == 'K'?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@charris what you're saying is like my original implementation. I modified this based on @eric-wieser's request here #13046 (comment). For me it's fine either way.

Copy link
Member
@eric-wieser eric-wieser Apr 5, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@charris: This shouldn't break any existing code once my comment below is addressed. The second check here makes no sense to me either.

The thinking is that requesting the order (of the strides) to be kept is nonsensical if the the number of strides is not the same.

!(PyArray_IS_C_CONTIGUOUS(prototype) || PyArray_IS_F_CONTIGUOUS(prototype)))) {
PyErr_SetString(PyExc_ValueError,
"mismatching ndim or non-C/F-layout arrays can not keep order");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you checking PyArray_IS_C_CONTIGUOUS and PyArray_IS_F_CONTIGUOUS here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because we can't keep the order when there's no particular memory layout, just like in the discussion here:
#13046 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My current understanding of 'K' is that we try to keep the memory order if possible, not in all circumstances. Indeed, the former code fell back on recomputing the strides when the prototype was neither c/f continguous.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, we should make sure that this does not switch to an error suddenly. I agree that in most cases K probably means if possible.

I think the default may actually be K and not something like None (to mean K if possible), so that K is the most common option? If K was passed explicitly, there might be an argument.

There are some strides where it would be unambiguous, but I am not sure it is worth to bother about those.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless I'm miserably misunderstanding some of the comments, I think that there are two clashing opinions:

  1. Keep order if possible (as my original implementation did), otherwise, fallback to something like C-contiguous (currently backed by @charris and @seberg);
  2. Raise an exception if the order is K but not possible to maintain it (backed by @eric-wieser, previously discussed in ENH: Add shape to *_like() array creation #13046 (comment)).

Feel free to correct me if I'm really misunderstanding it, otherwise, it would be nice to have here @charris, @eric-wieser and @seberg commenting here to reach an agreement before I do any further changes regarding this matter.

Copy link
Contributor Author
@pentschev pentschev Apr 5, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eric-wieser could you let me know about your findings then? I'm sorry, but I don't understand what you mean with your previous statement, it may be getting a little too complex for my understanding of NumPy handling of strides and orders.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, here's the case I'm thinking of:

>>> a = np.zeros((20, 30, 40)).transpose((1, 0, 2))[::10,::10,::10]

>>> a.strides
(3200, 96000, 80)

>>> np.copy(a, order='K').strides
(32, 96, 8)

note how the order of the strides is kept.

This is the behavior I would expect for empty_like too

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eric-wieser I believe the new commit I pushed covers now the case you're looking for. Extending the case from your previous comment, this is what we get with this new commit:

>>> np.empty_like(a, shape=(3, 2, 4), order='K').strides
(32, 96, 8)

Can you confirm this is what you had in mind?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That behavior looks correct to me now. Well take one more look at this tonight.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eric-wieser just checking, have you had the chance to have a look at this? Do you think we're good to merge this now?

return ret;
}

/* If no override data type, use the one from the prototype */
if (dtype == NULL) {
Expand Down
3 changes: 3 additions & 0 deletions numpy/core/src/multiarray/multiarraymodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1865,6 +1865,9 @@ array_empty_like(PyObject *NPY_UNUSED(ignored), PyObject *args, PyObject *kwds)
/* steals the reference to dtype if it's not NULL */
ret = (PyArrayObject *)PyArray_NewLikeArrayWithShape(prototype, order, dtype,
shape.len, shape.ptr, subok);
if (!ret) {
goto fail;
}
Py_DECREF(prototype);

return (PyObject *)ret;
Expand Down
0