8000 ENH: Add support for copy modes to NumPy by czgdp1807 · Pull Request #19173 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content

ENH: Add support for copy modes to NumPy #19173

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 65 commits into from
Nov 15, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
496bd1a
Added np.CopyMode
czgdp1807 Jun 5, 2021
3a3d31b
initial work for supporting never_copy
czgdp1807 Jun 5, 2021
be5823f
Addressed reviews and PyArray_CopyConverter defined
czgdp1807 Jun 7, 2021
e16f3ff
np.CopyMode.NEVER gives error if copy cannot be avoided in np.array.a…
czgdp1807 Jun 7, 2021
645fadb
Updated API ready for formal testing
czgdp1807 Jun 7, 2021
8538720
tests for astype added
czgdp1807 Jun 7, 2021
8c00223
Added tests for np.array
czgdp1807 Jun 7, 2021
4e59da1
Testing for copy argument for existing APIs complete
czgdp1807 Jun 7, 2021
6f0f6ac
resolved conflicts
czgdp1807 Jun 7, 2021
6bff3dd
Added some notes for future commits
czgdp1807 Jun 8, 2021
a90880a
RuntimeError -> ValueError
czgdp1807 Jun 8, 2021
5481a8a
enum.IntEnum -> enum.Enum
czgdp1807 Jun 8, 2021
f2e7a81
aligned astype with main branch
czgdp1807 Jun 8, 2021
e828f15
Add tests for scalars
czgdp1807 Jun 8, 2021
1a92ae9
Merge branch 'main' into never_copy
czgdp1807 Jun 8, 2021
91d6693
resolved linting issues
czgdp1807 Jun 8, 2021
7b53a02
Fixed blank line linting issue
czgdp1807 Jun 8, 2021
eedb300
Apply suggestions from code review
czgdp1807 Jun 9, 2021
bc8903d
renaming complete
czgdp1807 Jun 9, 2021
3268a48
fixed failures due to Python 3.8.2
czgdp1807 Jun 9, 2021
ab01330
Merge branch 'main' into never_copy
czgdp1807 Jun 9, 2021
782d823
fixed PYPY errors
czgdp1807 Jun 10, 2021
342e0c5
removed complicated checks for copy_mode
czgdp1807 Jun 10, 2021
a09929b
Merge branch 'main' into never_copy
czgdp1807 Jun 15, 2021
30e3472
interface shifted
czgdp1807 Jun 15, 2021
a6caf9c
Apply suggestions from code review
czgdp1807 Jun 15, 2021
3dcf3a9
resolved conflicts
czgdp1807 Aug 6, 2021
321e028
Shifted to CopyMode to np.array_api
czgdp1807 Aug 7, 2021
844883c
corrected linting issues
czgdp1807 Aug 7, 2021
d995ecc
resolved conflicts
czgdp1807 Aug 7, 2021
0f8d4c5
fixed linting issues
czgdp1807 Aug 7, 2021
698fb6d
Made _CopyMode private
czgdp1807 Aug 18, 2021
b341e4c
Fixed linting issues
czgdp1807 Aug 18, 2021
781d0a7
resolved conflicts
czgdp1807 Sep 3, 2021
45dbdc9
CopyMode added to np.array_api
czgdp1807 Sep 3, 2021
a39312c
fixed linting issues
czgdp1807 Sep 3, 2021
c2acd5b
fixed linting issues
czgdp1807 Sep 3, 2021
56647dd
Addressed reviews
czgdp1807 Sep 4, 2021
c04509e
resolved conflicts
czgdp1807 Nov 2, 2021
790f927
Addressed reviews
czgdp1807 Nov 2, 2021
2677788
Fixed type check
czgdp1807 Nov 2, 2021
d9a9785
Addressed reviews and increased code coverage
czgdp1807 Nov 4, 2021
5cb2d64
Addressed reviews and increased code coverage
czgdp1807 Nov 9, 2021
8b939c9
Intentional RuntimeError
czgdp1807 Nov 9, 2021
7658ad9
Removed dead code
czgdp1807 Nov 10, 2021
2cf561b
Prohibited calling ``__array__`` method in never copy mode
czgdp1807 Nov 10, 2021
37cd05e
Fixed warning and updated docs
czgdp1807 Nov 10, 2021
9f9a348
Apply suggestions from code review
czgdp1807 Nov 10, 2021
946ab24
Updated docs
czgdp1807 Nov 10, 2021
2c51b0a
Merge branch 'never_copy' of https://github.com/czgdp1807/numpy into …
czgdp1807 Nov 10, 2021
5ede7eb
Added release notes entry
czgdp1807 Nov 10, 2021
d1cb662
L[0]->L[False], L[1]->L[True]
czgdp1807 Nov 10, 2021
6c01915
Deleted release notes entry
czgdp1807 Nov 10, 2021
5f1965f
do_copy -> allow_copy
czgdp1807 Nov 10, 2021
05ff102
Apply suggestions from code review
czgdp1807 Nov 10, 2021
68dfa4b
Merge branch 'never_copy' of https://github.com/czgdp1807/numpy into …
czgdp1807 Nov 10, 2021
2db65c9
Addressed reviews
czgdp1807 Nov 10, 2021
d0d75f3
Reverted dead code removal
czgdp1807 Nov 10, 2021
eccb8df
Merge branch 'main' into never_copy
rgommers Nov 12, 2021
4b2cd27
STY: Small style fixups for never-copy changes
seberg Nov 12, 2021
84951a6
MAINT: Remove private CopyMode enum to private header
seberg Nov 12, 2021
f31c4a6
MAINT,BUG: Refactor __array__ and never-copy to move check later
seberg Nov 12, 2021
dea4055
MAINT: Rename `allow_copy` to `never_copy` in never-copy machinery
seberg Nov 12, 2021
9fee0f8
DOC: Slightly extend to docs to note that we assume no-copy buffer pr…
seberg Nov 12, 2021
f058aea
BUG: Fix refcounting issue and missed scalar special case for never-c…
seberg Nov 12, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion numpy/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,8 @@
import warnings

from ._globals import (
ModuleDeprecationWarning, VisibleDeprecationWarning, _NoValue
ModuleDeprecationWarning, VisibleDeprecationWarning,
_NoValue, _CopyMode
)

# We first need to detect if we're being called as part of the numpy setup
Expand Down
14 changes: 10 additions & 4 deletions numpy/__init__.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import mmap
import ctypes as ct
import array as _array
import datetime as dt
import enum
from abc import abstractmethod
from types import TracebackType, MappingProxyType
from contextlib import ContextDecorator
Expand Down Expand Up @@ -1745,7 +1746,7 @@ class ndarray(_ArrayOrScalarCommon, Generic[_ShapeType, _DType_co]):
order: _OrderKACF = ...,
casting: _CastingKind = ...,
subok: bool = ...,
copy: bool = ...,
copy: bool | _CopyMode = ...,
) -> NDArray[_ScalarType]: ...
@overload
def astype(
Expand All @@ -1754,7 +1755,7 @@ class ndarray(_ArrayOrScalarCommon, Generic[_ShapeType, _DType_co]):
order: _OrderKACF = ...,
casting: _CastingKind = ...,
subok: bool = ...,
copy: bool = ...,
copy: bool | _CopyMode = ...,
) -> NDArray[Any]: ...

@overload
Expand Down Expand Up @@ -2495,7 +2496,7 @@ class generic(_ArrayOrScalarCommon):
order: _OrderKACF = ...,
casting: _CastingKind = ...,
subok: bool = ...,
copy: bool = ...,
copy: bool | _CopyMode = ...,
) -> _ScalarType: ...
@overload
def astype(
Expand All @@ -2504,7 +2505,7 @@ class generic(_ArrayOrScalarCommon):
order: _OrderKACF = ...,
casting: _CastingKind = ...,
subok: bool = ...,
copy: bool = ...,
copy: bool | _CopyMode = ...,
) -> Any: ...

# NOTE: `view` will perform a 0D->scalar cast,
Expand Down Expand Up @@ -3253,6 +3254,11 @@ trunc: _UFunc_Nin1_Nout1[L['trunc'], L[7], None]

abs = absolute

class _CopyMode(enum.Enum):
ALWAYS: L[True]
IF_NEEDED: L[False]
NEVER: L[2]

# Warnings
class ModuleDeprecationWarning(DeprecationWarning): ...
class VisibleDeprecationWarning(UserWarning): ...
Expand Down
40 changes: 39 additions & 1 deletion numpy/_globals.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,11 @@ def foo(arg=np._NoValue):
motivated this module.

"""
import enum

__ALL__ = [
'ModuleDeprecationWarning', 'VisibleDeprecationWarning', '_NoValue'
'ModuleDeprecationWarning', 'VisibleDeprecationWarning',
'_NoValue', '_CopyMode'
]


Expand Down Expand Up @@ -89,3 +92,38 @@ def __repr__(self):


_NoValue = _NoValueType()


class _CopyMode(enum.Enum):
"""
An enumeration for the copy modes supported
by numpy.copy() and numpy.array(). The following three modes are supported,

- ALWAYS: This means that a deep copy of the input
array will always be taken.
- IF_NEEDED: This means that a deep copy of the input
array will be taken only if necessary.
- NEVER: This means that the deep copy will never be taken.
If a copy cannot be avoided then a `ValueError` will be
raised.

Note that the buffer-protocol could in theory do copies. NumPy currently
assumes an object exporting the buffer protocol will never do this.
"""

ALWAYS = True
IF_NEEDED = False
NEVER = 2

def __bool__(self):
# For backwards compatiblity
if self == _CopyMode.ALWAYS:
return True

if self == _CopyMode.IF_NEEDED:
return False

raise ValueError(f"{self} is neither True nor False.")


_CopyMode.__module__ = 'numpy'
6 changes: 3 additions & 3 deletions numpy/array_api/_creation_functions.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ def asarray(
*,
dtype: Optional[Dtype] = None,
device: Optional[Device] = None,
copy: Optional[bool] = None,
copy: Optional[Union[bool, np._CopyMode]] = None,
) -> Array:
"""
Array API compatible wrapper for :py:func:`np.asarray <numpy.asarray>`.
Expand All @@ -55,13 +55,13 @@ def asarray(
_check_valid_dtype(dtype)
if device not in ["cpu", None]:
raise ValueError(f"Unsupported device {device!r}")
if copy is False:
if copy in (False, np._CopyMode.IF_NEEDED):
# Note: copy=False is not yet implemented in np.asarray
raise NotImplementedError("copy=False is not yet implemented")
if isinstance(obj, Array):
if dtype is not None and obj.dtype != dtype:
copy = True
if copy is True:
if copy in (True, np._CopyMode.ALWAYS):
return Array._new(np.array(obj._array, copy=True, dtype=dtype))
return obj
if dtype is None and isinstance(obj, int) and (obj > 2 ** 64 or obj < -(2 ** 63)):
Expand Down
16 changes: 11 additions & 5 deletions numpy/array_api/tests/test_creation_functions.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,18 @@ def test_asarray_copy():
a[0] = 0
assert all(b[0] == 1)
assert all(a[0] == 0)
# Once copy=False is implemented, replace this with
# a = asarray([1])
# b = asarray(a, copy=False)
# a[0] = 0
# assert all(b[0] == 0)
a = asarray([1])
b = asarray(a, copy=np._CopyMode.ALWAYS)
a[0] = 0
assert all(b[0] == 1)
assert all(a[0] == 0)
a = asarray([1])
b = asarray(a, copy=np._CopyMode.NEVER)
a[0] = 0
assert all(b[0] == 0)
assert_raises(NotImplementedError, lambda: asarray(a, copy=False))
assert_raises(NotImplementedError,
lambda: asarray(a, copy=np._CopyMode.IF_NEEDED))


def test_arange_errors():
Expand Down
9 changes: 8 additions & 1 deletion numpy/core/include/numpy/ndarraytypes.h
Original file line number Diff line number Diff line change
Expand Up @@ -868,7 +868,7 @@ typedef int (PyArray_FinalizeFunc)(PyArrayObject *, PyObject *);

/*
* Always copy the array. Returned arrays are always CONTIGUOUS,
* ALIGNED, and WRITEABLE.
* ALIGNED, and WRITEABLE. See also: NPY_ARRAY_ENSURENOCOPY = 0x4000.
*
* This flag may be requested in constructor functions.
*/
Expand Down Expand Up @@ -937,6 +937,13 @@ typedef int (PyArray_FinalizeFunc)(PyArrayObject *, PyObject *);
#define NPY_ARRAY_UPDATEIFCOPY 0x1000 /* Deprecated in 1.14 */
#define NPY_ARRAY_WRITEBACKIFCOPY 0x2000

/*
* No copy may be made while converting from an object/array (result is a view)
*
* This flag may be requested in constructor functions.
*/
#define NPY_ARRAY_ENSURENOCOPY 0x4000
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it is bound to get outdated, but maybe we should add a comment which functions honor this? I guess most do, but it seems things are a bit confusing with many of these flags just honored by the CheckFromAny function and not by the FromAny version...


/*
* NOTE: there are also internal flags defined in multiarray/arrayobject.h,
* which start at bit 31 and work down.
Expand Down
11 changes: 6 additions & 5 deletions numpy/core/multiarray.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ from numpy import (
_ModeKind,
_SupportsBuffer,
_IOProtocol,
_CopyMode,
_NDIterFlagsKind,
_NDIterOpFlagsKind,
)
Expand Down Expand Up @@ -177,7 +178,7 @@ def array(
object: _ArrayType,
dtype: None = ...,
*,
copy: bool = ...,
copy: bool | _CopyMode = ...,
order: _OrderKACF = ...,
subok: L[True],
ndmin: int = ...,
Expand All @@ -188,7 +189,7 @@ def array(
object: _ArrayLike[_SCT],
dtype: None = ...,
*,
copy: bool = ...,
copy: bool | _CopyMode = ...,
order: _OrderKACF = ...,
subok: bool = ...,
ndmin: int = ...,
Expand All @@ -199,7 +200,7 @@ def array(
object: object,
dtype: None = ...,
*,
copy: bool = ...,
copy: bool | _CopyMode = ...,
order: _OrderKACF = ...,
subok: bool = ...,
ndmin: int = ...,
Expand All @@ -210,7 +211,7 @@ def array(
object: Any,
dtype: _DTypeLike[_SCT],
*,
copy: bool = ...,
copy: bool | _CopyMode = ...,
order: _OrderKACF = ...,
subok: bool = ...,
ndmin: int = ...,
Expand All @@ -221,7 +222,7 @@ def array(
object: Any,
dtype: DTypeLike,
*,
copy: bool = ...,
copy: bool | _CopyMode = ...,
order: _OrderKACF = ...,
subok: bool = ...,
ndmin: int = ...,
Expand Down
14 changes: 14 additions & 0 deletions numpy/core/src/multiarray/_multiarray_tests.c.src
Original file line number Diff line number Diff line change
Expand Up @@ -2363,6 +2363,17 @@ run_intp_converter(PyObject* NPY_UNUSED(self), PyObject *args)
return tup;
}

/* used to test NPY_ARRAY_ENSURENOCOPY raises ValueError */
static PyObject*
npy_ensurenocopy(PyObject* NPY_UNUSED(self), PyObject* args)
{
int flags = NPY_ARRAY_ENSURENOCOPY;
if (!PyArray_CheckFromAny(args, NULL, 0, 0, flags, NULL)) {
return NULL;
}
Py_RETURN_NONE;
}

static PyMethodDef Multiarray_TestsMethods[] = {
{"argparse_example_function",
(PyCFunction)argparse_example_function,
Expand Down Expand Up @@ -2424,6 +2435,9 @@ static PyMethodDef Multiarray_TestsMethods[] = {
{"npy_discard",
npy_discard,
METH_O, NULL},
{"npy_ensurenocopy",
npy_ensurenocopy,
METH_O, NULL},
{"get_buffer_info",
get_buffer_info,
METH_VARARGS, NULL},
Expand Down
15 changes: 9 additions & 6 deletions numpy/core/src/multiarray/array_coercion.c
F438
Original file line number Diff line number Diff line change
Expand Up @@ -858,6 +858,7 @@ PyArray_AdaptDescriptorToArray(PyArrayObject *arr, PyObject *dtype)
* (Initially it is a pointer to the user-provided head pointer).
* @param fixed_DType User provided fixed DType class
* @param flags Discovery flags (reporting and behaviour flags, see def.)
* @param never_copy Specifies if a copy is allowed during array creation.
* @return The updated number of maximum dimensions (i.e. scalars will set
* this to the current dimensions).
*/
Expand All @@ -866,7 +867,8 @@ PyArray_DiscoverDTypeAndShape_Recursive(
PyObject *obj, int curr_dims, int max_dims, PyArray_Descr**out_descr,
npy_intp out_shape[NPY_MAXDIMS],
coercion_cache_obj ***coercion_cache_tail_ptr,
PyArray_DTypeMeta *fixed_DType, enum _dtype_discovery_flags *flags)
PyArray_DTypeMeta *fixed_DType, enum _dtype_discovery_flags *flags,
int never_copy)
{
PyArrayObject *arr = NULL;
PyObject *seq;
Expand Down Expand Up @@ -924,7 +926,7 @@ PyArray_DiscoverDTypeAndShape_Recursive(
requested_descr = *out_descr;
}
arr = (PyArrayObject *)_array_from_array_like(obj,
requested_descr, 0, NULL);
requested_descr, 0, NULL, never_copy);
if (arr == NULL) {
return -1;
}
Expand Down Expand Up @@ -1118,7 +1120,7 @@ PyArray_DiscoverDTypeAndShape_Recursive(
max_dims = PyArray_DiscoverDTypeAndShape_Recursive(
objects[i], curr_dims + 1, max_dims,
out_descr, out_shape, coercion_cache_tail_ptr, fixed_DType,
flags);
flags, never_copy);

if (max_dims < 0) {
return -1;
Expand Down Expand Up @@ -1158,6 +1160,7 @@ PyArray_DiscoverDTypeAndShape_Recursive(
* The result may be unchanged (remain NULL) when converting a
* sequence with no elements. In this case it is callers responsibility
* to choose a default.
* @param never_copy Specifies that a copy is not allowed.
* @return dimensions of the discovered object or -1 on error.
* WARNING: If (and only if) the output is a single array, the ndim
* returned _can_ exceed the maximum allowed number of dimensions.
Expand All @@ -1170,7 +1173,7 @@ PyArray_DiscoverDTypeAndShape(
npy_intp out_shape[NPY_MAXDIMS],
coercion_cache_obj **coercion_cache,
PyArray_DTypeMeta *fixed_DType, PyArray_Descr *requested_descr,
PyArray_Descr **out_descr)
PyArray_Descr **out_descr, int never_copy)
{
coercion_cache_obj **coercion_cache_head = coercion_cache;
*coercion_cache = NULL;
Expand Down Expand Up @@ -1215,7 +1218,7 @@ PyArray_DiscoverDTypeAndShape(

int ndim = PyArray_DiscoverDTypeAndShape_Recursive(
obj, 0, max_dims, out_descr, out_shape, &coercion_cache,
fixed_DType, &flags);
fixed_DType, &flags, never_copy);
if (ndim < 0) {
goto fail;
}
Expand Down Expand Up @@ -1500,7 +1503,7 @@ _discover_array_parameters(PyObject *NPY_UNUSED(self),
int ndim = PyArray_DiscoverDTypeAndShape(
obj, NPY_MAXDIMS, shape,
&coercion_cache,
fixed_DType, fixed_descriptor, (PyArray_Descr **)&out_dtype);
fixed_DType, fixed_descriptor, (PyArray_Descr **)&out_dtype, 0);
Py_XDECREF(fixed_DType);
Py_XDECREF(fixed_descriptor);
if (ndim < 0) {
Expand Down
2 changes: 1 addition & 1 deletion numpy/core/src/multiarray/array_coercion.h
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ PyArray_DiscoverDTypeAndShape(
npy_intp out_shape[NPY_MAXDIMS],
coercion_cache_obj **coercion_cache,
PyArray_DTypeMeta *fixed_DType, PyArray_Descr *requested_descr,
PyArray_Descr **out_descr);
PyArray_Descr **out_descr, int never_copy);

NPY_NO_EXPORT int
PyArray_ExtractDTypeAndDescriptor(PyObject *dtype,
Expand Down
2 changes: 1 addition & 1 deletion numpy/core/src/multiarray/arrayobject.c
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,7 @@ PyArray_CopyObject(PyArrayObject *dest, PyObject *src_object)
*/
ndim = PyArray_DiscoverDTypeAndShape(src_object,
PyArray_NDIM(dest), dims, &cache,
NPY_DTYPE(PyArray_DESCR(dest)), PyArray_DESCR(dest), &dtype);
NPY_DTYPE(PyArray_DESCR(dest)), PyArray_DESCR(dest), &dtype, 0);
if (ndim < 0) {
return -1;
}
Expand Down
2 changes: 1 addition & 1 deletion numpy/core/src/multiarray/common.c
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ PyArray_DTypeFromObject(PyObject *obj, int maxdims, PyArray_Descr **out_dtype)
int ndim;

ndim = PyArray_DiscoverDTypeAndShape(
obj, maxdims, shape, &cache, NULL, NULL, out_dtype);
obj, maxdims, shape, &cache, NULL, NULL, out_dtype, 0);
if (ndim < 0) {
return -1;
}
Expand Down
Loading
0