From fc56a4d93943c8ac3b047b33b65e5bb554ae62ab Mon Sep 17 00:00:00 2001 From: melissawm Date: Fri, 22 Oct 2021 12:00:44 -0300 Subject: [PATCH 1/5] DOC: Added explanation document on interoperability --- doc/source/user/basics.interoperability.rst | 352 ++++++++++++++++++++ doc/source/user/basics.rst | 1 + tools/refguide_check.py | 1 + 3 files changed, 354 insertions(+) create mode 100644 doc/source/user/basics.interoperability.rst diff --git a/doc/source/user/basics.interoperability.rst b/doc/source/user/basics.interoperability.rst new file mode 100644 index 000000000000..444574e32990 --- /dev/null +++ b/doc/source/user/basics.interoperability.rst @@ -0,0 +1,352 @@ + +*************************** +Interoperability with NumPy +*************************** + +NumPy’s ndarray objects provide both a high-level API for operations on +array-structured data and a concrete implementation of the API based on +`strided in-RAM storage `__. +While this API is powerful and fairly general, its concrete implementation has +limitations. As datasets grow and NumPy becomes used in a variety of new +environments and architectures, there are cases where the strided in-RAM storage +strategy is inappropriate, which has caused different libraries to reimplement +this API for their own uses. This includes GPU arrays (CuPy_), Sparse arrays +(`scipy.sparse`, `PyData/Sparse `_) and parallel arrays (Dask_ arrays) +as well as various NumPy-like implementations in deep learning frameworks, like +TensorFlow_ and PyTorch_. Similarly, there are many projects that build on top +of the NumPy API for labeled and indexed arrays (XArray_), automatic +differentiation (JAX_), masked arrays (`numpy.ma`), physical units +(astropy.units_, pint_, unyt_), among others that add additional functionality +on top of the NumPy API. + +Yet, users still want to work with these arrays using the familiar NumPy API and +re-use existing code with minimal (ideally zero) porting overhead. With this +goal in mind, various protocols are defined for implementations of +multi-dimensional arrays with high-level APIs matching NumPy. + +Using arbitrary objects in NumPy +-------------------------------- + +When NumPy functions encounter a foreign object, they will try (in order): + +1. The buffer protocol, described `in the Python C-API documentation + `__. +2. The ``__array_interface__`` protocol, described + :ref:`in this page `. A precursor to Python’s buffer + protocol, it defines a way to access the contents of a NumPy array from other + C extensions. +3. The ``__array__`` protocol, which asks an arbitrary object to convert itself + into an array. + +For both the buffer and the ``__array_interface__`` protocols, the object +describes its memory layout and NumPy does everything else (zero-copy if +possible). If that’s not possible, the object itself is responsible for +returning a ``ndarray`` from ``__array__()``. + +The array interface +~~~~~~~~~~~~~~~~~~~ + +The :ref:`array interface ` defines a protocol for array-like +objects to re-use each other’s data buffers. Its implementation relies on the +existence of the following attributes or methods: + +- ``__array_interface__``: a Python dictionary containing the shape, the + element type, and optionally, the data buffer address and the strides of an + array-like object; +- ``__array__()``: a method returning the NumPy ndarray view of an array-like + object; +- ``__array_struct__``: a ``PyCapsule`` containing a pointer to a + ``PyArrayInterface`` C-structure. + +The ``__array_interface__`` and ``__array_struct__`` attributes can be inspected +directly: + + >>> import numpy as np + >>> x = np.array([1, 2, 5.0, 8]) + >>> x.__array_interface__ + {'data': (94708397920832, False), 'strides': None, 'descr': [('', '>> x.__array_struct__ + + +The ``__array_interface__`` attribute can also be used to manipulate the object +data in place: + + >>> class wrapper(): + ... pass + ... + >>> arr = np.array([1, 2, 3, 4]) + >>> buf = arr.__array_interface__ + >>> buf + {'data': (140497590272032, False), 'strides': None, 'descr': [('', '>> buf['shape'] = (2, 2) + >>> w = wrapper() + >>> w.__array_interface__ = buf + >>> new_arr = np.array(w, copy=False) + >>> new_arr + array([[1, 2], + [3, 4]]) + +We can check that ``arr`` and ``new_arr`` share the same data buffer: + + >>> new_arr[0, 0] = 1000 + >>> new_arr + array([[1000, 2], + [ 3, 4]]) + >>> arr + array([1000, 2, 3, 4]) + + +The ``__array__`` protocol +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``__array__`` protocol acts as a dispatch mechanism and ensures that any +NumPy-like object (an array, any object exposing the array interface, an object +whose ``__array__`` method returns an array or any nested sequence) that +implements it can be used as a NumPy array. If possible, this will mean using +``__array__`` to create a NumPy ndarray view of the array-like object. +Otherwise, this copies the data into a new ndarray object. This is not optimal, +as coercing arrays into ndarrays may cause performance problems or create the +need for copies and loss of metadata. + +To see an example of a custom array implementation including the use of the +``__array__`` protocol, see `Writing custom array containers +`__. + +Operating on foreign objects without converting +----------------------------------------------- + +Consider the following function. + + >>> import numpy as np + >>> def f(x): + ... return np.mean(np.exp(x)) + +We can apply it to a NumPy ndarray object directly: + + >>> x = np.array([1, 2, 3, 4]) + >>> f(x) + 21.1977562209304 + +We would like this function to work equally well with any NumPy-like array +object. Some of this is possible today with various protocol mechanisms within +NumPy. + +NumPy allows a class to indicate that it would like to handle computations in a +custom-defined way through the following interfaces: + +- ``__array_ufunc__``: allows third-party objects to support and override + :ref:`ufuncs `. +- ``__array_function__``: a catch-all for NumPy functionality that is not + covered by the ``__array_ufunc__`` protocol for universal functions. + +As long as foreign objects implement the ``__array_ufunc__`` or +``__array_function__`` protocols, it is possible to operate on them without the +need for explicit conversion. + +The ``__array_ufunc__`` protocol +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A :ref:`universal function (or ufunc for short) ` is a +“vectorized” wrapper for a function that takes a fixed number of specific inputs +and produces a fixed number of specific outputs. The output of the ufunc (and +its methods) is not necessarily an ndarray, if all input arguments are not +ndarrays. Indeed, if any input defines an ``__array_ufunc__`` method, control +will be passed completely to that function, i.e., the ufunc is overridden. + +A subclass can override what happens when executing NumPy ufuncs on it by +overriding the default ``ndarray.__array_ufunc__`` method. This method is +executed instead of the ufunc and should return either the result of the +operation, or ``NotImplemented`` if the operation requested is not implemented. + +The ``__array_function__`` protocol +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To achieve enough coverage of the NumPy API to support downstream projects, +there is a need to go beyond ``__array_ufunc__`` and implement a protocol that +allows arguments of a NumPy function to take control and divert execution to +another function (for example, a GPU or parallel implementation) in a way that +is safe and consistent across projects. + +The semantics of ``__array_function__`` are very similar to ``__array_ufunc__``, +except the operation is specified by an arbitrary callable object rather than a +ufunc instance and method. For more details, see `NEP 18 +`__. + + +Interoperability examples +------------------------- + +Example: Pandas ``Series`` objects +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Consider the following: + + >>> import pandas as pd + >>> ser = pd.Series([1, 2, 3, 4]) + >>> type(ser) + pandas.core.series.Series + +Now, ``ser`` is **not** a ndarray, but because it +`implements the __array_ufunc__ protocol +`__, +we can apply ufuncs to it as if it were a ndarray: + + >>> np.exp(ser) + 0 2.718282 + 1 7.389056 + 2 20.085537 + 3 54.598150 + dtype: float64 + >>> np.sin(ser) + 0 0.841471 + 1 0.909297 + 2 0.141120 + 3 -0.756802 + dtype: float64 + +We can even do operations with other ndarrays: + + >>> np.add(ser, np.array([5, 6, 7, 8])) + 0 6 + 1 8 + 2 10 + 3 12 + dtype: int64 + >>> f(ser) + 21.1977562209304 + >>> result = ser.__array__() + >>> type(result) + numpy.ndarray + +Example: PyTorch tensors +~~~~~~~~~~~~~~~~~~~~~~~~ + +`PyTorch `__ is an optimized tensor library for deep +learning using GPUs and CPUs. PyTorch arrays are commonly called *tensors*. +Tensors are similar to NumPy’s ndarrays, except that tensors can run on GPUs or +other hardware accelerators. In fact, tensors and NumPy arrays can often share +the same underlying memory, eliminating the need to copy data. + + >>> import torch + >>> data = [[1, 2],[3, 4]] + >>> x_np = np.array(data) + >>> x_tensor = torch.tensor(data) + +Note that ``x_np`` and ``x_tensor`` are different kinds of objects: + + >>> x_np + array([[1, 2], + [3, 4]]) + >>> x_tensor + tensor([[1, 2], + [3, 4]]) + +However, we can treat PyTorch tensors as NumPy arrays without the need for +explicit conversion: + + >>> np.exp(x_tensor) + tensor([[ 2.7183, 7.3891], + [20.0855, 54.5982]], dtype=torch.float64) + +Also, note that the return type of this function is compatible with the initial +data type. + +**Note** PyTorch does not implement ``__array_function__`` or +``__array_ufunc__``. Under the hood, the ``Tensor.__array__()`` method returns a +NumPy ndarray as a view of the tensor data buffer. See `this issue +`__ and the +`__torch_function__ implementation +`__ +for details. + +Example: CuPy arrays +~~~~~~~~~~~~~~~~~~~~ + +CuPy is a NumPy/SciPy-compatible array library for GPU-accelerated computing +with Python. CuPy implements a subset of the NumPy interface by implementing +``cupy.ndarray``, `a counterpart to NumPy ndarrays +`__. + + >>> import cupy as cp + >>> x_gpu = cp.array([1, 2, 3, 4]) + +The ``cupy.ndarray`` object implements the ``__array_ufunc__`` interface. This +enables NumPy ufuncs to be directly operated on CuPy arrays: + + >>> np.mean(np.exp(x_gpu)) + array(21.19775622) + +Note that the return type of these operations is still consistent with the +initial type: + + >>> arr = cp.random.randn(1, 2, 3, 4).astype(cp.float32) + >>> result = np.sum(arr) + >>> print(type(result)) + + +See `this page in the CuPy documentation for details +`__. + +``cupy.ndarray`` also implements the ``__array_function__`` interface, meaning +it is possible to do operations such as + + >>> a = np.random.randn(100, 100) + >>> a_gpu = cp.asarray(a) + >>> qr_gpu = np.linalg.qr(a_gpu) + +CuPy implements many NumPy functions on ``cupy.ndarray`` objects, but not all. +See `the CuPy documentation +`__ +for details. + +Example: Dask arrays +~~~~~~~~~~~~~~~~~~~~ + +Dask is a flexible library for parallel computing in Python. Dask Array +implements a subset of the NumPy ndarray interface using blocked algorithms, +cutting up the large array into many small arrays. This allows computations on +larger-than-memory arrays using multiple cores. + +Dask supports array protocols like ``__array__`` and +``__array_ufunc__``. + + >>> import dask.array as da + >>> x = da.random.normal(1, 0.1, size=(20, 20), chunks=(10, 10)) + >>> np.mean(np.exp(x)) + dask.array + >>> np.mean(np.exp(x)).compute() + 5.090097550553843 + +**Note** Dask is lazily evaluated, and the result from a computation isn’t +computed until you ask for it by invoking ``compute()``. + +See `the Dask array documentation +`__ +and the `scope of Dask arrays interoperability with NumPy arrays +`__ for details. + +Further reading +--------------- + +- `The Array interface + `__ +- `Writing custom array containers + `__. +- `Special array attributes + `__ + (details on the ``__array_ufunc__`` and ``__array_function__`` protocols) +- `NumPy roadmap: interoperability + `__ +- `PyTorch documentation on the Bridge with NumPy + `__ + +.. _CuPy: https://cupy.dev/ +.. _Sparse: https://sparse.pydata.org/ +.. _Dask: https://docs.dask.org/ +.. _TensorFlow: https://www.tensorflow.org/ +.. _PyTorch: https://pytorch.org/ +.. _XArray: http://xarray.pydata.org/ +.. _JAX: https://jax.readthedocs.io/ +.. _astropy.units: https://docs.astropy.org/en/stable/units/ +.. _pint: https://pint.readthedocs.io/ +.. _unyt: https://unyt.readthedocs.io/ diff --git a/doc/source/user/basics.rst b/doc/source/user/basics.rst index affb85db2f0c..c004d8978a46 100644 --- a/doc/source/user/basics.rst +++ b/doc/source/user/basics.rst @@ -20,3 +20,4 @@ fundamental NumPy ideas and philosophy. basics.subclassing basics.ufuncs basics.copies + basics.interoperability diff --git a/tools/refguide_check.py b/tools/refguide_check.py index 21ba5a448dcc..f8f601a04999 100644 --- a/tools/refguide_check.py +++ b/tools/refguide_check.py @@ -138,6 +138,7 @@ 'basics.indexing.rst', 'basics.subclassing.rst', 'basics.types.rst', + 'basics.interoperability.rst', 'misc.rst', ] From 7ce32d6188fcb76ad4790dd9679abdb3b7a6dacf Mon Sep 17 00:00:00 2001 From: melissawm Date: Tue, 9 Nov 2021 17:59:50 -0300 Subject: [PATCH 2/5] Addressing review comments --- doc/source/reference/arrays.classes.rst | 1 + doc/source/reference/arrays.interface.rst | 32 ++--- doc/source/user/basics.interoperability.rst | 122 +++++++++++--------- 3 files changed, 85 insertions(+), 70 deletions(-) diff --git a/doc/source/reference/arrays.classes.rst b/doc/source/reference/arrays.classes.rst index 92c271f6b964..d79c2e78a55d 100644 --- a/doc/source/reference/arrays.classes.rst +++ b/doc/source/reference/arrays.classes.rst @@ -42,6 +42,7 @@ however, of why your subroutine may not be able to handle an arbitrary subclass of an array is that matrices redefine the "*" operator to be matrix-multiplication, rather than element-by-element multiplication. +.. _special-attributes-and-methods: Special attributes and methods ============================== diff --git a/doc/source/reference/arrays.interface.rst b/doc/source/reference/arrays.interface.rst index 6a8c5f9c4d09..25cfa3de1136 100644 --- a/doc/source/reference/arrays.interface.rst +++ b/doc/source/reference/arrays.interface.rst @@ -4,18 +4,18 @@ .. _arrays.interface: -******************* -The Array Interface -******************* +**************************** +The array interface protocol +**************************** .. note:: - This page describes the numpy-specific API for accessing the contents of - a numpy array from other C extensions. :pep:`3118` -- + This page describes the NumPy-specific API for accessing the contents of + a NumPy array from other C extensions. :pep:`3118` -- :c:func:`The Revised Buffer Protocol ` introduces similar, standardized API to Python 2.6 and 3.0 for any extension module to use. Cython__'s buffer array support - uses the :pep:`3118` API; see the `Cython numpy + uses the :pep:`3118` API; see the `Cython NumPy tutorial`__. Cython provides a way to write code that supports the buffer protocol with Python versions older than 2.6 because it has a backward-compatible implementation utilizing the array interface @@ -81,7 +81,8 @@ This approach to the interface consists of the object having an ===== ================================================================ ``t`` Bit field (following integer gives the number of bits in the bit field). - ``b`` Boolean (integer type where all values are only True or False) + ``b`` Boolean (integer type where all values are only ``True`` or + ``False``) ``i`` Integer ``u`` Unsigned integer ``f`` Floating point @@ -141,11 +142,11 @@ This approach to the interface consists of the object having an must be stored by the new object if the memory area is to be secured. - **Default**: None + **Default**: ``None`` **strides** (optional) Either ``None`` to indicate a C-style contiguous array or - a Tuple of strides which provides the number of bytes needed + a tuple of strides which provides the number of bytes needed to jump to the next array element in the corresponding dimension. Each entry must be an integer (a Python :py:class:`int`). As with shape, the values may @@ -156,26 +157,26 @@ This approach to the interface consists of the object having an memory buffer. In this model, the last dimension of the array varies the fastest. For example, the default strides tuple for an object whose array entries are 8 bytes long and whose - shape is ``(10, 20, 30)`` would be ``(4800, 240, 8)`` + shape is ``(10, 20, 30)`` would be ``(4800, 240, 8)``. **Default**: ``None`` (C-style contiguous) **mask** (optional) - None or an object exposing the array interface. All + ``None`` or an object exposing the array interface. All elements of the mask array should be interpreted only as true or not true indicating which elements of this array are valid. The shape of this object should be `"broadcastable" ` to the shape of the original array. - **Default**: None (All array values are valid) + **Default**: ``None`` (All array values are valid) **offset** (optional) An integer offset into the array data region. This can only be used when data is ``None`` or returns a :class:`buffer` object. - **Default**: 0. + **Default**: ``0``. **version** (required) An integer showing the version of the interface (i.e. 3 for @@ -243,6 +244,11 @@ flag is present. returning the :c:type:`PyCapsule`, and configure a destructor to decref this reference. +.. note:: + + :obj:`__array_struct__` is considered legacy and should not be used for new + code. Use the :py:doc:`buffer protocol ` instead. + Type description examples ========================= diff --git a/doc/source/user/basics.interoperability.rst b/doc/source/user/basics.interoperability.rst index 444574e32990..eeb7492efeea 100644 --- a/doc/source/user/basics.interoperability.rst +++ b/doc/source/user/basics.interoperability.rst @@ -3,9 +3,9 @@ Interoperability with NumPy *************************** -NumPy’s ndarray objects provide both a high-level API for operations on +NumPy's ndarray objects provide both a high-level API for operations on array-structured data and a concrete implementation of the API based on -`strided in-RAM storage `__. +:ref:`strided in-RAM storage `. While this API is powerful and fairly general, its concrete implementation has limitations. As datasets grow and NumPy becomes used in a variety of new environments and architectures, there are cases where the strided in-RAM storage @@ -29,44 +29,39 @@ Using arbitrary objects in NumPy When NumPy functions encounter a foreign object, they will try (in order): -1. The buffer protocol, described `in the Python C-API documentation - `__. +1. The buffer protocol, described :py:doc:`in the Python C-API documentation + `. 2. The ``__array_interface__`` protocol, described - :ref:`in this page `. A precursor to Python’s buffer + :ref:`in this page `. A precursor to Python's buffer protocol, it defines a way to access the contents of a NumPy array from other C extensions. -3. The ``__array__`` protocol, which asks an arbitrary object to convert itself - into an array. +3. The ``__array__()`` method, which asks an arbitrary object to convert + itself into an array. For both the buffer and the ``__array_interface__`` protocols, the object describes its memory layout and NumPy does everything else (zero-copy if -possible). If that’s not possible, the object itself is responsible for +possible). If that's not possible, the object itself is responsible for returning a ``ndarray`` from ``__array__()``. -The array interface -~~~~~~~~~~~~~~~~~~~ +The array interface protocol +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The :ref:`array interface ` defines a protocol for array-like -objects to re-use each other’s data buffers. Its implementation relies on the -existence of the following attributes or methods: +The :ref:`array interface protocol ` defines a way for +array-like objects to re-use each other's data buffers. Its implementation +relies on the existence of the following attributes or methods: - ``__array_interface__``: a Python dictionary containing the shape, the element type, and optionally, the data buffer address and the strides of an array-like object; - ``__array__()``: a method returning the NumPy ndarray view of an array-like object; -- ``__array_struct__``: a ``PyCapsule`` containing a pointer to a - ``PyArrayInterface`` C-structure. -The ``__array_interface__`` and ``__array_struct__`` attributes can be inspected -directly: +The ``__array_interface__`` attribute can be inspected directly: >>> import numpy as np >>> x = np.array([1, 2, 5.0, 8]) >>> x.__array_interface__ {'data': (94708397920832, False), 'strides': None, 'descr': [('', '>> x.__array_struct__ - The ``__array_interface__`` attribute can also be used to manipulate the object data in place: @@ -96,21 +91,20 @@ We can check that ``arr`` and ``new_arr`` share the same data buffer: array([1000, 2, 3, 4]) -The ``__array__`` protocol +The ``__array__()`` method ~~~~~~~~~~~~~~~~~~~~~~~~~~ -The ``__array__`` protocol acts as a dispatch mechanism and ensures that any -NumPy-like object (an array, any object exposing the array interface, an object -whose ``__array__`` method returns an array or any nested sequence) that -implements it can be used as a NumPy array. If possible, this will mean using -``__array__`` to create a NumPy ndarray view of the array-like object. -Otherwise, this copies the data into a new ndarray object. This is not optimal, -as coercing arrays into ndarrays may cause performance problems or create the -need for copies and loss of metadata. +The ``__array__()`` method ensures that any NumPy-like object (an array, any +object exposing the array interface, an object whose ``__array__()`` method +returns an array or any nested sequence) that implements it can be used as a +NumPy array. If possible, this will mean using ``__array__()`` to create a NumPy +ndarray view of the array-like object. Otherwise, this copies the data into a +new ndarray object. This is not optimal, as coercing arrays into ndarrays may +cause performance problems or create the need for copies and loss of metadata, +as the original object and any attributes/behavior it may have had, is lost. -To see an example of a custom array implementation including the use of the -``__array__`` protocol, see `Writing custom array containers -`__. +To see an example of a custom array implementation including the use of +``__array__()``, see :ref:`basics.dispatch`. Operating on foreign objects without converting ----------------------------------------------- @@ -121,7 +115,11 @@ Consider the following function. >>> def f(x): ... return np.mean(np.exp(x)) -We can apply it to a NumPy ndarray object directly: +Note that `np.exp` is a :ref:`ufunc `, which means that it +operates on ndarrays in an element-by-element fashion. On the other hand, +`np.mean` operates along one of the array's axes. + +We can apply ``f`` to a NumPy ndarray object directly: >>> x = np.array([1, 2, 3, 4]) >>> f(x) @@ -149,9 +147,13 @@ The ``__array_ufunc__`` protocol A :ref:`universal function (or ufunc for short) ` is a “vectorized” wrapper for a function that takes a fixed number of specific inputs and produces a fixed number of specific outputs. The output of the ufunc (and -its methods) is not necessarily an ndarray, if all input arguments are not +its methods) is not necessarily an ndarray, if not all input arguments are ndarrays. Indeed, if any input defines an ``__array_ufunc__`` method, control -will be passed completely to that function, i.e., the ufunc is overridden. +will be passed completely to that function, i.e., the ufunc is overridden. The +``__array_ufunc__`` method defined on that (non-ndarray) object has access to +the NumPy ufunc. Because ufuncs have a well-defined structure, the foreign +``__array_ufunc__`` method may rely on ufunc attributes like ``.at()``, +``.reduce()``, and others. A subclass can override what happens when executing NumPy ufuncs on it by overriding the default ``ndarray.__array_ufunc__`` method. This method is @@ -169,9 +171,7 @@ is safe and consistent across projects. The semantics of ``__array_function__`` are very similar to ``__array_ufunc__``, except the operation is specified by an arbitrary callable object rather than a -ufunc instance and method. For more details, see `NEP 18 -`__. - +ufunc instance and method. For more details, see :ref:`NEP18`. Interoperability examples ------------------------- @@ -223,7 +223,7 @@ Example: PyTorch tensors `PyTorch `__ is an optimized tensor library for deep learning using GPUs and CPUs. PyTorch arrays are commonly called *tensors*. -Tensors are similar to NumPy’s ndarrays, except that tensors can run on GPUs or +Tensors are similar to NumPy's ndarrays, except that tensors can run on GPUs or other hardware accelerators. In fact, tensors and NumPy arrays can often share the same underlying memory, eliminating the need to copy data. @@ -251,13 +251,22 @@ explicit conversion: Also, note that the return type of this function is compatible with the initial data type. -**Note** PyTorch does not implement ``__array_function__`` or -``__array_ufunc__``. Under the hood, the ``Tensor.__array__()`` method returns a -NumPy ndarray as a view of the tensor data buffer. See `this issue -`__ and the -`__torch_function__ implementation -`__ -for details. +.. admonition:: Warning + + While this mixing of ndarrays and tensors may be convenient, it is not + recommended. It will not work for non-CPU tensors, and will have unexpected + behavior in corner cases. Users should prefer explicitly converting the + ndarray to a tensor. + +.. note:: + + PyTorch does not implement ``__array_function__`` or ``__array_ufunc__``. + Under the hood, the ``Tensor.__array__()`` method returns a NumPy ndarray as + a view of the tensor data buffer. See `this issue + `__ and the + `__torch_function__ implementation + `__ + for details. Example: CuPy arrays ~~~~~~~~~~~~~~~~~~~~ @@ -271,7 +280,8 @@ with Python. CuPy implements a subset of the NumPy interface by implementing >>> x_gpu = cp.array([1, 2, 3, 4]) The ``cupy.ndarray`` object implements the ``__array_ufunc__`` interface. This -enables NumPy ufuncs to be directly operated on CuPy arrays: +enables NumPy ufuncs to be applied to CuPy arrays (this will defer operation to +the matching CuPy CUDA/ROCm implementation of the ufunc): >>> np.mean(np.exp(x_gpu)) array(21.19775622) @@ -307,8 +317,7 @@ implements a subset of the NumPy ndarray interface using blocked algorithms, cutting up the large array into many small arrays. This allows computations on larger-than-memory arrays using multiple cores. -Dask supports array protocols like ``__array__`` and -``__array_ufunc__``. +Dask supports ``__array__()`` and ``__array_ufunc__``. >>> import dask.array as da >>> x = da.random.normal(1, 0.1, size=(20, 20), chunks=(10, 10)) @@ -317,8 +326,10 @@ Dask supports array protocols like ``__array__`` and >>> np.mean(np.exp(x)).compute() 5.090097550553843 -**Note** Dask is lazily evaluated, and the result from a computation isn’t -computed until you ask for it by invoking ``compute()``. +.. note:: + + Dask is lazily evaluated, and the result from a computation isn't computed + until you ask for it by invoking ``compute()``. See `the Dask array documentation `__ @@ -328,13 +339,10 @@ and the `scope of Dask arrays interoperability with NumPy arrays Further reading --------------- -- `The Array interface - `__ -- `Writing custom array containers - `__. -- `Special array attributes - `__ - (details on the ``__array_ufunc__`` and ``__array_function__`` protocols) +- :ref:`arrays.interface` +- :ref:`basics.dispatch` +- :ref:`special-attributes-and-methods` (details on the ``__array_ufunc__`` and + ``__array_function__`` protocols) - `NumPy roadmap: interoperability `__ - `PyTorch documentation on the Bridge with NumPy From c55b5072c09788ece4d67feeed3a7b9dec14d589 Mon Sep 17 00:00:00 2001 From: melissawm Date: Wed, 10 Nov 2021 09:32:15 -0300 Subject: [PATCH 3/5] Adding description of __array_finalize__ and __array_wrap__ --- doc/source/user/basics.interoperability.rst | 105 ++++++++++++++------ doc/source/user/c-info.beyond-basics.rst | 1 + 2 files changed, 78 insertions(+), 28 deletions(-) diff --git a/doc/source/user/basics.interoperability.rst b/doc/source/user/basics.interoperability.rst index eeb7492efeea..a59f8943109a 100644 --- a/doc/source/user/basics.interoperability.rst +++ b/doc/source/user/basics.interoperability.rst @@ -5,29 +5,41 @@ Interoperability with NumPy NumPy's ndarray objects provide both a high-level API for operations on array-structured data and a concrete implementation of the API based on -:ref:`strided in-RAM storage `. -While this API is powerful and fairly general, its concrete implementation has -limitations. As datasets grow and NumPy becomes used in a variety of new -environments and architectures, there are cases where the strided in-RAM storage -strategy is inappropriate, which has caused different libraries to reimplement -this API for their own uses. This includes GPU arrays (CuPy_), Sparse arrays -(`scipy.sparse`, `PyData/Sparse `_) and parallel arrays (Dask_ arrays) -as well as various NumPy-like implementations in deep learning frameworks, like -TensorFlow_ and PyTorch_. Similarly, there are many projects that build on top -of the NumPy API for labeled and indexed arrays (XArray_), automatic -differentiation (JAX_), masked arrays (`numpy.ma`), physical units -(astropy.units_, pint_, unyt_), among others that add additional functionality -on top of the NumPy API. +:ref:`strided in-RAM storage `. While this API is powerful and fairly +general, its concrete implementation has limitations. As datasets grow and NumPy +becomes used in a variety of new environments and architectures, there are cases +where the strided in-RAM storage strategy is inappropriate, which has caused +different libraries to reimplement this API for their own uses. This includes +GPU arrays (CuPy_), Sparse arrays (`scipy.sparse`, `PyData/Sparse `_) +and parallel arrays (Dask_ arrays) as well as various NumPy-like implementations +in deep learning frameworks, like TensorFlow_ and PyTorch_. Similarly, there are +many projects that build on top of the NumPy API for labeled and indexed arrays +(XArray_), automatic differentiation (JAX_), masked arrays (`numpy.ma`), +physical units (astropy.units_, pint_, unyt_), among others that add additional +functionality on top of the NumPy API. Yet, users still want to work with these arrays using the familiar NumPy API and re-use existing code with minimal (ideally zero) porting overhead. With this goal in mind, various protocols are defined for implementations of -multi-dimensional arrays with high-level APIs matching NumPy. +multi-dimensional arrays with high-level APIs matching NumPy. -Using arbitrary objects in NumPy --------------------------------- +Broadly speaking, there are three groups of features used for interoperability +with NumPy: -When NumPy functions encounter a foreign object, they will try (in order): +1. Methods of turning a foreign object into an ndarray; +2. Methods of deferring execution from a NumPy function to another array + library; +3. Methods that use NumPy functions and return an instance of a foreign object. + +We describe these features below. + + +1. Using arbitrary objects in NumPy +----------------------------------- + +The first set of interoperability features from the NumPy API allows foreign +objects to be treated as NumPy arrays whenever possible. When NumPy functions +encounter a foreign object, they will try (in order): 1. The buffer protocol, described :py:doc:`in the Python C-API documentation `. @@ -106,8 +118,12 @@ as the original object and any attributes/behavior it may have had, is lost. To see an example of a custom array implementation including the use of ``__array__()``, see :ref:`basics.dispatch`. -Operating on foreign objects without converting ------------------------------------------------ + +2. Operating on foreign objects without converting +-------------------------------------------------- + +A second set of methods defined by the NumPy API allows us to defer the +execution from a NumPy function to another array library. Consider the following function. @@ -115,9 +131,9 @@ Consider the following function. >>> def f(x): ... return np.mean(np.exp(x)) -Note that `np.exp` is a :ref:`ufunc `, which means that it -operates on ndarrays in an element-by-element fashion. On the other hand, -`np.mean` operates along one of the array's axes. +Note that `np.exp ` is a :ref:`ufunc `, which means +that it operates on ndarrays in an element-by-element fashion. On the other +hand, `np.mean ` operates along one of the array's axes. We can apply ``f`` to a NumPy ndarray object directly: @@ -126,8 +142,7 @@ We can apply ``f`` to a NumPy ndarray object directly: 21.1977562209304 We would like this function to work equally well with any NumPy-like array -object. Some of this is possible today with various protocol mechanisms within -NumPy. +object. NumPy allows a class to indicate that it would like to handle computations in a custom-defined way through the following interfaces: @@ -139,7 +154,7 @@ custom-defined way through the following interfaces: As long as foreign objects implement the ``__array_ufunc__`` or ``__array_function__`` protocols, it is possible to operate on them without the -need for explicit conversion. +need for explicit conversion. The ``__array_ufunc__`` protocol ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -147,7 +162,7 @@ The ``__array_ufunc__`` protocol A :ref:`universal function (or ufunc for short) ` is a “vectorized” wrapper for a function that takes a fixed number of specific inputs and produces a fixed number of specific outputs. The output of the ufunc (and -its methods) is not necessarily an ndarray, if not all input arguments are +its methods) is not necessarily a ndarray, if not all input arguments are ndarrays. Indeed, if any input defines an ``__array_ufunc__`` method, control will be passed completely to that function, i.e., the ufunc is overridden. The ``__array_ufunc__`` method defined on that (non-ndarray) object has access to @@ -173,6 +188,36 @@ The semantics of ``__array_function__`` are very similar to ``__array_ufunc__``, except the operation is specified by an arbitrary callable object rather than a ufunc instance and method. For more details, see :ref:`NEP18`. + +3. Returning foreign objects +---------------------------- + +A third type of feature set is meant to use the NumPy function implementation +and then convert the return value back into an instance of the foreign object. +The ``__array_finalize__`` and ``__array_wrap__`` methods act behind the scenes +to ensure that the return type of a NumPy function can be specified as needed. + +The ``__array_finalize__`` method is the mechanism that NumPy provides to allow +subclasses to handle the various ways that new instances get created. This +method is called whenever the system internally allocates a new array from an +object which is a subclass (subtype) of the ndarray. It can be used to change +attributes after construction, or to update meta-information from the “parent.” + +The ``__array_wrap__`` method “wraps up the action” in the sense of allowing a +subclass to set the type of the return value and update attributes and metadata. +This can be seen as the opposite of the ``__array__`` method. At the end of +every ufunc, this method is called on the input object with the +highest *array priority*, or the output object if one was specified. The +``__array_priority__`` attribute is used to determine what type of object to +return in situations where there is more than one possibility for the Python +type of the returned object. Subclasses may opt to use this method to transform +the output array into an instance of the subclass and update metadata before +returning the array to the user. + +For more information on these methods, see :ref:`basics.subclassing` and +:ref:`specific-array-subtyping`. + + Interoperability examples ------------------------- @@ -218,6 +263,7 @@ We can even do operations with other ndarrays: >>> type(result) numpy.ndarray + Example: PyTorch tensors ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -343,8 +389,11 @@ Further reading - :ref:`basics.dispatch` - :ref:`special-attributes-and-methods` (details on the ``__array_ufunc__`` and ``__array_function__`` protocols) -- `NumPy roadmap: interoperability - `__ +- :ref:`basics.subclassing` (details on the ``__array_wrap__`` and + ``__array_finalize__`` methods) +- :ref:`specific-array-subtyping` (more details on the implementation of + ``__array_finalize__``, ``__array_wrap__`` and ``__array_priority__``) +- :doc:`NumPy roadmap: interoperability ` - `PyTorch documentation on the Bridge with NumPy `__ diff --git a/doc/source/user/c-info.beyond-basics.rst b/doc/source/user/c-info.beyond-basics.rst index 7dd22afbf629..04ca834897e1 100644 --- a/doc/source/user/c-info.beyond-basics.rst +++ b/doc/source/user/c-info.beyond-basics.rst @@ -450,6 +450,7 @@ type(s). In particular, to create a sub-type in C follow these steps: More information on creating sub-types in C can be learned by reading PEP 253 (available at https://www.python.org/dev/peps/pep-0253). +.. _specific-array-subtyping: Specific features of ndarray sub-typing --------------------------------------- From a7ef06d6ceff0f3c618e37025696a79af2e8fed1 Mon Sep 17 00:00:00 2001 From: melissawm Date: Wed, 24 Nov 2021 10:06:44 -0300 Subject: [PATCH 4/5] Improved array_wrap description and added PyTorch example --- doc/source/user/basics.interoperability.rst | 36 +++++++++++++++------ 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/doc/source/user/basics.interoperability.rst b/doc/source/user/basics.interoperability.rst index a59f8943109a..46034e6bf76f 100644 --- a/doc/source/user/basics.interoperability.rst +++ b/doc/source/user/basics.interoperability.rst @@ -203,16 +203,17 @@ method is called whenever the system internally allocates a new array from an object which is a subclass (subtype) of the ndarray. It can be used to change attributes after construction, or to update meta-information from the “parent.” -The ``__array_wrap__`` method “wraps up the action” in the sense of allowing a -subclass to set the type of the return value and update attributes and metadata. -This can be seen as the opposite of the ``__array__`` method. At the end of -every ufunc, this method is called on the input object with the -highest *array priority*, or the output object if one was specified. The +The ``__array_wrap__`` method “wraps up the action” in the sense of allowing any +object (such as user-defined functions) to set the type of its return value and +update attributes and metadata. This can be seen as the opposite of the +``__array__`` method. At the end of every object that implements +``__array_wrap__``, this method is called on the input object with the highest +*array priority*, or the output object if one was specified. The ``__array_priority__`` attribute is used to determine what type of object to return in situations where there is more than one possibility for the Python -type of the returned object. Subclasses may opt to use this method to transform -the output array into an instance of the subclass and update metadata before -returning the array to the user. +type of the returned object. For example, subclasses may opt to use this method +to transform the output array into an instance of the subclass and update +metadata before returning the array to the user. For more information on these methods, see :ref:`basics.subclassing` and :ref:`specific-array-subtyping`. @@ -295,7 +296,7 @@ explicit conversion: [20.0855, 54.5982]], dtype=torch.float64) Also, note that the return type of this function is compatible with the initial -data type. +data type. .. admonition:: Warning @@ -314,6 +315,23 @@ data type. `__ for details. +Note also that we can see ``__array_wrap__`` in action here, even though +``torch.Tensor`` is not a subclass of ndarray:: + + >>> import torch + >>> t = torch.arange(4) + >>> np.abs(t) + tensor([0, 1, 2, 3]) + >>> wrap = lambda self, array: np.asarray(array) + >>> torch.Tensor.__array_wrap__ = wrap + >>> t = torch.arange(4) + >>> np.abs(t) + array([0, 1, 2, 3]) + +PyTorch implements ``__array_wrap__`` to be able to get tensors back from NumPy +functions, and we can modify it directly to control which type of objects are +returned from these functions. + Example: CuPy arrays ~~~~~~~~~~~~~~~~~~~~ From 5eb658519989cab3850d6e4df17ff09587a159b4 Mon Sep 17 00:00:00 2001 From: melissawm Date: Wed, 5 Jan 2022 13:56:18 -0300 Subject: [PATCH 5/5] Remove extra pytorch example --- doc/source/user/basics.interoperability.rst | 5 ----- 1 file changed, 5 deletions(-) diff --git a/doc/source/user/basics.interoperability.rst b/doc/source/user/basics.interoperability.rst index 46034e6bf76f..adad4dab92dd 100644 --- a/doc/source/user/basics.interoperability.rst +++ b/doc/source/user/basics.interoperability.rst @@ -322,11 +322,6 @@ Note also that we can see ``__array_wrap__`` in action here, even though >>> t = torch.arange(4) >>> np.abs(t) tensor([0, 1, 2, 3]) - >>> wrap = lambda self, array: np.asarray(array) - >>> torch.Tensor.__array_wrap__ = wrap - >>> t = torch.arange(4) - >>> np.abs(t) - array([0, 1, 2, 3]) PyTorch implements ``__array_wrap__`` to be able to get tensors back from NumPy functions, and we can modify it directly to control which type of objects are