DOC: Describe __array_func__ in subclassing

This includes the use of super everywhere, and in the brief description of __array_ufunc__ in the reference section.
numpy · charris · Apr 27, 2017 · Nov 6, 2016 · Nov 9, 2016 · Nov 12, 2016
commit 71201d286c38157f25325096286b2bc6089fbf84
diff --git a/doc/source/reference/arrays.classes.rst b/doc/source/reference/arrays.classes.rst
@@ -43,6 +43,10 @@ NumPy provides several hooks that classes can customize:
 
    .. versionadded:: 1.13
 
+   .. note:: The API is `provisional
+             <https://docs.python.org/3/glossary.html#term-provisional-api>`_,
+             i.e., we do not yet guarantee backward compatibility.
+
    Any class, ndarray subclass or not, can define this method or set it to
    :obj:`None` in order to override the behavior of NumPy's ufuncs. This works
    quite similarly to Python's ``__mul__`` and other binary operation routines.
@@ -70,8 +74,11 @@ NumPy provides several hooks that classes can customize:
    raised.
 
    .. note:: In addition to ufuncs, :func:`__array_ufunc__` also
-      overrides the behavior of :func:`numpy.dot` and :func:`numpy.matmul`
-      even though these are not ufuncs.
+      overrides the behavior of :func:`numpy.dot` and :func:`numpy.matmul`.
+      This even though these are not ufuncs, but they can be thought of as
+      :ref:`generalized universal functions<c-api.generalized-ufuncs>`
+      (which are overridden). We intend to extend this behaviour to other
+      relevant functions.
 
    Like with other methods in python, such as ``__hash__`` and
    ``__iter__``, it is possible to indicate that your class does *not*
@@ -92,17 +99,26 @@ NumPy provides several hooks that classes can customize:
    :class:`ndarray` will unconditionally return :obj:`NotImplemented`,
    so that your reverse methods will get called.
 
-   .. note:: If you subclass :class:`ndarray`, we strongly recommend
-      that you avoid confusion by neither setting
-      :func:`__array_ufunc__` to :obj:`None`, which makes no
-      sense for an array subclass, nor by defining it and also defining
-      reverse methods, which methods will be called by ``CPython`` in
-      preference over the :class:`ndarray` forward methods.
+   .. note:: If you subclass :class:`ndarray`:
+
+      - We strongly recommend that you avoid confusion by neither setting
+        :func:`__array_ufunc__` to :obj:`None`, which makes no sense for
+        an array subclass, nor by defining it and also defining reverse
+        methods, which methods will be called by ``CPython`` in
+        preference over the :class:`ndarray` forward methods.
+      - :class:`ndarray` defines its own :func:`__array_ufunc__`, which
+        corresponds to ``getattr(ufunc, method)(*inputs, **kwargs)``. Hence,
+        a typical override of :func:`__array_ufunc__` would convert any
+        instances of one's own class, pass these on to its superclass using
+        ``super().__array_ufunc__(*inputs, **kwargs)``, and finally return
+        the results after possible back-conversion. This practice ensures
+        that it is possible to have a hierarchy of subclasses. See
+        :ref:`Subclassing ndarray <basics.subclassing>` for details.
 
    .. note:: If a class defines the :func:`__array_ufunc__` method,
       this disables the :func:`__array_wrap__`,
       :func:`__array_prepare__`, :data:`__array_priority__` mechanism
-      described below (which may eventually be deprecated).
+      described below for ufuncs (which may eventually be deprecated).
 
 .. py:method:: class.__array_finalize__(obj)
 

diff --git a/numpy/doc/subclassing.py b/numpy/doc/subclassing.py
@@ -1,5 +1,4 @@
-"""
-=============================
+"""=============================
 Subclassing ndarray in python
 =============================
 
@@ -220,8 +219,9 @@ class other than the class in which it is defined, the ``__init__``
 * For the explicit constructor call, our subclass will need to create a
   new ndarray instance of its own class.  In practice this means that
   we, the authors of the code, will need to make a call to
-  ``ndarray.__new__(MySubClass,...)``, or do view casting of an existing
-  array (see below)
+  ``ndarray.__new__(MySubClass,...)``, a class-hierarchy prepared call to
+  ``super(MySubClass, cls).__new__(cls, ...)``, or do view casting of an
+  existing array (see below)
 * For view casting and new-from-template, the equivalent of
   ``ndarray.__new__(MySubClass,...`` is called, at the C level.
 
@@ -237,7 +237,7 @@ class other than the class in which it is defined, the ``__init__``
    class C(np.ndarray):
        def __new__(cls, *args, **kwargs):
            print('In __new__ with class %s' % cls)
-           return np.ndarray.__new__(cls, *args, **kwargs)
+           return super(C, cls).__new__(cls, *args, **kwargs)
 
        def __init__(self, *args, **kwargs):
            # in practice you probably will not need or want an __init__
@@ -275,7 +275,8 @@ def __array_finalize__(self, obj):
 
     def __array_finalize__(self, obj):
 
-``ndarray.__new__`` passes ``__array_finalize__`` the new object, of our
+One sees that the ``super`` call, which goes to
+``ndarray.__new__``, passes ``__array_finalize__`` the new object, of our
 own class (``self``) as well as the object from which the view has been
 taken (``obj``).  As you can see from the output above, the ``self`` is
 always a newly created instance of our subclass, and the type of ``obj``
@@ -303,13 +304,14 @@ def __array_finalize__(self, obj):
   class InfoArray(np.ndarray):
 
       def __new__(subtype, shape, dtype=float, buffer=None, offset=0,
-            strides=None, order=None, info=None):
+                  strides=None, order=None, info=None):
           # Create the ndarray instance of our type, given the usual
           # ndarray input arguments.  This will call the standard
           # ndarray constructor, but return an object of our type.
           # It also triggers a call to InfoArray.__array_finalize__
-          obj = np.ndarray.__new__(subtype, shape, dtype, buffer, offset, strides,
-                           order)
+          obj = super(InfoArray, subtype).__new__(subtype, shape, dtype,
+                                                  buffer, offset, strides,
+                                                  order)
           # set the new 'info' attribute to the value passed
           obj.info = info
           # Finally, we must return the newly created object:
@@ -412,15 +414,132 @@ def __array_finalize__(self, obj):
   >>> v.info
   'information'
 
-.. _array-wrap:
+.. _array-ufunc:
+
+``__array_ufunc__`` for ufuncs
+------------------------------
+
+  .. versionadded:: 1.13
+
+A subclass can override what happens when executing numpy ufuncs on it by
+overriding the default ``ndarray.__array_ufunc__`` method. This method is
+executed *instead* of the ufunc and should return either the result of the
+operation, or :obj:`NotImplemented` if the operation requested is not
+implemented.
+
+The signature of ``__array_ufunc__`` is::
+
+    def __array_ufunc__(ufunc, method, *inputs, **kwargs):
 
-``__array_wrap__`` for ufuncs
--------------------------------------------------------
+   - *ufunc* is the ufunc object that was called.
+   - *method* is a string indicating which Ufunc method was called
+     (one of ``"__call__"``, ``"reduce"``, ``"reduceat"``,
+     ``"accumulate"``, ``"outer"``, ``"inner"``).
+   - *inputs* is a tuple of the input arguments to the ``ufunc``.
+   - *kwargs* is a dictionary containing the optional input arguments
+     of the ufunc. If given, any ``out`` arguments, both positional
+     and keyword, are passed as a :obj:`tuple` in *kwargs*.
 
-``__array_wrap__`` gets called at the end of numpy ufuncs and other numpy
-functions, to allow a subclass to set the type of the return value
-and update attributes and metadata. Let's show how this works with an example.
-First we make the same subclass as above, but with a different name and
+A typical implementation would convert any inputs or ouputs that are
+instances of one's own class, pass everything on to a superclass using
+``super()``, and finally return the results after possible
+back-conversion. An example, taken from the test case
+``test_ufunc_override_with_super`` in ``core/tests/test_umath.pu``, is the
+following.
+
+.. testcode::
+
+    input numpy as np
+
+    class A(np.ndarray):
+        def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
+            args = []
+            in_no = []
+            for i, input_ in enumerate(inputs):
+                if isinstance(input_, A):
+                    in_no.append(i)
+                    args.append(input_.view(np.ndarray))
+                else:
+                    args.append(input_)
+
+            outputs = kwargs.pop('out', [])
+            out_no = []
+            if outputs:
+                out_args = []
+                for j, output in enumerate(outputs):
+                    if isinstance(output, A):
+                        out_no.append(j)
+                        out_args.append(output.view(np.ndarray))
+                    else:
+                        out_args.append(output)
+                kwargs['out'] = tuple(out_args)
+
+            info = {key: no for (key, no) in (('inputs', in_no),
+                                              ('outputs', out_no))
+                    if no != []}
+
+            results = super(A, self).__array_ufunc__(ufunc, method,
+                                                     *args, **kwargs)
+            if not isinstance(results, tuple):
+                if not isinstance(results, np.ndarray):
+                    return results
+                results = (results,)
+
+            if outputs == []:
+                outputs = [None] * len(results)
+            results = tuple(result.view(A) if output is None else output
+                            for result, output in zip(results, outputs))
+            if isinstance(results[0], A):
+                results[0].info = info
+
+            return results[0] if len(results) == 1 else results
+
+So, this class does not actually do anything interesting: it just
+converts any instances of its own to regular ndarray (otherwise, we'd
+get infinite recursion!), and adds an ``info`` dictionary that tells
+which inputs and outputs it converted. Hence, e.g.,
+
+>>> a = np.arange(5.).view(A)
+>>> b = np.sin(a)
+>>> b.info
+{'inputs': [0]}
+>>> b = np.sin(np.arange(5.), out=(a,))
+>>> b.info
+{'outputs': [0]}
+>>> a = np.arange(5.).view(A)
+>>> b = np.ones(1).view(A)
+>>> a += b
+>>> a.info
+{'inputs': [0, 1], 'outputs': [0]}
+
+Note that one might also consider just doing ``getattr(ufunc,
+methods)(*inputs, **kwargs)`` instead of the ``super`` call. This would
+work (indeed, ``ndarray.__array_ufunc__`` effectively does just that), but
+by using ``super`` one can more easily have a class hierarchy.  E.g.,
+suppose we had another class ``B`` that defined ``__array_ufunc__`` and
+then made a subclass ``C`` depending on both, i.e., ``class C(A, B)``
+without yet another ``__array_ufunc__`` override.  Then any ufunc on an
+instance of ``C`` would pass on to ``A.__array_ufunc__``, the ``super``
+call in ``A`` would go to ``B.__array_ufunc__``, and the ``super`` call in
+``B`` would go to ``ndarray.__array_ufunc__``.
+
+.. _array-wrap:
+
+``__array_wrap__`` for ufuncs and other functions
+-------------------------------------------------
+
+Prior to numpy 1.13, the behaviour of ufuncs could be tuned using
+``__array_wrap__`` and ``__array_prepare__``. These two allowed one to
+change the output type of a ufunc, but, in constrast to
+``__array_ufunc__``, did not allow one to make any changes to the inputs.
+It is hoped to eventually deprecate these, but ``__array_wrap__`` is also
+used by other numpy functions and methods, such as ``squeeze``, so at the
+present time is still needed for full functionality.
+
+Conceptually, ``__array_wrap__`` "wraps up the action" in the sense of
+allowing a subclass to set the type of the return value and update
+attributes and metadata.  Let's show how this works with an example.  First
+we return to the simpler example subclass, but with a different name and
 some print statements:
 
 .. testcode::
@@ -446,7 +565,7 @@ def __array_wrap__(self, out_arr, context=None):
           print('   self is %s' % repr(self))
           print('   arr is %s' % repr(out_arr))
           # then just call the parent
-          return np.ndarray.__array_wrap__(self, out_arr, context)
+          return super(MySubClass, self).__array_wrap__(self, out_arr, context)
 
 We run a ufunc on an instance of our new a
4D1F
rray:
 
@@ -467,13 +586,12 @@ def __array_wrap__(self, out_arr, context=None):
 >>> ret.info
 'spam'
 
-Note that the ufunc (``np.add``) has called the ``__array_wrap__`` method of the
-input with the highest ``__array_priority__`` value, in this case
-``MySubClass.__array_wrap__``, with arguments ``self`` as ``obj``, and
-``out_arr`` as the (ndarray) result of the addition.  In turn, the
-default ``__array_wrap__`` (``ndarray.__array_wrap__``) has cast the
-result to class ``MySubClass``, and called ``__array_finalize__`` -
-hence the copying of the ``info`` attribute.  This has all happened at the C level.
+Note that the ufunc (``np.add``) has called the ``__array_wrap__`` method
+with arguments ``self`` as ``obj``, and ``out_arr`` as the (ndarray) result
+of the addition.  In turn, the default ``__array_wrap__``
+(``ndarray.__array_wrap__``) has cast the result to class ``MySubClass``,
+and called ``__array_finalize__`` - hence the copying of the ``info``
+attribute.  This has all happened at the C level.
 
 But, we could do anything we wanted:
 
@@ -494,11 +612,12 @@ def __array_wrap__(self, arr, context=None):
 So, by defining a specific ``__array_wrap__`` method for our subclass,
 we can tweak the output from ufuncs. The ``__array_wrap__`` method
 requires ``self``, then an argument - which is the result of the ufunc -
-and an optional parameter *context*. This parameter is returned by some
-ufuncs as a 3-element tuple: (name of the ufunc, argument of the ufunc,
-domain of the ufunc). ``__array_wrap__`` should return an instance of
-its containing class.  See the masked array subclass for an
-implementation.
+and an optional parameter *context*. This parameter is returned by
+ufuncs as a 3-element tuple: (name of the ufunc, arguments of the ufunc,
+domain of the ufunc), but is not set by other numpy functions. Though,
+as seen above, it is possible to do otherwise, ``__array_wrap__`` should
+return an instance of its containing class.  See the masked array
+subclass for an implementation.
 
 In addition to ``__array_wrap__``, which is called on the way out of the
 ufunc, there is also an ``__array_prepare__`` method which is called on
@@ -511,10 +630,6 @@ def __array_wrap__(self, arr, context=None):
 Like ``__array_wrap__``, ``__array_prepare__`` must return an ndarray or
 subclass thereof or raise an error.
 
-.. note:: As of numpy 1.13, there also is a new, more powerful method to
-    handle how a subclass deals with ufuncs, ``__array_ufunc__``.  For details,
-    see the reference section.
-
 Extra gotchas - custom ``__del__`` methods and ndarray.base
 -----------------------------------------------------------