10000 Merge pull request #11175 from mhvk/gufunc-signature-modification2 · numpy/numpy@a2fb23a · GitHub
[go: up one dir, main page]
Skip to content

Commit a2fb23a

Browse files
mattipmattip
authored
Merge pull request #11175 from mhvk/gufunc-signature-modification2
 ENH: Generalized ufunc signature expansion for frozen and flexible dimensions
2 parents 1ba4173 + c8e15ba commit a2fb23a

File tree

13 files changed

+810
-226
lines changed

13 files changed

+810
-226
lines changed

‎doc/release/1.16.0-notes.rst

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,8 @@ for unraveling. ``dims`` remains supported, but is now deprecated.
102102
C API changes
103103
=============
104104

105+
The :c:data:`NPY_API_VERSION` was incremented to 0x0000D since
106+
``core_dim_flags`` and ``core_dim_sizes`` were added to :c:type:`PyUFuncObject`.
105107

106108
New Features
107109
============
@@ -172,6 +174,45 @@ behavior will be appending. This applied to: `LDFLAGS`, `F77FLAGS`,
172174
`F90FLAGS`, `FREEFLAGS`, `FOPT`, `FDEBUG`, and `FFLAGS`. See gh-11525 for more
173175
details.
174176

177+
Generalized ufunc signatures now allow fixed-size dimensions
178+
------------------------------------------------------------
179+
By using a numerical value in the signature of a generalized ufunc, one can
180+
indicate that the given function requires input or output to have dimensions
181+
with the given size. E.g., the signature of a function that converts a polar
182+
angle to a two-dimensional cartesian unit vector would be ``()->(2)``; that
183+
for one that converts two spherical angles to a three-dimensional unit vector
184+
would be ``(),()->(3)``; and that for the cross product of two
185+
three-dimensional vectors would be ``(3),(3)->(3)``.
186+
187+
Note that to the elementary function these dimensions are not treated any
188+
differently from variable ones indicated with a name starting with a letter;
189+
the loop still is passed the corresponding size, but it can now count on that
190+
size being equal to the fixed one given in the signature.
191+
192+
Generalized ufunc signatures now allow flexible dimensions
193+
----------------------------------------------------------
194+
195+
Some functions, in particular numpy's implementation of ``@`` as ``matmul``,
196+
are very similar to generalized ufuncs in that they operate over core
197+
dimensions, but one could not present them as such because they were able to
198+
deal with inputs in which a dimension is missing. To support this, it is now
199+
allowed to postfix a dimension name with a question mark to indicate that the
200+
dimension does not necessarily have to be present.
201+
202+
With this addition, the signature for ``matmul`` can be expressed as
203+
``(m?,n),(n,p?)->(m?,p?)``. This indicates that if, e.g., the second operand
204+
has only one dimension, for the purposes of the elementary function it will be
205+
treated as if that input has core shape ``(n, 1)``, and the output has the
206+
corresponding core shape of ``(m, 1)``. The actual output array, however, has
207+
the flexible dimension removed, i.e., it will have shape ``(..., m)``.
208+
Similarly, if both arguments have only a single dimension, the inputs will be
209+
presented as having shapes ``(1, n)`` and ``(n, 1)`` to the elementary
210+
function, and the output as ``(1, 1)``, while the actual output array returned
211+
will have shape ``()``. In this way, the signature allows one to use a
212+
single elementary function for four related but different signatures,
213+
``(m,n),(n,p)->(m,p)``, ``(n),(n,p)->(p)``, ``(m,n),(n)->(m)`` and
214+
``(n),(n)->()``.
215+
175216
``np.clip`` and the ``clip`` method check for memory overlap
176217
------------------------------------------------------------
177218
The ``out`` argument to these functions is now always tested for memory overlap

‎doc/source/reference/c-api.generalized-ufuncs.rst

Lines changed: 40 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -127,38 +127,56 @@ The formal syntax of signatures is as follows::
127127
<Output arguments> ::= <Argument list>
128128
<Argument list> ::= nil | <Argument> | <Argument> "," <Argument list>
129129
<Argument> ::= "(" <Core dimension list> ")"
130-
<Core dimension list> ::= nil | <Core dimension name> |
131-
<Core dimension name> "," <Core dimension list>
132-
<Core dimension name> ::= valid Python variable name
133-
130+
<Core dimension list> ::= nil | <Core dimension> |
131+
<Core dimension> "," <Core dimension list>
132+
<Core dimension> ::= <Dimension name> <Dimension modifier>
133+
<Dimension name> ::= valid Python variable name | valid integer
134+
<Dimension modifier> ::= nil | "?"
134135

135136
Notes:
136137

137138
#. All quotes are for clarity.
138-
#. Core dimensions that share the same name must have the exact same size.
139+
#. Unmodified core dimensions that share the same name must have the same size.
139140
Each dimension name typically corresponds to one level of looping in the
140141
elementary function's implementation.
141142
#. White spaces are ignored.
143+
#. An integer as a dimension name freezes that dimension to the value.
144+
#. If the name is suffixed with the "?" modifier, the dimension is a core
145+
dimension only if it exists on all inputs and outputs that share it;
146+
otherwise it is ignored (and replaced by a dimension of size 1 for the
147+
elementary function).
142148

143149
Here are some examples of signatures:
144150

145-
+-------------+------------------------+-----------------------------------+
146-
| add | ``(),()->()`` | |
147-
+-------------+------------------------+-----------------------------------+
148-
| sum1d | ``(i)->()`` | |
149-
+-------------+------------------------+-----------------------------------+
150-
| inner1d | ``(i),(i)->()`` | |
151-
+-------------+------------------------+-----------------------------------+
152-
| matmat | ``(m,n),(n,p)->(m,p)`` | matrix multiplication |
153-
+-------------+------------------------+-----------------------------------+
154-
| vecmat | ``(n),(n,p)->(p)`` | vector-matrix multiplication |
155-
+-------------+------------------------+-----------------------------------+
156-
| matvec | ``(m,n),(n)->(m)`` | matrix-vector multiplication |
157-
+-------------+------------------------+-----------------------------------+
158-
| outer_inner | ``(i,t),(j,t)->(i,j)`` | inner over the last dimension, |
159-
| | | outer over the second to last, |
160-
| | | and loop/broadcast over the rest. |
161-
+-------------+------------------------+-----------------------------------+
151+
+-------------+----------------------------+-----------------------------------+
152+
| name | signature | common usage |
153+
+=============+============================+===================================+
154+
| add | ``(),()->()`` | binary ufunc |
155+
+-------------+----------------------------+-----------------------------------+
156+
| sum1d | ``(i)->()`` | reduction |
157+
+-------------+----------------------------+-----------------------------------+
158+
| inner1d | ``(i),(i)->()`` | vector-vector multiplication |
159+
+-------------+----------------------------+-----------------------------------+
160+
| matmat | ``(m,n),(n,p)->(m,p)`` | matrix multiplication |
161+
+-------------+----------------------------+-----------------------------------+
162+
| vecmat | ``(n),(n,p)->(p)`` | vector-matrix multiplication |
163+
+-------------+----------------------------+-----------------------------------+
164+
| matvec | ``(m,n),(n)->(m)`` | matrix-vector multiplication |
165+
+-------------+----------------------------+-----------------------------------+
166+
| matmul | ``(m?,n),(n,p?)->(m?,p?)`` | combination of the four above |
167+
+-------------+----------------------------+-----------------------------------+
168+
| outer_inner | ``(i,t),(j,t)->(i,j)`` | inner over the last dimension, |
169+
| | | outer over the second to last, |
170+
| | | and loop/broadcast over the rest. |
171+
+-------------+----------------------------+-----------------------------------+
172+
| cross1d | ``(3),(3)->(3)`` | cross product where the last |
173+
| | | dimension is frozen and must be 3 |
174+
+-------------+----------------------------+-----------------------------------+
175+
176+
.. _frozen:
177+
178+
The last is an instance of freezing a core dimension and can be used to
179+
improve ufunc performance
162180

163181
C-API for implementing Elementary Functions
164182
-------------------------------------------

‎doc/source/reference/c-api.types-and-structures.rst

Lines changed: 43 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -182,8 +182,18 @@ PyArrayDescr_Type
182182

183183
.. c:type:: PyArray_Descr
184184
185-
The format of the :c:type:`PyArray_Descr` structure that lies at the
186-
heart of the :c:data:`PyArrayDescr_Type` is
185+
The :c:type:`PyArray_Descr` structure lies at the heart of the
186+
:c:data:`PyArrayDescr_Type`. While it is described here for
187+
completeness, it should be considered internal to NumPy and manipulated via
188+
``PyArrayDescr_*`` or ``PyDataType*`` functions and macros. The size of this
189+
structure is subject to change across versions of NumPy. To ensure
190+
compatibility:
191+
192+
- Never declare a non-pointer instance of the struct
193+
- Never perform pointer arithmatic
194+
- Never use ``sizof(PyArray_Descr)``
195+
196+
It has the following structure:
187197

188198
.. code-block:: c
189199
@@ -685,7 +695,14 @@ PyUFunc_Type
685695
the information needed to call the underlying C-code loops that
686696
perform the actual work. While it is described here for completeness, it
687697
should be considered internal to NumPy and manipulated via ``PyUFunc_*``
688-
functions. It has the following structure:
698+
functions. The size of this structure is subject to change across versions
699+
of NumPy. To ensure compatibility:
700+
701+
- Never declare a non-pointer instance of the struct
702+
- Never perform pointer arithmetic
703+
- Never use ``sizeof(PyUFuncObject)``
704+
705+
It has the following structure:
689706
690707
.. code-block:: c
691708
@@ -713,10 +730,13 @@ PyUFunc_Type
713730
char *core_signature;
714731
PyUFunc_TypeResolutionFunc *type_resolver;
715732
PyUFunc_LegacyInnerLoopSelectionFunc *legacy_inner_loop_selector;
716-
void *reserved2;
717733
PyUFunc_MaskedInnerLoopSelectionFunc *masked_inner_loop_selector;
718734
npy_uint32 *op_flags;
719735
npy_uint32 *iter_flags;
736+
/* new in API version 0x0000000D */
737+
npy_intp *core_dim_sizes;
738+
npy_intp *core_dim_flags;
739+
720740
} PyUFuncObject;
721741
722742
.. c:macro: PyUFuncObject.PyObject_HEAD
@@ -776,6 +796,10 @@ PyUFunc_Type
776796
specifies how many different 1-d loops (of the builtin data
777797
types) are available.
778798
799+
.. c:member:: int PyUFuncObject.reserved1
800+
801+
Unused.
802+
779803
.. c:member:: char *PyUFuncObject.name
780804
781805
A string name for the ufunc. This is used dynamically to build
@@ -870,6 +894,21 @@ PyUFunc_Type
870894
871895
Override the default nditer flags for the ufunc.
872896
897+
Added in API version 0x0000000D
898+
899+
.. c:member:: npy_intp *PyUFuncObject.core_dim_sizes
900+
901+
For each distinct core dimension, the possible
902+
:ref:`frozen <frozen>` size if :c:data:`UFUNC_CORE_DIM_SIZE_INFERRED` is 0
903+
904+
.. c:member:: npy_uint32 *PyUFuncObject.core_dim_flags
905+
906+
For each distinct core dimension, a set of ``UFUNC_CORE_DIM*`` flags
907+
908+
- :c:data:`UFUNC_CORE_DIM_CAN_IGNORE` if the dim name ends in ``?``
909+
- :c:data:`UFUNC_CORE_DIM_SIZE_INFERRED` if the dim size will be
910+
determined from the operands and not from a :ref:`frozen <frozen>` signature
911+
873912
PyArrayIter_Type
874913
----------------
875914

‎numpy/core/code_generators/cversions.txt

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,3 +43,5 @@
4343
# PyArray_SetWritebackIfCopyBase and deprecated PyArray_SetUpdateIfCopyBase.
4444
0x0000000c = a1bc756c5782853ec2e3616cf66869d8
4545

46+
# Version 13 (Numpy 1.16) Added fields core_dim_flags and core_dim_sizes to PyUFuncObject
47+
0x0000000d = a1bc756c5782853ec2e3616cf66869d8

‎numpy/core/include/numpy/ufuncobject.h

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,9 +209,32 @@ typedef struct _tagPyUFuncObject {
209209
* set by nditer object.
210210
*/
211211
npy_uint32 iter_flags;
212+
213+
/* New in NPY_API_VERSION 0x0000000D and above */
214+
215+
/*
216+
* for each core_num_dim_ix distinct dimension names,
217+
* the possible "frozen" size (-1 if not frozen).
218+
*/
219+
npy_intp *core_dim_sizes;
220+
221+
/*
222+
* for each distinct core dimension, a set of UFUNC_CORE_DIM* flags
223+
*/
224+
npy_uint32 *core_dim_flags;
225+
226+
227+
212228
} PyUFuncObject;
213229

214230
#include "arrayobject.h"
231+
/* Generalized ufunc; 0x0001 reserved for possible use as CORE_ENABLED */
232+
/* the core dimension's size will be determined by the operands. */
233+
#define UFUNC_CORE_DIM_SIZE_INFERRED 0x0002
234+
/* the core dimension may be absent */
235+
#define UFUNC_CORE_DIM_CAN_IGNORE 0x0004
236+
/* flags inferred during execution */
237+
#define UFUNC_CORE_DIM_MISSING 0x00040000
215238

216239
#define UFUNC_ERR_IGNORE 0
217240
#define UFUNC_ERR_WARN 1

‎numpy/core/setup.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -737,6 +737,7 @@ def get_mathlib_info(*args):
737737
join('src', 'common', 'ucsnarrow.h'),
738738
join('src', 'common', 'ufunc_override.h'),
739739
join('src', 'common', 'umathmodule.h'),
740+
join('src', 'common', 'numpyos.h'),
740741
]
741742

742743
common_src = [
@@ -746,6 +747,7 @@ def get_mathlib_info(*args):
746747
join('src', 'common', 'templ_common.h.src'),
747748
join('src', 'common', 'ucsnarrow.c'),
748749
join('src', 'common', 'ufunc_override.c'),
750+
join('src', 'common', 'numpyos.c'),
749751
]
750752

751753
blas_info = get_info('blas_opt', 0)
@@ -785,7 +787,6 @@ def get_mathlib_info(*args):
785787
join('src', 'multiarray', 'multiarraymodule.h'),
786788
join('src', 'multiarray', 'nditer_impl.h'),
787789
join('src', 'multiarray', 'number.h'),
788-
join('src', 'multiarray', 'numpyos.h'),
789790
join('src', 'multiarray', 'refcount.h'),
790791
join('src', 'multiarray', 'scalartypes.h'),
791792
join('src', 'multiarray', 'sequence.h'),
@@ -851,7 +852,6 @@ def get_mathlib_info(*args):
851852
join('src', 'multiarray', 'nditer_constr.c'),
852853
join('src', 'multiarray', 'nditer_pywrap.c'),
853854
join('src', 'multiarray', 'number.c'),
854-
join('src', 'multiarray', 'numpyos.c'),
855855
join('src', 'multiarray', 'refcount.c'),
856856
join('src', 'multiarray', 'sequence.c'),
857857
join('src', 'multiarray', 'shape.c'),

‎numpy/core/setup_common.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,8 @@
4141
# 0x0000000b - 1.13.x
4242
# 0x0000000c - 1.14.x
4343
# 0x0000000c - 1.15.x
44-
C_API_VERSION = 0x0000000c
44+
# 0x0000000d - 1.16.x
45+
C_API_VERSION = 0x0000000d
4546

4647
class MismatchCAPIWarning(Warning):
4748
pass

‎numpy/core/src/multiarray/numpyos.c renamed to ‎numpy/core/src/common/numpyos.c

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -769,3 +769,31 @@ NumPyOS_ascii_ftoLf(FILE *fp, long double *value)
769769
}
770770
return r;
771771
}
772+
773+
NPY_NO_EXPORT npy_longlong
774+
NumPyOS_strtoll(const char *str, char **endptr, int base)
775+
{
776+
#if defined HAVE_STRTOLL
777+
return strtoll(str, endptr, base);
778+
#elif defined _MSC_VER
779+
return _strtoi64(str, endptr, base);
780+
#else
781+
/* ok on 64 bit posix */
782+
return PyOS_strtol(str, endptr, base);
783+
#endif
784+
}
785+
786+
NPY_NO_EXPORT npy_ulonglong
787+
NumPyOS_strtoull(const char *str, char **endptr, int base)
788+
{
789+
#if defined HAVE_STRTOULL
790+
return strtoull(str, endptr, base);
791+
#elif defined _MSC_VER
792+
return _strtoui64(str, endptr, base);
793+
#else
794+
/* ok on 64 bit posix */
795+
return PyOS_strtoul(str, endptr, base);
796+
#endif
797+
}
798+
799+

‎numpy/core/src/multiarray/numpyos.h renamed to ‎numpy/core/src/common/numpyos.h

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,4 +31,11 @@ NumPyOS_ascii_ftoLf(FILE *fp, long double *value);
3131
NPY_NO_EXPORT int
3232
NumPyOS_ascii_isspace(int c);
3333

34+
/* Convert a string to an int in an arbitrary base */
35+
NPY_NO_EXPORT npy_longlong
36+
NumPyOS_strtoll(const char *str, char **endptr, int base);
37+
38+
/* Convert a string to an int in an arbitrary base */
39+
NPY_NO_EXPORT npy_ulonglong
40+
NumPyOS_strtoull(const char *str, char **endptr, int base);
3441
#endif

‎numpy/core/src/multiarray/arraytypes.c.src

Lines changed: 2 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -150,32 +150,6 @@ MyPyLong_AsUnsigned@Type@ (PyObject *obj)
150150

151151
/**end repeat**/
152152

153-
static npy_longlong
154-
npy_strtoll(const char *str, char **endptr, int base)
155-
{
156-
#if defined HAVE_STRTOLL
157-
return strtoll(str, endptr, base);
158-
#elif defined _MSC_VER
159-
return _strtoi64(str, endptr, base);
160-
#else
161-
/* ok on 64 bit posix */
162-
return PyOS_strtol(str, endptr, base);
163-
#endif
164-
}
165-
166-
static npy_ulonglong
167-
npy_strtoull(const char *str, char **endptr, int base)
168-
{
169-
#if defined HAVE_STRTOULL
170-
return strtoull(str, endptr, base);
171-
#elif defined _MSC_VER
172-
return _strtoui64(str, endptr, base);
173-
#else
174-
/* ok on 64 bit posix */
175-
return PyOS_strtoul(str, endptr, base);
176-
#endif
177-
}
178-
179153
/*
180154
*****************************************************************************
181155
** GETITEM AND SETITEM **
@@ -1796,8 +1770,8 @@ BOOL_scan(FILE *fp, npy_bool *ip, void *NPY_UNUSED(ignore),
17961770
* #type = npy_byte, npy_ubyte, npy_short, npy_ushort, npy_int, npy_uint,
17971771
* npy_long, npy_ulong, npy_longlong, npy_ulonglong,
17981772
* npy_datetime, npy_timedelta#
1799-
* #func = (PyOS_strtol, PyOS_strtoul)*4, npy_strtoll, npy_strtoull,
1800-
* npy_strtoll*2#
1773+
* #func = (PyOS_strtol, PyOS_strtoul)*4, NumPyOS_strtoll, NumPyOS_strtoull,
1774+
* NumPyOS_strtoll*2#
18011775
* #btype = (npy_long, npy_ulong)*4, npy_longlong, npy_ulonglong,
18021776
* npy_longlong*2#
18031777
*/

0 commit comments

Comments
 (0)
0