8000 MAINT,ENH: Reorganize buffered iteration setup by seberg · Pull Request #27883 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content

MAINT,ENH: Reorganize buffered iteration setup #27883

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
Dec 17, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
8d69f05
MAINT: Refactor buffered loop to do an initial setup
seberg Nov 28, 2024
7d19f2d
TST: Fixup tests to pass (for now not correct)
seberg Nov 30, 2024
809cf0c
BUG: Avoid a coresize of 0
seberg Dec 2, 2024
9302b30
TST: The nullpointer doesn't print uniformly accross platforms
seberg Dec 2, 2024
54d967f
BUG: Add missing pointer dereference
seberg Dec 2, 2024
87ad5a2
BUG: Fix coreoffset mechanism and unset buf-reuse on different (parti…
seberg Dec 2, 2024
67cddf7
TST: Forgot one more nditer debug_print() fixup
seberg Dec 2, 2024
facd237
Apply suggestions from code review
seberg Dec 2, 2024
22f335c
MAINT: Remove bufnever logic from allocate-arrays (we do it later any…
seberg Dec 2, 2024
ac282fc
MAINT: Rename to BUF_SINGLESTRIDE and use the new gap in the flags
seberg Dec 2, 2024
6ed3b99
MAINT: Remove unnecessary fixed-stride logic
seberg Dec 3, 2024
162a951
MAINT: Fill in correct buffer transferstride always
seberg Dec 3, 2024
0cc37b9
MAINT: Don't use static zero (initialize by caller instead)
seberg Dec 3, 2024
786e546
DOC: Improve the function documentation somewhat.
seberg Dec 3, 2024
d09be24
BUG: Fix typo in new assert
seberg Dec 3, 2024
92ecbaf
MAINT: Reinstante the CONTIG flag (as buggy as it was)
seberg Dec 5, 2024
5bad357
remove debug print (was this part of the sanitizer problem?!)
seberg Dec 5, 2024
67cb5c5
TST: See if deleting a, b makes santizer tests pass
seberg Dec 6, 2024
5508cc5
MAINT: Move buffersize init (new code always limits it to itersize)
seberg Dec 10, 2024
be6749d
DOC: Explain buffer-setup cost more (maybe should rethink it...)
seberg Dec 12, 2024
426934d
DOC: Expand/change iteration explanation based on Marten's review
seberg Dec 15, 2024
1836fd1
MAINT: More review comments (mostly code clarifications/simplifications)
seberg Dec 15, 2024
e758983
MAINT: Smaller changes based on review
seberg Dec 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Apply suggestions from code review
  • Loading branch information
seberg committed Dec 2, 2024
commit facd23702fa3aa2aff5638483c01ebf1a929bb1a
1 change: 0 additions & 1 deletion numpy/_core/src/multiarray/nditer_api.c
Original file line number Diff line number Diff line change
Expand Up @@ -808,7 +808,6 @@ NpyIter_IsFirstVisit(NpyIter *iter, int iop)
* We only need to check the outer level of this two-level loop,
* because of the requirement that EXTERNAL_LOOP be enabled.
*/
// TODO: Do I need to change this? Chance is no, so long REDUCE_OUTERSTRIDES still makes sense!
if (itflags&NPY_ITFLAG_BUFFER) {
NpyIter_BufferData *bufferdata = NIT_BUFFERDATA(iter);
/* The outer reduce loop */
Expand Down
14 changes: 2 additions & 12 deletions numpy/_core/src/multiarray/nditer_constr.c
Original file line number Diff line number Diff line change
Expand Up @@ -2018,7 +2018,6 @@ npyiter_find_buffering_setup(NpyIter *iter)
/*
* Once a reduce operand reaches a ==0/!=0 stride flip, this dimension
* becomes the outer reduce dimension.
* We may continue growing, but only if strides align for any such operand.
*/
int outer_reduce_dim = 0;

Expand Down Expand Up @@ -2051,10 +2050,7 @@ npyiter_find_buffering_setup(NpyIter *iter)
for (iop = 0; iop < nop; iop++) {
/* Check that we set things up nicely (if shape is ever 1) */
assert((axisdata->shape == 1) ? (prev_strides[iop] == strides[iop]) : 1);
/*
* Best case: the strides collaps for this operand, all fine.
* Keep track at which single-stride or outer dims we are.
*/
/* Best case: the strides collapse for this operand. */
if (prev_strides[iop] * prev_shape == strides[iop]) {
if (op_single_stride_dims[iop] == idim) {
op_single_stride_dims[iop] += 1;
Expand Down Expand Up @@ -2095,7 +2091,7 @@ npyiter_find_buffering_setup(NpyIter *iter)
assert(!op_reduce_outer_dim[iop] || op_reduce_outer_dim[iop] == outer_reduce_dim);
}
if (iop != nop) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can this be hit? I don't see a break in the for loop above...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just forgot to delete this. The first version I had broke immediately on finding a reduce op. But I wanted the info for all operands, so now it's breaking later.

/* Including this dimension is invalid due to a reduction. */
/* Including this dimension was invalid due to a reduction. */
break;
}

Expand Down Expand Up @@ -2282,12 +2278,6 @@ npyiter_find_buffering_setup(NpyIter *iter)
best_size = best_coresize * (maximum_size / best_coresize);
}
}
/*
* Set the buffersize to either the:
* - the largest we amount trivially iterate (no buffering!).
* - the largest multiple of the coresize that is smaller than the
* requested/default buffersize.
*/
NIT_BUFFERDATA(iter)->buffersize = best_size;
/* Core size is 0 (unless the user applies a range explicitly). */
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

type? "Core starts at 0 (unless..."

NIT_BUFFERDATA(iter)->coreoffset = 0;
Expand Down
1 change: 0 additions & 1 deletion numpy/_core/src/multiarray/nditer_impl.h
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,6 @@
#define NPY_ITFLAG_REDUCE (1 << 12)
/* Reduce iteration doesn't need to recalculate reduce loops next time */
#define NPY_ITFLAG_REUSE_REDUCE_LOOPS (1 << 13)

/*
* Offset of (combined) ArrayMethod flags for all transfer functions.
* For now, we use the top 8 bits.
Expand Down
1 change: 1 addition & 0 deletions numpy/_core/src/umath/ufunc_object.c
Original file line number Diff line number Diff line change
Expand Up @@ -2493,6 +2493,7 @@ reduce_loop(PyArrayMethod_Context *context,
dataptrs_copy[3] = dataptrs[2];
strides_copy[3] = strides[2];
}

retval = strided_loop(context,
dataptrs_copy, countptr, strides_copy, auxdata);
if (retval < 0) {
Expand Down
3 changes: 1 addition & 2 deletions numpy/_core/tests/test_einsum.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@
import numpy as np
from numpy.testing import (
assert_, assert_equal, assert_array_equal, assert_almost_equal,
assert_raises, suppress_warnings, assert_raises_regex, assert_allclose,
assert_array_almost_equal_nulp
assert_raises, suppress_warnings, assert_raises_regex, assert_allclose
)

# Setup for optimize einsum
Expand Down
4 changes: 2 additions & 2 deletions numpy/_core/tests/test_nditer.py
Original file line number Diff line number Diff line change
Expand Up @@ -2681,8 +2681,8 @@ def test_iter_buffering_reduction():
it = np.nditer([p, None],
['delay_bufalloc', 'reduce_ok', 'buffered', 'external_loop'],
[['readonly'], ['readwrite', 'allocate']],
op_axes=[[-1, 0], [1, 0]],
itershape=(2, 2), op_dtypes=["float64", "int64"])
op_axes=[[-1, 0], [-1, -1]],
itershape=(2, 2))
with it:
it.operands[1].fill(0)
it.reset()
Expand Down
0