8000 BUG: don't check alignment of size=0 arrays (RELAXED_STRIDES) by ahaldane · Pull Request #12511 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content

BUG: don't check alignment of size=0 arrays (RELAXED_STRIDES) #12511

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 10, 2018

Conversation

ahaldane
Copy link
Member
@ahaldane ahaldane commented Dec 8, 2018

The copy-loops shouldn't assert pointer alignment for size-0 arrays, especially when RELAXED_STRIDES is turned on.

Fixes #12503

@ahaldane ahaldane 8000 added this to the 1.16.0 release milestone Dec 8, 2018
@charris charris added the 09 - Backport-Candidate PRs tagged should be backported label Dec 8, 2018
# check RELAXED_STRIDES don't trip assertions in NDEBUG mode (gh-12503)
buf = np.empty(4, np.uint8)
buf = buf[2:3][:-1]
data = np.ndarray((0,), np.double, buf, order='C')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably should try this for all the affected dtypes.

@charris
Copy link
Member
charris commented Dec 8, 2018

I think the test needs more coverage but otherwise LGTM. Thanks for the quick patch.

@oleksandr-pavlyk
Copy link
Contributor

The example I provided comes from scipy's test. Specifically scipy._lib.tests.test__utils::test__aligned_zeros. It covers different alignments and different dtypes.

I just thought that having a concise direct example would allow to get to the bottom of the issue faster.

@ahaldane
Copy link
Member Author
ahaldane commented Dec 9, 2018

That seems like test code worth having in numpy itself. I'll see if I can adapt it here...

@ahaldane ahaldane force-pushed the fix_relaxstride_loops branch from 85acd2f to b91f3ef Compare December 9, 2018 19:39
@ahaldane
Copy link
Member Author

Updated with more elaborate test.

Copy link
Contributor
@oleksandr-pavlyk oleksandr-pavlyk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @ahaldane

8000

@tylerjereddy
Copy link
Contributor

Can we / should we update our Travis CI debug run to catch this? I guess we need a few more flags there?

for align in [1, 2, 3, 4, 8, 16, 32, 64, None]:
for n in [0, 1, 3, 11]:
for order in ["C", "F", None]:
for dtype in [np.uint8, np.float64]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd probably test every C type just on general principles, it is a good way to discover unnoticed bugs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you made it a test class that would be easier to do, these last loops could be in a test function parametrized by dtype.

@ahaldane
Copy link
Member Author

@tylerjereddy the CI debug run would have caught this if this new test was present. It seems none of our existing tests tested misaligned copies. Thankfully scipy covered for us.

align = dtype.alignment
if not hasattr(shape, '__len__'):
shape = (shape,)
size = functools.reduce(operator.mul, shape) * dtype.itemsize
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be rewritten as np.prod(shape) * dtype.itemsize

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotta be careful of overflow, merged a fix for that just a bit ago that used dtype=np.intp in the prod, although I might just go with int64.

# In particular, check RELAXED_STRIDES don't trip alignment assertions in
# NDEBUG mode for size-0 arrays (gh-12503)

def aligned_zeros(shape, dtype=float, order="C", align=None):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function exists in 'test_multiarray', can we refactor the two into one?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, good catch.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interestingly, numpy's _aligned_zeros has an extra clause for object types which is missing from scipy's version. Removing it causes a segfault.

So any scipy devs reading this, consider adding that clause.

@ahaldane ahaldane force-pushed the fix_relaxstride_loops branch 2 times, most recently from 11f7055 to 483b5cb Compare December 10, 2018 21:37
@ahaldane
Copy link
Member Author

Updated to test all dtypes except object.

The new test takes 1.8s to run on my system, which is a little long but still acceptable I think. It's a lot of nested loops.

@@ -7923,6 +7923,44 @@ def test_uintalignment_and_alignment():
dst = np.zeros((2,2), dtype='c8')
dst[:,1] = src[:,1] # assert in lowlevel_strided_loops fails?

class TestAlignemnt(object):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: typo in class name

@mattip
Copy link
Member
mattip commented Dec 10, 2018

LGTM check is stuck?

Don't check alignment of size-0 arrays in copy-loops, because of
RELAXED_STRIDES.

Fixes numpy#12503
@ahaldane ahaldane force-pushed the fix_relaxstride_loops branch from 483b5cb to d7ab72d Compare December 10, 2018 22:24
@charris charris changed the title MAINT: don't check alignment size=0 arrays (RELAXED_STRIDES) BUG: don't check alignment size=0 arrays (RELAXED_STRIDES) Dec 10, 2018
@charris charris changed the title BUG: don't check alignment size=0 arrays (RELAXED_STRIDES) BUG: don't check alignment of size=0 arrays (RELAXED_STRIDES) Dec 10, 2018
@charris charris merged commit b40321f into numpy:master Dec 10, 2018
@charris
Copy link
Member
charris commented Dec 10, 2018

Thanks Allan.

@charris charris removed this from the 1.16.0 release milestone Dec 16, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants
0