8000 BUG: Fix broken pickle in MaskedArray when dtype is object (Return list instead of string when dtype is object) by raghavrv · Pull Request #8122 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

BUG: Fix broken pickle in MaskedArray when dtype is object (Return list instead of string when dtype is object) #8122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Oct 6, 2016
Merged

BUG: Fix broken pickle in MaskedArray when dtype is object (Return list instead of string when dtype is object) #8122

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6b0d1e3
TST add test to check pickling of object dtyped masked array
raghavrv Oct 4, 2016
3669bfa
Serialize to list instead of bytestring for the data array
raghavrv Oct 5, 2016
2ed5002
Test for different masks
raghavrv Oct 5, 2016
66993e0
Test fill value for object dtype too
raghavrv Oct 6, 2016
1fa49f8
Use ndarray.__reduce__ to extract the bytestring/list
raghavrv Oct 6, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 2 additions & 11 deletions numpy/ma/core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5860,17 +5860,8 @@ def __getstate__(self):

"""
cf = 'CF'[self.flags.fnc]
state = (1,
self.shape,
self.dtype,
self.flags.fnc,
self._data.tobytes(cf),
# self._data.tolist(),
getmaskarray(self).tobytes(cf),
# getmaskarray(self).tolist(),
self._fill_value,
)
return state
data_state = super(MaskedArray, self).__reduce__()[2]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shoyer, does this really make more sense given that we're making assumptions about super.__reduce__'s return value, and discarding all elements but one?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty comfortable with this, given that (1) the spec of __reduce__ is mostly prescribed by Python and this MaskedArray is defined in NumPy itself, which means that if any ever changes how ndarray.__reduce__ works our unit tests will catch the issue for MaskedArray.

return data_state + (getmaskarray(self).tobytes(cf), self._fill_value)

def __setstate__(self, state):
"""Restore the internal state of the masked array, for
Expand Down
25 changes: 18 additions & 7 deletions numpy/ma/tests/test_core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -476,13 +476,24 @@ def test_str_repr(self):

def test_pickling(self):
# Tests pickling
a = arange(10)
a[::3] = masked
a.fill_value = 999
a_pickled = pickle.loads(a.dumps())
assert_equal(a_pickled._mask, a._mask)
assert_equal(a_pickled._data, a._data)
assert_equal(a_pickled.fill_value, 999)
for dtype in (int, float, str, object):
a = arange(10).astype(dtype)
a.fill_value = 999

masks = ([0, 0, 0, 1, 0, 1, 0, 1, 0, 1], # partially masked
True, # Fully masked
False) # Fully unmasked

for mask in masks:
a.mask = mask
a_pickled = pickle.loads(a.dumps())
assert_equal(a_pickled._mask, a._mask)
assert_equal(a_pickled._data, a._data)
if dtype in (object, int):
assert_equal(a_pickled.fill_value, 999)
else:
assert_equal(a_pickled.fill_value, dtype(999))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason not to test this for object?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the catch... I missed an else here :( Fixed it in the recent commit...

assert_array_equal(a_pickled.mask, mask)

def test_pickling_subbaseclass(self):
# Test pickling w/ a subclass of ndarray
Expand Down
0