8000 BUG: Made SparseDataFrame.fillna() fill all NaNs by kernc · Pull Request #16892 · pandas-dev/pandas · GitHub
[go: up one dir, main page]
Skip to content

BUG: Made SparseDataFrame.fillna() fill all NaNs #16892

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

BUG: Made SparseDataFrame.fillna() fill all NaNs #16892

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4bc01a1
BUG: Made SparseDataFrame.fillna() fill all NaNs
kristopheryahoo Apr 29, 2017
2974232
fixup! BUG: Made SparseDataFrame.fillna() fill all NaNs
kernc Jul 12, 2017
c1cd33e
fixup! BUG: Made SparseDataFrame.fillna() fill all NaNs
kernc Jul 13, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixup! BUG: Made SparseDataFrame.fillna() fill all NaNs
  • Loading branch information
@kernc
kernc committed Jul 12, 2017
commit 297423209cdc5b157f8e1ae2468ee547b51ae043
1 change: 1 addition & 0 deletions doc/source/whatsnew/v0.21.0.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ Groupby/Resample/Rolling
Sparse
^^^^^^

- Bug in :func:`SparseDataFrame.fillna` not filling all NaNs when frame was instantiated from SciPy sparse matrix (:issue:`16112`)


Reshaping
Expand Down
5 changes: 2 additions & 3 deletions pandas/core/sparse/array.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -595,9 +595,8 @@ def fillna(self, value, downcast=None):
if issubclass(self.dtype.type, np.floating):
value = float(value)

new_values = self.sp_values.copy()
new_values[isnull(new_values)] = value
fill_value = value if isnull(self.fill_value) else self.fill_value
new_values = np.where(isnull(self.sp_values), value, self.sp_values)
fill_value = value if self._null_fill_value else self.fill_value

return self._simple_new(new_values, self.sp_index,
fill_value=fill_value)
Expand Down
14 changes: 6 additions & 8 deletions pandas/tests/sparse/test_frame.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1252,7 +1252,6 @@ def test_from_scipy_correct_ordering(spmatrix):
tm.skip_if_no_package('scipy')

arr = np.arange(1, 5).reshape(2, 2)

try:
spm = spmatrix(arr)
assert spm.dtype == arr.dtype
Expand All @@ -1268,9 +1267,9 @@ def test_from_scipy_correct_ordering(spmatrix):
tm.assert_frame_equal(sdf.to_dense(), expected.to_dense())


def test_from_scipy_object_fillna(spmatrix):
def test_from_scipy_fillna(spmatrix):
# GH 16112
tm.skip_if_no_package('scipy', max_version='0.19.0')
tm.skip_if_no_package('scipy')

arr = np.eye(3)
arr[1:, 0] = np.nan
Expand All @@ -1287,12 +1286,11 @@ def test_from_scipy_object_fillna(spmatrix):
sdf = pd.SparseDataFrame(spm).fillna(-1.0)

# Returning frame should fill all nan values with -1.0
expected = pd.SparseDataFrame({0: {0: 1.0, 1: np.nan, 2: np.nan},
1: {0: np.nan, 1: 1.0, 2: np.nan},
2: {0: np.nan, 1: np.nan, 2: 1.0}}
).fillna(-1.0)
expected = pd.SparseDataFrame([[1, -1, -1],
[-1, 1, -1],
[-1, -1, 1.]])

tm.assert_frame_equal(sdf.to_dense(), expected.to_dense())
tm.assert_numpy_array_equal(sdf.values, expected.values)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you change this to assert_sparse_frame_equal

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No because they are not. Or I can if I may either densify them first, or use expected.fillna(). The expected frame, as constructed, contains SparseArrays of shape (3,) whereas sdf of shape (1,) (even after filling). The internals seem intricate, and I'd prefer not to touch them here. :S

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure what you mean. What exactly does the result look like here? IOW show the internal structure. If these are not equal then that is a bigger problem.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, SparseSeries {0: nan, 1: 1, 2: nan} can have the underlying SparseArray of shape (3,) ([nan, 1, nan], sparse block of length 3, start index 0) or of shape (1,) ([1], sparse block of length 1, start index 1).

I did manage something.



class TestSparseDataFrameArithmetic(object):
Expand Down
0