8000 BUG: fix concat of Sparse with non-sparse dtypes by jorisvandenbossche · Pull Request #34338 · pandas-dev/pandas · GitHub
[go: up one dir, main page]
Skip to content

BUG: fix concat of Sparse with non-sparse dtypes #34338

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
May 29, 2020
Merged

BUG: fix concat of Sparse with non-sparse dtypes #34338

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
76618a8
add test with behaviour on released pandas
jorisvandenbossche May 23, 2020
29e63a5
fix _get_common_dtype to preserve sparse/categorical behaviour
jorisvandenbossche May 23, 2020
d6373f8
Update pandas/core/dtypes/concat.py
jorisvandenbossche May 29, 2020
7db5a9b
add link
jorisvandenbossche May 29, 2020
b4b48f2
Merge remote-tracking branch 'upstream/master' into concat-sparse-object
jorisvandenbossche May 29, 2020
20e7e4c
fix typing issue
jorisvandenbossche May 29, 2020
50fb8bc
fix import
jorisvandenbossche May 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix typing issue
  • Loading branch information
@jorisvandenbossche
jorisvandenbossche committed May 29, 2020
commit 20e7e4c91cb7e5b46a3c589bd1c356e589c4e405
6 changes: 4 additions & 2 deletions pandas/core/dtypes/concat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
"""
Utility functions related to concat.
"""
from typing import cast

import numpy as np

Expand All @@ -20,7 +21,7 @@
)
from pandas.core.dtypes.generic import ABCCategoricalIndex, ABCRangeIndex, ABCSeries

from pandas.core.arrays import ExtensionArray
from pandas.core.arrays import ExtensionArray, SparseArray
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be creating a circular import? https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=36392&view=logs&j=bef1c175-2c1b-51ae-044a-2437c76fc339&t=770e7bb1-09f5-5ebf-b63b-578d2906aac9

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I already pushed a fix.
I don't really understand why, though, as SparseArray is included in the pandas.core.arrays init, just as ExtensionArray.

from pandas.core.construction import array


Expand Down Expand Up @@ -81,10 +82,11 @@ def _cast_to_common_type(arr: ArrayLike, dtype: DtypeObj) -> ArrayLike:
except ValueError:
return arr.astype(object, copy=False)

if is_sparse(arr.dtype) and not is_sparse(dtype):
if is_sparse(arr) and not is_sparse(dtype):
# problem case: SparseArray.astype(dtype) doesn't follow the specified
# dtype exactly, but converts this to Sparse[dtype] -> first manually
# convert to dense array
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened #34457 for this

arr = cast(SparseArray, arr)
return arr.to_dense().astype(dtype, copy=False)

if (
Expand Down
0