8000 ENH: general concat with ExtensionArrays through find_common_type by jorisvandenbossche · Pull Request #33607 · pandas-dev/pandas · GitHub
[go: up one dir, main page]
Skip to content

ENH: general concat with ExtensionArrays through find_common_type #33607

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
May 2, 2020
Merged

ENH: general concat with ExtensionArrays through find_common_type #33607

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
3464e95
initial find_common_type/_get_common_type + tests for IntegerDtype
jorisvandenbossche Apr 17, 2020
b1d9d68
handle categoricals
jorisvandenbossche Apr 17, 2020
bb398e7
handle sparse
jorisvandenbossche Apr 17, 2020
83fdc91
handle non-initialized CategoricalDtype in find_common_type
jorisvandenbossche Apr 17, 2020
7f2ac2a
handle datetimelike special case
jorisvandenbossche Apr 17, 2020
d0f90de
update docstring and comment
jorisvandenbossche Apr 17, 2020
2d5fcb0
ignore mypy
jorisvandenbossche Apr 17, 2020
a68206b
Merge remote-tracking branch 'upstream/master' into concat-protocol
jorisvandenbossche May 1, 2020
fc98b65
common_type -> commong_dtype
jorisvandenbossche May 1, 2020
b072591
ensure deduplicated list of dtypes is passed
jorisvandenbossche May 1, 2020
91c984a
add very basic base extension test
jorisvandenbossche May 1, 2020
2a2b9d5
document API change
jorisvandenbossche May 1, 2020
8893165
update EA interface docs
jorisvandenbossche May 1, 2020
e19e3ef
add type annotation on find_common_type
jorisvandenbossche May 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
handle datetimelike special case
  • Loading branch information
@jorisvandenbossche
jorisvandenbossche committed Apr 17, 2020
commit 7f2ac2ae43789caf15afd3d77c8e8bca95324031
13 changes: 11 additions & 2 deletions pandas/core/dtypes/concat.py
View file
Open in desktop
DB55
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,16 @@ def _cast_to_common_type(arr: ArrayLike, dtype: DtypeObj) -> ArrayLike:
except ValueError:
return arr.astype(object, copy=False)

if (
isinstance(arr, np.ndarray)
and arr.dtype.kind in ["m", "M"]
and dtype is np.dtype("object")
):
# wrap datetime-likes in EA to ensure astype(object) gives Timestamp/Timedelta
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 quality comment

# this can happen when concat_compat is called directly on arrays (when arrays
# are not coming from Index/Series._values), eg in BlockManager.quantile
arr = array(arr)

if is_extension_array_dtype(dtype):
if isinstance(arr, np.ndarray):
# numpy's astype cannot handle ExtensionDtypes
Expand Down Expand Up @@ -123,7 +133,6 @@ def is_nonempty(x) -> bool:

typs = get_dtype_kinds(to_concat)
_contains_datetime = any(typ.startswith("datetime") for typ in typs)
_contains_period = any(typ.startswith("period") for typ in typs)

all_empty = not len(non_empties)
single_dtype = len({x.dtype for x in to_concat}) == 1
Expand All @@ -140,7 +149,7 @@ def is_nonempty(x) -> bool:
else:
return np.concatenate(to_concat)

elif _contains_datetime or "timedelta" in typs or _contains_period:
elif _contains_datetime or "timedelta" in typs:
return concat_datetime(to_concat, axis=axis, typs=typs)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we do the DTA/TDA casting above, and do isinstance(obj, ExtensionArray) checks, can all of the dt64/td64 cases be handled by the EA code above?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, because they are not using ExtensionDtype.


elif any_ea and axis == 1:
Expand Down
0