8000 REF/BUG/API: factorizing categorical data by TomAugspurger · Pull Request #19938 · pandas-dev/pandas · GitHub
[go: up one dir, main page]
Skip to content

REF/BUG/API: factorizing categorical data #19938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Mar 15, 2018
Merged

REF/BUG/API: factorizing categorical data #19938

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
18de376
REF/BUG/API: factorizing categorical data
TomAugspurger Feb 23, 2018
9ef5be2
Explicit dtype for expected
TomAugspurger Feb 28, 2018
5e52b6f
Clean : imports / remove sort
TomAugspurger Mar 6, 2018
e19ae86
Merge remote-tracking branch 'upstream/master' into categorical-facto…
TomAugspurger Mar 6, 2018
121b682
Restore sort
TomAugspurger Mar 6, 2018
a6bc405
REF: remove sort from Categorical.factorize
TomAugspurger Mar 9, 2018
0bfbc47
Updated comment
TomAugspurger Mar 9, 2018
b25f383
Merge remote-tracking branch 'upstream/master' into categorical-facto…
TomAugspurger Mar 12, 2018
2688c4f
Fixed new sort algo
TomAugspurger Mar 12, 2018
2395a99
Merge remote-tracking branch 'upstream/master' into categorical-facto…
TomAugspurger Mar 12, 2018
ab4f01c
Implement interface
TomAugspurger Mar 13, 2018
65150f4
Merge remote-tracking branch 'upstream/master' into categorical-facto…
TomAugspurger Mar 13, 2018
ad8173b
Merge remote-tracking branch 'upstream/master' into categorical-facto…
TomAugspurger Mar 13, 2018
1e006d1
Merge remote-tracking branch 'upstream/master' into categorical-facto…
TomAugspurger Mar 14, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Restore sort
  • Loading branch information
@TomAugspurger
TomAugspurger committed Mar 6, 2018
commit 121b682ab4c7b912df2a814305560b23467154f4
16 changes: 9 additions & 7 deletions pandas/core/arrays/categorical.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,7 @@
is_scalar,
is_dict_like)

from pandas.core.algorithms import (
factorize, take_1d, unique1d, _factorize_array)
from pandas.core.algorithms import factorize, take_1d, unique1d
from pandas.core.accessor import PandasDelegate
from pandas.core.base import (PandasObject,
NoNewAttributesMixin, _shared_docs)
Expand Down Expand Up @@ -366,10 +365,6 @@ def __init__(self, values, categories=None, ordered=None, dtype=None,
self._dtype = self._dtype.update_dtype(dtype)
self._codes = coerce_indexer_dtype(codes, dtype.categories)

@classmethod
def _constructor_from_sequence(cls, scalars):
return cls(scalars)

@property
def categories(self):
"""The categories of this categorical.
Expand Down Expand Up @@ -2074,11 +2069,13 @@ def unique(self):
take_codes = sorted(take_codes)
return cat.set_categories(cat.categories.take(take_codes))

def factorize(self, na_sentinel=-1):
def factorize(self, sort=False, na_sentinel=-1):
"""Encode the Categorical as an enumerated type.

Parameters
----------
sort : boolean, default False
Sort by values
na_sentinel: int, default -1
Value to mark "not found"

Expand Down Expand Up @@ -2113,6 +2110,7 @@ def factorize(self, na_sentinel=-1):
[a, b]
Categories (2, object): [a, b]
"""
from pandas.core.algorithms import _factorize_array, take_1d

codes = self.codes.astype('int64')
codes[codes == -1] = iNaT
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The interface we have to hashtable.get_labels() is very odd right now, IOW we have a check_null flag which then makes the caller know to substitute values to iNaT (for int64) and know which are the sentinels. This is breaking the abstrastion. Rather would either like to be able to pass in the actual sentinel (not the output sentinel, but that's another confusion). e.g . you would simply pass -1 here.

I think its worth re-factoring this (maybe before this PR), though I suppose could be after.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#20328

Yes, that'd be nicer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we actually want this to be public?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

factorize in general? I don’t see why not. It’s present on series and index.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#19938 (comment) was in reference to the API docs. We whitelist the methods on Categorical that are included in the API docs (just __array__ and from_codes for now).

Expand All @@ -2123,6 +2121,10 @@ def factorize(self, na_sentinel=-1):
uniques = self._constructor(self.categories.take(uniques),
categories=self.categories,
ordered=self.ordered)
if sort:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would remove the sort from here and just do it in factorize part (after the else), otherwise logic is in multiple places here

order = uniques.argsort()
labels = take_1d(order, labels, fill_value=na_sentinel)
uniques = uniques.take(order)
return labels, uniques

def equals(self, other):
Expand Down
0