8000 pivot_table very slow on Categorical data; how about an observed keyword argument? #24923 by benjaminr · Pull Request #24953 · pandas-dev/pandas · GitHub
[go: up one dir, main page]

Skip to content

pivot_table very slow on Categorical data; how about an observed keyword argument? #24923 #24953

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
d1554c2
Update to frame and pivot to accept observed kwarg to pass to groupby…
benjaminr Jan 26, 2019
0662fa3
Addition of whatsnew entry.
benjaminr Jan 26, 2019
6121313
Added change to pass observed fixture to pivot_table test, added chan…
benjaminr Jan 26, 2019
9f93ab9
Added "an" to whatsnew and added example from original issue to the t…
benjaminr Jan 30, 2019
ebe5972
Removed unnecessary sentence.
benjaminr Jan 31, 2019
416e9c8
Test separated into own test. Added issue comment. Updated df var nam…
benjaminr Jan 31, 2019
5c62063
Addition of asv for pivot_table of categorical data with observed key…
benjaminr Jan 31, 2019
8663be2
Resolve PEP8 issue.
benjaminr Jan 31, 2019
a1e3afe
Merge in master from upstream.
benjaminr Feb 2, 2019
22637a3
Minor adjustment to asv entry.
benjaminr Feb 2, 2019
088f277
Merge branch 'master' of github.com:pandas-dev/pandas into feature/pi…
benjaminr Feb 8, 2019
672847b
Merge branch 'master' into PR_TOOL_MERGE_PR_24953
jreback Feb 9, 2019
d97a077
Merge branch 'master' of github.com:pandas-dev/pandas into feature/pi…
benjaminr Feb 9, 2019
9de99fa
Merge branch 'master' of github.com:pandas-dev/pandas into feature/pi…
benjaminr Feb 12, 2019
9a9569f
Merge branch 'master' of github.com:pandas-dev/pandas into feature/pi…
benjaminr Feb 13, 2019
c8e085d
Triggering CI tests.
benjaminr Feb 13, 2019
2516386
Triggering CI tests - attempt 2.
benjaminr Feb 13, 2019
0efeed8
Merge branch 'master' of github.com:pandas-dev/pandas into feature/pi…
benjaminr Mar 2, 2019
13168d2
Merge branch 'master' of github.com:pandas-dev/pandas into feature/pi…
benjaminr Mar 2, 2019
8518833
Merge branch 'master' of github.com:pandas-dev/pandas into feature/pi…
benjaminr Mar 20, 2019
58a8f6e
Merge branch 'master' of github.com:pandas-dev/pandas into feature/pi…
benjaminr Mar 20, 2019
12b8fac
Triggering CI.
benjaminr Mar 20, 2019
09af30b
Addition of test to ensure observed pivots on categorial data are ind…
benjaminr Mar 22, 2019
6df9e6d
Removal of test that is otherwise handled by asv.
benjaminr Mar 23, 2019
a23b5d0
Extra asv benchmark to see difference between pivots on categorical d…
benjaminr Mar 23, 2019
8d50e85
Removal of import time.
benjaminr Mar 23, 2019
3d39dff
Fix pep8 issue.
benjaminr Mar 23, 2019
ee696d9
Merge branch 'master' of github.com:pandas-dev/pandas into feature/pi…
Apr 5, 2019
12c0f82
Merge branch 'master' of github.com:pandas-dev/pandas into feature/pi…
Apr 6, 2019
f586e42
Manual merging of conflicts.
benjaminr Apr 20, 2019
cf7e8f5
Trailing whitespace fix.
benjaminr Apr 20, 2019
a3bcf1a
Setting categorical datatype after calc on expected.
benjaminr Apr 21, 2019
bb7cfef
Merge branch 'master' of github.com:pandas-dev/pandas into feature/pi…
benjaminr Apr 22, 2019
5921646
Empty commit to trigger CI.
benjaminr Apr 22, 2019
3c1720c
Merge branch 'master' of github.com:pandas-dev/pandas into feature/pi…
benjaminr Apr 23, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Test separated into own test. Added issue comment. Updated df var nam…
…es so more explicit in tm assertion call.
  • Loading branch information
benjaminr committed Jan 31, 2019
commit 416e9c85cc8ff676fed92f1760cc9de0158a40d9
12 changes: 7 additions & 5 deletions pandas/tests/reshape/test_pivot.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,22 +65,24 @@ def test_pivot_table(self, observed):
index + [columns])['D'].agg(np.mean).unstack()
tm.assert_frame_equal(table, expected)

def test_pivot_table_categorical_observed(self, observed):
# issue #24923
df = pd.DataFrame({'col1': list('abcde'),
'col2': list('fghij'),
'col3': [1, 2, 3, 4, 5]})

df.col1 = df.col1.astype('category')
df.col2 = df.col1.astype('category')

df_pivot = df.pivot_table(index='col1', values='col3',
expected = df.pivot_table(index='col1', values='col3',
columns='col2', aggfunc=np.sum,
fill_value=0)

df_pivot_observed = df.pivot_table(index='col1', values='col3',
columns='col2', aggfunc=np.sum,
fill_value=0, observed=observed)
result = df.pivot_table(index='col1', values='col3',
columns='col2', aggfunc=np.sum,
fill_value=0, observed=observed)

tm.assert_frame_equal(df_pivot, df_pivot_observed)
tm.assert_frame_equal(result, expected)

def test_pivot_table_nocols(self):
df = DataFrame({'rows': ['a', 'b', 'c'],
Expand Down
0