10000 PERF: Add cache keyword to to_datetime (#11665) by mroeschke · Pull Request #17077 · pandas-dev/pandas · GitHub
[go: up one dir, main page]
Skip to content

PERF: Add cache keyword to to_datetime (#11665) #17077

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 25 commits into from
Nov 11, 2017
Merged

PERF: Add cache keyword to to_datetime (#11665) #17077

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
7f67ac9
move cache into convert_listlike
mroeschke Sep 6, 2017
a7c65f7
Move cache down the stack, explore threshold to trigger cache
mroeschke Sep 9, 2017
243349a
Add more cache conditions
mroeschke Sep 11, 2017
d154a6d
Add some benchmarks
mroeschke Sep 11, 2017
b5e71d2
Some performance testing
mroeschke Sep 14, 2017
fb2e831
Add asvs, modify tests for caches
mroeschke Sep 25, 2017
33c79d3
Fix asv errors and condition
mroeschke Sep 26, 2017
dcaafb6
Pep8 fixes
mroeschke Sep 26, 2017
04df9d9
Remove unused import
mroeschke Sep 26, 2017
34b468f
Wrap cache logic in a function
mroeschke Sep 26, 2017
d287cc6
Fix Series test
mroeschke Sep 26, 2017
1bf4c9d
Add whatsnew and small documentation fix
mroeschke Sep 26, 2017
3ffdd46
pep 8 fixes
mroeschke Sep 26, 2017
a093b88
Move box logic into maybe_convert_cache
mroeschke Oct 9, 2017
d1fc211
Use quicker unique check
mroeschke Oct 20, 2017
9486df3
Move caching function outside to_datetime
mroeschke Nov 4, 2017
d059d44
Pass most tests
mroeschke Nov 5, 2017
02ab4f3
Skip test related to GH 18111, lint
mroeschke Nov 5, 2017
82f36d3
Update docstring
mroeschke Nov 6, 2017
76547e1
adjust imports, docs and move whatsnew
mroeschke Nov 8, 2017
590c9cc
Remove whitespace
mroeschke Nov 8, 2017
9a985ac
Address comments
mroeschke Nov 9, 2017
85a1f2d
Lint fix
mroeschke Nov 9, 2017
49f5850
Move docs and adjust test
mroeschke Nov 10, 2017
07fa22d
Lint
mroeschke Nov 11, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix asv errors and condition
  • Loading branch information
@mroeschke
mroeschke committed Nov 11, 2017
commit 33c79d3218cdf61006ccfdffdd717d72ef0218eb
15 changes: 6 additions & 9 deletions asv_bench/benchmarks/timeseries.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,23 +386,20 @@ def time_format_exact(self):
def time_format_no_exact(self):
to_datetime(self.s, format='%d%b%y', exact=False)

def time_cache_with_unique_seconds_and unit(self):
to_datetime(self.unique_numeric_seconds, unit='s')
def time_cache_with_unique_seconds_and_unit(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think these now need to matrix cache=True/False

to_datetime(self.unique_numeric_seconds, unit='s')

def time_cache_with_dup_seconds_and_unit(self):
to_datetime(self.dup_numeric_seconds, unit='s')
to_datetime(self.dup_numeric_seconds, unit='s')

def time_cache_with_dup_string_dates(self):
to_datetime(self.dup_string_dates)
to_datetime(self.dup_string_dates)

def time_cache_with_dup_string_dates_and_format(self):
to_datetime(self.dup_string_dates, format='%Y-%m-%d')
to_datetime(self.dup_string_dates, format='%Y-%m-%d')

def time_cache_with_dup_string_tzoffset_dates(self):
to_datetime(self.dup_string_with_tz)

def time_cache_with_dup_string_tzoffset_dates_and_format(self):
to_datetim(self.dup_string_with_tz, format='%Y-%m-%d %H:%M:%S%z')
to_datetime(self.dup_string_with_tz)


class Offsets(object):
Expand Down
13 changes: 6 additions & 7 deletions pandas/core/tools/datetimes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -373,13 +373,12 @@ def _convert_listlike(arg, box, format, name=None, tz=tz):
arg = arg + offset

convert_cache = None
if cache and is_list_like(arg):
if len(arg) >= 1000:
unique_dates = algorithms.unique(arg)
if len(unique_dates) != len(arg):
from pandas import Series
cache_dates = _convert_listlike(unique_dates, False, format)
convert_cache = Series(cache_dates, index=unique_dates)
if cache and is_list_like(arg) and len(arg) >= 1000:
unique_dates = algorithms.unique(arg)
if len(unique_dates) != len(arg):
from pandas import Series
cache_dates = _convert_listlike(unique_dates, False, format)
convert_cache = Series(cache_dates, index=unique_dates)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so its better to actually to make convert_cache a function, which can then take the unique_dates and return the converted data, avoids lots of code duplication.


if isinstance(arg, tslib.Timestamp):
result = arg
Expand Down
0