8000 BUG: _convert_and_box_cache raise ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True by anmyachev · Pull Request #26097 · pandas-dev/pandas · GitHub
[go: up one dir, main page]
Skip to content

BUG: _convert_and_box_cache raise ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True #26097

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 31 commits into from
Jul 3, 2019
Merged

BUG: _convert_and_box_cache raise ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True #26097

Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
655ec31
fixed _convert_and_box_cache that raised ValueError: Tz-aware datetim…
anmyachev Apr 15, 2019
6a9856e
Revert bandaid workaround
vnlitvinov Apr 16, 2019
ffd9ecf
Add test that fails only when to_datetime gets cache=True
vnlitvinov Apr 16, 2019
d6c584e
Fix to_datetime caching logic so test_to_datetime_offset passes
vnlitvinov Apr 16, 2019
00f72e0
Fix flake8 issues
vnlitvinov Apr 16, 2019
5ad9911
removed debugging stuff; 'name' is default argument now
anmyachev Apr 17, 2019
428cae0
added 'whatsnew'
anmyachev Apr 17, 2019
7ed05f2
Test that to_datetime produces equal result for cache on and off
vnlitvinov Apr 18, 2019
b60f1d5
fixed isort errors
anmyachev Apr 18, 2019
3e2df79
Rework _box_if_needed into _box_as_indexlike
vnlitvinov Apr 19, 2019
3f0285e
Clarify added tests intention
vnlitvinov Apr 19, 2019
d19c2cf
first using notations
anmyachev Apr 22, 2019
b1cf140
changed wildcard import in '/pandas/core/index.py' to explicit import
anmyachev Apr 22, 2019
56db677
changed 'notations' -> 'annotations'
anmyachev Apr 22, 2019
4f9ea36
fixed isort errors
anmyachev Apr 22, 2019
c72a561
rollback of a certain style for annotations
anmyachev Apr 23, 2019
67a0c40
added 'Scalar' and 'DatetimeScalar' unions
anmyachev Apr 24, 2019
1e0d953
added annotations for some arguments; changed formatting
anmyachev Apr 26, 2019
1942bbe
using 'assert_almost_equal' now
anmyachev Apr 28, 2019
97e4548
rerun CI tests
anmyachev Jun 12, 2019
342d7d0
fixed problems found by review
anmyachev Jun 12, 2019
12f9853
replaced '# noqa' statement for flake8 linter
anmyachev Jun 14, 2019
62e75f8
removed 'Optional' using
anmyachev Jun 14, 2019
71ca9be
using TypeVar for 'DatetimeScalar' definition
anmyachev Jun 14, 2019
c35e124
fixed isort error
anmyachev Jun 14, 2019
d3412e2
fixed bug: 'UTC' and 'Etc/GMT' should be the same
anmyachev Jun 16, 2019
3141cb6
added comment about timezones
anmyachev Jun 16, 2019
ca200cd
removed 'UTC_EQ_STR', 'UTC_EQ'
anmyachev Jun 16, 2019
1cc469e
removing extra stuff
anmyachev Jul 2, 2019
137395f
fixed mypy errors
anmyachev Jul 2, 2019
2d8921b
renamed arg in '_box_as_indexlike' func: tz -> utc
anmyachev Jul 3, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Rework _box_if_needed into _box_as_indexlike
  • Loading branch information
@vnlitvinov @anmyachev
vnlitvinov authored and anmyachev committed Jul 2, 2019
commit 3e2df79f9e8b5ec5b82263c91f5009ffa667ef7c
31 changes: 14 additions & 17 deletions pandas/core/tools/datetimes.py
View file
Open in desktop
10000
Original file line number Diff line number Diff line change
Expand Up @@ -60,17 +60,15 @@ def _maybe_cache(arg, format, cache, convert_listlike):
return cache_array


def _box_if_needed(dt_array, box, default, tz, name=None):
def _box_as_indexlike(dt_array, tz=None, name=None):
"""
Properly boxes the ndarray of datetimes (if requested) to DatetimeIndex
Properly boxes the ndarray of datetimes to DatetimeIndex
if it is possible or to generic Index instead

Parameters
----------
dt_array: 1-d array
array of datetimes to be boxed
box : boolean
True boxes result as an Index-like, False returns an ndarray
tz : object
None or 'utc'
name : string, default None
Expand All @@ -79,18 +77,13 @@ def _box_if_needed(dt_array, box, default, tz, name=None):
Returns
-------
result : datetime of converted dates
Returns:

- Index-like if box=True
- ndarray if box=False
- DatetimeIndex if convertible to sole datetime64 type
- general Index otherwise
"""
if box:
from pandas import DatetimeIndex, Index
if is_datetime64_dtype(dt_array):
return DatetimeIndex(dt_array, tz=tz, name=name)
# e.g. an Index of datetime objects
return Index(dt_array, name=name)
return default
from pandas import DatetimeIndex, Index
if is_datetime64_dtype(dt_array):
return DatetimeIndex(dt_array, tz=tz, name=name)
return Index(dt_array, name=name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to apply tz to this result if Index(dt_array, name=name) returns a DatetimeIndex

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jreback @mroeschke After a close look at the Index constructor(pandas/core/indexes/base.py), I had a question about dtype default value. The documentation says that it should be object, but actually None is used. What case is correct?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its default is None, however it will attempt to infer the type of thing passed, so you may need to explicity pass dtype='object' to force it to object here.



def _convert_and_box_cache(arg, cache_array, box, name=None):
Expand All @@ -117,7 +110,9 @@ def _convert_and_box_cache(arg, cache_array, box, name=None):
"""
from pandas import Series
result = Series(arg).map(cache_array)
return _box_if_needed(result, box, result.values, None, name)
if box:
return _box_as_indexlike(result, tz=None, name=name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a test where cache=True and utc=True? I would think tz needs to be passed here instead of None if that were the case.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anmyachev do we have a test like this?

you are passing tz in the call below

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anmyachev do we have a test like this?

@jreback yes. For example, test_to_datetime_tz_pytz (pandas\tests\indexes\datetimes\test_tools.py::519)

    @pytest.mark.parametrize('cache', [True, False])
    def test_to_datetime_tz_pytz(self, cache):
        # see gh-8260
        us_eastern = pytz.timezone('US/Eastern')
        arr = np.array([us_eastern.localize(datetime(year=2000, month=1, day=1,
                                                     hour=3, minute=0)),
                        us_eastern.localize(datetime(year=2000, month=6, day=1,
                                                     hour=3, minute=0))],
                       dtype=object)
        result = pd.to_datetime(arr, utc=True, cache=cache)
        expected = DatetimeIndex(['2000-01-01 08:00:00+00:00',
                                  '2000-06-01 07:00:00+00:00'],
                                 dtype='datetime64[ns, UTC]', freq=None)
        tm.assert_index_equal(result, expected)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I c, we call the passed parameter tz, which is sligthly confusing.

return result.values


def _return_parsed_timezone_results(result, timezones, box, tz, name):
Expand Down Expand Up @@ -349,7 +344,9 @@ def _convert_listlike_datetimes(arg, box, format, name=None, tz=None,
for ts in result]
return np.array(result, dtype=object)

return _box_if_needed(result, box, result, tz, name)
if box:
return _box_as_indexlike(result, tz=tz, name=name)
return result


def _adjust_to_origin(arg, origin, unit):
Expand Down
3819
0