8000 BUG/API: to_datetime preserves UTC offsets when parsing datetime strings by mroeschke · Pull Request #21822 · pandas-dev/pandas · GitHub
[go: up one dir, main page]

Skip to content

BUG/API: to_datetime preserves UTC offsets when parsing datetime strings #21822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 43 commits into from
Jul 30, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
ac5a3d1
BUG: to_datetime no longer converts offsets to UTC
Jul 7, 2018
b81a8e9
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 8, 2018
6bf46a8
Document and now return offset
Jul 8, 2018
678b337
Add some tests, start converting some existing uses of array_to_datetime
Jul 8, 2018
1917148
Add more tests
Jul 8, 2018
581a33e
Adjust test
Jul 8, 2018
a1bc8f9
Flake8
Jul 8, 2018
80042e6
Add tests confirming new behavior
Jul 8, 2018
7c4135e
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 10, 2018
0651416
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 11, 2018
bacb6e3
Lint
Jul 11, 2018
a2f4aad
adjust a test
Jul 11, 2018
d48f341
Ensure box object index, pass tests
Jul 11, 2018
7efb25c
Adjust tests
Jul 11, 2018
1d527ff
Adjust test
Jul 11, 2018
f89d6b6
Cleanup and add comments
Jul 12, 2018
d91c63f
address comments
Jul 12, 2018
1054e8b
adjust test to be closer to the original behavior
Jul 12, 2018
d9cdc91
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 12, 2018
7d04613
Add TypeError clause
Jul 12, 2018
031284c
Add TypeError not ValueError
Jul 12, 2018
749e62e
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 12, 2018
23cbf75
fix typo
Jul 12, 2018
1e6f87a
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 18, 2018
7539bcf
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 19, 2018
c1f51cd
New implimentation
Jul 19, 2018
db75a24
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 20, 2018
4733ac5
Change implimentation and add some tests
Jul 20, 2018
e3db735
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 20, 2018
2fa681f
Add missing commas
Jul 20, 2018
5f36c98
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 24, 2018
d7ff275
Change implimentation since tzoffsets cannot be hashed
Jul 25, 2018
4ff7cb3
Add whatsnew
Jul 25, 2018
8463d91
Address review
Jul 25, 2018
dddc6b3
Address tzlocal
Jul 25, 2018
cca3983
Change is to == for older dateutil compat
Jul 26, 2018
e441be0
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 26, 2018
a8a65f7
Modify example in whatsnew to display
Jul 26, 2018
75f9fd9
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 26, 2018
6052475
Add more specific errors
Jul 27, 2018
f916c69
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 28, 2018
807a251
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 29, 2018
1cbd9b9
Add some benchmarks and reformat tests
Jul 30, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Flake8
  • Loading branch information
Matt Roeschke committed Jul 8, 2018
commit a1bc8f916947faca3d5b08d1d2cd0b2aaae04f57
3 changes: 2 additions & 1 deletion pandas/core/tools/datetimes.py
Original file line number Diff line number Diff line change
Expand Up @@ -275,7 +275,8 @@ def _convert_listlike_datetimes(arg, box, format, name=None, tz=None,
require_iso8601=require_iso8601
)
if tz_parsed is not None:
return DatetimeIndex._simple_new(result, name=name, tz=tz_parsed)
return DatetimeIndex._simple_new(result, name=name,
tz=tz_parsed)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

case with multiple tzs that has to get wrapped in object-dtype?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That case will result in tz_parsed = None so this branch will not be hit.


if is_datetime64_dtype(result) and box:
result = DatetimeIndex(result, tz=tz, name=name)
Expand Down
3 changes: 2 additions & 1 deletion pandas/tests/indexes/datetimes/test_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -1500,7 +1500,8 @@ def test_parsers_time(self):
"Timestamp('2013-01-01 05:45:00+0545', tz='pytz.FixedOffset(345)')"),
('2013-01-01 05:30+0530', pytz.FixedOffset(330),
"Timestamp('2013-01-01 05:30:00+0530', tz='pytz.FixedOffset(330)')")])
def test_parsers_timezone_minute_offsets_roundtrip(self, cache, dt_string, tz, dt_string_repr):
def test_parsers_timezone_minute_offsets_roundtrip(self, cache, dt_string,
tz, dt_string_repr):
# GH11708
base = to_datetime("2013-01-01 00:00:00", cache=cache)
base = base.tz_localize('UTC').tz_convert(tz)
Expand Down
16 changes: 10 additions & 6 deletions pandas/tests/tslibs/test_array_to_datetime.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,8 +78,9 @@ def test_parsing_timezone_offsets(self, dt_string, expected_tz):
# All of these datetime strings with offsets are equivalent
# to the same datetime after the timezone offset is added

# TODO: Appears that the dateparser doesnt return offset info if string is non-ISO
# maybe something in the np_datetime_strings parser is not catching this?
# TODO: Appears that parsing non-ISO strings adjust the date to UTC
# but don't return the offset. Not sure if this is the intended
# behavior of non-iso strings in np_datetime_strings
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

np_datetime_strings doesn't handle non-ISO. That case ends up going through dateutil (via parse_datetime_string). I'm not totally sure what the TODO is for. '01-01-2013 00:00:00' comes back tz-naive, right?

arr = np.array(['01-01-2013 00:00:00'], dtype=object)
expected = tslib.array_to_datetime(arr)[0]

Expand All @@ -89,11 +90,14 @@ def test_parsing_timezone_offsets(self, dt_string, expected_tz):
assert result_tz is expected_tz

def test_parsing_different_timezone_offsets(self):
#GH 17697
# GH 17697
data = ["2015-11-18 15:30:00+05:30", "2015-11-18 15:30:00+06:30"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have these same tests using to_datetime? testing the cython function is obviously useful as well

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

result, result_tz = tslib.array_to_datetime(np.array(data, dtype=object))
expected = np.array([datetime(2015, 11, 18, 15, 30, tzinfo=tzoffset(None, 19800)),
datetime(2015, 11, 18, 15, 30, tzinfo=tzoffset(None, 23400))],
data = np.array(data, dtype=object)
result, result_tz = tslib.array_to_datetime(data)
expected = np.array([datetime(2015, 11, 18, 15, 30,
tzinfo=tzoffset(None, 19800)),
datetime(2015, 11, 18, 15, 30,
tzinfo=tzoffset(None, 23400))],
dtype=object)
tm.assert_numpy_array_equal(result, expected)
assert result_tz is None
Expand Down
0