8000 BUG/API: to_datetime preserves UTC offsets when parsing datetime strings by mroeschke · Pull Request #21822 · pandas-dev/pandas · GitHub
[go: up one dir, main page]

Skip to content

BUG/API: to_datetime preserves UTC offsets when parsing datetime strings #21822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 43 commits into from
Jul 30, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
ac5a3d1
BUG: to_datetime no longer converts offsets to UTC
Jul 7, 2018
b81a8e9
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 8, 2018
6bf46a8
Document and now return offset
Jul 8, 2018
678b337
Add some tests, start converting some existing uses of array_to_datetime
Jul 8, 2018
1917148
Add more tests
Jul 8, 2018
581a33e
Adjust test
Jul 8, 2018
a1bc8f9
Flake8
Jul 8, 2018
80042e6
Add tests confirming new behavior
Jul 8, 2018
7c4135e
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 10, 2018
0651416
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 11, 2018
bacb6e3
Lint
Jul 11, 2018
a2f4aad
adjust a test
Jul 11, 2018
d48f341
Ensure box object index, pass tests
Jul 11, 2018
7efb25c
Adjust tests
Jul 11, 2018
1d527ff
Adjust test
Jul 11, 2018
f89d6b6
Cleanup and add comments
Jul 12, 2018
d91c63f
address comments
Jul 12, 2018
1054e8b
adjust test to be closer to the original behavior
Jul 12, 2018
d9cdc91
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 12, 2018
7d04613
Add TypeError clause
Jul 12, 2018
031284c
Add TypeError not ValueError
Jul 12, 2018
749e62e
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 12, 2018
23cbf75
fix typo
Jul 12, 2018
1e6f87a
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 18, 2018
7539bcf
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 19, 2018
c1f51cd
New implimentation
Jul 19, 2018
db75a24
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 20, 2018
4733ac5
Change implimentation and add some tests
Jul 20, 2018
e3db735
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 20, 2018
2fa681f
Add missing commas
Jul 20, 2018
5f36c98
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 24, 2018
d7ff275
Change implimentation since tzoffsets cannot be hashed
Jul 25, 2018
4ff7cb3
Add whatsnew
Jul 25, 2018
8463d91
Address review
Jul 25, 2018
dddc6b3
Address tzlocal
Jul 25, 2018
cca3983
Change is to == for older dateutil compat
Jul 26, 2018
e441be0
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 26, 2018
a8a65f7
Modify example in whatsnew to display
Jul 26, 2018
75f9fd9
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 26, 2018
6052475
Add more specific errors
Jul 27, 2018
f916c69
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 28, 2018
807a251
Merge remote-tracking branch 'upstream/master' into parse_tz_offsets
Jul 29, 2018
1cbd9b9
Add some benchmarks and reformat tests
Jul 30, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Cleanup and add comments
  • Loading branch information
Matt Roeschke committed Jul 12, 2018
commit f89d6b6b02a0e750124430720bcab1509cf05bd5
16 changes: 5 additions & 11 deletions pandas/_libs/tslib.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -456,14 +456,15 @@ cpdef array_to_datetime(ndarray[object] values, errors='raise',
is encountered

Also returns a pytz.FixedOffset if an array of strings with the same
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In principle other tzinfos could be returned, specifically if it falls back to dateutil

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is specifying that array_to_datetime function itself can return a pytz.FixedOffset or None from the C parser output. If it goes through the dateutil parser in the non-ValueError branch, I don't think there's a way to recover the timezone?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When parse_datetime_string is called if there's a timezone it should return a tz-aware datetime object, so the tzinfo can be pulled off that can't it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh right that's true, good catch. Sure I will try to add some tests and functionality to hit the dateutil parser before the object branch.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I am using a set to store the parsed timezone offsets (which should be more performant that using an array in theory / I was having some trouble using an array due to duplicates), however dateutil.tz.tzoffsets cannot be hashed: dateutil/dateutil#792

So instead, I am saving the total_seconds() of the dateutil tzoffset in the set instead and reconstructing the offsets as pytz.FixedOffsets

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add Parameters here

timezone offset if passed and utc=True is not passed
timezone offset is passed and utc=True is not passed. Otherwise, None
is returned

Handles datetime.date, datetime.datetime, np.datetime64 objects, numeric,
strings

Returns
-------
(ndarray, timezone offset)
tuple (ndarray, timezone offset)
"""
cdef:
Py_ssize_t i, n = len(values)
Expand All @@ -481,18 +482,12 @@ cpdef array_to_datetime(ndarray[object] values, errors='raise',
bint is_coerce = errors=='coerce'
_TSObject _ts
int out_local=0, out_tzoffset=0
# Can't directly create a ndarray[int] out_local,
# since most np.array constructors expect a long dtype
# while _string_to_dts expects purely int
# maybe something I am missing?
ndarray[int64_t] out_local_values
ndarray[int64_t] out_tzoffset_vals

# specify error conditions
assert is_raise or is_ignore or is_coerce

try:
out_local_values = np.empty(n, dtype=np.int64)
out_tzoffset_vals = np.empty(n, dtype=np.int64)
result = np.empty(n, dtype='M8[ns]')
iresult = result.view('i8')
Expand Down Expand Up @@ -630,7 +625,6 @@ cpdef array_to_datetime(ndarray[object] values, errors='raise',
# No error raised by string_to_dts, pick back up
# where we left off
out_tzoffset_vals[i] = out_tzoffset
out_local_values[i] = out_local
value = dtstruct_to_dt64(&dts)
if out_local == 1:
seen_datetime_offset = 1
Expand Down Expand Up @@ -685,12 +679,12 @@ cpdef array_to_datetime(ndarray[object] values, errors='raise',
# object path where an array of datetimes
# (with individual datutil.tzoffsets) are returned
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo datutil


# Faster to compare integers than to compare objects
# Faster to compare integers than to compare pytz objects
is_same_offsets = (out_tzoffset_vals[0] == out_tzoffset_vals).all()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There may be a perf tradeoff here, specifically in the case where we have all-strings, all of which are ISO, but that don't have matching timezones. Going through the parse_datetime_string path below is much slower than _string_to_dts. Going through the python path entails a big hit.

The various paths (including require_iso8859 ugh) make this a giant hassle. @jreback one way to simplify this hassle would be to strengthen the requirement on require_iso8859. ATM it raises if it sees a non-ISO string, but is fine with datetime/np.datetime64 objects. If it were strings-only, then a bunch of logic could be simplified (not necessarily this PR). Thoughts?

if not is_same_offsets:
raise TypeError
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This (pre-existing) pattern is pretty opaque to a first-time reader. What if instead of raising TypeError the fallback block became its own function that gets called from here?

else:
tz_out = pytz.FixedOffset(out_tzoffset_vals[0])
tz_out = pytz.FixedOffset(out_tzoffset)

return result, tz_out
except OutOfBoundsDatetime:
Expand Down
2 changes: 1 addition & 1 deletion pandas/core/tools/datetimes.py
Original file line number Diff line number Diff line change
Expand Up @@ -412,7 +412,7 @@ def to_datetime(arg, errors='raise', dayfirst=False, yearfirst=False,
datetime.datetime objects as well).
box : boolean, default True

- If True returns a DatetimeIndex or Index
- If True returns a DatetimeIndex or Index-like object
- If False returns ndarray of values.
format : string, default None
strftime to parse time, eg "%d/%m/%Y", note that "%f" will parse
Expand Down
4 changes: 2 additions & 2 deletions pandas/tests/indexes/datetimes/test_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -580,7 +580,7 @@ def test_week_without_day_and_calendar_year(self, date, format):
with tm.assert_raises_regex(ValueError, msg):
pd.to_datetime(date, format=format)

def test_ts_strings_with_same_offset(self):
def test_iso_8601_strings_with_same_offset(self):
# GH 17697, 11736
ts_str = "2015-11-18 15:30:00+05:30"
result = to_datetime(ts_str)
Expand All @@ -594,7 +594,7 @@ def test_ts_strings_with_same_offset(self):
result = DatetimeIndex([ts_str] * 2)
tm.assert_index_equal(result, expected)

def test_ts_strings_with_different_offsets(self):
def test_iso_8601_strings_with_different_offsets(self):
# GH 17697, 11736
ts_strings = ["2015-11-18 15:30:00+05:30",
"2015-11-18 16:30:00+06:30"]
Expand Down
5 changes: 2 additions & 3 deletions pandas/tests/tslibs/test_array_to_datetime.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,9 +78,8 @@ def test_parsing_timezone_offsets(self, dt_string, expected_tz):
# All of these datetime strings with offsets are equivalent
# to the same datetime after the timezone offset is added

# TODO: Appears that parsing non-ISO strings adjust the date to UTC
# but don't return the offset. Not sure if this is the intended
# behavior of non-iso strings in np_datetime_strings
# Non-ISO 8601 datetime strings will not return a timezone offset
# as a limitation of the C parser
arr = np.array(['01-01-2013 00:00:00'], dtype=object)
expected = tslib.array_to_datetime(arr)[0]

Expand Down
0