8000 datetime64 breaks equality and hash invariant. · Issue #3836 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

datetime64 breaks equality and hash invariant. #3836

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
mdickinson opened this issue Sep 30, 2013 · 10 comments
Open

datetime64 breaks equality and hash invariant. #3836

mdickinson opened this issue Sep 30, 2013 · 10 comments
Labels
00 - Bug component: numpy.datetime64 (and timedelta64)

Comments

@mdickinson
Copy link
Contributor

It looks as though the datetime64 dtype breaks the Python rule that x == y should imply hash(x) == hash(y). This broke a Pandas application that was grouping on dates, and then doing a dictionary lookup to find the lines of a DataFrame associated to a particular date.

Python 3.3.2 (default, May 21 2013, 11:48:51) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.7.1'
>>> import datetime
>>> x = numpy.datetime64(datetime.datetime.now())
>>> y = numpy.datetime64(x, 'ns')
>>> x == y  # gives the expected 'True'
True
>>> hash(x) == hash(y)  # expected True
False
@juliantaylor
Copy link
Contributor

very likely related to #3800 which fixes integer hashing. Probably a similar fix can be applied.

@juliantaylor
Copy link
Contributor

currently numpy just hashes the int64, which breaks the invariant if two datetime represent the same time in different units.
preserving this invariant is actually pretty tricky as one has to convert all datetimes to a common unit which is not always possible with int64.
Best I can think of is doing the conversion in PyLong for arbitrary precision, it could possibly be improved in speed by using a int128 type if available (e.g. gcc >= 4.6)

can timezone based times (Y/M/W/D) ever compare equal to non zone units (s/ms...)?

@njsmith
Copy link
Member
njsmith commented Oct 2, 2013

I don't think there's any real concern about speed here, e.g. just blowing
everything up into two 64 bit integers would be totally fine?

@juliantaylor
Copy link
Contributor

I don't know anything about datetime (for what is this undocumented num variable in the metadata(?!)), but I think this should work:

def hash(input)
  basemeta = smallest_unit // attoseconds
  PyLong num, PyLong denom = get_base_conversion(input, basemeta)
  if timezone:
     base = get_timezone_to_D_base_conversion() // 356 12, 7
     num, denom = mix(base, num, denom)

  PyLong result = convert_to_base_unit(num, denom, input.asPyLong)
  return hash(result)

which basically means reimplementing a bunch of the existing conversion stuff (for casting) in PyLong.

@juliantaylor
Copy link
Contributor

do we still want to do this for 1.8 or defer it to a 1.8.1?

@njsmith
Cop 8000 y link
Member
njsmith commented Oct 5, 2013

This bug is already there in 1.7, right? And has no PR yet? I think we just leave it for 1.8.1/1.9.

@knutae
Copy link
knutae commented May 24, 2019

Ah, this explains some unexpected behavior I've been seeing.

Python 3.6.8 (default, Apr  9 2019, 04:59:38) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> x = numpy.datetime64(1512345678910, "ms")
>>> y = numpy.datetime64(x, "ns")
>>> x == y
True
>>> [x] == [y]
True
>>> {x} == {y}
False
>>> y in [x]
True
>>> y in {x}
False

It means that datetime64 objects, or tuples containing them, cannot consistently be used in sets, or as keys in dictionaries.

@seberg seberg self-assigned this May 24, 2019
@walshb walshb mentioned this issue Oct 1, 2019
Merged
@seberg seberg removed their assignment Oct 1, 2019
@eric-wieser
Copy link
Member

Another way in which this fails today:

now = datetime.datetime.now()
np_now = np.datetime64(now)
assert now == np_now
assert hash(now) == hash(np_now)  # fails

walshb added a commit to walshb/numpy that referenced this issue May 26, 2020
@walshb
BUG: datetime64 hash.
bf3b995 
numpy#3836
@mattip mattip mentioned this issue Aug 22, 2021
Closed
@jbrockmendel jbrockmendel mentioned this issue Dec 28, 2021
Open
3 tasks
@jbrockmendel jbrockmendel mentioned this issue Jan 12, 2023
Open
@seberg
Copy link
Member
seberg commented Jan 24, 2023 •
edited
Loading

Pandas has a PR open now about this here: pandas-dev/pandas#50960 which may make sense to upstream as is or similar. (Sorry had the wrong link first)

@jbrockmendel jbrockmendel mentioned this issue Jan 25, 2023
Closed
5 tasks
@seberg
Copy link
Member
seberg commented Feb 15, 2023

For (a bit) of visibility: The PR currently converts to datetime and if that is out of range does something similar to a tuple hash: hash((year, month, day, minute, second, us, ps, as)).
I don't think this is very fixed, I am relatively happy with that scheme, although I might consider combining multiple values (i.e. hash((..., minute * 60 + second))) when we upstream.

But, if anyone has a thought on a better scheme that would also be good. We do need to extract the year anyway to decide whether we need to use the builtin python hashing.
(The builtin datetime hash, hashes the raw bytes of the datetime struct, so isn't nice to steal.)

walshb added a commit to walshb/numpy that referenced this issue Oct 28, 2024
@walshb
BUG: datetime64 hash.
fa643d6 
numpy#3836
walshb added a commit to walshb/numpy that referenced this issue Oct 29, 2024
@walshb
BUG: datetime64 hash.
e000644 
numpy#3836
walshb added a commit to walshb/numpy that referenced this issue Oct 31, 2024
@walshb
BUG: datetime64 hash.
6f4ec86 
numpy#3836
walshb added a commit to walshb/numpy that referenced this issue Nov 1, 2024
@walshb
BUG: datetime64 hash.
f2eff67 
numpy#3836
walshb added a commit to walshb/numpy that referenced this issue Nov 8, 2024
@walshb
BUG: datetime64 hash.
7ec91ca 
numpy#3836
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug component: numpy.datetime64 (and timedelta64)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@seberg @charris @eric-wieser @juliantaylor @njsmith @mdickinson @knutae @mattip
0