8000 BUG: Fix uint->timedelta promotion to raise TypeError by anirudh2290 · Pull Request #16639 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

BUG: Fix uint->timedelta promotion to raise TypeError #16639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 20, 2020
Merged

BUG: Fix uint->timedelta promotion to raise TypeError #16639

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2990d6e
BUG: Fix promote_types uint<->m8 behavior
anirudh2290 Jun 19, 2020
27edfbb
MAINT: Add platform specific umath tests to gitignore
anirudh2290 Jun 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,9 @@ numpy/core/src/umath/simd.inc
numpy/core/src/umath/struct_ufunc_test.c
numpy/core/src/umath/test_rational.c
numpy/core/src/umath/umath_tests.c
numpy/core/src/umath/_umath_tests.dispatch.avx2.c
numpy/core/src/umath/_umath_tests.dispatch.h
numpy/core/src/umath/_umath_tests.dispatch.sse41.c
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a few files to gitignore, maybe got added as part of #13516 ?

numpy/distutils/__config__.py
numpy/linalg/umath_linalg.c
doc/source/**/generated/
Expand Down
10 changes: 8 additions & 2 deletions doc/release/upcoming_changes/16592.compatibility.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
float->timedelta promotion will raise a TypeError
-------------------------------------------------
float->timedelta and uint64->timedelta promotion will raise a TypeError
-----------------------------------------------------------------------
Float and timedelta promotion consistently raises a TypeError.
``np.promote_types("float32", "m8")`` aligns with
``np.promote_types("m8", "float32")`` now and both raise a TypeError.
Previously, ``np.promote_types("float32", "m8")`` returned ``"m8"`` which
was considered a bug.

Uint64 and timedelta promotion consistently raises a TypeError.
``np.promote_types("uint64", "m8")`` aligns with
``np.promote_types("m8", "uint64")`` now and both raise a TypeError.
Previously, ``np.promote_types("uint64", "m8")`` returned ``"m8"`` which
was considered a bug.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please let me know if this should go into a seperate file 16639.compatibility.rst

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, i think both of this and the above gitignore additions are fine here.

This affects not just uint64, but all unsigned integers, with only signed integers remaining to promote (for the time being).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so uint32 works fine, since it already finds the casting mapping in _npy_type_promotion_table .

>>> np.promote_types("uint32", "m8")
dtype('<m8')
>>> np.promote_types("m8", "uint32")
dtype('<m8')
>>> np.promote_types("m8", "uint64")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: invalid type promotion
>>> np.promote_types("uint64", "m8")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: invalid type promotion
>>>

My understanding was that it should work fine for uint32, it should only fail for large uints since it maybe too big in value to be cast accurately ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, ok, right, thanks for the explanation. Yes, I think it is fine here then.

I do think we probably should go all the way, and outright deprecate all integer+timedelta promotions. However, that is better for a new PR. We should also make the casting safety of these at least unsafe, probably even no-cast unless it has generic units. But both of those are different issues.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think we probably should go all the way, and outright deprecate all integer+timedelta promotions.

Maybe i am missing something here, but why should this "not be allowed" if it can be cast accurately to timedelta ? Even if we had non generic units we can always convert the integer to the timedelta of that many units right ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, I keep going in circles. I think the promotion should definitely not be allowed, because an integer and a timedelta represent completely different things. If we concatenate a timedelta64 and an integer array, then the result being a timedelta may be a misrepresentation of the integers. I think we can expect users to do the cast manually?

Casting safety is a bit of a different issue. Because they are completely different things, the casting should possibly not be considered "safe". This is what astropy.units do as far as I know. They refuse to cast, unless the value is unit-less.

On the other hand, there was the opinion that it could be considered "safe" if it will definitely round-trip correctly (which it does). Which is a good point. Possibly we need to add new casting safety parameters...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway, thanks for this! I will put this in, since it does not hurt for now and the asymmetric definition is not helpful to begin with.

Whether the other promotion cases should also be deprecated is a bit of a different issue, I am pretty sure I want that in the long run. And I am starting to be more and more happy with decoupling that from the type-safety discussion.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Sebastian for the explanation and the merge !

Whether the other promotion cases should also be deprecated is a bit of a different issue, I am pretty sure I want that in the long run. And I am starting to be more and more happy with decoupling that from the type-safety discussion.

The reason I don't quite understand how we will be able to decouple type-safety from promotion is because the docs mention the following : """Returns the data type with the smallest size and smallest scalar kind to which both type1 and type2 may be safely cast.""" so, atleast according to documentation promotion depends on casting safety to a common type.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, its the current rule, and it has its arguments. I am not quite sure on it yet.

I think I prefer the coupling version, with these type of casts just not being designated "safe".

There is things like strings. We currently promote string+number to string, which seems plain wrong to me. But we currently also consider it a safe cast.
The "safe" cast part could at least be motivated by the fact that you can convert any number into a string, and it will roundtrip back correctly. But I do not see how you can possibly motivate promotion to work. There is no operation which works the same on strings and integers, so storing both into a string array (e.g. concatenation) is just wrong. And I think for promotion to make sense, it should not be wrong.

Long story short: For numbers that rule is clean and clear. I am not really convinced it was ever quite thought when the rule was generalized to non-numbers.

2 changes: 1 addition & 1 deletion numpy/core/src/multiarray/convert_datatype.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1419,7 +1419,7 @@ PyArray_PromoteTypes(PyArray_Descr *type1, PyArray_Descr *type2)
}
break;
case NPY_TIMEDELTA:
if (PyTypeNum_ISINTEGER(type_num1)) {
if (PyTypeNum_ISSIGNED(type_num1)) {
return ensure_dtype_nbo(type2);
}
break;
Expand Down
2 changes: 2 additions & 0 deletions numpy/core/tests/test_datetime.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -778,6 +778,8 @@ def test_dtype_promotion(self):
# timedelta and float cannot be safely cast with each other
assert_raises(TypeError, np.promote_types, "float32", "m8")
assert_raises(TypeError, np.promote_types, "m8", "float32")
assert_raises(TypeError, np.promote_types, "uint64", "m8")
assert_raises(TypeError, np.promote_types, "m8", "uint64")

# timedelta <op> timedelta may overflow with big unit ranges
assert_raises(OverflowError, np.promote_types,
Expand Down
0