8000 randint over all of uint64 raises OverflowError if high=... is uint64 but not if it is int · Issue #9256 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

randint over all of uint64 raises OverflowError if high=... is uint64 but not if it is int #9256

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
vrtsig opened this issue Jun 15, 2017 · 17 comments
Closed

randint over all of uint64 raises OverflowError if high=... is uint64 but not if it is int #9256

vrtsig opened this issue Jun 15, 2017 · 17 comments
Labels
component: numpy.random

Comments

@vrtsig
Copy link
vrtsig commented Jun 15, 2017

Stumbled over this earlier today:

from numpy import uint64, iinfo
from numpy.random import RandomState

uint64_max = iinfo(uint64).max
RandomState(0).randint(uint64_max, dtype=uint64)
# returns 10123822223749065263

RandomState(0).randint(uint64(uint64_max), dtype=uint64)
# raises OverflowError

Traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "mtrand.pyx", line 979, in mtrand.RandomState.randint (numpy/random/mtrand/mtrand.c:16422)
  File "mtrand.pyx", line 980, in mtrand.RandomState.randint (numpy/random/mtrand/mtrand.c:16227)
  File "randint_helpers.pxi", line 450, in mtrand._rand_uint64 (numpy/random/mtrand/mtrand.c:5637)
OverflowError: Python int too large to convert to C unsigned long

conda 4.3.21
Python 3.5.3.final.0
numpy 1.12.1
osx-64 (El Capitan)
@eric-wieser
Copy link
Member
eric-wieser commented Jun 15, 2017 •
edited
Loading

This is bizarre:

>>> np.random.randint(np.uint64(0xfffffffffffffBFF), dtype=np.uint64)
10123822223749065263
>>> np.random.randint(np.uint64(0xfffffffffffffC00), dtype=np.uint64)
OverflowError: int too big to convert

I don't think this bug is related to #8846

@charris
Copy link
Member
charris commented Jun 15, 2017

I don't see either problem. I take it this is windows?

@vrtsig
Copy link
Author
vrtsig commented Jun 15, 2017

nope, OSX.

@charris
Copy link
Member
charris commented Jun 15, 2017 •
edited
Loading

I think this is fixed in master and 1.13.

+        # TODO: Do not cast these inputs to Python int
+        #
+        # This is a workaround until gh-8851 is resolved (bug in NumPy
+        # integer comparison and subtraction involving uint64 and non-
+        # uint64). Afterwards, remove these two lines.
+        ilow = int(low)
+        ihigh = int(high)

@vrtsig
Copy link
Author
vrtsig commented Jun 15, 2017

@charris you are right. It seems fine under 1.13.0. Thanks for looking into it 👍

@charris
Copy link
Member
charris commented Jun 15, 2017

@vrtsig Note that 1.13.0 has other, new bugs that should get fixed in 1.13.1

@charris
Copy link
Member
charris commented Jun 15, 2017

@eric-wieser What platform are you on?

@eric-wieser
Copy link
Member

1.12.1, but I think the fix in 1.13 is only coincidentally a solution, and there's likely a bug somewhere now morr hidden

@eric-wieser
Copy link
Member
eric-wieser commented Jun 15, 2017 •
edited
Loading

Yep, the issue originates in randint_helpers.pxi.in, which has not been modified since 1.12.

In particular, I think this might be a cython bug, and some problem converting from a np.uint64 to a npy_uint64, due to somehow going through a long in the process.

I also think the changes in #8717 might help fix/hide the problem, by trying to avoid PyInt/int round-tripping

@charris
Copy link
Member
charris commented Jun 16, 2017

I think you should try it with master/1.13. The fix changes the type of the arguments passed to the functions in randint_helpers.pxi.in. That said, cython could get this wrong on windows, where long is 32 bits.

@charris
Copy link
Member
charris commented Jun 16, 2017

You could probably look in the cython generated file.

@bashtage
Copy link
Contributor

I think you should try it with master/1.13. The fix changes the type of the arguments passed to the functions in randint_helpers.pxi.in. That said, cython could get this wrong on windows, where long is 32 bits.

Seems to be mostly OK:

In [31]: np.max(rs.randint(uint64_max, dtype=uint64,size=100000)) / 2 **64
Out[31]: 0.9999990152188338

@bashtage
Copy link
Contributor
bashtage commented Aug 8, 2017

@charris @eric-wieser This is fixed and can be closed.

@mattip mattip closed this as completed Jul 18, 2018
@kitchoi
Copy link
Contributor
kitchoi commented May 31, 2019

I ran into the following error with Windows (64bit), Python 3.6, numpy 1.15.4, distributed by EDM

Enthought Deployment Manager -- https://www.enthought.com
Python 3.6.0 |Enthought, Inc. (x86_64)| (default, Mar  3 2017, 11:40:20) [MSC v.
1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import numpy as np
>>> print(sys.platform)
win32
>>> print(sys.version_info)
sys.version_info(major=3, minor=6, micro=0, releaselevel='final', serial=0)
>>> np.random.RandomState(1).randint(np.iinfo(np.uint32).max)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "mtrand.pyx", line 991, in mtrand.RandomState.randint
ValueError: high is out of bounds for int32
>>> np.random.RandomState(1).choice(np.iinfo(np.uint32).max, 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "mtrand.pyx", line 1163, in mtrand.RandomState.choice
  File "mtrand.pyx", line 991, in mtrand.RandomState.randint
ValueError: high is out of bounds for int32

(The same code does not fail on Linux nor OSX).

@charris @eric-wieser Should this be re-opened or should I open a new issue?

@bashtage
Copy link
Contributor

This is fixed in master

@kitchoi
Copy link
Contributor
kitchoi commented May 31, 2019

Thank you @bashtage for your quick response. I am sorry for not having checked master before commenting. Just wanted to confirm, did you try this on Windows?

@bashtage
Copy link
Contributor
bashtage commented Jun 1, 2019 •
edited
Loading

This is not a bug. The default dtype is 'l', which is C long. On Windows 32 and 64 this is int32. If you use Generator that is the new API for random generation (out in 1.17), then you won't have this issue since the default dtype is int64.

Try:

np.random.RandomState(1).randint(np.iinfo(np.uint32).max, dtype='uint32')
Out[5]: 1791095845

or

np.random.RandomState(1).randint(np.iinfo(np.int32).max, dtype='int32')
Out[5]:  1791095845

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component: numpy.random
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@charris @eric-wieser @mattip @kitchoi @bashtage @vrtsig
0