8000 Geometric, negative binomial and poisson fail for extreme arguments (Trac #896) · Issue #1494 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

Geometric, negative binomial and poisson fail for extreme arguments (Trac #896) #1494

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
numpy-gitbot opened this issue Oct 19, 2012 · 24 comments
Closed

Geometric, negative binomial and poisson fail for extreme arguments (Trac #896) #1494

numpy-gitbot opened this issue Oct 19, 2012 · 24 comments
Labels
00 - Bug component: numpy.random

Comments

@numpy-gitbot
Copy link

Original ticket http://projects.scipy.org/numpy/ticket/896 on 2008-08-29 by trac user anand, assigned to @rkern.

I'm guessing these are all related and have something to do with overflow:

In [1]: import numpy          

In [2]: p=1e-13

In [3]: numpy.random.geometric(p)
Out[3]: -2147483648

In [4]: numpy.random.poisson((1-p)/p)
Out[4]: -2147483648

In [5]: numpy.random.negative_binomial(1,p)
Out[5]: -2147483648
@numpy-gitbot
Copy link
Author

Attachment added by trac user anand on 2008-08-29: distributions.patch

@numpy-gitbot
Copy link
Author

trac user anand wrote on 2008-08-29

File distributions.patch 'fixes' it by setting doubles to LONG_MAX if they can't be cast to long... I know it's debatable whether this is actually better.

@numpy-gitbot
Copy link
Author

Attachment added by trac user anand on 2008-08-29: double.patch.zip

@numpy-gitbot
Copy link
Author

trac user anand wrote on 2008-08-29

Double.patch.zip makes the problem a little better by making geometric, poisson and negative_binomial return doubles.

@numpy-gitbot
Copy link
Author

Attachment added by trac user anand on 2008-08-29: check_bounds.patch

@numpy-gitbot
Copy link
Author

trac user anand wrote on 2008-08-29

And check_bounds.patch raises an error if the absolute value of the return value is LONG_MAX.

@numpy-gitbot
Copy link
Author

@teoliphant wrote on 2008-08-29

Do we want to be raising an error if any of the entries overflow? This is different array behavior than usual.

@numpy-gitbot
Copy link
Author

@rkern wrote on 2008-08-29

The way I have been fixing these problems is to simply reject the result if it is out of bounds and resample until I do get something in bounds. The generators then sample a slightly modified form of the distribution which is truncated to the limits of the output representation. Since this is basically the deal you make with any computerized non-uniform RNG, I'm fairly comfortable with it.

@numpy-gitbot
Copy link
Author

@rkern wrote on 2008-08-29

But I do want to get 1.2.0 out first. I think these can wait.

@numpy-gitbot
Copy link
Author

Milestone changed to 1.2.1 by @rkern on 2008-08-29

@numpy-gitbot
Copy link
Author

trac user anand wrote on 2008-09-01

If the standard array behavior is to set to -LONG_MAX-1 when casting to long fails, it makes sense for the RNG's to conform... but isn't that a C gotcha that numpy should protect against like it does with array bounds checking?

@numpy-gitbot
Copy link
Author

Milestone changed to 1.4.0 by @charris on 2009-03-15

@numpy-gitbot
Copy link
Author

@charris wrote on 2010-05-25

Has this been fixed? I recall some discussion of this on the list.

@numpy-gitbot
Copy link
Author

Milestone changed to NumPy 2.0 by @charris on 2010-05-25

@numpy-gitbot
Copy link
Author

@mwiebe wrote on 2011-03-23

Except for poisson, it's still basically the same (but LONGLONG_MIN instead on a 64-bit system):

In [64]: numpy.random.geometric(p)
Out[64]: -9223372036854775808

In [65]: numpy.random.poisson((1-p)/p)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

/home/mwiebe/<ipython console> in <module>()

/home/mwiebe/installtest/lib64/python2.7/site-packages/numpy/random/mtrand.so in mtrand.RandomState.poisson (numpy/random/mtrand/mtrand.c:14605)()

ValueError: lam value too large

In [66]: numpy.random.negative_binomial(1,p)
Out[66]: -9223372036854775808

@numpy-gitbot
Copy link
Author

Milestone changed to Unscheduled by @mwiebe on 2011-03-23

@numpy-gitbot numpy-gitbot mentioned this issue Oct 19, 2012
Closed
@charris
Copy link
Member
charris commented Feb 18, 2014

These problems look to be fixed. @joseph-pkt, @rkern Comments?

@rkern
Copy link
Member
rkern commented Feb 18, 2014

I don't have a 32-bit build handy to test.

@rkern
Copy link
Member
rkern commented Feb 18, 2014

It's still an issue.

[~]
|11> import numpy

[~]
|12> p = 1e-26

[~]
|13> numpy.random.geometric(p)
-9223372036854775808

@charris
Copy link
Member
charris commented Feb 18, 2014

Shoot :(

@bashtage bashtage mentioned this issue Apr 14, 2019
Closed
16 tasks
@mattip 8000
Copy link
Member
mattip commented May 29, 2019

Still producing -2**31 on 32 bit, but np.random.Generator().* produces valid long` values. Closing, since we will not change the legacy generator and the new API fixes this. Please reopen if you think we should fix this on legacy.

@mattip mattip closed this as completed May 29, 2019
@hnajjar0
Copy link

What's the new API??

@rkern
Copy link
Member
rkern commented Sep 19, 2019

Here is the current documentation.

@SV-97
Copy link
SV-97 commented May 5, 2021 •
edited
Loading

There still seem to be overflow/underflow problems even in the new API:

In [1]: from numpy.random import default_rng                                    

In [2]: default_rng(1).negative_binomial(1859539395891982336, 0.1)              
Out[2]: -9223372036854775808
...
In [14]: default_rng(seed=1).negative_binomial(10e20, 0.1)                      
Out[14]: -9223372036854775808

In [15]: default_rng(seed=1).negative_binomial(10e20, 0.5)                    
Out[15]: -9223372036854775808

In particular the "tipover"-point depends on p. For p=0.5 it's at 9223372037481356287 (so default_rng(1).negative_binomial(9223372037481356287, 0.5) > 0 and default_rng(1).negative_binomial(9223372037481356287+1, 0.5) < 0) and for p=0 (which shouldn't even be allowed and raise an error) at exactly 2**63.

This has been verified on python 3.8.3, numpy version 1.18.5 and 1.17.4 running on linux mint 20 as well as python 3.8.5, numpy version 1.19.2 running on macOS Big Sur (Version 11.3).

The code samples don't raise an error regardless of np.seterr(all="raise").

@SV-97 SV-97 mentioned this issue May 12, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug component: numpy.random
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@rkern @charris @mattip @numpy-gitbot @SV-97 @hnajjar0
0