8000 genfromtxt problem with long integers (Trac #1428) · Issue #2026 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

genfromtxt problem with long integers (Trac #1428) #2026

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
thouis opened this issue Oct 19, 2012 · 3 comments
Closed

genfromtxt problem with long integers (Trac #1428) #2026

thouis opened this issue Oct 19, 2012 · 3 comments
Labels
00 - Bug component: Other

Comments

@thouis
Copy link
Contributor
thouis commented Oct 19, 2012

Original ticket http://projects.scipy.org/numpy/ticket/1428 on 2010-03-15 by @josef-pkt, assigned to unknown.

It seems genfromtxt has problems identifying long integers (at
least on Windows 32)

I have a csv file with large integers:

>>> np.array(4160680000,int)
Traceback (most recent call last):
 File "<pyshell#4>", line 1, in <module>
   np.array(4160680000,int)
OverflowError: long int too large to convert to int

genfromtxt fails with non-informative exception

>>> s = '''Date,Open,High,Low,Close,Volume,Adj Close
... 2010-02-12,1075.95,1077.81,1062.97,1075.51,4160680000,1075.51
... 2010-02-11,1067.10,1080.04,1060.59,1078.47,4400870000,1078.47'''


>>> sh = StringIO(s)
>>> data = np.genfromtxt(sh, delimiter=",", dtype=None, names=True)
Traceback (most recent call last):
 File "C:\Programs\Python25\Lib\site-packages\numpy\lib\io.py", line
1367, in genfromtxt
   output = np.array(data, dtype=ddtype)
TypeError: expected a readable buffer object

same with explicit dtypes using int for the long integer

>>> dt= [(_,'S10'),(_,float),(_,float),(_,float),(_,float),(_,int),('',float)]
>>> sh = StringIO(s)
>>> data = np.genfromtxt(sh, delimiter=",", dtype=dt, names=True)
Traceback (most recent call last):
 File "C:\Programs\Python25\Lib\site-packages\numpy\lib\io.py", line
1388, in genfromtxt
   rows = np.array(data, dtype=[('', _) for _ in dtype_flat])
TypeError: expected a readable buffer object

using float works:

>>> dt= [(_,'S10'),(_,float),(_,float),(_,float),(_,float),(_,float),('',float)]
>>> sh = StringIO(s)
>>> data = np.genfromtxt(sh, delimiter=",", dtype=dt, names=True)
>>> data
array([ ('2010-02-12', 1075.95, 1077.8099999999999, 1062.97, 1075.51,
4160680000.0, 1075.51),
      ('2010-02-11', 1067.0999999999999, 1080.04, 1060.5899999999999,
1078.47, 4400870000.0, 1078.47)],
     dtype=[('Date', '|S10'), ('Open', '<f8'), ('High', '<f8'),
('Low', '<f8'), ('Close', '<f8'), ('Volume', '<f8'), ('Adj_Close',
'<f8')])
>>>
@thouis thouis closed this as completed Oct 19, 2012
@numpy-gitbot
Copy link

trac user xscript wrote on 2010-09-06

Replying to [ticket:1428 josefpktd]:

It seems genfromtxt has problems identifying long integers (at
least on Windows 32)

I can confirm this is also happening on a Debian box with numpy 1.4.1, although an integer larger than the first example is needed to trigger a long int (e.g., 10000000000000000000).

@numpy-gitbot
Copy link

@rgommers wrote on 2011-04-02

#2162 has a patch for this for loadtxt, the same can then be done for genfromtxt.

@numpy-gitbot
Copy link

@charris wrote on 2011-04-02

You can now fix this by explicitly specifying int64 or uint64. Floats will raise errors if those types are specified and the automatic detection case will still fail. I suppose this could all be fixed up a bit more but that would require considerably more work.

@numpy-gitbot numpy-gitbot mentioned this issue Oct 23, 2012
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug component: Other
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@thouis @numpy-gitbot
0