8000 numpy.fromfile OSError: first argument must be an open file (Python 3) · Issue #6246 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content
8000

numpy.fromfile OSError: first argument must be an open file (Python 3) #6246

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
cjermain opened this issue Aug 25, 2015 · 12 comments
Closed

numpy.fromfile OSError: first argument must be an open file (Python 3) #6246

cjermain opened this issue Aug 25, 2015 · 12 comments

Comments

@cjermain
Copy link
Contributor

Using Python 3 and Numpy 1.9.2 there is an issue with numpy.fromfile when called on an existing file object. This code works fine in Python 2.7.

$ fh
<_io.FileIO name=8 mode='rb+'>
$ np.fromfile(fh, dtype=dtype, count=length)
OSError: first argument must be an open file

I've checked fh.closed is False, so the file is still open. The error is thrown by numpy.multiarraymodule.c#2119.

Someone mentioned these issues on Stack Overflow, but no clear solutions are present.

Any suggestions?
@cjermain
Copy link
Contributor Author

I tried a few other io objects and had no issues.

The io.FileIO object that I'm passing to np.fromfile is different from other io objects in Python 3.4 (which I'm using), because the file descriptor is non-inheritable following PEP 0446. Since np.fromfile uses npy_PyFile_Dup2 to duplicate the file descriptor, ever since Numpy 1.8.1 with commit ea89757 by @juliantaylor , I suspect the issue is in the os.dup call on line 174. It looks like using os.dup2 will make inheritable file descriptors and is not affected by PEP 0446. I'm not sure yet if this makes sense -- any thoughts?

I'll try using Python 3.3 to see if the changes to os.dup cause the affect.

@pv
Copy link
Member
pv commented Aug 26, 2015 via email

@charris charris mentioned this issue Aug 26, 2015
Closed
@cjermain
Copy link
Contributor Author

I tried on a different computer with Python 3.4 to reproduce the error by constructing a io.FileIO object to read an empty file -- this works fine.

Perhaps the error instead has to do with the type of file I am reading from. Its '/dev/comedi0', a data acquisition board, which has user permissions from the a udev .rules file. Previously with numpy 1.9.2 and Python 2.7, I had no issues reading from it using np.fromfile.

There are a few error cases in npy_PyFile_Dup2 that return NULL without reporting the specific error. I'll try adding some more error reporting to this function to diagnose the function call that is failing.

@cjermain
Copy link
Contributor Author
cjermain commented Sep 3, 2015

After commenting out the error string setting in multiarraymodule.c#L2121, I was given the correct error message from npy_3kcompat.h#L250. This issue is an unintended bug that cropped up after the fix for issue #4118. Since my file can not preform npy_ftell(), the npy_PyFile_Dup2 function fails. I checked manually and confirmed that the tell() function raises an OSError.

First off, the line in multiarraymodule.c#L2121 should be removed, because it defeats the purpose of having the specific error messages in npy_3kcompat.h -- PR to follow.

Secondly, I don't think the tell operation should be a hard requirement for all files. Perhaps there could be an argument set in np.fromfile to ignore this operation. Any thoughts on the best design?

@cjermain cjermain mentioned this issue Sep 3, 2015
Closed
@pv
Copy link
Member
pv commented Sep 3, 2015

The problem here is in synchronizing the Python3 file handle position and its potential buffering with the OS-level handle. For files that don't have random access, it's not very clear how to do this.
8000

@cjermain
Copy link
Contributor Author
cjermain commented Sep 8, 2015

The file is opened as an unbuffered I/O using the buffering keyword.

open(filename, "rb+", buffering=0)

Since there should be no buffering, can this situation be handled?

@pv
Copy link
Member
pv commented Sep 8, 2015

I don't know. This depends on implementation details of CPython, and would have to be checked.

@cjermain cjermain mentioned this issue Sep 8, 2015
Open
@cjermain cjermain mentioned this issue Sep 18, 2015
Merged
@rpratt20
Copy link
rpratt20 commented Oct 2, 2015

I have a related issue using older versions.
I am using Python 2.7 and reading a binary data file. If I just have a main program with a file open statement and then use np.fromfile the operation works perfect. If I then move the identical code into a function def and pass the file name it fails with the same error you are receiving. OSError: first argument must be an open file. I am very new to Python and can use any advice.

@wking
Copy link
wking commented Oct 7, 2015

Is this closable now that #6330 has landed, or should it stay open until @rpratt20's Python 2 issue has been addressed?

@charris
Copy link
Member
charris commented Oct 7, 2015

I don't know. @rpratt20 does your problem persist with current numpy? If so, could you open an new issue and paste in a code snippet that illustrates the problem.

@rpratt20
Copy link
rpratt20 commented Oct 8, 2015

Someone pointed out to me that fromfile is designed to read straight from a file and not buffered. Using this:
trace.data = np.frombuffer(fpin.read(record_dtype.itemsize * npts), record_dtype) gives me correct input so there probably is no issue. Thanks.

Randy
----- Original Message -----
From: W. Trevor King
To: numpy/numpy
Cc: rpratt20
Sent: Wednesday, October 07, 2015 2:35 PM
Subject: Re: [numpy] numpy.fromfile OSError: first argument must be an open file (Python 3) (#6246)

Is this closable now that #6330 has landed, or should it stay open until @rpratt20's Python 2 issue has been addressed?

—
Reply to this email directly or view it on GitHub.

@charris charris closed this as completed Oct 8, 2015
@cjermain
Copy link
Contributor Author
cjermain commented Oct 8, 2015

Thanks @charris!

@kohr-h kohr-h mentioned this issue Nov 24, 2016
Merged
@jannikmi jannikmi mentioned this issue Feb 3, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@pv @charris @wking @cjermain @rpratt20
0