8000 ENH: pathlib support for fromfile(), .tofile() and .dump() by sorenrasmussenai · Pull Request #12915 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

ENH: pathlib support for fromfile(), .tofile() and .dump() #12915

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jun 4, 2019
Merged

ENH: pathlib support for fromfile(), .tofile() and .dump() #12915

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4a94ee4
ENH: pathlib support for fromfile(), .tofile() and .dump()
Feb 1, 2019
eea6b8c
Switched to using numpy.compat
Feb 4, 2019
3c060d5
Fixed remaining issues, pointed out by @eric-weiser
Feb 5, 2019
703c59c
Added release note
Mar 12, 2019
998daea
Merge branch 'master' into pathlib_support
sorenrasmussenai Mar 12, 2019
0933c15
Update 1.17.0-notes.rst
sorenrasmussenai Mar 19, 2019
9a968f3
Merge branch 'master' into pathlib_support
eric-wieser May 31, 2019
6c8f7fb
TST, DOC: Fix tests, add versionchanged sections
eric-wieser May 31, 2019
3c618a3
Merge remote-tracking branch 'upstream/master' into pr/n12915_pathlib…
eric-wieser Jun 4, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
ENH: pathlib support for fromfile(), .tofile() and .dump() 
See #8576
  • Loading branch information
Soren Rasmussen committed Feb 1, 2019
commit 4a94ee439bc237a7790635a7e8391c4aaf891f7f
3 changes: 3 additions & 0 deletions numpy/core/src/multiarray/methods.c
View file
Open in desktop
10000
Original file line number Diff line number Diff line change
Expand Up @@ -577,6 +577,7 @@ array_tofile(PyArrayObject *self, PyObject *args, PyObject *kwds)
return NULL;
}

file = NpyPath_PathlikeToFspath(file);
if (PyBytes_Check(file) || PyUnicode_Check(file)) {
file = npy_PyFile_OpenFile(file, "wb");
if (file == NULL) {
Expand Down Expand Up @@ -2110,6 +2111,8 @@ PyArray_Dump(PyObject *self, PyObject *file, int protocol)
if (cpick == NULL) {
return -1;
}

file = NpyPath_PathlikeToFspath(file);
if (PyBytes_Check(file) || PyUnicode_Check(file)) {
file = npy_PyFile_OpenFile(file, "wb");
if (file == NULL) {
Expand Down
33 changes: 33 additions & 0 deletions numpy/core/src/multiarray/methods.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,37 @@ extern NPY_NO_EXPORT PyMethodDef array_methods[];
NPY_NO_EXPORT const char *
npy_casting_to_string(NPY_CASTING casting);

/*
* Pathlib support.
* For Python >= 3.6, use the os.Pathlike interface.
* Else, for Python >= 3.4, use file = str(file) if file is a PurePath
* For older Python, do nothing.
*/
static inline PyObject *
NpyPath_PathlikeToFspath(PyObject *file)
{
#if PY_VERSION_HEX >= 0x03060000 /* os.pathlike arrived in 3.6 */
if (PyObject_HasAttrString(file, "__fspath__")) {
file = PyOS_FSPath(file);
}
return file;
#elif PY_VERSION_HEX >= 0x03040000 /* pathlib arrived in 3.4 */
PyObject *pathlib, *pathlib_PurePath;
int fileIsPurePath;

pathlib = PyImport_ImportModule("pathlib");
if (!pathlib) {
return NULL;
}
pathlib_PurePath = PyObject_GetAttrString(pathlib, "PurePath");
fileIsPurePath = PyObject_IsInstance(file, pathlib_PurePath);
Py_XDECREF(pathlib);
Py_XDECREF(pathlib_PurePath);
if (fileIsPurePath) {
file = PyObject_Str(file);
}
#endif
return file;
}

#endif
2 changes: 2 additions & 0 deletions numpy/core/src/multiarray/multiarraymodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2071,6 +2071,8 @@ array_fromfile(PyObject *NPY_UNUSED(ignored), PyObject *args, PyObject *keywds)
Py_XDECREF(type);
return NULL;
}

file = NpyPath_PathlikeToFspath(file);
if (PyString_Check(file) || PyUnicode_Check(file)) {
file = npy_PyFile_OpenFile(file, "rb");
if (file == NULL) {
Expand Down
19 changes: 19 additions & 0 deletions numpy/core/tests/test_multiarray.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@

from numpy.core.numeric import pickle

if sys.version_info[:2] >= (3, 4):
import pathlib
if sys.version_info[0] >= 3:
import builtins
else:
Expand Down Expand Up @@ -4540,6 +4542,23 @@ def test_roundtrip_filename(self):
y = np.fromfile(self.filename, dtype=self.dtype)
assert_array_equal(y, self.x.flat)

def test_roundtrip_pathlib(self):
if sys.version_info[:2] >= (3, 4):
p = pathlib.Path(self.filename)
self.x.tofile(p)
y = np.fromfile(p, dtype=self.dtype)
assert_array_equal(y, self.x.flat)

def test_roundtrip_dump_pathlib(self):
if sys.version_info[:2] >= (3, 4):
p = pathlib.Path(self.filename)
self.x.dump(p)
if sys.version_info[:2] >= (3, 6):
y = np.load(p)
else:
y = np.load(str(p))
assert_array_equal(y, self.x)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self.x.dump(p) is failing to write to the file on PyPy, while self.x.tofile(p) is working fine. Is there a call to flush missing somewhere, or a missing close to trigger the write?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, #13684.

def test_roundtrip_binary_str(self):
s = self.x.tobytes()
y = np.frombuffer(s, dtype=self.dtype)
Expand Down
0