8000 ENH: pathlib support for fromfile(), .tofile() and .dump() by sorenrasmussenai · Pull Request #12915 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content

ENH: pathlib support for fromfile(), .tofile() and .dump() #12915

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jun 4, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions doc/release/1.17.0-notes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,11 @@ concatenation.
In some cases where ``np.interp`` would previously return ``np.nan``, it now
returns an appropriate infinity.

Pathlib support for ``np.fromfile``, ``ndarray.tofile`` and ``ndarray.dump``
----------------------------------------------------------------------------
``np.fromfile``, ``np.ndarray.tofile`` and ``np.ndarray.dump`` now support
the `pathlib.Path` type for the ``file``/``fid`` parameter.

Specialized ``np.isnan``, ``np.isinf``, and ``np.isfinite`` ufuncs for bool and int types
-----------------------------------------------------------------------------------------
The boolean and integer types are incapable of storing ``np.nan`` and
Expand Down
17 changes: 14 additions & 3 deletions numpy/core/_add_newdocs.py
Original file line number Diff line number Diff line change
Expand Up @@ -1152,8 +1152,12 @@

Parameters
----------
file : file or str
file : file or str or Path
Open file object or filename.

.. versionchanged:: 1.17.0
`pathlib.Path` objects are now accepted.

dtype : data-type
Data type of the returned array.
For binary files, it is used to determine the size and byte-order
Expand Down Expand Up @@ -2962,9 +2966,12 @@

Parameters
----------
file : str
file : str or Path
A string naming the dump file.

.. versionchanged:: 1.17.0
`pathlib.Path` objects are now accepted.

"""))


Expand Down Expand Up @@ -4009,8 +4016,12 @@

Parameters
----------
fid : file or str
fid : file or str or Path
An open file object, or a string containing a filename.

.. versionchanged:: 1.17.0
`pathlib.Path` objects are now accepted.

sep : str
Separator between array items for text output.
If "" (empty), a binary file is written, equivalent to
Expand Down
4 changes: 4 additions & 0 deletions numpy/core/src/multiarray/methods.c
Original file line number Diff line number Diff line change
Expand Up @@ -577,6 +577,10 @@ array_tofile(PyArrayObject *self, PyObject *args, PyObject *kwds)
return NULL;
}

file = NpyPath_PathlikeToFspath(file);
if (file == NULL) {
return NULL;
}
if (PyBytes_Check(file) || PyUnicode_Check(file)) {
file = npy_PyFile_OpenFile(file, "wb");
if (file == NULL) {
Expand Down
23 changes: 23 additions & 0 deletions numpy/core/src/multiarray/methods.h
Original file line number Diff line number Diff line change
@@ -1,9 +1,32 @@
#ifndef _NPY_ARRAY_METHODS_H_
#define _NPY_ARRAY_METHODS_H_

#include "npy_import.h"

extern NPY_NO_EXPORT PyMethodDef array_methods[];

NPY_NO_EXPORT const char *
npy_casting_to_string(NPY_CASTING casting);

/* Pathlib support */
static inline PyObject *
NpyPath_PathlikeToFspath(PyObject *file)
{
static PyObject *os_PathLike = NULL;
static PyObject *os_fspath = NULL;
npy_cache_import("numpy.compat", "os_PathLike", &os_PathLike);
if (os_PathLike == NULL) {
return NULL;
}
npy_cache_import("numpy.compat", "os_fspath", &os_fspath);
if (os_fspath == NULL) {
return NULL;
}

if (!PyObject_IsInstance(file, os_PathLike)) {
return file;
}
return PyObject_CallFunctionObjArgs(os_fspath, file, NULL);
}

#endif
6 changes: 6 additions & 0 deletions numpy/core/src/multiarray/multiarraymodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -2078,6 +2078,12 @@ array_fromfile(PyObject *NPY_UNUSED(ignored), PyObject *args, PyObject *keywds)
Py_XDECREF(type);
return NULL;
}

file = NpyPath_PathlikeToFspath(file);
if (file == NULL) {
return NULL;
}

if (offset != 0 && strcmp(sep, "") != 0) {
PyErr_SetString(PyExc_TypeError, "'offset' argument only permitted for binary files");
return NULL;
Expand Down
22 changes: 22 additions & 0 deletions numpy/core/tests/test_multiarray.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,14 @@

from numpy.compat import pickle

try:
import pathlib
except ImportError:
try:
import pathlib2 as pathlib
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not actually sure our stuff works on pathlib2, or whether our CI exercises this code path.

Either way, if it turns out this test fails on systems using pathlib2, at least with the test present we'll likely get a bug report.

except ImportError:
pathlib = None

if sys.version_info[0] >= 3:
import builtins
else:
Expand Down Expand Up @@ -4639,6 +4647,20 @@ def test_roundtrip_filename(self):
y = np.fromfile(self.filename, dtype=self.dtype)
assert_array_equal(y, self.x.flat)

@pytest.mark.skipif(pathlib is None, reason="pathlib not found")
def test_roundtrip_pathlib(self):
p = pathlib.Path(self.filename)
self.x.tofile(p)
y = np.fromfile(p, dtype=self.dtype)
assert_array_equal(y, self.x.flat)

@pytest.mark.skipif(pathlib is None, reason="pathlib not found")
def test_roundtrip_dump_pathlib(self):
p = pathlib.Path(self.filename)
self.x.dump(p)
y = np.load(p, allow_pickle=True)
assert_array_equal(y, self.x)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self.x.dump(p) is failing to write to the file on PyPy, while self.x.tofile(p) is working fine. Is there a call to flush missing somewhere, or a missing close to trigger the write?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, #13684.

def test_roundtrip_binary_str(self):
s = self.x.tobytes()
y = np.frombuffer(s, dtype=self.dtype)
Expand Down
0