8000 gh-117151: IO performance improvement, increase io.DEFAULT_BUFFER_SIZE to 128k by morotti · Pull Request #118144 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

gh-117151: IO performance improvement, increase io.DEFAULT_BUFFER_SIZE to 128k #118144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Mar 7, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
remove constant _MAXIMUM_BUFFER_SIZE
  • Loading branch information
rmmancom committed Feb 10, 2025
commit 7ce54c4c7627e579c2e1c52b9918eeff3375e6c0
2 changes: 1 addition & 1 deletion Doc/library/functions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1406,7 +1406,7 @@ are always available. They are listed here in alphabetical order.
given, the default buffering policy works as follows:

* Binary files are buffered in fixed-size chunks; the size of the buffer
is max(min(blocksize, _MAXIMUM_BUFFER_SIZE), DEFAULT_BUFFER_SIZE)
is max(min(blocksize, 8 MiB), DEFAULT_BUFFER_SIZE)
when the device block size is available.
On most systems, the buffer will typically be 128 kilobytes long.

Expand Down
7 changes: 3 additions & 4 deletions 7 Lib/_pyio.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,9 @@
valid_seek_flags.add(os.SEEK_HOLE)
valid_seek_flags.add(os.SEEK_DATA)

# open() uses max(min(blocksize, _MAXIMUM_BUFFER_SIZE), DEFAULT_BUFFER_SIZE)
# open() uses max(min(blocksize, 8 MiB), DEFAULT_BUFFER_SIZE)
# when the device block size is available.
DEFAULT_BUFFER_SIZE = 128 * 1024 # bytes
_MAXIMUM_BUFFER_SIZE = 8192 * 1024 # bytes

# NOTE: Base classes defined here are registered with the "official" ABCs
# defined in io.py. We don't use real inheritance though, because we don't want
Expand Down Expand Up @@ -126,7 +125,7 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
given, the default buffering policy works as follows:

* Binary files are buffered in fixed-size chunks; the size of the buffer
is max(min(blocksize, _MAXIMUM_BUFFER_SIZE), DEFAULT_BUFFER_SIZE)
is max(min(blocksize, 8 MiB), DEFAULT_BUFFER_SIZE)
when the device block size is available.
On most systems, the buffer will typically be 128 kilobytes long.

Expand Down Expand Up @@ -244,7 +243,7 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
buffering = -1
line_buffering = True
if buffering < 0:
buffering = max(min(raw._blksize, _MAXIMUM_BUFFER_SIZE), DEFAULT_BUFFER_SIZE)
buffering = max(min(raw._blksize, 8192 * 1024), DEFAULT_BUFFER_SIZE)
if buffering < 0:
raise ValueError("invalid buffering size")
if buffering == 0:
Expand Down
11 changes: 3 additions & 8 deletions Modules/_io/_iomodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -60,11 +60,6 @@ PyDoc_STRVAR(module_doc,
"DEFAULT_BUFFER_SIZE\n"
"\n"
" An int containing the default buffer size used by the module's buffered\n"
" I/O classes.\n"
"\n"
"_MAXIMUM_BUFFER_SIZE\n"
"\n"
" An int containing the maximum buffer size used by the module's buffered\n"
" I/O classes.\n"
);

Expand Down Expand Up @@ -136,7 +131,7 @@ the size of a fixed-size chunk buffer. When no buffering argument is
given, the default buffering policy works as follows:

* Binary files are buffered in fixed-size chunks; the size of the buffer
is max(min(blocksize, _MAXIMUM_BUFFER_SIZE), DEFAULT_BUFFER_SIZE)
is max(min(blocksize, 8 MiB), DEFAULT_BUFFER_SIZE)
when the device block size is available.
On most systems, the buffer will typically be 128 kilobytes long.

Expand Down Expand Up @@ -204,7 +199,7 @@ static PyObject *
_io_open_impl(PyObject *module, PyObject *file, const char *mode,
int buffering, const char *encoding, const char *errors,
const char *newline, int closefd, PyObject *opener)
/*[clinic end generated code: output=aefafc4ce2b46dc0 input=a35153cf2829c537]*/
/*[clinic end generated code: output=aefafc4ce2b46dc0 input=28027fdaabb8d744]*/
{
size_t i;

Expand Down Expand Up @@ -372,7 +367,7 @@ _io_open_impl(PyObject *module, PyObject *file, const char *mode,
if (blksize_obj == NULL)
goto error;
buffering = PyLong_AsLong(blksize_obj);
buffering = Py_MAX(Py_MIN(buffering, _MAXIMUM_BUFFER_SIZE), DEFAULT_BUFFER_SIZE);
buffering = Py_MAX(Py_MIN(buffering, 8192 * 1024), DEFAULT_BUFFER_SIZE);
Py_DECREF(blksize_obj);
if (buffering == -1 && PyErr_Occurred())
goto error;
Expand Down
1 change: 0 additions & 1 deletion Modules/_io/_iomodule.h
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,6 @@ extern Py_ssize_t _PyIO_find_line_ending(
extern int _PyIO_trap_eintr(void);

#define DEFAULT_BUFFER_SIZE (128 * 1024) /* bytes */
#define _MAXIMUM_BUFFER_SIZE (8192 * 1024) /* bytes */

/*
* Offset type for positioning.
Expand Down
4 changes: 2 additions & 2 deletions Modules/_io/clinic/_iomodule.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0