8000 gh-120754: Reduce system calls in full-file readall case by cmaloney · Pull Request #120755 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-120754: Reduce system calls in full-file readall case #120755

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Jul 4, 2024
Merged

gh-120754: Reduce system calls in full-file readall case #120755

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
78c4de0
Reduce system calls in full-file readall case
cmaloney Jun 18, 2024
30d335e
📜🤖 Added by blurb_it.
blurb-it[bot] Jun 19, 2024
9d7f925
Update news to pass lint
cmaloney Jun 19, 2024
dd0b294
Fix warning around type coercion on 64 bit windows; range is explicit…
cmaloney Jun 19, 2024
7ad6fa8
Change constant to named constant per review
cmaloney Jun 27, 2024
fa9ac6a
Move downcast to Py_SAFE_DOWNCAST
cmaloney Jun 27, 2024
93aee47
Update Misc/NEWS.d/next/Core and Builtins/2024-06-19-19-54-35.gh-issu…
hauntsaninja Jun 28, 2024
39e48ee
Update pyio to try and fstat and seek less
cmaloney Jun 29, 2024
b7d3880
Cap read size at _PY_READ_MAX to fix windows x86
cmaloney Jun 29, 2024
a4c2cb6
Cap initial bufsize on pyio
cmaloney Jun 29, 2024
84bd2d8
Pyio was relying on getting the size after truncate for test_largefil…
cmaloney Jun 29, 2024
b505334
Comment formatting
erlend-aasland Jul 2, 2024
7e276ec
size_estimated -> estimated_size
cmaloney Jul 3, 2024
9be6d1d
Py_SAFE_DOWNCAST -> C casts
cmaloney Jul 3, 2024
dc8e910
Apply suggestions from code review
cmaloney Jul 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Change constant to named constant per review
  • Loading branch information
@cmaloney
cmaloney committed Jun 27, 2024
commit 7ad6fa81faf35f194c7daf0373c2aef571c56fdc
7 changes: 5 additions & 2 deletions Modules/_io/fileio.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@
# define SMALLCHUNK BUFSIZ
#endif

/* Size at which a buffer is considered "large" and behavior should change to
avoid excessive memory allocation */
#define LARGE_BUFFER_CUTOFF_SIZE 65536

/*[clinic input]
module _io
Expand Down Expand Up @@ -689,7 +692,7 @@ new_buffersize(fileio *self, size_t currentsize)
giving us amortized linear-time behavior. For bigger sizes, use a
less-than-double growth factor to avoid excessive allocation. */
assert(currentsize <= PY_SSIZE_T_MAX);
if (currentsize > 65536)
if (currentsize > LARGE_BUFFER_CUTOFF_SIZE)
addend = currentsize >> 3;
else
addend = 256 + currentsize;
Expand Down Expand Up @@ -739,7 +742,7 @@ _io_FileIO_readall_impl(fileio *self)
then calls readall() to get the rest, which would result in allocating
more than required. Guard against that for larger files where we expect
the I/O time to dominate anyways while keeping small files fast. */
if (bufsize > 65536) {
if (bufsize > LARGE_BUFFER_CUTOFF_SIZE) {
Py_BEGIN_ALLOW_THREADS
_Py_BEGIN_SUPPRESS_IPH
#ifdef MS_WINDOWS
Expand Down
0