8000 gh-89727: Fix os.walk RecursionError on deep trees by jonburdo · Pull Request #99803 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-89727: Fix os.walk RecursionError on deep trees #99803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Dec 19, 2022
Merged

gh-89727: Fix os.walk RecursionError on deep trees #99803

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
99f4691
gh-89727: Fix os.walk RecursionError on deep trees
jonburdo Nov 26, 2022
3552d49
add unit test to ensure os.walk is not affect by recursion limit
jonburdo Dec 16, 2022
3078ea6
move os._walk code to os.walk
jonburdo Dec 16, 2022
f37bbe8
fix blurb to mention os.walk
jonburdo Dec 16, 2022
f26a5b8
use deque in os.walk
jonburdo Dec 16, 2022
955d21b
Merge branch 'main' into iterative-os-walk
jonburdo Dec 16, 2022
8fbfb2e
Merge branch 'main' into iterative-os-walk
jonburdo Dec 16, 2022
ef46eda
move deque import inside os.walk
jonburdo Dec 16, 2022
e261b9f
rename temp_recursion_limit to set_recursion_limit
jonburdo Dec 16, 2022
5ec53f7
Merge branch 'main' into iterative-os-walk
jonburdo Dec 16, 2022
507b650
switch back to list instead of deque for os.walk stack
jonburdo Dec 16, 2022
1c35610
Apply suggestions from code review
jonburdo Dec 18, 2022
73138a6
clean up os.walk comments
jonburdo Dec 18, 2022
2814cc5
test correct value in test_walk_above_recursion_limit
jonburdo Dec 18, 2022
37f3cc7
set islink and join before loop in os.walk
jonburdo Dec 18, 2022
d9c766c
Merge branch 'main' into iterative-os-walk
jonburdo Dec 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Apply suggestions from code review 
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
  • Loading branch information
@jonburdo @JelleZijlstra
jonburdo and JelleZijlstra authored Dec 18, 2022
commit 1c356102d80ac154b9379c86603d975d2dfe20e0
12 changes: 5 additions & 7 deletions Lib/os.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,13 +353,11 @@ def walk(top, topdown=True, onerror=None, followlinks=False):
walk_dirs = []

# We may not have read permission for top, in which case we can't
# get a list of the files the directory contains. os.walk
# always suppressed the exception then, rather than blow up for a
# get a list of the files the directory contains.
# We suppressed the exception here, rather than blow up for a
# minor reason when (say) a thousand readable directories are still
# left to visit. That logic is copied here.
# left to visit.
try:
# Note that scandir is global in this module due
# to earlier import-*.
scandir_it = scandir(top)
except OSError as error:
if onerror is not None:
Expand All @@ -384,7 +382,7 @@ def walk(top, topdown=True, onerror=None, followlinks=False):
is_dir = entry.is_dir()
except OSError:
# If is_dir() raises an OSError, consider that the entry is not
# a directory, same behaviour than os.path.isdir().
# a directory, same behaviour as os.path.isdir().
is_dir = False

if is_dir:
Expand Down Expand Up @@ -419,7 +417,7 @@ def walk(top, topdown=True, onerror=None, followlinks=False):
islink, join = path.islink, path.join
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're caching these functions in locals, might as well do it outside the main while loop to micro-optimize a bit more.

for dirname in reversed(dirs):
new_path = join(top, dirname)
# Issue #23605: os.path.islink() is used instead of caching
# bpo-23605: os.path.islink() is used instead of caching
# entry.is_symlink() result during the loop on os.scandir() because
# the caller can replace the directory entry during the "yield"
# above.
Expand Down
0