GH-115060: Speed up `pathlib.Path.glob()` by not scanning literal parts #117732

barneygale · 2024-04-10T21:43:21Z

Don't bother calling os.scandir() to scan for literal pattern segments, like foo in foo/*.py. Instead, append the segment(s) as-is and call through to the next selector with exists=False, which signals that the path might not exist. Subsequent selectors will call os.scandir() or os.lstat() to filter out missing paths as needed.

Timings:

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib'))"
5000 loops, best of 5: 69.4 usec per loop
20000 loops, best of 5: 13.6 usec per loop
# --> 5.1x faster

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/'))"
5000 loops, best of 5: 73.3 usec per loop
20000 loops, best of 5: 14.2 usec per loop
# --> 5.16x faster

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/*'))"
1000 loops, best of 5: 362 usec per loop
1000 loops, best of 5: 301 usec per loop
# --> 1.2x faster

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/*/__init__.py'))"
200 loops, best of 5: 1.18 msec per loop
1000 loops, best of 5: 273 usec per loop
# --> 4.32x faster

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/**/__init__.py'))"
50 loops, best of 5: 9.46 msec per loop
50 loops, best of 5: 5.72 msec per loop
# --> 1.65x faster

$ ./python -m timeit -s "from pathlib import Path; p = Path.cwd()" "list(p.glob('Lib/pathlib/__init__.py'))"
1000 loops, best of 5: 210 usec per loop
20000 loops, best of 5: 14.9 usec per loop
# --> 14.1x faster

Issue: Speed up pathlib.Path.glob() by removing redundant regex matching #115060

…al parts Don't bother calling `os.scandir()` to scan for literal pattern segments, like `foo` in `foo/*.py`. Instead, append the segment(s) as-is and call through to the next selector with `exists=False`, which signals that the path might not exist. Subsequent selectors will call `os.scandir()` or `os.lstat()` to filter out missing paths as needed.

…al parts (python#117732) Don't bother calling `os.scandir()` to scan for literal pattern segments, like `foo` in `foo/*.py`. Instead, append the segment(s) as-is and call through to the next selector with `exists=False`, which signals that the path might not exist. Subsequent selectors will call `os.scandir()` or `os.lstat()` to filter out missing paths as needed.

barneygale added performance Performance or resource usage topic-pathlib labels Apr 10, 2024

bedevere-app bot added the awaiting core review label Apr 10, 2024

bedevere-app bot mentioned this pull request Apr 10, 2024

Speed up pathlib.Path.glob() by removing redundant regex matching #115060

Closed

Fix tests, add comment.

4c48dc0

barneygale mentioned this pull request Apr 10, 2024

GH-116380: Speed up glob.glob() by removing some system calls #116392

Merged

barneygale added 2 commits April 11, 2024 00:35

More test fixes

0e7aee7

Once more with feeling

c4e3edc

barneygale merged commit 0eb52f5 into python:main Apr 12, 2024

bedevere-app bot removed the awaiting core review label Apr 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

GH-115060: Speed up `pathlib.Path.glob()` by not scanning literal parts #117732

GH-115060: Speed up `pathlib.Path.glob()` by not scanning literal parts #117732

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GH-115060: Speed up pathlib.Path.glob() by not scanning literal parts #117732

GH-115060: Speed up pathlib.Path.glob() by not scanning literal parts #117732

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

GH-115060: Speed up `pathlib.Path.glob()` by not scanning literal parts #117732

GH-115060: Speed up `pathlib.Path.glob()` by not scanning literal parts #117732