8000 gh-74028: `concurrent.futures.Executor.map`: avoid reference cycles when an exception is raised by ebonnal · Pull Request #131701 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

gh-74028: concurrent.futures.Executor.map: avoid reference cycles when an exception is raised #131701

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 49 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
7dc850b
`test_concurrent_futures.test_map_exception`: assert no reference cyc…
ebonnal Mar 24, 2025
19edced
test there is no future in referrers
ebonnal Mar 25, 2025
1c3d01a
switch to assertTrue(all(...))
ebonnal Mar 25, 2025
fa790f1
add test msg
ebonnal Mar 25, 2025
e48d0e0
format msgs
ebonnal Mar 25, 2025
99850b8
switch to assert list empty for a better assertion error message
ebonnal Mar 25, 2025
6e78014
edit comment
ebonnal Mar 26, 2025
48ed15d
Merge remote-tracking branch 'cpython/main' into test/executor-map-re…
ebonnal Mar 26, 2025
f59c749
Merge branch 'main' into test/executor-map-ref-cycles
graingert Mar 30, 2025
3dc2461
assert there are no reference cycles at all
graingert Mar 30, 2025
4dce280
avoid a reference cycle in subinterpreter pools
graingert Mar 30, 2025
9bc69b9
test that traceback frames should not contain any variables referring…
ebonnal Mar 30, 2025
9c781c1
interpreter.WorkerContext.run: finally unassign exc_wrapper
ebonnal Mar 30, 2025
a438301
fix msg grammar
ebonnal Mar 30, 2025
ada6140
fix access to exception
ebonnal Mar 30, 2025
d5e8c7a
narrow test: traceback frames should not contain any exception
ebonnal Mar 30, 2025
c4b771a
clean up exc_wrapper with minimal indentation
ebonnal Mar 31, 2025
03f8ab4
test_map_exception: only search for exception that captures itself in…
ebonnal Mar 31, 2025
ef30122
concurrent.futures.process: avoid ref cycle in _sendback_result
ebonnal Mar 31, 2025
09b0819
concurrent.futures.process: avoid ref cycle in _process_worker
ebonnal Mar 31, 2025
4fab860
format test msg
ebonnal Mar 31, 2025
0714013
Merge remote-tracking branch 'cpython/main' into test/executor-map-re…
ebonnal Mar 31, 2025
f5135ab
refactor ZeroDivisionError assert
ebonnal Mar 31, 2025
855c427
Revert "concurrent.futures.process: avoid ref cycle in _process_worker"
ebonnal Mar 31, 2025
4504516
Revert "concurrent.futures.process: avoid ref cycle in _sendback_result"
ebonnal Mar 31, 2025
fc7a569
skip referrers test for free-threading build on win/linux
ebonnal Mar 31, 2025
cae6474
call gc.collect for free-threading on linux/windows
ebonnal Mar 31, 2025
4abb239
Revert "call gc.collect for free-threading on linux/windows"
ebonnal Apr 1, 2025
4bc07c0
Revert "skip referrers test for free-threading build on win/linux"
ebonnal Apr 1, 2025
7b7a5e6
Merge branch 'main' into test/executor-map-ref-cycles
graingert Apr 3, 2025
3b00c5f
more gc referrers
graingert Apr 3, 2025
1948eb3
even more gc referrers
graingert Apr 3, 2025
37b7cb3
Merge branch 'main' into test/executor-map-ref-cycles
graingert Apr 4, 2025
9184964
delete a ref to work_item
graingert Apr 4, 2025
7e5300c
delete reference to result_item before joining process
graingert Apr 4, 2025
c5e39ee
unpack ResultItem and WorkItem
graingert Apr 4, 2025
548d517
fix typo
graingert Apr 5, 2025
fac7e89
Merge branch 'main' into test/executor-map-ref-cycles
graingert Apr 5, 2025
fd9a89e
bit of a hack?
graingert Apr 5, 2025
dbaa044
Revert investigations
ebonnal Apr 7, 2025
869e4b7
pause before checking referrers
ebonnal Apr 7, 2025
a40b9b8
Merge remote-tracking branch 'cpython/main' into test/executor-map-re…
ebonnal Apr 7, 2025
1ee9bf4
add comment about the avoided ref cycle with exc_wrapper
ebonnal Apr 7, 2025
95765f6
format test
ebonnal Apr 7, 2025
9020971
Merge remote-tracking branch 'cpython/main' into test/executor-map-re…
ebonnal Apr 7, 2025
a5c6b17
make the self-captured exceptions check not rely on var name
ebonnal Apr 7, 2025
0f8748c
use traceback.walk_tb
ebonnal Apr 7, 2025
85a3c36
comment the skipping of the current frame
ebonnal Apr 7, 2025
82542f3
test_map_exception: improve readability
ebonnal Apr 23, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
test_map_exception: improve readability
  • Loading branch information
ebonnal committed Apr 23, 2025
commit 82542f398da251e3d3fd9fad0ba8b93a1a92b92f
25 changes: 13 additions & 12 deletions Lib/test/test_concurrent_futures/executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
import traceback
import weakref
from concurrent import futures
from operator import add, itemgetter
from operator import add
from test import support
from test.support import Py_GIL_DISABLED

Expand Down Expand Up @@ -60,36 +60,37 @@ def test_map_exception(self):

exception = None
try:
next(i)
i.__next__()
except Exception as e:
exception = e
self.assertTrue(
isinstance(exception, ZeroDivisionError),
msg="next should raise a ZeroDivisionError",
msg="should raise a ZeroDivisionError",
)

# free-threading builds need this pause on Ubuntu (ARM) and Windows
# pause needed for free-threading builds on Ubuntu (ARM) and Windows
time.sleep(1)

self.assertFalse(
gc.get_referrers(exception),
msg="the exception should not have any referrers",
)

frames = map(itemgetter(0), traceback.walk_tb(exception.__traceback__))

# skip current frame
next(frames)

self.assertFalse(
[
(var, val)
for frame in frames
# go through the frames of the exception's traceback
for frame, _ in traceback.walk_tb(exception.__traceback__)
# skipping the current frame
if frame is not exception.__traceback__.tb_frame
# go through the locals captured in that frame
for var, val in frame.f_locals.items()
# check if one of them is an exception
if isinstance(val, Exception)
and frame in map(itemgetter(0), traceback.walk_tb(val.__traceback__))
# check if it is captured in its own traceback
and frame is val.__traceback__.tb_frame
],
msg=f"the exception's traceback should not contain an exception that captures itself in its own traceback",
msg=f"the exception's traceback should not contain an exception captured in its own traceback",
)

@support.requires_resource('walltime')
Expand Down
Loading
0