8000 bpo-37658: Actually return result in race condition by Dreamsorcerer · Pull Request #29202 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

bpo-37658: Actually return result in race condition #29202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Nov 29, 2021
Merged

bpo-37658: Actually return result in race condition #29202

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
019ed9d
Actually fix issue37658
Dreamsorcerer Oct 24, 2021
ffe5106
Correct test
Dreamsorcerer Oct 24, 2021
722fe3a
Update test_tasks.py
Dreamsorcerer Oct 24, 2021
54d0118
Remove redundant else condition.
Dreamsorcerer Oct 24, 2021
169294e
Avoid timing issues with test.
Dreamsorcerer Oct 25, 2021
5cfc8b7
Create 2021-11-28-15-30-34.bpo-37658.8Hno7d.rst
Dreamsorcerer Nov 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 2 additions & 6 deletions Lib/asyncio/tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -415,11 +415,9 @@ async def wait_for(fut, timeout):

await _cancel_and_wait(fut, loop=loop)
try:
fut.result()
return fut.result()
except exceptions.CancelledError as exc:
raise exceptions.TimeoutError() from exc
else:
raise exceptions.TimeoutError()

waiter = loop.create_future()
timeout_handle = loop.call_later(timeout, _release_waiter, waiter)
Expand Down Expand Up @@ -455,11 +453,9 @@ async def wait_for(fut, timeout):
# exception, we should re-raise it
# See https://bugs.python.org/issue40607
try:
fut.result()
return fut.result()
except exceptions.CancelledError as exc:
raise exceptions.TimeoutError() from exc
else:
raise exceptions.TimeoutError()
finally:
timeout_handle.cancel()

Expand Down
38 changes: 8 additions & 30 deletions Lib/test/test_asyncio/test_tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1009,20 +1009,16 @@ def gen():
self.assertEqual(res, "ok")

def test_wait_for_cancellation_race_condition(self):
def gen():
yield 0.1
yield 0.1
yield 0.1
yield 0.1
async def inner():
with contextlib.suppress(asyncio.CancelledError):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially I just emulated the race condition in the bug report, but because it's a race condition, it seems that the test occasionally failed due to actually hitting the timeout.

Instead, I've just suppressed the cancellation, which makes the behaviour 100% reproducible.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not an equivalent test. As I understand, the whole point of that test was to inject the cancellation at the exact point to cause the race on the return to ‘await waiter’ once fut is done. The new test is not checking this race at all but instead checking that a cancellation ignoring task can skip the timeout.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not an equivalent test, because the previous test is completely wrong. It does not emulate a race condition, and it does not reproduce the issue in the bug report.

I wrote a test which actually reproduces the race condition as given in the bug report, and the issue was not fixed. As mentioned above, I had to use this suppress cancellation trick to make the test reproducible (it will still run things in the same order as the race condition), the original test worked correctly everytime I ran it, but failed in one of the CI runs, and we can't have flaky tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can see the old version which was timing dependent in the commit history:
169294e

8000
await asyncio.sleep(1)
return 1

loop = self.new_test_loop(gen)
async def main():
result = await asyncio.wait_for(inner(), timeout=.01)
assert result == 1

fut = self.new_future(loop)
loop.call_later(0.1, fut.set_result, "ok")
task = loop.create_task(asyncio.wait_for(fut, timeout=1))
loop.call_later(0.1, task.cancel)
res = loop.run_until_complete(task)
self.assertEqual(res, "ok")
asyncio.run(main())

def test_wait_for_waits_for_task_cancellation(self):
loop = asyncio.new_event_loop()
Expand Down Expand Up @@ -1101,24 +1097,6 @@ async def inner():
with self.assertRaises(FooException):
loop.run_until_complete(foo())

def test_wait_for_raises_timeout_error_if_returned_during_cancellation(self):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test literally tests that we don't return a value when it has completed successfully...

I can't see any reason for this test, the issue linked to the PR does not mention anything that warrants this test existing: https://bugs.python.org/issue40607

So, as far as I can tell, this test should be binned and the behaviour should be as defined in the above changes.

loop = asyncio.new_event_loop()
self.addCleanup(loop.close)

async def foo():
async def inner():
try:
await asyncio.sleep(0.2)
except asyncio.CancelledError:
return 42

inner_task = self.new_task(loop, inner())

await asyncio.wait_for(inner_task, timeout=_EPSILON)

with self.assertRaises(asyncio.TimeoutError):
loop.run_until_complete(foo())

def test_wait_for_self_cancellation(self):
loop = asyncio.new_event_loop()
self.addCleanup(loop.close)
Expand Down
3 changes: 3 additions & 0 deletions Misc/NEWS.d/next/Library/2021-11-28-15-30-34.bpo-37658.8Hno7d.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Fix issue when on certain conditions ``asyncio.wait_for()`` may allow a
coroutine to complete successfully, but fail to return the result,
potentially causing memory leaks or other issues.
0