8000 GH-111693: Propagate correct asyncio.CancelledError instance out of asyncio.Condition.wait() by kristjanvalur · Pull Request #111694 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

GH-111693: Propagate correct asyncio.CancelledError instance out of asyncio.Condition.wait() #111694

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Jan 8, 2024
Merged

GH-111693: Propagate correct asyncio.CancelledError instance out of asyncio.Condition.wait() #111694

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
c3aff47
Add test and fix case where original CancelledError was not re-raised.
kristjanvalur Nov 3, 2023
111e74a
Simplify Lock/Semaphore error handling, also deal with any error.
kristjanvalur Nov 3, 2023
6d41771
Remove vestigial code from Future
kristjanvalur Nov 4, 2023
4e16fa7
fix whitespace
kristjanvalur Nov 4, 2023
ad56f29
Apply suggestions from code review
kristjanvalur Nov 20, 2023
30e0ec4
Restructure try/except, and improve comments
kristjanvalur Nov 21, 2023
6d59820
Merge remote-tracking branch 'kristjan/kristjan/CancelledError' into …
kristjanvalur Nov 21, 2023
c060f40
Apply suggestions from code review
kristjanvalur Nov 21, 2023
62bd6cd
Suggestsions from code review.
kristjanvalur Nov 26, 2023
01046c5
comment and add tests for waking up multiple FIFO tasks in the Semaphore
kristjanvalur Nov 26, 2023
8802dfa
Fix comment typo
gvanrossum Dec 30, 2023
f7c103c
📜🤖 Added by blurb_it.
blurb-it[bot] Jan 7, 2024
1572b2b
Format, capitalize, punctuate comments.
kristjanvalur Jan 7, 2024
dce420e
Merge branch 'main& 8000 #39; into kristjan/CancelledError
kristjanvalur Jan 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions Lib/asyncio/futures.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,9 +138,6 @@ def _make_cancelled_error(self):
exc = exceptions.CancelledError()
else:
exc = exceptions.CancelledError(self._cancel_message)
exc.__context__ = self._cancelled_exc
# Remove the reference since we don't need this anymore.
self._cancelled_exc = None
return exc

def cancel(self, msg=None):
Expand Down
49 changes: 26 additions & 23 deletions Lib/asyncio/locks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,9 @@ async def acquire(self):
This method blocks until the lock is unlocked, then sets it to
locked and returns True.
"""
# implement fair scheduling, where thread always waits
# its turn.
# Jumping the queue if all are cancelled is an optimization.
if (not self._locked and (self._waiters is None or
all(w.cancelled() for w in self._waiters))):
self._locked = True
Expand All @@ -105,19 +108,16 @@ async def acquire(self):
fut = self._get_loop().create_future()
self._waiters.append(fut)

# Finally block should be called before the CancelledError
# handling as we don't want CancelledError to call
# _wake_up_first() and attempt to wake up itself.
try:
try:
await fut
finally:
self._waiters.remove(fut)
except exceptions.CancelledError:
await fut
except BaseException:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still not very comfortable with widening this exception net for the purpose of sneaking in support for your library. If we want that to be supported we should make it an explicit feature, everywhere, rather than just tweaking an except clause here and there until your tests pass. Without any comments explaining the intended guarantee, what's to stop the next clever maintainer from narrowing the exception being caught to CancelledError again?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point. I quite understand that you are reluctant to touch this just to humour an eccentric experimental library which is trying to push the envelope of the original design. And I'm happy to have this rejected as long as we have had the chance to discuss it. I'll add a comment here in the mean time, just for safety.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC you're planning to make your library to use a subclass of CancelledError, so you don't need this to be more general anyway. So I'd like to see CancelledError here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Of course, IMO, it would be even greater if in some future version, we would have something like:

class InterruptError(BaseException):
    pass

class CancelledError(InterruptError):
   pass

but that is future music :)

self._waiters.remove(fut)
if not self._locked:
# Error occurred after release() was called, must re-do release
self._wake_up_first()
raise

self._waiters.remove(fut)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I look at this more carefully, I think I like the original version using nested try blocks better. Reading the new code I have to correlate the two separate removals to prove to myself that as soon as we've awaited fut (whether it got a return value or was cancelled) it is removed from the list of waiters. And that's why it was written like that originally.

(Note that try blocks cause no overhead unless an exception is actually raised -- the compiler even duplicates the finally block to ensure this.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I guess it makes it simpler to reason about, good to know that nested blocks have no overhead anymore.

self._locked = True
return True

Expand Down Expand Up @@ -269,17 +269,22 @@ async def wait(self):
self._waiters.remove(fut)

finally:
# Must reacquire lock even if wait is cancelled
cancelled = False
# Must reacquire lock even if wait is cancelled.
# We only catch CancelledError here, since we don't want any
# other (fatal) errors with the future to cause us to spin.
err = None
while True:
try:
await self.acquire()
break
except exceptions.CancelledError:
cancelled = True
except exceptions.CancelledError as e:
err = e

if cancelled:
raise exceptions.CancelledError
if err:
try:
raise err # re-raise same exception instance
finally:
err = None # brake reference cycles

async def wait_for(self, predicate):
"""Wait until a predicate becomes true.
Expand Down Expand Up @@ -378,20 +383,18 @@ async def acquire(self):
fut = self._get_loop().create_future()
self._waiters.append(fut)

# Finally block should be called before the CancelledError
# handling as we don't want CancelledError to call
# _wake_up_first() and attempt to wake up itself.
try:
try:
await fut
finally:
self._waiters.remove(fut)
except exceptions.CancelledError:
if not fut.cancelled():
await fut
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thing as before -- I actually like the nested try/finally phrasing better. Also, again, sneaky about catching all errors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point, especially if it costs nothing.

except BaseException:
self._waiters.remove(fut)
if fut.done() and not fut.cancelled():
# Error occurred after release() was called for us. Must undo
# the bookkeeping done there and retry.
self._value += 1
self._wake_up_next()
raise

self._waiters.remove(fut)
if self._value > 0:
self._wake_up_next()
return True
Expand Down
57 changes: 57 additions & 0 deletions Lib/test/test_asyncio/test_locks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -758,6 +758,63 @@ async def test_timeout_in_block(self):
with self.assertRaises(asyncio.TimeoutError):
await asyncio.wait_for(condition.wait(), timeout=0.5)

async def test_cancelled_error_wakeup(self):
"""Test that a cancelled error, received when awaiting wakeup
will be re-raised un-modified.
"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please convert the docstrings on the tests to comments -- test docstrings tend to be printed by the test runner, messing up the neat output. Note how no other tests here have docstrings.

wake = False
raised = None
cond = asyncio.Condition()

async def func():
nonlocal raised
async with cond:
with self.assertRaises(asyncio.CancelledError) as err:
await cond.wait_for(lambda: wake)
raised = err.exception
raise raised

task = asyncio.create_task(func())
await asyncio.sleep(0)
# Task is waiting on the condition, cancel it there
task.cancel(msg="foo")
with self.assertRaises(asyncio.CancelledError) as err:
await task
self.assertEqual(err.exception.args, ("foo",))
# we should have got the _same_ exception instance as the one originally raised
self.assertIs(err.exception, raised)

async def test_cancelled_error_re_aquire(self):
"""Test that a cancelled error, received when re-aquiring lock,
will be re-raised un-modified.
"""
wake = False
raised = None
cond = asyncio.Condition()

async def func():
nonlocal raised
async with cond:
with self.assertRaises(asyncio.CancelledError) as err:
await cond.wait_for(lambda: wake)
raised = err.exception
raise raised

task = asyncio.create_task(func())
await asyncio.sleep(0)
# Task is waiting on the condition
await cond.acquire()
wake = True
cond.notify()
await asyncio.sleep(0)
# task is now trying to re-acquire the lock, cancel it there
task.cancel(msg="foo")
cond.release()
with self.assertRaises(asyncio.CancelledError) as err:
await task
self.assertEqual(err.exception.args, ("foo",))
# we should have got the _same_ exception instance as the one originally raised
self.assertIs(err.exception, raised)

class SemaphoreTests(unittest.IsolatedAsyncioTestCase):

Expand Down
0