8000 gh-96471: Add asyncio queue shutdown by EpicWink · Pull Request #104228 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-96471: Add asyncio queue shutdown #104228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
Apr 6, 2024
Merged

gh-96471: Add asyncio queue shutdown #104228

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
440a702
Add asyncio queue shutdown
EpicWink Sep 1, 2022
fb458db
Fix queue shutdown
YvesDup Feb 10, 2023
e5951ac
📜🤖 Added by blurb_it.
blurb-it[bot] May 6, 2023
a72aedd
Merge remote-tracking branch 'upstream/main' into asyncio-queue-shutdown
EpicWink Feb 20, 2024
d5e925d
Add references in docs and news entry
EpicWink Feb 20, 2024
f3517fb
Merge remote-tracking branch 'upstream/main' into asyncio-queue-shutdown
EpicWink Mar 20, 2024
bd2a7c3
Improve docs
EpicWink Mar 20, 2024
e9ac8de
Consume queue on immediate shutdown
EpicWink Mar 20, 2024
1e7813a
Fix links in what's-new
EpicWink Mar 22, 2024
1275bb6
Merge remote-tracking branch 'upstream/main' into asyncio-queue-shutdown
EpicWink Mar 22, 2024
eec29bb
Fix formatting in news entry
EpicWink Mar 22, 2024
2c6156f
Merge remote-tracking branch 'upstream/main' into asyncio-queue-shutdown
EpicWink Mar 22, 2024
17f1f32
Merge remote-tracking branch 'upstream/main' into asyncio-queue-shutdown
EpicWink Mar 26, 2024
a233830
Improve tests
EpicWink Mar 26, 2024
420a247
Improve tests even more
EpicWink Mar 26, 2024
25ad2ac
Merge remote-tracking branch 'upstream/main' into asyncio-queue-shutdown
EpicWink Mar 26, 2024
f3321b4
Merge remote-tracking branch 'upstream/main' into asyncio-queue-shutdown
EpicWink Mar 27, 2024
6d9edd6
Document tests
EpicWink Mar 27, 2024
1135d85
Merge remote-tracking branch 'upstream/main' into asyncio-queue-shutdown
EpicWink Mar 28, 2024
ddc6ad6
Always allow getters to re-check queue empty
EpicWink Mar 28, 2024
2fa1bd9
Merge branch 'main' into asyncio-queue-shutdown
gvanrossum Apr 3, 2024
aef4063
Simplify shutdown-check in put and get
EpicWink Apr 4, 2024
d49c6dd
Format shutdown docstring
EpicWink Apr 4, 2024
5a435a6
Check for 0 unfinised tasks in shutdown
EpicWink Apr 4, 2024
c8db40e
Use asyncio.sleep to run other tasks
EpicWink Apr 4, 2024
ca01ee1
Use public method to shut down queue in format test
EpicWink Apr 4, 2024
b02c4dd
Only start queue join after shutdown in test
EpicWink Apr 4, 2024
8deca77
Test join before failing task-done
EpicWink Apr 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Improve docs
  • Loading branch information
@EpicWink
EpicWink committed Mar 20, 2024
commit bd2a7c397d75a0297e01dbaf2b19c3d313204a1a
14 changes: 7 additions & 7 deletions Doc/library/asyncio-queue.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,13 +105,13 @@ Queue
Shut down the queue, making :meth:`~Queue.get` and :meth:`~Queue.put`
raise :exc:`QueueShutDown`.

By default, :meth:`~Queue.get` on a shut down queue will only raise once
the queue is empty. Set *immediate* to true to make gets raise
immediately instead.
By default, :meth:`~Queue.get` on a shut down queue will only
raise once the queue is empty. Set *immediate* to true to make
:meth:`~Queue.get` raise immediately instead.

All blocked callers of :meth:`~Queue.put` will be unblocked. If
*immediate* is true, also unblock callers of :meth:`~Queue.get` and
:meth:`~Queue.join`.
*immediate* is true, also unblock callers of :meth:`~Queue.get`
and :meth:`~Queue.join`.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry but I have a doubt, shouldn't this documentation block be rather:

    All blocked callers of :meth:`~Queue.put` and :meth:`~Queue.get` 
    will be unblocked. If *immediate* is true, also unblock callers of 
    :meth:`~Queue.join`.

In event of change, the docstring of the shutdown method must be updated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

join callers aren't necessarily even unblocked anyway, if consumers are processing any items. I should probably say that a task is marked as done for each item in the queue if immediate shutdown.

Also, I think the threading queue docs are the same.

Copy link
Contributor
@YvesDup YvesDup Apr 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a task is marked as done for each item in the queue if immediate shutdown.

It's very precise, better.

Also, I think the threading queue docs are the same.

Yes, I commented here so as not to forget (see #117532 (comment)).

English is your native language, I think it'is best if you update documentations and docstrings.

Update: but I can create the follow-up PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've made a PR: #117621

.. versionadded:: 3.13

Expand Down Expand Up @@ -173,8 +173,8 @@ Exceptions

.. exception:: QueueShutDown

Exception raised when getting an item from or putting an item onto a
queue which has been shut down.
Exception raised when :meth:`~Queue.put` or :meth:`~Queue.get` is
called on a queue which has been shut down.

.. versionadded:: 3.13

Expand Down
4 changes: 4 additions & 0 deletions Doc/whatsnew/3.13.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,10 @@ asyncio
forcefully close an asyncio server.
(Contributed by Pierre Ossman in :gh:`113538`.)

* Add :meth:`asyncio.queues.Queue.shutdown` (along with
:exc:`asyncio.queues.QueueShutDown`) for queue termination.
(Contributed by Laurie Opperman and Yves Duprat in :gh:`104228`.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've worked well together, I'd be nice if you'd mention me.
:-)

base64
---

Expand Down
3 changes: 1 addition & 2 deletions Lib/asyncio/queues.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,8 +202,7 @@ async def get(self):
self._getters.remove(getter)
except ValueError:
# The getter could be removed from self._getters by a
# previous put_nowait call,
# or a shutdown call.
# previous put_nowait call, or a shutdown call.
pass
if not self.empty() and not getter.cancelled():
# We were woken up by put_nowait(), but can't take
Expand Down
0