-
-
Notifications
You must be signed in to change notification settings - Fork 32.4k
gh-124694: Add concurrent.futures.InterpreterPoolExecutor #124548
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
ericsnowcurrently
merged 39 commits into
python:main
from
ericsnowcurrently:interpreter-pool-executor
Oct 16, 2024
Merged
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit
Hold shift + click to select a range
5c69d38
Make ThreadPoolExecutor extensible.
ericsnowcurrently 01789be
Add InterpreterPoolExecutor.
ericsnowcurrently 6def4be
Clean up the interpreter if initialize() fails.
ericsnowcurrently 84993a5
Add a missing import.
ericsnowcurrently c540cf0
Fix some typos.
ericsnowcurrently 45d584d
Add more tests.
ericsnowcurrently c90c016
Add docs.
ericsnowcurrently 1cb4657
Add a NEwS entry.
ericsnowcurrently 4dc0989
Fix the last test.
ericsnowcurrently 57b2db6
Add more tests.
ericsnowcurrently 75e11d2
Simplify ExecutionFailed.
ericsnowcurrently 69c2b8e
Fix the signature of resolve_task().
ericsnowcurrently f03c314
Capture any uncaught exception.
ericsnowcurrently 4806d9f
Add TODO comments.
ericsnowcurrently efc0395
Docs fixes.
ericsnowcurrently a29aee3
Automatically apply textwrap.dedent() to scripts.
ericsnowcurrently 8bab457
Fix the WASI build.
ericsnowcurrently cd29914
wasi
ericsnowcurrently 0287f3b
Ignore race in test.
er
8000
icsnowcurrently 80cd7b1
Add BrokenInterpreterPool.
ericsnowcurrently f8d4273
Tweak the docs.
ericsnowcurrently 3a8bfce
Clarify the InterpreterPoolExecutor docs.
ericsnowcurrently af6c27a
Catch all exceptions.
ericsnowcurrently 8c0a405
Factor out exception serialization helpers.
ericsnowcurrently 1ae7ca2
Set the ExecutionFailed error as __cause__.
ericsnowcurrently d24e85d
Drop the exception serialization helpers.
ericsnowcurrently 05a03ad
Always finalize if there is an error in initialize().
ericsnowcurrently f150931
Explicitly note the problem with functions defined in __main__.
ericsnowcurrently 97d0292
Handle the case where interpreters.queues doesn't exist.
ericsnowcurrently baf0504
Merge branch 'main' into interpreter-pool-executor
ericsnowcurrently 5c3a327
Add a What's New entry about InterpreterPoolExecutor.
ericsnowcurrently a2032a8
Fix a typo.
ericsnowcurrently 54119b8
Fix the documented signature.
ericsnowcurrently 744dca7
Test and document asyncio support.
ericsnowcurrently f61d62d
Apply suggestions from code review
ericsnowcurrently ee65bb2
Expand the docs.
ericsnowcurrently a7f5c50
For now, drop support for scripts.
ericsnowcurrently b148e09
Fix a TODO comment.
ericsnowcurrently e365ae7
Fix the docs.
ericsnowcurrently File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Clarify the InterpreterPoolExecutor docs.
- Loading branch information
commit 3a8bfcef5567c54ada62f6c6d712d7557ef43143
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -246,37 +246,33 @@ objects can be executed and returned. | |||||
using a pool of at most *max_workers* threads. Each thread runs | ||||||
tasks in its own interpreter. | ||||||
|
||||||
*initializer* and *initargs* are the same as with | ||||||
:class:`ThreadPoolExecutor`, though they are pickled like with | ||||||
:class:`ProcessPoolExecutor`. Additionally, you can pass a script | ||||||
(:class:`str`) for *initiazer*, which will be executed in the | ||||||
interpreter's ``__main__`` module. In that case, *initargs* must | ||||||
not be passed in. | ||||||
|
||||||
Similarly you can pass a script to :meth:`~Executor.submit`, which | ||||||
will be executed in the interpreter's ``__main__`` module. In that | ||||||
case no arguments may be provided and the return value is always | ||||||
``None``. Functions (and arguments) are pickled like we do with | ||||||
the initializer. | ||||||
|
||||||
For both *initializer* and :meth:`~Executor.submit`, if a script | ||||||
is passed in then it will automatically have :func:`textwrap.dedent` | ||||||
applied to it. That means you don't have to do so. | ||||||
|
||||||
:meth:`~Executor.map` does *not* support passing in a script. | ||||||
|
||||||
In each of those cases, an uncaught exception from the initializer | ||||||
or task might not be suitable to send between interpreters, to be | ||||||
raised as is. In tha case, an | ||||||
:class:`~concurrent.futures.interpreter.ExecutionFailed` exception | ||||||
is raised instead which contains a summary of the original exception. | ||||||
*initializer* may be a callable and *initargs* a tuple of arguments, | ||||||
just like with :class:`ThreadPoolExecutor`. However, they are pickled | ||||||
like with :class:`ProcessPoolExecutor`. Likewise, functions (and | ||||||
arguments) passed to :meth:`~Executor.submit` are pickled. | ||||||
|
||||||
*shared* is an optional dict of objects shared by all interpreters | ||||||
in the pool. The items are added to each interpreter's ``__main__`` | ||||||
module. Not all objects are shareable. Those that are include | ||||||
the builtin singletons, :class:`str` and :class:`bytes`, | ||||||
and :class:`memoryview`. See :pep:`734` for more info. | ||||||
|
||||||
You can also pass a script (:class:`str`) for *initiazer* or to | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
I noticed one small thing. I'm looking forward to using subinterpreters! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. fixed |
||||||
:meth:`~Executor.submit` (but not to :meth:`~Executor.map`). | ||||||
In both cases, the script will be executed in the interpreter's | ||||||
``__main__`` module. The executor will automatically apply | ||||||
:func:`textwrap.dedent` to the script, so you don't have to do so. | ||||||
With a script, arguments must not be passed in. | ||||||
For :meth:`!Executor.submit`, the return value for a script | ||||||
is always ``None``. | ||||||
|
||||||
Normally an uncaught exception from *initializer* or from a submitted | ||||||
task will be sent back between interpreters to be raised as is. | ||||||
ericsnowcurrently marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
However, in some situations the exception might not be suitable to be | ||||||
sent back. In that case, the executor raises an | ||||||
:class:`~concurrent.futures.interpreter.ExecutionFailed` exception | ||||||
instead, which contains a summary of the original exception. | ||||||
|
||||||
The other caveats that apply to :class:`ThreadPoolExecutor` apply here. | ||||||
ericsnowcurrently marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
.. versionadded:: next | ||||||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This motivated me to add a bit more clarity in the docs. 😄