8000 gh-124694: Add concurrent.futures.InterpreterPoolExecutor by ericsnowcurrently · Pull Request #124548 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

gh-124694: Add concurrent.futures.InterpreterPoolExecutor #124548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
5c69d38
Make ThreadPoolExecutor extensible.
ericsnowcurrently Sep 25, 2024
01789be
Add InterpreterPoolExecutor.
ericsnowcurrently Sep 25, 2024
6def4be
Clean up the interpreter if initialize() fails.
ericsnowcurrently Sep 27, 2024
84993a5
Add a missing import.
ericsnowcurrently Sep 27, 2024
c540cf0
Fix some typos.
ericsnowcurrently Sep 27, 2024
45d584d
Add more tests.
ericsnowcurrently Sep 27, 2024
c90c016
Add docs.
ericsnowcurrently Sep 27, 2024
1cb4657
Add a NEwS entry.
ericsnowcurrently Sep 27, 2024
4dc0989
Fix the last test.
ericsnowcurrently Sep 27, 2024
57b2db6
Add more tests.
ericsnowcurrently Sep 27, 2024
75e11d2
Simplify ExecutionFailed.
ericsnowcurrently Sep 30, 2024
69c2b8e
Fix the signature of resolve_task().
ericsnowcurrently Sep 30, 2024
f03c314
Capture any uncaught exception.
ericsnowcurrently Sep 30, 2024
4806d9f
Add TODO comments.
ericsnowcurrently Sep 30, 2024
efc0395
Docs fixes.
ericsnowcurrently Sep 30, 2024
a29aee3
Automatically apply textwrap.dedent() to scripts.
ericsnowcurrently Sep 30, 2024
8bab457
Fix the WASI build.
ericsnowcurrently Sep 30, 2024
cd29914
wasi
ericsnowcurrently Oct 1, 2024
0287f3b
Ignore race in test.
er 8000 icsnowcurrently Oct 1, 2024
80cd7b1
Add BrokenInterpreterPool.
ericsnowcurrently Oct 8, 2024
f8d4273
Tweak the docs.
ericsnowcurrently Oct 8, 2024
3a8bfce
Clarify the InterpreterPoolExecutor docs.
ericsnowcurrently Oct 8, 2024
af6c27a
Catch all exceptions.
ericsnowcurrently Oct 8, 2024
8c0a405
Factor out exception serialization helpers.
ericsnowcurrently Oct 8, 2024
1ae7ca2
Set the ExecutionFailed error as __cause__.
ericsnowcurrently Oct 8, 2024
d24e85d
Drop the exception serialization helpers.
ericsnowcurrently Oct 8, 2024
05a03ad
Always finalize if there is an error in initialize().
ericsnowcurrently Oct 8, 2024
f150931
Explicitly note the problem with functions defined in __main__.
ericsnowcurrently Oct 8, 2024
97d0292
Handle the case where interpreters.queues doesn't exist.
ericsnowcurrently Oct 8, 2024
baf0504
Merge branch 'main' into interpreter-pool-executor
ericsnowcurrently Oct 15, 2024
5c3a327
Add a What's New entry about InterpreterPoolExecutor.
ericsnowcurrently Oct 15, 2024
a2032a8
Fix a typo.
ericsnowcurrently Oct 15, 2024
54119b8
Fix the documented signature.
ericsnowcurrently Oct 15, 2024
744dca7
Test and document asyncio support.
ericsnowcurrently Oct 15, 2024
f61d62d
Apply suggestions from code review
ericsnowcurrently Oct 16, 2024
ee65bb2
Expand the docs.
ericsnowcurrently Oct 16, 2024
a7f5c50
For now, drop support for scripts.
ericsnowcurrently Oct 16, 2024
b148e09
Fix a TODO comment.
ericsnowcurrently Oct 16, 2024
e365ae7
Fix the docs.
ericsnowcurrently Oct 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Clarify the InterpreterPoolExecutor docs.
  • Loading branch information
ericsnowcurrently committed Oct 8, 2024
commit 3a8bfcef5567c54ada62f6c6d712d7557ef43143
44 changes: 20 additions & 24 deletions Doc/library/concurrent.futures.rst
Original file line number Diff line number Diff line change
Expand Up @@ -246,37 +246,33 @@ objects can be executed and returned.
using a pool of at most *max_workers* threads. Each thread runs
tasks in its own interpreter.

*initializer* and *initargs* are the same as with
:class:`ThreadPoolExecutor`, though they are pickled like with
:class:`ProcessPoolExecutor`. Additionally, you can pass a script
(:class:`str`) for *initiazer*, which will be executed in the
interpreter's ``__main__`` module. In that case, *initargs* must
not be passed in.

Similarly you can pass a script to :meth:`~Executor.submit`, which
will be executed in the interpreter's ``__main__`` module. In that
case no arguments may be provided and the return value is always
``None``. Functions (and arguments) are pickled like we do with
the initializer.

For both *initializer* and :meth:`~Executor.submit`, if a script
is passed in then it will automatically have :func:`textwrap.dedent`
applied to it. That means you don't have to do so.

:meth:`~Executor.map` does *not* support passing in a script.

In each of those cases, an uncaught exception from the initializer
or task might not be suitable to send between interpreters, to be
raised as is. In tha case, an
:class:`~concurrent.futures.interpreter.ExecutionFailed` exception
is raised instead which contains a summary of the original exception.
*initializer* may be a callable and *initargs* a tuple of arguments,
just like with :class:`ThreadPoolExecutor`. However, they are pickled
like with :class:`ProcessPoolExecutor`. Likewise, functions (and
arguments) passed to :meth:`~Executor.submit` are pickled.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
*initializer* may be a callable and *initargs* a tuple of arguments,
just like with :class:`ThreadPoolExecutor`. However, they are pickled
like with :class:`ProcessPoolExecutor`. Likewise, functions (and
arguments) passed to :meth:`~Executor.submit` are pickled.
An *initializer* may be a callable and *initargs* a tuple of arguments,
similar to the behavior of :class:`ThreadPoolExecutor`. Additionally, they are pickled
like with :class:`ProcessPoolExecutor`. Likewise, functions (and
arguments) passed to :meth:`~Executor.submit` are pickled.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This motivated me to add a bit more clarity in the docs. 😄


*shared* is an optional dict of objects shared by all interpreters
in the pool. The items are added to each interpreter's ``__main__``
module. Not all objects are shareable. Those that are include
the builtin singletons, :class:`str` and :class:`bytes`,
and :class:`memoryview`. See :pep:`734` for more info.

You can also pass a script (:class:`str`) for *initiazer* or to
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can also pass a script (:class:`str`) for *initiazer* or to
You can also pass a script (:class:`str`) for *initializer* or to

I noticed one small thing. I'm looking forward to using subinterpreters!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

:meth:`~Executor.submit` (but not to :meth:`~Executor.map`).
In both cases, the script will be executed in the interpreter's
``__main__`` module. The executor will automatically apply
:func:`textwrap.dedent` to the script, so you don't have to do so.
With a script, arguments must not be passed in.
For :meth:`!Executor.submit`, the return value for a script
is always ``None``.

Normally an uncaught exception from *initializer* or from a submitted
task will be sent back between interpreters to be raised as is.
However, in some situations the exception might not be suitable to be
sent back. In that case, the executor raises an
:class:`~concurrent.futures.interpreter.ExecutionFailed` exception
instead, which contains a summary of the original exception.

The other caveats that apply to :class:`ThreadPoolExecutor` apply here.

.. versionadded:: next
Expand Down
0