8000 bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread by asvetlov · Pull Request #14344 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

bpo-35621: Support running subprocesses in asyncio when loop is executed in non-main thread #14344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Jun 30, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update documentation
  • Loading branch information
asvetlov committed Jun 24, 2019
commit efb19c08449ee2ab29c68e63f73c3d05f4b3ebf1
62 changes: 54 additions & 8 deletions Doc/library/asyncio-policy.rst
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ asyncio ships with the following built-in policies:

.. availability:: Windows.

.. _asyncio-watchers:

Process Watchers
================
Expand All @@ -129,10 +130,11 @@ In asyncio, child processes are created with
:func:`create_subprocess_exec` and :meth:`loop.subprocess_exec`
functions.

asyncio defines the :class:`AbstractChildWatcher` abstract base class,
which child watchers should implement, and has two different
implementations: :class:`SafeChildWatcher` (configured to be used
by default) and :class:`FastChildWatcher`.
asyncio defines the :class:`AbstractChildWatcher` abstract base class, which child
watchers should implement, and has four different implementations:
:class:`ThreadedChildWatcher` (configured to be used by default),
:class:`MultiLoopChildWatcher`, :class:`SafeChildWatcher`, and
:class:`FastChildWatcher`.

See also the :ref:`Subprocess and Threads <asyncio-subprocess-threads>`
section.
Expand Down Expand Up @@ -184,23 +186,64 @@ implementation used by the asyncio event loop:

Note: loop may be ``None``.

.. method:: is_active()

Returns ``True`` is the watcher is ready to use, e.g. :py:data:`SIGCHLD`
handler is installed, a loop is attached etc. Spawning a subprocess with
*inactive* current child watcher raises :exc:`RuntimeError`.

.. versionadded:: 3.8

.. method:: close()

Close the watcher.

This method has to be called to ensure that underlying
resources are cleaned-up.

.. class:: SafeChildWatcher
.. class:: ThreadedChildWatcher

This implementation starts a new waiting thread for every subprocess spawn.

The watcher can be used by asyncio code executed in non-main thread without any
limitation.

There is no noticeable overhead when handling a big number of children (*O(1)* each
time a child terminates) but stating a thread per process is not free.

asyncio uses this safe implementation by default.

.. versionadded:: 3.8

.. class:: MultiLoopChildWatcher

This implementation avoids disrupting other code spawning processes
This implementation registers a :py:data:`SIGCHLD` signal handler on
instantiation (which may conflict with other code that install own handler for this
signal).

The watcher avoids disrupting other code spawning processes
by polling every process explicitly on a :py:data:`SIGCHLD` signal.

This is a safe solution but it has a significant overhead when
There is no limitation for running subprocesses from different threads once the
watcher is installed.

The solution is safe but it has a significant overhead when
handling a big number of processes (*O(n)* each time a
:py:data:`SIGCHLD` is received).

asyncio uses this safe implementation by default.
.. versionadded:: 3.8

.. class:: SafeChildWatcher

This implementation uses active event loop from the main thread to handle
:py:data:`SIGCHLD` signal. If the main thread has no running event loop another
thread cannot spawn a subprocess (:exc:`RuntimeError` is raised).

The watcher avoids disrupting other code spawning processes
by polling every process explicitly on a :py:data:`SIGCHLD` signal.

This solution is as safe as :class:`MultiLoopChildWatcher` and has the same *O(N)*
complexity but requires a running event loop in the main thread to work.

.. class:: FastChildWatcher

Expand All @@ -211,6 +254,9 @@ implementation used by the asyncio event loop:
There is no noticeable overhead when handling a big number of
children (*O(1)* each time a child terminates).

This solution requires a running event loop in the main thread to work, as
:class:`SafeChildWatcher`.


Custom Policies
===============
Expand Down
23 changes: 14 additions & 9 deletions Doc/library/asyncio-subprocess.rst
Original file line number Diff line number Diff line change
Expand Up @@ -293,18 +293,23 @@ their completion.
Subprocess and Threads
----------------------

Standard asyncio event loop supports running subprocesses from
different threads, but there are limitations:
Standard asyncio event loop supports running subprocesses from different threads by
default.

* An event loop must run in the main thread.
On Windows subprocesses are provided by :class:`ProactorEventLoop` only (default),
:class:`SelectorEventLoop` has no subprocess support.

* The child watcher must be instantiated in the main thread
before executing subprocesses from other threads. Call the
:func:`get_child_watcher` function in the main thread to instantiate
the child watcher.
On UNIX *child watchers* are used for subprocess finish waiting, see
:ref:`asyncio-watchers` for more info.

Note that alternative event loop implementations might not share
the above limitations; please refer to their documentation.

.. versionchanged:: 3.8

UNIX switched to use :class:`ThreadedChildWatcher` for spawning subprocesses from
different threads without any limitation.

Note that alternative event loop implementations might have own limitations;
please refer to their documentation.

.. seealso::

Expand Down
0