gh-88050: Fix asyncio subprocess kill process cleanly when process is blocked #32073
Conversation
98f282f to
485b26b
Compare
|
What's wrong with Windows? |
…OC_m.rst Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>
This is the version I am most confident with. I wrote a summary about this with and without fix which would hopefully help you in reviewing. I have spent weeks on trying to get the best fix for this and this is the best I could come up with.
Regarding windows, if you see the tests in this file, you'll see that there are literally no tests testing killing of subprocess with |
|
Thanks @kumaraditya303 for the PR, and @gvanrossum for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11. |
|
GH-97915 is a backport of this pull request to the 3.11 branch. |
…rocess is blocked (pythonGH-32073) (cherry picked from commit 7015e13) Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
|
GH-97916 is a backport of this pull request to the 3.10 branch. |
…rocess is blocked (pythonGH-32073) (cherry picked from commit 7015e13) Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
* main: (66 commits) pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879) docs(typing): add "see PEP 675" to LiteralString (python#97926) pythongh-97850: Remove all known instances of module_repr() (python#97876) I changed my surname early this year (python#96671) pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768) pythongh-91539: improve performance of get_proxies_environment (python#91566) build(deps): bump actions/stale from 5 to 6 (python#97701) pythonGH-95172 Make the same version `versionadded` oneline (python#95172) pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073) pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774) pythongh-93357: Port test cases to IsolatedAsyncioTestCase, part 2 (python#97896) pythongh-95196: Disable incorrect pickling of the C implemented classmethod descriptors (pythonGH-96383) pythongh-97758: Fix a crash in getpath_joinpath() called without arguments (pythonGH-97759) pythongh-74696: Pass root_dir to custom archivers which support it (pythonGH-94251) pythongh-97661: Improve accuracy of sqlite3.Cursor.fetchone docs (python#97662) pythongh-87092: bring compiler code closer to a preprocessing-opt-assembler organisation (pythonGH-97644) pythonGH-96704: Add {Task,Handle}.get_context(), use it in call_exception_handler() (python#96756) pythongh-93738: Documentation C syntax (:c:type:`PyTypeObject*` -> :c:expr:`PyTypeObject*`) (python#97778) pythongh-97825: fix AttributeError when calling subprocess.check_output(input=None) with encoding or errors args (python#97826) Add re.VERBOSE flag documentation example (python#97678) ...
* main: (53 commits) pythongh-94808: Coverage: Test that maximum indentation level is handled (python#95926) pythonGH-88050: fix race in closing subprocess pipe in asyncio (python#97951) pythongh-93738: Disallow pre-v3 syntax in the C domain (python#97962) pythongh-95986: Fix the example using match keyword (python#95989) pythongh-97897: Prevent os.mkfifo and os.mknod segfaults with macOS 13 SDK (pythonGH-97944) pythongh-94808: Cover `PyUnicode_Count` in CAPI (python#96929) pythongh-94808: Cover `PyObject_PyBytes` case with custom `__bytes__` method (python#96610) pythongh-95691: Doc BufferedWriter and BufferedReader (python#95703) pythonGH-88968: Add notes about socket ownership transfers (python#97936) pythongh-96865: [Enum] fix Flag to use CONFORM boundary (pythonGH-97528) pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879) docs(typing): add "see PEP 675" to LiteralString (python#97926) pythongh-97850: Remove all known instances of module_repr() (python#97876) I changed my surname early this year (python#96671) pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768) pythongh-91539: improve performance of get_proxies_environment (python#91566) build(deps): bump actions/stale from 5 to 6 (python#97701) pythonGH-95172 Make the same version `versionadded` oneline (python#95172) pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073) pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774) ...
…rocess is blocked (python#32073)
|
AMD64 Ubuntu Shared 3.x buildbot started to fail recently with Out-of-Memory (OOM) errors: Python test processes are killed with SIGKILL by the Linux kernel. David Bolen, owner of the buildbot worker, wrote that there's currently an awful lot (340) of processes like: Maybe this change supposed to kill asyncio processes more cleanly is now making the situation worse? Or maybe it's a different regression? I ran the following command in 3 terminals on Linux to stress test the test: While tests are running, I see Python processes running Maybe the issue was fixed in the meanwhile? I don't know. |
|
So, there are already a new set of hung processes on my buildbot since I killed everything a short while ago. There's an active 3.11 build going on, but the asyncio test already completed, so it sounds like those processes should be gone. There are two clusters (3 processes) about 16 minutes apart, so I'm guessing one from the last build and one from this one. test_asyncio did run near the end of the last build and near the beginning of the current one. As soon as I have some free cycles, I'll try some manual tests on the buildbot to see if I can reproduce. But in the meantime I'll monitor to ensure they don't accumulate enough to interfere again with subsequent builds. |
|
I created issue #98407: "AMD64 Ubuntu Shared 3.x: python processes killed with SIGKILL by Linux Out-of-Memory (OOM), maybe related to test_asyncio". |
This PR fixes the issue by calling the waiters after all the callbacks are executed so that no callbacks are scheduled after the method returns.