8000 gh-84559: Change the multiprocessing start method default to `forkser… · python/cpython@b65f2cd · GitHub
[go: up one dir, main page]
Skip to content

Commit b65f2cd

Browse files
gpsheadblurb-it[bot]hugovk
authored
gh-84559: Change the multiprocessing start method default to forkserver (GH-101556)
Change the default multiprocessing start method away from fork to forkserver or spawn on the remaining platforms where it was fork. See the issue for context. This makes the default far more thread safe (other than for people spawning threads at import time... - don't do that!). Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
1 parent 83e5dc0 commit b65f2cd

File tree

7 files changed

+75
-33
lines changed

7 files changed

+75
-33
lines changed

‎Doc/library/concurrent.futures.rst

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -286,14 +286,6 @@ to a :class:`ProcessPoolExecutor` will result in deadlock.
286286

287287
Added the *initializer* and *initargs* arguments.
288288

289-
.. note::
290-
The default :mod:`multiprocessing` start method
291-
(see :ref:`multiprocessing-start-methods`) will change away from
292-
*fork* in Python 3.14. Code that requires *fork* be used for their
293-
:class:`ProcessPoolExecutor` should explicitly specify that by
294-
passing a ``mp_context=multiprocessing.get_context("fork")``
295-
parameter.
296-
297289
.. versionchanged:: 3.11
298290
The *max_tasks_per_child* argument was added to allow users to
299291
control the lifetime of workers in the pool.
@@ -310,6 +302,12 @@ to a :class:`ProcessPoolExecutor` will result in deadlock.
310302
*max_workers* uses :func:`os.process_cpu_count` by default, instead of
311303
:func:`os.cpu_count`.
312304

305+
.. versionchanged:: 3.14
306+
The default process start method (see
307+
:ref:`multiprocessing-start-methods`) changed away from *fork*. If you
308+
require the *fork* start method for :class:`ProcessPoolExecutor` you must
309+
explicitly pass ``mp_context=multiprocessing.get_context("fork")``.
310+
313311
.. _processpoolexecutor-example:
314312

315313
ProcessPoolExecutor Example

‎Doc/library/multiprocessing.rst

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -124,11 +124,11 @@ to start a process. These *start methods* are
124124
inherited by the child process. Note that safely forking a
125125
multithreaded process is problematic.
126126

127-
Available on POSIX systems. Currently the default on POSIX except macOS.
127+
Available on POSIX systems.
128128

129-
.. note::
130-
The default start method will change away from *fork* in Python 3.14.
131-
Code that requires *fork* should explicitly specify that via
129+
.. versionchanged:: 3.14
130+
This is no longer the default start method on any platform.
131+
Code that requires *fork* must explicitly specify that via
132132
:func:`get_context` or :func:`set_start_method`.
133133

134134
.. versionchanged:: 3.12
@@ -146,9 +146,11 @@ to start a process. These *start methods* are
146146
side-effect so it is generally safe for it to use :func:`os.fork`.
147147
No unnecessary resources are inherited.
148148

149-
Available on POSIX platforms which support passing file descriptors
150-
over Unix pipes such as Linux.
149+
Available on POSIX platforms which support passing file descriptors over
150+
Unix pipes such as Linux. The default on those.
151151

152+
.. versionchanged:: 3.14
153+
This became the default start method on POSIX platforms.
152154

153155
.. versionchanged:: 3.4
154156
*spawn* added on all POSIX platforms, and *forkserver* added for
@@ -162,6 +164,13 @@ to start a process. These *start methods* are
162164
method should be considered unsafe as it can lead to crashes of the
163165
subprocess as macOS system libraries may start threads. See :issue:`33725`.
164166

167+
.. versionchanged:: 3.14
168+
169+
On POSIX platforms the default start method was changed from *fork* to
170+
*forkserver* to retain the performance but avoid common multithreaded
171+
process incompatibilities. See :gh:`84559`.
172+
173+
165174
On POSIX using the *spawn* or *forkserver* start methods will also
166175
start a *resource tracker* process which tracks the unlinked named
167176
system resources (such as named semaphores or

‎Doc/whatsnew/3.14.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -385,6 +385,14 @@ Deprecated
385385
as a single positional argument.
386386
(Contributed by Serhiy Storchaka in :gh:`109218`.)
387387

388+
* :mod:`multiprocessing` and :mod:`concurrent.futures`:
389+
The default start method (see :ref:`multiprocessing-start-methods`) changed
390+
away from *fork* to *forkserver* on platforms where it was not already
391+
*spawn* (Windows & macOS). If you require the threading incompatible *fork*
392+
start method you must explicitly specify it when using :mod:`multiprocessing`
393+
or :mod:`concurrent.futures` APIs.
394+
(Contributed by Gregory P. Smith in :gh:`84559`.)
395+
388396
* :mod:`os`:
389397
:term:`Soft deprecate <soft deprecated>` :func:`os.popen` and
390398
:func:`os.spawn* <os.spawnl>` functions. They should no longer be used to

‎Lib/multiprocessing/context.py

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -259,13 +259,12 @@ def get_start_method(self, allow_none=False):
259259

260260
def get_all_start_methods(self):
261261
"""Returns a list of the supported start methods, default first."""
262-
if sys.platform == 'win32':
263-
return ['spawn']
264-
else:
265-
methods = ['spawn', 'fork'] if sys.platform == 'darwin' else ['fork', 'spawn']
266-
if reduction.HAVE_SEND_HANDLE:
267-
methods.append('forkserver')
268-
return methods
262+
default = self._default_context.get_start_method()
263+
start_method_names = [default]
264+
start_method_names.extend(
265+
name for name in _concrete_contexts if name != default
266+
)
267+
return start_method_names
269268

270269

271270
#
@@ -320,14 +319,15 @@ def _check_available(self):
320319
'spawn': SpawnContext(),
321320
'forkserver': ForkServerContext(),
322321
}
323-
if sys.platform == 'darwin':
324-
# bpo-33725: running arbitrary code after fork() is no longer reliable
325-
# on macOS since macOS 10.14 (Mojave). Use spawn by default instead.
326-
_default_context = DefaultContext(_concrete_contexts['spawn'])
322+
# bpo-33725: running arbitrary code after fork() is no longer reliable
323+
# on macOS since macOS 10.14 (Mojave). Use spawn by default instead.
324+
# gh-84559: We changed everyones default to a thread safeish one in 3.14.
325+
if reduction.HAVE_SEND_HANDLE and sys.platform != 'darwin':
326+
_default_context = DefaultContext(_concrete_contexts['forkserver'])
327327
else:
328-
_default_context = DefaultContext(_concrete_contexts['fork'])
328+
_default_context = DefaultContext(_concrete_contexts['spawn'])
329329

330-
else:
330+
else: # Windows
331331

332332
class SpawnProcess(process.BaseProcess):
333333
_start_method = 'spawn'

‎Lib/test/_test_multiprocessing.py

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5553,15 +5553,29 @@ def test_set_get(self):
55535553
multiprocessing.set_start_method(old_method, force=True)
55545554
self.assertGreaterEqual(count, 1)
55555555

5556-
def test_get_all(self):
5556+
def test_get_all_start_methods(self):
55575557
methods = multiprocessing.get_all_start_methods()
5558+
self.assertIn('spawn', methods)
55585559
if sys.platform == 'win32':
55595560
self.assertEqual(methods, ['spawn'])
5561+
elif sys.platform == 'darwin':
5562+
self.assertEqual(methods[0], 'spawn') # The default is first.
5563+
# Whether these work or not, they remain available on macOS.
5564+
self.assertIn('fork', methods)
5565+
self.assertIn('forkserver', methods)
55605566
else:
5561-
self.assertTrue(methods == ['fork', 'spawn'] or
5562-
methods == ['spawn', 'fork'] or
5563-
methods == ['fork', 'spawn', 'forkserver'] or
5564-
methods == ['spawn', 'fork', 'forkserver'])
5567+
# POSIX
5568+
self.assertIn('fork', methods)
5569+
if other_methods := set(methods) - {'fork', 'spawn'}:
5570+
# If there are more than those two, forkserver must be one.
5571+
self.assertEqual({'forkserver'}, other_methods)
5572+
# The default is the first method in the list.
5573+
self.assertIn(methods[0], {'forkserver', 'spawn'},
5574+
msg='3.14+ default must not be fork')
5575+
if methods[0] == 'spawn':
5576+
# Confirm that the current default selection logic prefers
5577+
# forkserver vs spawn when available.
5578+
self.assertNotIn('forkserver', methods)
55655579

55665580
def test_preload_resources(self):
55675581
if multiprocessing.get_start_method() != 'forkserver':

‎Lib/test/support/__init__.py

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2209,7 +2209,15 @@ def skip_if_broken_multiprocessing_synchronize():
22092209
# bpo-38377: On Linux, creating a semaphore fails with OSError
22102210
# if the current user does not have the permission to create
22112211
# a file in /dev/shm/ directory.
2212-
synchronize.Lock(ctx=None)
2212+
import multiprocessing
2213+
synchronize.Lock(ctx=multiprocessing.get_context('fork'))
2214+
# The explicit fork mp context is required in order for
2215+
# TestResourceTracker.test_resource_tracker_reused to work.
2216+
# synchronize creates a new multiprocessing.resource_tracker
2217+
# process at module import time via the above call in that
2218+
# scenario. Awkward. This enables gh-84559. No code involved
2219+
# should have threads at that point so fork() should be safe.
2220+
22132221
except OSError as exc:
22142222
raise unittest.SkipTest(f"broken multiprocessing SemLock: {exc!r}")
22152223

‎Misc/NEWS.d/next/Library/2024-09-19-00-09-48.gh-issue-84559.IrxvQe.rst

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
The default :mod:`multiprocessing` start method on Linux and other POSIX
2+
systems has been changed away from often unsafe ``"fork"`` to ``"forkserver"``
3+
(when the platform supports sending file handles over pipes as most do) or
4+
``"spawn"``. Mac and Windows are unchanged as they already default to
5+
``"spawn"``.

0 commit comments

Comments
 (0)
0