8000 Merge branch 'main' into integer-overflow-assert · rialbat/cpython@e76ba4b · GitHub
[go: up one dir, main page]
Skip to content

Commit e76ba4b

Browse files
rialbatrialbat
authored
Merge branch 'main' into integer-overflow-assert
2 parents d3bc2af + 99712c4 commit e76ba4b

36 files changed

+886
-762
lines changed

‎Doc/c-api/arg.rst

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -686,6 +686,12 @@ Building values
686686
``p`` (:class:`bool`) [int]
687687
Convert a C :c:expr:`int` to a Python :class:`bool` object.
688688
689+
Be aware that this format requires an ``int`` argument.
690+
Unlike most other contexts in C, variadic arguments are not coerced to
691+
a suitable type automatically.
692+
You can convert another type (for example, a pointer or a float) to a
693+
suitable ``int`` value using ``(x) ? 1 : 0`` or ``!!x``.
694+
689695
.. versionadded:: 3.14
690696
691697
``c`` (:class:`bytes` of length 1) [char]

‎Doc/c-api/extension-modules.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,6 @@ in the following ways:
242242
* Single-phase modules support module lookup functions like
243243
:c:func:`PyState_FindModule`.
244244

245-
.. [#testsinglephase] ``_testsinglephase`` is an internal module used \
246-
in CPython's self-test suite; your installation may or may not \
245+
.. [#testsinglephase] ``_testsinglephase`` is an internal module used
246+
in CPython's self-test suite; your installation may or may not
247247
include it.

‎Doc/library/dialog.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -220,7 +220,7 @@ is the base class for dialogs defined in other supporting modules.
220220

221221
.. class:: Dialog(master=None, **options)
222222

223-
.. method:: show(color=None, **options)
223+
.. method:: show(**options)
224224

225225
Render the Dialog window.
226226

‎Doc/library/netrc.rst

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,14 @@ the Unix :program:`ftp` program and other FTP clients.
2424
a :exc:`FileNotFoundError` exception will be raised.
2525
Parse errors will raise :exc:`NetrcParseError` with diagnostic
2626
information including the file name, line number, and terminating token.
27+
2728
If no argument is specified on a POSIX system, the presence of passwords in
2829
the :file:`.netrc` file will raise a :exc:`NetrcParseError` if the file
2930
ownership or permissions are insecure (owned by a user other than the user
3031
running the process, or accessible for read or write by any other user).
3132
This implements security behavior equivalent to that of ftp and other
32-
programs that use :file:`.netrc`.
33+
programs that use :file:`.netrc`. Such security checks are not available
34+
on platforms that do not support :func:`os.getuid`.
3335

3436
.. versionchanged:: 3.4 Added the POSIX permission check.
3537

‎Doc/library/sys.rst

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1185,6 +1185,15 @@ always available. Unless explicitly noted otherwise, all variables are read-only
11851185
``cache_tag`` is set to ``None``, it indicates that module caching should
11861186
be disabled.
11871187

1188+
*supports_isolated_interpreters* is a boolean value, whether
1189+
this implementation supports multiple isolated interpreters.
1190+
It is ``True`` for CPython on most platforms. Platforms with
1191+
this support implement the low-level :mod:`!_interpreters` module.
1192+
1193+
.. seealso::
1194+
1195+
:pep:`684`, :pep:`734`, and :mod:`concurrent.interpreters`.
1196+
11881197
:data:`sys.implementation` may contain additional attributes specific to
11891198
the Python implementation. These non-standard attributes must start with
11901199
an underscore, and are not described here. Regardless of its contents,
@@ -1194,6 +1203,9 @@ always available. Unless explicitly noted otherwise, all variables are read-only
11941203

11951204
.. versionadded:: 3.3
11961205

1206+
.. versionchanged:: 3.14
1207+
Added ``supports_isolated_interpreters`` field.
1208+
11971209
.. note::
11981210

11991211
The addition of new required attributes must go through the normal PEP

‎InternalDocs/asyncio.md

Lines changed: 122 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,12 @@ asyncio
22
=======
33

44

5-
This document describes the working and implementation details of C
6-
implementation of the
5+
This document describes the working and implementation details of the
76
[`asyncio`](https://docs.python.org/3/library/asyncio.html) module.
87

8+
**The following section describes the implementation details of the C implementation**.
9+
10+
# Task management
911

1012
## Pre-Python 3.14 implementation
1113

@@ -158,7 +160,8 @@ flowchart TD
158160
subgraph two["Thread deallocating"]
159161
A1{"thread's task list empty? <br> llist_empty(tstate->asyncio_tasks_head)"}
160162
A1 --> |true| B1["deallocate thread<br>free_threadstate(tstate)"]
161-
A1 --> |false| C1["add tasks to interpreter's task list<br> llist_concat(&tstate->interp->asyncio_tasks_head,tstate->asyncio_tasks_head)"]
163+
A1 --> |false| C1["add tasks to interpreter's task list<br> llist_concat(&tstate->interp->asyncio_tasks_head,
164+
&tstate->asyncio_tasks_head)"]
162165
C1 --> B1
163166
end
164167
@@ -205,6 +208,121 @@ In free-threading, it avoids contention on a global dictionary as
205208
threads can access the current task of thier running loop without any
206209
locking.
207210

211+
---
212+
213+
**The following section describes the implementation details of the Python implementation**.
214+
215+
# async generators
216+
217+
This section describes the implementation details of async generators in `asyncio`.
218+
219+
Since async generators are meant to be used from coroutines,
220+
their finalization (execution of finally blocks) needs
221+
to be done while the loop is running.
222+
Most async generators are closed automatically
223+
when they are fully iterated over and exhausted; however,
224+
if the async generator is not fully iterated over,
225+
it may not be closed properly, leading to the `finally` blocks not being executed.
226+
227+
Consider the following code:
228+
```py
229+
import asyncio
230+
231+
async def agen():
232+
try:
233+
yield 1
234+
finally:
235+
await asyncio.sleep(1)
236+
print("finally executed")
237+
238+
239+
async def main():
240+
async for i in agen():
241+
break
242+
243+
loop = asyncio.EventLoop()
244+
loop.run_until_complete(main())
245+
```
246+
247+
The above code will not print "finally executed", because the
248+
async generator `agen` is not fully iterated over
249+
and it is not closed manually by awaiting `agen.aclose()`.
250+
251+
To solve this, `asyncio` uses the `sys.set_asyncgen_hooks` function to
252+
set hooks for finalizing async generators as described in
253+
[PEP 525](https://peps.python.org/pep-0525/).
254+
255+
- **firstiter hook**: When the async generator is iterated over for the first time,
256+
the *firstiter hook* is called. The async generator is added to `loop._asyncgens` WeakSet
257+
and the event loop tracks all active async generators.
258+
259+
- **finalizer hook**: When the async generator is about to be finalized,
260+
the *finalizer hook* is called. The event loop removes the async generator
261+
from `loop._asyncgens` WeakSet, and schedules the finalization of the async
262+
generator by creating a task calling `agen.aclose()`. This ensures that the
263+
finally block is executed while the event loop is running. When the loop is
264+
shutting down, the loop checks if there are active async generators and if so,
265+
it similarly schedules the finalization of all active async generators by calling
266+
`agen.aclose()` on each of them and waits for them to complete before shutting
267+
down the loop.
268+
269+
This ensures that the async generator's `finally` blocks are executed even
270+
if the generator is not explicitly closed.
271+
272+
Consider the following example:
273+
274+
```python
275+
import asyncio
276+
277+
async def agen():
278+
try:
279+
yield 1
280+
yield 2
281+
finally:
282+
print("executing finally block")
283+
284+
async def main():
285+
async for item in agen():
286+
print(item)
287+
break # not fully iterated
288+
289+
asyncio.run(main())
290+
```
291+
292+
```mermaid
293+
flowchart TD
294+
subgraph one["Loop running"]
295+
A["asyncio.run(main())"] --> B
296+
B["set async generator hooks <br> sys.set_asyncgen_hooks()"] --> C
297+
C["async for item in agen"] --> F
298+
F{"first iteration?"} --> |true|D
299+
F{"first iteration?"} --> |false|H
300+
D["calls firstiter hook<br>loop._asyncgen_firstiter_hook(agen)"] --> E
301+
E["add agen to WeakSet<br> loop._asyncgens.add(agen)"] --> H
302+
H["item = await agen.\_\_anext\_\_()"] --> J
303+
J{"StopAsyncIteration?"} --> |true|M
304+
J{"StopAsyncIteration?"} --> |false|I
305+
I["print(item)"] --> S
306+
S{"continue iterating?"} --> |true|C
307+
S{"continue iterating?"} --> |false|M
308+
M{"agen is no longer referenced?"} --> |true|N
309+
M{"agen is no longer referenced?"} --> |false|two
310+
N["finalize agen<br>_PyGen_Finalize(agen)"] --> O
311+
O["calls finalizer hook<br>loop._asyncgen_finalizer_hook(agen)"] --> P
312+
P["remove agen from WeakSet<br>loop._asyncgens.discard(agen)"] --> Q
313+
Q["schedule task to close it<br>self.create_task(agen.aclose())"] --> R
314+
R["print('executing finally block')"] --> E1
315+
316+
end
317+
318+
subgraph two["Loop shutting down"]
319+
A1{"check for alive async generators?"} --> |true|B1
320+
B1["close all async generators <br> await asyncio.gather\(*\[ag.aclose\(\) for ag in loop._asyncgens\]"] --> R
321+
A1{"check for alive async generators?"} --> |false|E1
322+
E1["loop.close()"]
323+
end
324+
325+
```
208326

209327
[^1]: https://github.com/python/cpython/issues/123089
210-
[^2]: https://github.com/python/cpython/issues/80788
328+
[^2]: https://github.com/python/cpython/issues/80788

‎Lib/_pyrepl/base_eventqueue.py

Lines changed: 3 additions & 3 deletions

‎Lib/netrc.py

Lines changed: 20 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ def push(self, char: int | bytes) -> None:
8787
if isinstance(k, dict):
8888
self.keymap = k
8989
else:
90-
self.insert(Event('key', k, self.flush_buf()))
90+
self.insert(Event('key', k, bytes(self.flush_buf())))
9191
self.keymap = self.compiled_keymap
9292

9393
elif self.buf and self.buf[0] == 27: # escape
@@ -96,7 +96,7 @@ def push(self, char: int | bytes) -> None:
9696
# the docstring in keymap.py
9797
trace('unrecognized escape sequence, propagating...')
9898
self.keymap = self.compiled_keymap
99-
self.insert(Event('key', '\033', bytearray(b'\033')))
99+
self.insert(Event('key', '\033', b'\033'))
100100
for _c in self.flush_buf()[1:]:
101101
self.push(_c)
102102

@@ -106,5 +106,5 @@ def push(self, char: int | bytes) -> None:
106106
except UnicodeError:
107107
return
108108
else:
109-
self.insert(Event('key', decoded, self.flush_buf()))
109+
self.insert(Event('key', decoded, bytes(self.flush_buf())))
110110
self.keymap = self.compiled_keymap
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,19 @@
77
__all__ = ["netrc", "NetrcParseError"]
88

99

10+
def _can_security_check():
11+
# On WASI, getuid() is indicated as a stub but it may also be missing.
12+
return os.name == 'posix' and hasattr(os, 'getuid')
13+
14+
15+
def _getpwuid(uid):
16+
try:
17+
import pwd
18+
return pwd.getpwuid(uid)[0]
19+
except (ImportError, LookupError):
20+
return f'uid {uid}'
21+
22+
1023
class NetrcParseError(Exception):
1124
"""Exception raised on syntax errors in the .netrc file."""
1225
def __init__(self, msg, filename=None, lineno=None):
@@ -142,21 +155,15 @@ def _parse(self, file, fp, default_netrc):
142155
self._security_check(fp, default_netrc, self.hosts[entryname][0])
143156

144157
def _security_check(self, fp, default_netrc, login):
145-
if os.name == 'posix' and default_netrc and login != "anonymous":
158+
if _can_security_check() and default_netrc and login != "anonymous":
146159
prop = os.fstat(fp.fileno())
147-
if prop.st_uid != os.getuid():
148-
import pwd
149-
try:
150-
fowner = pwd.getpwuid(prop.st_uid)[0]
151-
except KeyError:
152-
fowner = 'uid %s' % prop.st_uid
153-
try:
154-
user = pwd.getpwuid(os.getuid())[0]
155-
except KeyError:
156-
user = 'uid %s' % os.getuid()
160+
current_user_id = os.getuid()
161+
if prop.st_uid != current_user_id:
162+
fowner = _getpwuid(prop.st_uid)
163+
user = _getpwuid(current_user_id)
157164
raise NetrcParseError(
158-
(f"~/.netrc file owner ({fowner}, {user}) does not match"
159-
" current user"))
165+
f"~/.netrc file owner ({fowner}) does not match"
166+
f" current user ({user})")
160167
if (prop.st_mode & (stat.S_IRWXG | stat.S_IRWXO)):
161168
raise NetrcParseError(
162169
"~/.netrc access too permissive: access"

‎Lib/test/libregrtest/tsan.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
'test_capi.test_pyatomic',
99
'test_code',
1010
'test_ctypes',
11-
# 'test_concurrent_futures', # gh-130605: too many data races
11+
'test_concurrent_futures',
1212
'test_enum',
1313
'test_functools',
1414
'test_httpservers',

‎Lib/test/test_free_threading/test_heapq.py

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
import heapq
44

55
from enum import Enum
6-
from threading import Thread, Barrier
6+
from threading import Thread, Barrier, Lock
77
from random import shuffle, randint
88

99
from test.support import threading_helper
@@ -178,6 +178,33 @@ def heapreplace_max_func(max_heap, replace_items):
178178
self.assertEqual(len(max_heap), OBJECT_COUNT)
179179
self.test_heapq.check_max_invariant(max_heap)
180180

181+
def test_lock_free_list_read(self):
182+
n, n_threads = 1_000, 10
183+
l = []
184+
barrier = Barrier(n_threads * 2)
185+
186+
count = 0
187+
lock = Lock()
188+
189+
def worker():
190+
with lock:
191+
nonlocal count
192+
x = count
193+
count += 1
194+
195+
barrier.wait()
196+
for i in range(n):
197+
if x % 2:
198+
heapq.heappush(l, 1)
199+
heapq.heappop(l)
200+
else:
201+
try:
202+
l[0]
203+
except IndexError:
204+
pass
205+
206+
self.run_concurrently(worker, (), n_threads * 2)
207+
181208
@staticmethod
182209
def is_sorted_ascending(lst):
183210
"""

0 commit comments

Comments
 (0)
0