8000 Merge branch 'main' into gh-121735/zip-module-resources · python/cpython@b9ce84f · GitHub
[go: up one dir, main page]
Skip to content

Commit b9ce84f

Browse files
jaracojaraco
authored
Merge branch 'main' into gh-121735/zip-module-resources
2 parents 0bd9acc + e913d2c commit b9ce84f

30 files changed

+645
-333
lines changed

‎Doc/c-api/init.rst

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -394,8 +394,7 @@ Initializing and finalizing the interpreter
394394
Undo all initializations made by :c:func:`Py_Initialize` and subsequent use of
395395
Python/C API functions, and destroy all sub-interpreters (see
396396
:c:func:`Py_NewInterpreter` below) that were created and not yet destroyed since
397-
the last call to :c:func:`Py_Initialize`. Ideally, this frees all memory
398-
allocated by the Python interpreter. This is a no-op when called for a second
397+
the last call to :c:func:`Py_Initialize`. This is a no-op when called for a second
399398
time (without calling :c:func:`Py_Initialize` again first).
400399
401400
Since this is the reverse of :c:func:`Py_Initialize`, it should be called
@@ -407,6 +406,12 @@ Initializing and finalizing the interpreter
407406
If there were errors during finalization (flushing buffered data),
408407
``-1`` is returned.
409408
409+
Note that Python will do a best effort at freeing all memory allocated by the Python
410+
interpreter. Therefore, any C-Extension should make sure to correctly clean up all
411+
of the preveiously allocated PyObjects before using them in subsequent calls to
412+
:c:func:`Py_Initialize`. Otherwise it could introduce vulnerabilities and incorrect
413+
behavior.
414+
410415
This function is provided for a number of reasons. An embedding application
411416
might want to restart Python without having to restart the application itself.
412417
An application that has loaded the Python interpreter from a dynamically
@@ -421,10 +426,11 @@ Initializing and finalizing the interpreter
421426
loaded extension modules loaded by Python are not unloaded. Small amounts of
422427
memory allocated by the Python interpreter may not be freed (if you find a leak,
423428
please report it). Memory tied up in circular references between objects is not
424-
freed. Some memory allocated by extension modules may not be freed. Some
425-
extensions may not work properly if their initialization routine is called more
426-
than once; this can happen if an application calls :c:func:`Py_Initialize` and
427-
:c:func:`Py_FinalizeEx` more than once.
429+
freed. Interned strings will all be deallocated regarldess of their reference count.
430+
Some memory allocated by extension modules may not be freed. Some extensions may not
431+
work properly if their initialization routine is called more than once; this can
432+
happen if an application calls :c:func:`Py_Initialize` and :c:func:`Py_FinalizeEx`
433+
more than once.
428434

429435
.. audit-event:: cpython._PySys_ClearAuditHooks "" c.Py_FinalizeEx
430436

‎Doc/conf.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -546,6 +546,8 @@
546546
r'https://msdn.microsoft.com/.*': 'https://learn.microsoft.com/.*',
547547
r'https://docs.microsoft.com/.*': 'https://learn.microsoft.com/.*',
548548
r'https://go.microsoft.com/fwlink/\?LinkID=\d+': 'https://learn.microsoft.com/.*',
549+
# Debian's man page redirects to its current stable version
550+
r'https://manpages.debian.org/\w+\(\d(\w+)?\)': r'https://manpages.debian.org/\w+/[\w/\-\.]*\.\d(\w+)?\.en\.html',
549551
# Language redirects
550552
r'https://toml.io': 'https://toml.io/en/',
551553
r'https://www.redhat.com': 'https://www.redhat.com/en',

‎Doc/library/enum.rst

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -570,6 +570,8 @@ Data Types
570570
>>> len(white)
571571
3
572572

573+
.. versionadded:: 3.11
574+
573575
.. method:: __bool__(self):
574576

575577
Returns *True* if any members in flag, *False* otherwise::

‎Doc/tutorial/appendix.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,8 @@ There are two variants of the interactive :term:`REPL`. The classic
1414
basic interpreter is supported on all platforms with minimal line
1515
control capabilities.
1616

17-
On Unix-like systems (e.g. Linux or macOS) with :mod:`curses` and
18-
:mod:`readline` support, a new interactive shell is used by default.
17+
On Windows, or Unix-like systems with :mod:`curses` support,
18+
a new interactive shell is used by default.
1919
This one supports color, multiline editing, history browsing, and
2020
paste mode. To disable color, see :ref:`using-on-controlling-color` for
2121
details. Function keys provide some additional functionality.

‎Doc/whatsnew/3.13.rst

Lines changed: 197 additions & 179 deletions
Large diffs are not rendered by default.

‎Doc/whatsnew/3.14.rst

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -419,6 +419,16 @@ New Features
419419
which has an ambiguous return value.
420420
(Contributed by Irit Katriel and Erlend Aasland in :gh:`105201`.)
421421

422+
* :c:func:`Py_Finalize` now deletes all interned strings. This
423+
is backwards incompatible to any C-Extension that holds onto an interned
424+
string after a call to :c:func:`Py_Finalize` and is then reused after a
425+
call to :c:func:`Py_Initialize`. Any issues arising from this behavior will
426+
normally result in crashes during the exectuion of the subsequent call to
427+
:c:func:`Py_Initialize` from accessing uninitialized memory. To fix, use
428+
an address sanitizer to identify any use-after-free coming from
429+
an interned string and deallocate it during module shutdown.
430+
(Contribued by Eddie Elizondo in :gh:`113601`.)
431+
422432
Porting to Python 3.14
423433
----------------------
424434

‎Include/internal/pycore_frame.h

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,13 @@ static inline void _PyFrame_Copy(_PyInterpreterFrame *src, _PyInterpreterFrame *
128128
// Don't leave a dangling pointer to the old frame when creating generators
129129
// and coroutines:
130130
dest->previous = NULL;
131+
132+
#ifdef Py_GIL_DISABLED
133+
PyCodeObject *co = (PyCodeObject *)dest->f_executable;
134+
for (int i = stacktop; i < co->co_nlocalsplus + co->co_stacksize; i++) {
135+
dest->localsplus[i] = PyStackRef_NULL;
136+
}
137+
#endif
131138
}
132139

133140
/* Consumes reference to func and locals.
@@ -153,6 +160,16 @@ _PyFrame_Initialize(
153160
for (int i = null_locals_from; i < code->co_nlocalsplus; i++) {
154161
frame->localsplus[i] = PyStackRef_NULL;
155162
}
163+
164+
#ifdef Py_GIL_DISABLED
165+
// On GIL disabled, we walk the entire stack in GC. Since stacktop
166+
// is not always in sync with the real stack pointer, we have
167+
// no choice but to traverse the entire stack.
168+
// This just makes sure we don't pass the GC invalid stack values.
169+
for (int i = code->co_nlocalsplus; i < code->co_nlocalsplus + code->co_stacksize; i++) {
170+
frame->localsplus[i] = PyStackRef_NULL;
171+
}
172+
#endif
156173
}
157174

158175
/* Gets the pointer to the locals array
@@ -314,6 +331,13 @@ _PyFrame_PushTrampolineUnchecked(PyThreadState *tstate, PyCodeObject *code, int
314331
frame->instr_ptr = _PyCode_CODE(code);
315332
frame->owner = FRAME_OWNED_BY_THREAD;
316333
frame->return_offset = 0;
334+
335+
#ifdef Py_GIL_DISABLED
336+
assert(code->co_nlocalsplus == 0);
337+
for (int i = 0; i < code->co_stacksize; i++) {
338+
frame->localsplus[i] = PyStackRef_NULL;
339+
}
340+
#endif
317341
return frame;
318342
}
319343

‎Include/internal/pycore_gc.h

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -381,6 +381,8 @@ extern void _PyGC_ClearAllFreeLists(PyInterpreterState *interp);
381381
extern void _Py_ScheduleGC(PyThreadState *tstate);
382382
extern void _Py_RunGC(PyThreadState *tstate);
383383

384+
// GC visit callback for tracked interpreter frames
385+
extern int _PyGC_VisitFrameStack(struct _PyInterpreterFrame *frame, visitproc visit, void *arg);
384386

385387
#ifdef __cplusplus
386388
}

‎Include/internal/pycore_mimalloc.h

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,9 +36,18 @@ typedef enum {
3636
# define MI_TSAN 1
3737
#endif
3838

39+
#ifdef __cplusplus
40+
extern "C++" {
41+
#endif
42+
3943
#include "mimalloc/mimalloc.h"
4044
#include "mimalloc/mimalloc/types.h"
4145
#include "mimalloc/mimalloc/internal.h"
46+
47+
#ifdef __cplusplus
48+
}
49+
#endif
50+
4251
#endif
4352

4453
#ifdef Py_GIL_DISABLED

‎Include/internal/pycore_runtime.h

Lines changed: 50 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,15 @@ struct _gilstate_runtime_state {
4444

4545
/* Runtime audit hook state */
4646

47+
#define _Py_Debug_Cookie "xdebugpy"
48+
49+
#ifdef Py_GIL_DISABLED
50+
# define _Py_Debug_gilruntimestate_enabled offsetof(struct _gil_runtime_state, enabled)
51+
# define _Py_Debug_Free_Threaded 1
52+
#else
53+
# define _Py_Debug_gilruntimestate_enabled 0
54+
# define _Py_Debug_Free_Threaded 0
55+
#endif
4756
typedef struct _Py_AuditHookEntry {
4857
struct _Py_AuditHookEntry *next;
4958
Py_AuditHookFunction hookCFunction;
@@ -53,6 +62,7 @@ typedef struct _Py_AuditHookEntry {
5362
typedef struct _Py_DebugOffsets {
5463
char cookie[8];
5564
uint64_t version;
65+
uint64_t free_threaded;
5666
// Runtime state offset;
5767
struct _runtime_state {
5868
uint64_t size;
@@ -71,6 +81,8 @@ typedef struct _Py_DebugOffsets {
7181
uint64_t sysdict;
7282
uint64_t builtins;
7383
uint64_t ceval_gil;
84+
uint64_t gil_runtime_state;
85+
uint64_t gil_runtime_state_enabled;
7486
uint64_t gil_runtime_state_locked;
7587
uint64_t gil_runtime_state_holder;
7688
} interpreter_state;
@@ -122,20 +134,57 @@ typedef struct _Py_DebugOffsets {
122134
struct _type_object {
123135
uint64_t size;
124136
uint64_t tp_name;
137+
uint64_t tp_repr;
138+
uint64_t tp_flags;
125139
} type_object;
126140

127141
// PyTuple object offset;
128142
struct _tuple_object {
129143
uint64_t size;
130144
uint64_t ob_item;
145+
uint64_t ob_size;
131146
} tuple_object;
132147

148+
// PyList object offset;
149+
struct _list_object {
150+
uint64_t size;
151+
uint64_t ob_item;
152+
uint64_t ob_size;
153+
} list_object;
154+
155+
// PyDict object offset;
156+
struct _dict_object {
157+
uint64_t size;
158+
uint64_t ma_keys;
159+
uint64_t ma_values;
160+
} dict_object;
161+
162+
// PyFloat object offset;
163+
struct _float_object {
164+
uint64_t size;
165+
uint64_t ob_fval;
166+
} float_object;
167+
168+
// PyLong object offset;
169+
struct _long_object {
170+
uint64_t size;
171+
uint64_t lv_tag;
172+
uint64_t ob_digit;
173+
} long_object;
174+
175+
// PyBytes object offset;
176+
struct _bytes_object {
177+
uint64_t size;
178+
uint64_t ob_size;
179+
uint64_t ob_sval;
180+
} bytes_object;
181+
133182
// Unicode object offset;
134183
struct _unicode_object {
135184
uint64_t size;
136185
uint64_t state;
137186
uint64_t length;
138-
size_t asciiobject_size;
187+
uint64_t asciiobject_size;
139188
} unicode_object;
140189

141190
// GC runtime state offset;

‎Include/internal/pycore_runtime_init.h

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,11 +29,12 @@ extern PyTypeObject _PyExc_MemoryError;
2929
/* The static initializers defined here should only be used
3030
in the runtime init code (in pystate.c and pylifecycle.c). */
3131

32-
#define _PyRuntimeState_INIT(runtime) \
32+
#define _PyRuntimeState_INIT(runtime, debug_cookie) \
3333
{ \
3434
.debug_offsets = { \
35-
.cookie = "xdebugpy", \
35+
.cookie = debug_cookie, \
3636
.version = PY_VERSION_HEX, \
37+
.free_threaded = _Py_Debug_Free_Threaded, \
3738
.runtime_state = { \
3839
.size = sizeof(_PyRuntimeState), \
3940
.finalizing = offsetof(_PyRuntimeState, _finalizing), \
@@ -49,6 +50,8 @@ extern PyTypeObject _PyExc_MemoryError;
4950
.sysdict = offsetof(PyInterpreterState, sysdict), \
5051
.builtins = offsetof(PyInterpreterState, builtins), \
5152
.ceval_gil = offsetof(PyInterpreterState, ceval.gil), \
53+
.gil_runtime_state = offsetof(PyInterpreterState, _gil), \
54+
.gil_runtime_state_enabled = _Py_Debug_gilruntimestate_enabled, \
5255
.gil_runtime_state_locked = offsetof(PyInterpreterState, _gil.locked), \
5356
.gil_runtime_state_holder = offsetof(PyInterpreterState, _gil.last_holder), \
5457
}, \
@@ -90,10 +93,37 @@ extern PyTypeObject _PyExc_MemoryError;
9093
.type_object = { \
9194
.size = sizeof(PyTypeObject), \
9295
.tp_name = offsetof(PyTypeObject, tp_name), \
96+
.tp_repr = offsetof(PyTypeObject, tp_repr), \
97+
.tp_flags = offsetof(PyTypeObject, tp_flags), \
9398
}, \
9499
.tuple_object = { \
95100
.size = sizeof(PyTupleObject), \
96101
.ob_item = offsetof(PyTupleObject, ob_item), \
102+
.ob_size = offsetof(PyTupleObject, ob_base.ob_size), \
103+
}, \
104+
.list_object = { \
105+
.size = sizeof(PyListObject), \
106+
.ob_item = offsetof(PyListObject, ob_item), \
107+
.ob_size = offsetof(PyListObject, ob_base.ob_size), \
108+
}, \
109+
.dict_object = { \
110+
.size = sizeof(PyDictObject), \
111+
.ma_keys = offsetof(PyDictObject, ma_keys), \
112+
.ma_values = offsetof(PyDictObject, ma_values), \
113+
}, \
114+
.float_object = { \
115+
.size = sizeof(PyFloatObject), \
116+
.ob_fval = offsetof(PyFloatObject, ob_fval), \
117+
}, \
118+
.long_object = { \
119+
.size = sizeof(PyLongObject), \
120+
.lv_tag = offsetof(_PyLongValue, lv_tag), \
121+
.ob_digit = offsetof(_PyLongValue, ob_digit), \
122+
}, \
123+
.bytes_object = { \
124+
.size = sizeof(PyBytesObject), \
125+
.ob_size = offsetof(PyBytesObject, ob_base.ob_size), \
126+
.ob_sval = offsetof(PyBytesObject, ob_sval), \
97127
}, \
98128
.unicode_object = { \
99129
.size = sizeof(PyUnicodeObject), \

‎Include/internal/pycore_stackref.h

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -150,8 +150,7 @@ PyStackRef_FromPyObjectNew(PyObject *obj)
150150
// Make sure we don't take an already tagged value.
151151
assert(((uintptr_t)obj & Py_TAG_BITS) == 0);
152152
assert(obj != NULL);
153-
// TODO (gh-117139): Add deferred objects later.
154-
if (_Py_IsImmortal(obj)) {
153+
if (_Py_IsImmortal(obj) || _PyObject_HasDeferredRefcount(obj)) {
155154
return (_PyStackRef){ .bits = (uintptr_t)obj | Py_TAG_DEFERRED };
156155
}
157156
else {
@@ -220,7 +219,8 @@ PyStackRef_DUP(_PyStackRef stackref)
220219
{
221220
if (PyStackRef_IsDeferred(stackref)) {
222221
assert(PyStackRef_IsNull(stackref) ||
223-
_Py_IsImmortal(PyStackRef_AsPyObjectBorrow(stackref)));
222+
_Py_IsImmortal(PyStackRef_AsPyObjectBorrow(stackref)) ||
223+
_PyObject_HasDeferredRefcount(PyStackRef_AsPyObjectBorrow(stackref)));
224224
return stackref;
225225
}
226226
Py_INCREF(PyStackRef_AsPyObjectBorrow(stackref));

0 commit comments

Comments
 (0)
0