8000 gh-108634: Py_TRACE_REFS uses a hash table · python/cpython@a44bea2 · GitHub
[go: up one dir, main page]
Skip to content

Commit a44bea2

Browse files
vstinnervstinner
committed
gh-108634: Py_TRACE_REFS uses a hash table
Python built with "configure --with-trace-refs" (tracing references) is now ABI compatible with Python release build and debug build. Moreover, it now also supports the Limited API. Change Py_TRACE_REFS build: * Remove _PyObject_EXTRA_INIT macro. * The PyObject structure no longer has two extra members (_ob_prev and _ob_next). * Use a hash table (_Py_hashtable_t) to trace references (all objects): PyInterpreterState.object_state.refchain. * Py_TRACE_REFS build is now ABI compatible with release build and debug build. * Limited C API extensions can now be built with Py_TRACE_REFS: xxlimited, xxlimited_35, _testclinic_limited. * No longer rename PyModule_Create2() and PyModule_FromDefAndSpec2() functions to PyModule_Create2TraceRefs() and PyModule_FromDefAndSpec2TraceRefs(). * _Py_PrintReferenceAddresses() is now called before finalize_interp_delete() which deletes the refchain hash table. Test changes for Py_TRACE_REFS: * Add test.support.Py_TRACE_REFS constant. * Add test_sys.test_getobjects() to test sys.getobjects() function. * test_exceptions skips test_recursion_normalizing_with_no_memory() and test_memory_error_in_PyErr_PrintEx() if Python is built with Py_TRACE_REFS. * test_repl skips test_no_memory(). * test_capi skisp test_set_nomemory().
1 parent 14ec0bb commit a44bea2

31 files changed

+269
-245
lines changed

‎Doc/c-api/typeobj.rst

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -528,28 +528,6 @@ type objects) *must* have the :c:member:`~PyVarObject.ob_size` field.
528528
This field is inherited by subtypes.
529529

530530

531-
.. c:member:: PyObject* PyObject._ob_next
532-
PyObject* PyObject._ob_prev
533-
534-
These fields are only present when the macro ``Py_TRACE_REFS`` is defined
535-
(see the :option:`configure --with-trace-refs option <--with-trace-refs>`).
536-
537-
Their initialization to ``NULL`` is taken care of by the
538-
``PyObject_HEAD_INIT`` macro. For :ref:`statically allocated objects
539-
<static-types>`, these fields always remain ``NULL``. For :ref:`dynamically
540-
allocated objects <heap-types>`, these two fields are used to link the
541-
object into a doubly linked list of *all* live objects on the heap.
542-
543-
This could be used for various debugging purposes; currently the only uses
544-
are the :func:`sys.getobjects` function and to print the objects that are
545-
still alive at the end of a run when the environment variable
546-
:envvar:`PYTHONDUMPREFS` is set.
547-
548-
**Inheritance:**
549-
550-
These fields are not inherited by subtypes.
551-
552-
553531
PyVarObject Slots
554532
-----------------
555533

‎Doc/using/configure.rst

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -408,8 +408,7 @@ See also the :ref:`Python Development Mode <devmode>` and the
408408
.. versionchanged:: 3.8
409409
Release builds and debug builds are now ABI compatible: defining the
410410
``Py_DEBUG`` macro no longer implies the ``Py_TRACE_REFS`` macro (see the
411-
:option:`--with-trace-refs` option), which introduces the only ABI
412-
incompatibility.
411+
:option:`--with-trace-refs` option).
413412

414413

415414
Debug options
@@ -430,8 +429,14 @@ Debug options
430429
* Add :func:`sys.getobjects` function.
431430
* Add :envvar:`PYTHONDUMPREFS` environment variable.
432431

433-
This build is not ABI compatible with release build (default build) or debug
434-
build (``Py_DEBUG`` and ``Py_REF_DEBUG`` macros).
432+
The :envvar:`PYTHONDUMPREFS` environment variable can be used to dump
433+
objects and reference counts still alive at Python exit.
434+
435+
:ref:`Statically allocated objects <static-types>` are not traced.
436+
437+
.. versionchanged:: 3.13
438+
This build is now ABI compatible with release build and :ref:`debug build
439+
<debug-build>`.
435440

436441
.. versionadded:: 3.8
437442

‎Doc/whatsnew/3.13.rst

Lines changed: 9 additions & 0 deletions
< 10000 td data-grid-cell-id="diff-0eef04c1aae0cf8389945464f271ff6789259ef8985f5fdbb21c9a99c4b5021f-905-914-2" data-line-anchor="diff-0eef04c1aae0cf8389945464f271ff6789259ef8985f5fdbb21c9a99c4b5021fR914" data-selected="false" role="gridcell" style="background-color:var(--bgColor-default);padding-right:24px" tabindex="-1" valign="top" class="focusable-grid-cell diff-text-cell right-side-diff-cell left-side">

Original file line numberDiff line numberDiff line change
@@ -828,6 +828,11 @@ Build Changes
828828
* SQLite 3.15.2 or newer is required to build the :mod:`sqlite3` extension module.
829829
(Contributed by Erlend Aasland in :gh:`105875`.)
830830

831+
* Python built with :file:`configure` :option:`--with-trace-refs` (tracing
832+
references) is now ABI compatible with Python release build and
833+
:ref:`debug build <debug-build>`.
834+
(Contributed by Victor Stinner in :gh:`108634`.)
835+
831836

832837
C API Changes
833838
=============
@@ -900,6 +905,10 @@ New Features
900905
(with an underscore prefix).
901906
(Contributed by Victor Stinner in :gh:`108014`.)
902907

908+
* Python built with :file:`configure` :option:`--with-trace-refs` (tracing
909+
references) now supports the :ref:`Limited API <limited-c-api>`.
910+
(Contributed by Victor Stinner in :gh:`108634`.)
911+
903912
Porting to Python 3.13
904913
----------------------
905914

‎Include/internal/pycore_global_objects_fini_generated.h

Lines changed: 0 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Include/internal/pycore_global_strings.h

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -276,10 +276,8 @@ struct _Py_global_strings {
276276
STRUCT_FOR_ID(after_in_child)
277277
STRUCT_FOR_ID(after_in_parent)
278278
STRUCT_FOR_ID(aggregate_class)
279-
STRUCT_FOR_ID(alias)
280279
STRUCT_FOR_ID(append)
281280
STRUCT_FOR_ID(argdefs)
282-
STRUCT_FOR_ID(args)
283281
STRUCT_FOR_ID(arguments)
284282
STRUCT_FOR_ID(argv)
285283
STRUCT_FOR_ID(as_integer_ratio)
@@ -403,8 +401,6 @@ struct _Py_global_strings {
403401
STRUCT_FOR_ID(errors)
404402
STRUCT_FOR_ID(event)
405403
STRUCT_FOR_ID(eventmask)
406-
STRUCT_FOR_ID(exc_type)
407-
STRUCT_FOR_ID(exc_value)
408404
STRUCT_FOR_ID(excepthook)
409405
STRUCT_FOR_ID(exception)
410406
STRUCT_FOR_ID(existing_file_name)
@@ -720,7 +716,6 @@ struct _Py_global_strings {
720716
STRUCT_FOR_ID(timetuple)
721717
STRUCT_FOR_ID(top)
722718
STRUCT_FOR_ID(trace_callback)
723-
STRUCT_FOR_ID(traceback)
724719
STRUCT_FOR_ID(trailers)
725720
STRUCT_FOR_ID(translate)
726721
STRUCT_FOR_ID(true)

‎Include/internal/pycore_object.h

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,6 @@ PyAPI_FUNC(int) _PyObject_IsFreed(PyObject *);
4444
backwards compatible solution */
4545
#define _PyObject_HEAD_INIT(type) \
4646
{ \
47-
_PyObject_EXTRA_INIT \
4847
.ob_refcnt = _Py_IMMORTAL_REFCNT, \
4948
.ob_type = (type) \
5049
},
@@ -173,6 +172,8 @@ _PyType_HasFeature(PyTypeObject *type, unsigned long feature) {
173172
extern void _PyType_InitCache(PyInterpreterState *interp);
174173

175174
extern void _PyObject_InitState(PyInterpreterState *interp);
175+
extern void _PyObject_FiniState(PyInterpreterState *interp);
176+
extern bool _PyRefchain_IsTraced(PyInterpreterState *interp, PyObject *obj);
176177

177178
/* Inline functions trading binary compatibility for speed:
178179
_PyObject_Init() is the fast version of PyObject_Init(), and

‎Include/internal/pycore_object_state.h

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ extern "C" {
88
# error "this header requires Py_BUILD_CORE define"
99
#endif
1010

11+
#include "pycore_hashtable.h" // _Py_hashtable_t
12+
1113
struct _py_object_runtime_state {
1214
#ifdef Py_REF_DEBUG
1315
Py_ssize_t interpreter_leaks;
@@ -20,11 +22,10 @@ struct _py_object_state {
2022
Py_ssize_t reftotal;
2123
#endif
2224
#ifdef Py_TRACE_REFS
23-
/* Head of circular doubly-linked list of all objects. These are linked
24-
* together via the _ob_prev and _ob_next members of a PyObject, which
25-
* exist only in a Py_TRACE_REFS build.
26-
*/
27-
PyObject refchain;
25+
// Hash table storing all objects. The key is the object pointer
26+
// (PyObject*) and the value is always the number 1 (as uintptr_t).
27+
// See _PyRefchain_IsTraced() and _PyRefchain_Trace() functions.
28+
_Py_hashtable_t *refchain;
2829
#endif
2930
int _not_used;
3031
};

‎Include/internal/pycore_runtime_init.h

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -192,7 +192,7 @@ extern PyTypeObject _PyExc_MemoryError;
192192
#ifdef Py_TRACE_REFS
193193
# define _py_object_state_INIT(INTERP) \
194194
{ \
195-
.refchain = {&INTERP.object_state.refchain, &INTERP.object_state.refchain}, \
195+
.refchain = NULL, \
196196
}
197197
#else
198198
# define _py_object_state_INIT(INTERP) \

‎Include/internal/pycore_runtime_init_generated.h

Lines changed: 0 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Include/internal/pycore_unicodeobject_generated.h

Lines changed: 0 additions & 15 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Include/modsupport.h

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -111,14 +111,6 @@ PyAPI_FUNC(int) PyModule_ExecDef(PyObject *module, PyModuleDef *def);
111111
#define PYTHON_ABI_VERSION 3
112112
#define PYTHON_ABI_STRING "3"
113113

114-
#ifdef Py_TRACE_REFS
115-
/* When we are tracing reference counts, rename module creation functions so
116-
modules compiled with incompatible settings will generate a
117-
link-time error. */
118-
#define PyModule_Create2 PyModule_Create2TraceRefs
119-
#define PyModule_FromDefAndSpec2 PyModule_FromDefAndSpec2TraceRefs
120-
#endif
121-
122114
PyAPI_FUNC(PyObject *) PyModule_Create2(PyModuleDef*, int apiver);
123115

124116
#ifdef Py_LIMITED_API

‎Include/object.h

Lines changed: 0 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -58,23 +58,6 @@ whose size is determined when the object is allocated.
5858
# define Py_REF_DEBUG
5959
#endif
6060

61-
#if defined(Py_LIMITED_API) && defined(Py_TRACE_REFS)
62-
# error Py_LIMITED_API is incompatible with Py_TRACE_REFS
63-
#endif
64-
65-
#ifdef Py_TRACE_REFS
66-
/* Define pointers to support a doubly-linked list of all live heap objects. */
67-
#define _PyObject_HEAD_EXTRA \
68-
PyObject *_ob_next; \
69-
PyObject *_ob_prev;
70-
71-
#define _PyObject_EXTRA_INIT _Py_NULL, _Py_NULL,
72-
73-
#else
74-
# define _PyObject_HEAD_EXTRA
75-
# define _PyObject_EXTRA_INIT
76-
#endif
77-
7861
/* PyObject_HEAD defines the initial segment of every PyObject. */
7962
#define PyObject_HEAD PyObject ob_base;
8063

@@ -130,14 +113,12 @@ check by comparing the reference count field to the immortality reference count.
130113
#ifdef Py_BUILD_CORE
131114
#define PyObject_HEAD_INIT(type) \
132115
{ \
133-
_PyObject_EXTRA_INIT \
134116
{ _Py_IMMORTAL_REFCNT }, \
135117
(type) \
136118
},
137119
#else
138120
#define PyObject_HEAD_INIT(type) \
139121
{ \
140-
_PyObject_EXTRA_INIT \
141122
{ 1 }, \
142123
(type) \
143124
},
@@ -164,8 +145,6 @@ check by comparing the reference count field to the immortality reference count.
164145
* in addition, be cast to PyVarObject*.
165146
*/
166147
struct _object {
167-
_PyObject_HEAD_EXTRA
168-
169148
#if (defined(__GNUC__) || defined(__clang__)) \
170149
&& !(defined __STDC_VERSION__ && __STDC_VERSION__ >= 201112L)
171150
// On C99 and older, anonymous union is a GCC and clang extension

‎Include/pyport.h

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -684,12 +684,6 @@ extern char * _getpty(int *, int, mode_t, int);
684684
# endif
685685
#endif
686686

687-
/* Check that ALT_SOABI is consistent with Py_TRACE_REFS:
688-
./configure --with-trace-refs should must be used to define Py_TRACE_REFS */
689-
#if defined(ALT_SOABI) && defined(Py_TRACE_REFS)
690-
# error "Py_TRACE_REFS ABI is not compatible with release and debug ABI"
691-
#endif
692-
693687
#if defined(__ANDROID__) || defined(__VXWORKS__)
694688
// Use UTF-8 as the locale encoding, ignore the LC_CTYPE locale.
695689
// See _Py_GetLocaleEncoding(), PyUnicode_DecodeLocale()

‎Lib/test/support/__init__.py

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -779,9 +779,6 @@ def python_is_optimized():
779779

780780
_header = 'nP'
781781
_align = '0n'
782-
if hasattr(sys, "getobjects"):
783-
_header = '2P' + _header
784-
_align = '0P'
785782
_vheader = _header + 'n'
786783

787784
def calcobjsize(fmt):
@@ -2469,3 +2466,5 @@ def adjust_int_max_str_digits(max_digits):
24692466
#Windows doesn't have os.uname() but it doesn't support s390x.
24702467
skip_on_s390x = unittest.skipIf(hasattr(os, 'uname') and os.uname().machine == 's390x',
24712468
'skipped on s390x')
2469+
2470+
Py_TRACE_REFS = hasattr(sys, 'getobjects')

‎Lib/test/test_capi/test_mem.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,9 @@ def test_pyobject_forbidden_bytes_is_freed(self):
112112
def test_pyobject_freed_is_freed(self):
113113
self.check_pyobject_is_freed('check_pyobject_freed_is_freed')
114114

115+
# Python built with Py_TRACE_REFS fail with a fatal error in
116+
# _PyRefchain_Trace() on memory allocation error.
117+
@unittest.skipIf(support.Py_TRACE_REFS, 'cannot test Py_TRACE_REFS build')
115118
def test_set_nomemory(self):
116119
code = """if 1:
117120
import _testcapi

‎Lib/test/test_exceptions.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1484,6 +1484,9 @@ def recurse_in_body_and_except():
14841484

14851485

14861486
@cpython_only
1487+
# Python built with Py_TRACE_REFS fail with a fatal error in
1488+
# _PyRefchain_Trace() on memory allocation error.
1489+
@unittest.skipIf(support.Py_TRACE_REFS, 'cannot test Py_TRACE_REFS build')
14871490
def test_recursion_normalizing_with_no_memory(self):
14881491
# Issue #30697. Test that in the abort that occurs when there is no
14891492
# memory left and the size of the Python frames stack is greater than
@@ -1652,6 +1655,9 @@ def test_unhandled(self):
16521655
self.assertTrue(report.endswith("\n"))
16531656

16541657
@cpython_only
1658+
# Python built with Py_TRACE_REFS fail with a fatal error in
1659+
# _PyRefchain_Trace() on memory allocation error.
1660+
@unittest.skipIf(support.Py_TRACE_REFS, 'cannot test Py_TRACE_REFS build')
16551661
def test_memory_error_in_PyErr_PrintEx(self):
16561662
code = """if 1:
16571663
import _testcapi

‎Lib/test/test_import/__init__.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@
2929
from test.support import os_helper
3030
from test.support import (
3131
STDLIB_DIR, swap_attr, swap_item, cpython_only, is_emscripten,
32-
is_wasi, run_in_subinterp, run_in_subinterp_with_config)
32+
is_wasi, run_in_subinterp, run_in_subinterp_with_config, Py_TRACE_REFS)
3333
from test.support.import_helper import (
3434
forget, make_legacy_pyc, unlink, unload, DirsOnSysPath, CleanImport)
3535
from test.support.os_helper import (
@@ -2556,7 +2556,7 @@ def test_basic_multiple_interpreters_main_no_reset(self):
25562556
def test_basic_multiple_interpreters_deleted_no_reset(self):
25572557
# without resetting; already loaded in a deleted interpreter
25582558

2559-
if hasattr(sys, 'getobjects'):
2559+
if Py_TRACE_REFS:
25602560
# It's a Py_TRACE_REFS build.
25612561
# This test breaks interpreter isolation a little,
25622562
# which causes problems on Py_TRACE_REF builds.

0 commit comments

Comments
 (0)
0