diff --git a/.gitattributes b/.gitattributes
index 2f5a030981fb94..5682b9150a36e2 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -10,6 +10,7 @@
*.ico binary
*.jpg binary
*.pck binary
+*.pdf binary
*.png binary
*.psd binary
*.tar binary
@@ -67,6 +68,7 @@ PCbuild/readme.txt dos
**/clinic/*.cpp.h generated
**/clinic/*.h.h generated
*_db.h generated
+Doc/c-api/lifecycle.dot.svg generated
Doc/data/stable_abi.dat generated
Doc/library/token-list.inc generated
Include/internal/pycore_ast.h generated
diff --git a/Doc/c-api/allocation.rst b/Doc/c-api/allocation.rst
index 7cbc99ad145ad4..f8d01a3f29b30e 100644
--- a/Doc/c-api/allocation.rst
+++ b/Doc/c-api/allocation.rst
@@ -16,7 +16,20 @@ Allocating Objects on the Heap
Initialize a newly allocated object *op* with its type and initial
reference. Returns the initialized object. Other fields of the object are
- not affected.
+ not initialized. Despite its name, this function is unrelated to the
+ object's :meth:`~object.__init__` method (:c:member:`~PyTypeObject.tp_init`
+ slot). Specifically, this function does **not** call the object's
+ :meth:`!__init__` method.
+
+ In general, consider this function to be a low-level routine. Use
+ :c:member:`~PyTypeObject.tp_alloc` where possible.
+ For implementing :c:member:`!tp_alloc` for your type, prefer
+ :c:func:`PyType_GenericAlloc` or :c:func:`PyObject_New`.
+
+ .. note::
+
+ This function only initializes the object's memory corresponding to the
+ initial :c:type:`PyObject` structure. It does not zero the rest.
.. c:function:: PyVarObject* PyObject_InitVar(PyVarObject *op, PyTypeObject *type, Py_ssize_t size)
@@ -24,38 +37,107 @@ Allocating Objects on the Heap
This does everything :c:func:`PyObject_Init` does, and also initializes the
length information for a variable-size object.
+ .. note::
+
+ This function only initializes some of the object's memory. It does not
+ zero the rest.
+
.. c:macro:: PyObject_New(TYPE, typeobj)
- Allocate a new Python object using the C structure type *TYPE*
- and the Python type object *typeobj* (``PyTypeObject*``).
- Fields not defined by the Python object header are not initialized.
- The caller will own the only reference to the object
- (i.e. its reference count will be one).
- The size of the memory allocation is determined from the
- :c:member:`~PyTypeObject.tp_basicsize` field of the type object.
+ Allocates a new Python object using the C structure type *TYPE* and the
+ Python type object *typeobj* (``PyTypeObject*``) by calling
+ :c:func:`PyObject_Malloc` to allocate memory and initializing it like
+ :c:func:`PyObject_Init`. The caller will own the only reference to the
+ object (i.e. its reference count will be one).
+
+ Avoid calling this directly to allocate memory for an object; call the type's
+ :c:member:`~PyTypeObject.tp_alloc` slot instead.
+
+ When populating a type's :c:member:`~PyTypeObject.tp_alloc` slot,
+ :c:func:`PyType_GenericAlloc` is preferred over a custom function that
+ simply calls this macro.
+
+ This macro does not call :c:member:`~PyTypeObject.tp_alloc`,
+ :c:member:`~PyTypeObject.tp_new` (:meth:`~object.__new__`), or
+ :c:member:`~PyTypeObject.tp_init` (:meth:`~object.__init__`).
+
+ This cannot be used for objects with :c:macro:`Py_TPFLAGS_HAVE_GC` set in
+ :c:member:`~PyTypeObject.tp_flags`; use :c:macro:`PyObject_GC_New` instead.
+
+ Memory allocated by this macro must be freed with :c:func:`PyObject_Free`
+ (usually called via the object's :c:member:`~PyTypeObject.tp_free` slot).
+
+ .. note::
+
+ The returned memory is not guaranteed to have been completely zeroed
+ before it was initialized.
+
+ .. note::
+
+ This macro does not construct a fully initialized object of the given
+ type; it merely allocates memory and prepares it for further
+ initialization by :c:member:`~PyTypeObject.tp_init`. To construct a
+ fully initialized object, call *typeobj* instead. For example::
+
+ PyObject *foo = PyObject_CallNoArgs((PyObject *)&PyFoo_Type);
- Note that this function is unsuitable if *typeobj* has
- :c:macro:`Py_TPFLAGS_HAVE_GC` set. For such objects,
- use :c:func:`PyObject_GC_New` instead.
+ .. seealso::
+
+ * :c:func:`PyObject_Free`
+ * :c:macro:`PyObject_GC_New`
+ * :c:func:`PyType_GenericAlloc`
+ * :c:member:`~PyTypeObject.tp_alloc`
.. c:macro:: PyObject_NewVar(TYPE, typeobj, size)
- Allocate a new Python object using the C structure type *TYPE* and the
- Python type object *typeobj* (``PyTypeObject*``).
- Fields not defined by the Python object header
- are not initialized. The allocated memory allows for the *TYPE* structure
- plus *size* (``Py_ssize_t``) fields of the size
- given by the :c:member:`~PyTypeObject.tp_itemsize` field of
- *typeobj*. This is useful for implementing objects like tuples, which are
- able to determine their size at construction time. Embedding the array of
- fields into the same allocation decreases the number of allocations,
- improving the memory management efficiency.
-
- Note that this function is unsuitable if *typeobj* has
- :c:macro:`Py_TPFLAGS_HAVE_GC` set. For such objects,
- use :c:func:`PyObject_GC_NewVar` instead.
+ Like :c:macro:`PyObject_New` except:
+
+ * It allocates enough memory for the *TYPE* structure plus *size*
+ (``Py_ssize_t``) fields of the size given by the
+ :c:member:`~PyTypeObject.tp_itemsize` field of *typeobj*.
+ * The memory is initialized like :c:func:`PyObject_InitVar`.
+
+ This is useful for implementing objects like tuples, which are able to
+ determine their size at construction time. Embedding the array of fields
+ into the same allocation decreases the number of allocations, improving the
+ memory management efficiency.
+
+ Avoid calling this directly to allocate memory for an object; call the type's
+ :c:member:`~PyTypeObject.tp_alloc` slot instead.
+
+ When populating a type's :c:member:`~PyTypeObject.tp_alloc` slot,
+ :c:func:`PyType_GenericAlloc` is preferred over a custom function that
+ simply calls this macro.
+
+ This cannot be used for objects with :c:macro:`Py_TPFLAGS_HAVE_GC` set in
+ :c:member:`~PyTypeObject.tp_flags`; use :c:macro:`PyObject_GC_NewVar`
+ instead.
+
+ Memory allocated by this function must be freed with :c:func:`PyObject_Free`
+ (usually called via the object's :c:member:`~PyTypeObject.tp_free` slot).
+
+ .. note::
+
+ The returned memory is not guaranteed to have been completely zeroed
+ before it was initialized.
+
+ .. note::
+
+ This macro does not construct a fully initialized object of the given
+ type; it merely allocates memory and prepares it for further
+ initialization by :c:member:`~PyTypeObject.tp_init`. To construct a
+ fully initialized object, call *typeobj* instead. For example::
+
+ PyObject *list_instance = PyObject_CallNoArgs((PyObject *)&PyList_Type);
+
+ .. seealso::
+
+ * :c:func:`PyObject_Free`
+ * :c:macro:`PyObject_GC_NewVar`
+ * :c:func:`PyType_GenericAlloc`
+ * :c:member:`~PyTypeObject.tp_alloc`
.. c:function:: void PyObject_Del(void *op)
diff --git a/Doc/c-api/gcsupport.rst b/Doc/c-api/gcsupport.rst
index 3e23605778f05a..f6fa52b36c5ab3 100644
--- a/Doc/c-api/gcsupport.rst
+++ b/Doc/c-api/gcsupport.rst
@@ -57,11 +57,49 @@ rules:
Analogous to :c:macro:`PyObject_New` but for container objects with the
:c:macro:`Py_TPFLAGS_HAVE_GC` flag set.
+ Do not call this directly to allocate memory for an object; call the type's
+ :c:member:`~PyTypeObject.tp_alloc` slot instead.
+
+ When populating a type's :c:member:`~PyTypeObject.tp_alloc` slot,
+ :c:func:`PyType_GenericAlloc` is preferred over a custom function that
+ simply calls this macro.
+
+ Memory allocated by this macro must be freed with
+ :c:func:`PyObject_GC_Del` (usually called via the object's
+ :c:member:`~PyTypeObject.tp_free` slot).
+
+ .. seealso::
+
+ * :c:func:`PyObject_GC_Del`
+ * :c:macro:`PyObject_New`
+ * :c:func:`PyType_GenericAlloc`
+ * :c:member:`~PyTypeObject.tp_alloc`
+
+
.. c:macro:: PyObject_GC_NewVar(TYPE, typeobj, size)
Analogous to :c:macro:`PyObject_NewVar` but for container objects with the
:c:macro:`Py_TPFLAGS_HAVE_GC` flag set.
+ Do not call this directly to allocate memory for an object; call the type's
+ :c:member:`~PyTypeObject.tp_alloc` slot instead.
+
+ When populating a type's :c:member:`~PyTypeObject.tp_alloc` slot,
+ :c:func:`PyType_GenericAlloc` is preferred over a custom function that
+ simply calls this macro.
+
+ Memory allocated by this macro must be freed with
+ :c:func:`PyObject_GC_Del` (usually called via the object's
+ :c:member:`~PyTypeObject.tp_free` slot).
+
+ .. seealso::
+
+ * :c:func:`PyObject_GC_Del`
+ * :c:macro:`PyObject_NewVar`
+ * :c:func:`PyType_GenericAlloc`
+ * :c:member:`~PyTypeObject.tp_alloc`
+
+
.. c:function:: PyObject* PyUnstable_Object_GC_NewWithExtraData(PyTypeObject *type, size_t extra_size)
Analogous to :c:macro:`PyObject_GC_New` but allocates *extra_size*
@@ -73,6 +111,10 @@ rules:
The extra data will be deallocated with the object, but otherwise it is
not managed by Python.
+ Memory allocated by this function must be freed with
+ :c:func:`PyObject_GC_Del` (usually called via the object's
+ :c:member:`~PyTypeObject.tp_free` slot).
+
.. warning::
The function is marked as unstable because the final mechanism
for reserving extra data after an instance is not yet decided.
@@ -136,6 +178,21 @@ rules:
Releases memory allocated to an object using :c:macro:`PyObject_GC_New` or
:c:macro:`PyObject_GC_NewVar`.
+ Do not call this directly to free an object's memory; call the type's
+ :c:member:`~PyTypeObject.tp_free` slot instead.
+
+ Do not use this for memory allocated by :c:macro:`PyObject_New`,
+ :c:macro:`PyObject_NewVar`, or related allocation functions; use
+ :c:func:`PyObject_Free` instead.
+
+ .. seealso::
+
+ * :c:func:`PyObject_Free` is the non-GC equivalent of this function.
+ * :c:macro:`PyObject_GC_New`
+ * :c:macro:`PyObject_GC_NewVar`
+ * :c:func:`PyType_GenericAlloc`
+ * :c:member:`~PyTypeObject.tp_free`
+
.. c:function:: void PyObject_GC_UnTrack(void *op)
diff --git a/Doc/c-api/lifecycle.dot b/Doc/c-api/lifecycle.dot
new file mode 100644
index 00000000000000..dca9f87e9e0aca
--- /dev/null
+++ b/Doc/c-api/lifecycle.dot
@@ -0,0 +1,156 @@
+digraph "Life Events" {
+ graph [
+ fontnames="svg"
+ fontsize=12.0
+ id="life_events_graph"
+ layout="dot"
+ margin="0,0"
+ ranksep=0.25
+ stylesheet="lifecycle.dot.css"
+ ]
+ node [
+ fontname="Courier"
+ fontsize=12.0
+ ]
+ edge [
+ fontname="Times-Italic"
+ fontsize=12.0
+ ]
+
+ "start" [fontname="Times-Italic" shape=plain label=< start > style=invis]
+ {
+ rank="same"
+ "tp_new" [href="typeobj.html#c.PyTypeObject.tp_new" target="_top"]
+ "tp_alloc" [href="typeobj.html#c.PyTypeObject.tp_alloc" target="_top"]
+ }
+ "tp_init" [href="typeobj.html#c.PyTypeObject.tp_init" target="_top"]
+ "reachable" [fontname="Times-Italic" shape=box]
+ "tp_traverse" [
+ href="typeobj.html#c.PyTypeObject.tp_traverse"
+ ordering="in"
+ target="_top"
+ ]
+ "finalized?" [
+ fontname="Times-Italic"
+ label=finalized?>
+ ordering="in"
+ shape=diamond
+ tooltip="marked as finalized?"
+ ]
+ "tp_finalize" [
+ href="typeobj.html#c.PyTypeObject.tp_finalize"
+ ordering="in"
+ target="_top"
+ ]
+ "tp_clear" [href="typeobj.html#c.PyTypeObject.tp_clear" target="_top"]
+ "uncollectable" [
+ fontname="Times-Italic"
+ label=(leaked)>
+ shape=box
+ tooltip="uncollectable (leaked)"
+ ]
+ "tp_dealloc" [
+ href="typeobj.html#c.PyTypeObject.tp_dealloc"
+ ordering="in"
+ target="_top"
+ ]
+ "tp_free" [href="typeobj.html#c.PyTypeObject.tp_free" target="_top"]
+
+ "start" -> "tp_new" [
+ label=< type call >
+ ]
+ "tp_new" -> "tp_alloc" [
+ label=< direct call > arrowhead=empty
+ labeltooltip="tp_new to tp_alloc: direct call"
+ tooltip="tp_new to tp_alloc: direct call"
+ ]
+ "tp_new" -> "tp_init" [tooltip="tp_new to tp_init"]
+ "tp_init" -> "reachable" [tooltip="tp_init to reachable"]
+ "reachable" -> "tp_traverse" [
+ dir="back"
+ label=< not in a cyclic isolate >
+ labeltooltip="tp_traverse to reachable: not in a cyclic isolate"
+ tooltip="tp_traverse to reachable: not in a cyclic isolate"
+ ]
+ "reachable" -> "tp_traverse" [
+ label=< periodic cyclic isolate detection >
+ labeltooltip="reachable to tp_traverse: periodic cyclic isolate detection"
+ tooltip="reachable to tp_traverse: periodic cyclic isolate detection"
+ ]
+ "reachable" -> "tp_init" [tooltip="reachable to tp_init"]
+ "reachable" -> "tp_finalize" [
+ dir="back"
+ label=< resurrected (maybe remove finalized mark) >
+ labeltooltip="tp_finalize to reachable: resurrected (maybe remove finalized mark)"
+ tooltip="tp_finalize to reachable: resurrected (maybe remove finalized mark)"
+ ]
+ "tp_traverse" -> "finalized?" [
+ label=< cyclic isolate >
+ labeltooltip="tp_traverse to finalized?: cyclic isolate"
+ tooltip="tp_traverse to finalized?: cyclic isolate"
+ ]
+ "reachable" -> "finalized?" [
+ label=< no refs >
+ labeltooltip="reachable to finalized?: no refs"
+ tooltip="reachable to finalized?: no refs"
+ ]
+ "finalized?" -> "tp_finalize" [
+ label=< no (mark as finalized) >
+ labeltooltip="finalized? to tp_finalize: no (mark as finalized)"
+ tooltip="finalized? to tp_finalize: no (mark as finalized)"
+ ]
+ "finalized?" -> "tp_clear" [
+ label=< yes >
+ labeltooltip="finalized? to tp_clear: yes"
+ tooltip="finalized? to tp_clear: yes"
+ ]
+ "tp_finalize" -> "tp_clear" [
+ label=< no refs or cyclic isolate >
+ labeltooltip="tp_finalize to tp_clear: no refs or cyclic isolate"
+ tooltip="tp_finalize to tp_clear: no refs or cyclic isolate"
+ ]
+ "tp_finalize" -> "tp_dealloc" [
+ arrowtail=empty
+ dir="back"
+ href="lifecycle.html#c.PyObject_CallFinalizerFromDealloc"
+ style=dashed
+ label=< recommended call (see explanation)>
+ labeltooltip="tp_dealloc to tp_finalize: recommended call (see explanation)"
+ target="_top"
+ tooltip="tp_dealloc to tp_finalize: recommended call (see explanation)"
+ ]
+ "tp_finalize" -> "tp_dealloc" [
+ label=< no refs >
+ labeltooltip="tp_finalize to tp_dealloc: no refs"
+ tooltip="tp_finalize to tp_dealloc: no refs"
+ ]
+ "tp_clear" -> "tp_dealloc" [
+ label=< no refs >
+ labeltooltip="tp_clear to tp_dealloc: no refs"
+ tooltip="tp_clear to tp_dealloc: no refs"
+ ]
+ "tp_clear" -> "uncollectable" [
+ label=< cyclic isolate >
+ labeltooltip="tp_clear to uncollectable: cyclic isolate"
+ tooltip="tp_clear to uncollectable: cyclic isolate"
+ ]
+ "uncollectable" -> "tp_dealloc" [
+ style=invis
+ tooltip="uncollectable to tp_dealloc"
+ ]
+ "reachable" -> "uncollectable" [
+ label=< cyclic isolate (no GC support) >
+ labeltooltip="reachable to uncollectable: cyclic isolate (no GC support)"
+ tooltip="reachable to uncollectable: cyclic isolate (no GC support)"
+ ]
+ "reachable" -> "tp_dealloc" [
+ label=< no refs>
+ labeltooltip="reachable to tp_dealloc: no refs"
+ ]
+ "tp_dealloc" -> "tp_free" [
+ arrowhead=empty
+ label=< direct call >
+ labeltooltip="tp_dealloc to tp_free: direct call"
+ tooltip="tp_dealloc to tp_free: direct call"
+ ]
+}
diff --git a/Doc/c-api/lifecycle.dot.css b/Doc/c-api/lifecycle.dot.css
new file mode 100644
index 00000000000000..3abf95b74da6ba
--- /dev/null
+++ b/Doc/c-api/lifecycle.dot.css
@@ -0,0 +1,21 @@
+#life_events_graph {
+ --svg-fgcolor: currentcolor;
+ --svg-bgcolor: transparent;
+}
+#life_events_graph a {
+ color: inherit;
+}
+#life_events_graph [stroke="black"] {
+ stroke: var(--svg-fgcolor);
+}
+#life_events_graph text,
+#life_events_graph [fill="black"] {
+ fill: var(--svg-fgcolor);
+}
+#life_events_graph [fill="white"] {
+ fill: var(--svg-bgcolor);
+}
+#life_events_graph [fill="none"] {
+ /* On links, setting fill will make the entire shape clickable */
+ fill: var(--svg-bgcolor);
+}
diff --git a/Doc/c-api/lifecycle.dot.pdf b/Doc/c-api/lifecycle.dot.pdf
new file mode 100644
index 00000000000000..ed5b5039c83e2c
Binary files /dev/null and b/Doc/c-api/lifecycle.dot.pdf differ
diff --git a/Doc/c-api/lifecycle.dot.svg b/Doc/c-api/lifecycle.dot.svg
new file mode 100644
index 00000000000000..7ace27dfcba113
--- /dev/null
+++ b/Doc/c-api/lifecycle.dot.svg
@@ -0,0 +1,374 @@
+
+
+
+
+
+
diff --git a/Doc/c-api/lifecycle.rst b/Doc/c-api/lifecycle.rst
new file mode 100644
index 00000000000000..0e2ffc096caba7
--- /dev/null
+++ b/Doc/c-api/lifecycle.rst
@@ -0,0 +1,273 @@
+.. highlight:: c
+
+.. _life-cycle:
+
+Object Life Cycle
+=================
+
+This section explains how a type's slots relate to each other throughout the
+life of an object. It is not intended to be a complete canonical reference for
+the slots; instead, refer to the slot-specific documentation in
+:ref:`type-structs` for details about a particular slot.
+
+
+Life Events
+-----------
+
+The figure below illustrates the order of events that can occur throughout an
+object's life. An arrow from *A* to *B* indicates that event *B* can occur
+after event *A* has occurred, with the arrow's label indicating the condition
+that must be true for *B* to occur after *A*.
+
+.. only:: html and not epub
+
+ .. raw:: html
+
+
+
+ .. raw:: html
+ :file: lifecycle.dot.svg
+
+ .. raw:: html
+
+
+
+.. only:: epub or not (html or latex)
+
+ .. image:: lifecycle.dot.svg
+ :align: center
+ :class: invert-in-dark-mode
+ :alt: Diagram showing events in an object's life. Explained in detail
+ below.
+
+.. only:: latex
+
+ .. image:: lifecycle.dot.pdf
+ :align: center
+ :class: invert-in-dark-mode
+ :alt: Diagram showing events in an object's life. Explained in detail
+ below.
+
+.. container::
+ :name: life-events-graph-description
+
+ Explanation:
+
+ * When a new object is constructed by calling its type:
+
+ #. :c:member:`~PyTypeObject.tp_new` is called to create a new object.
+ #. :c:member:`~PyTypeObject.tp_alloc` is directly called by
+ :c:member:`~PyTypeObject.tp_new` to allocate the memory for the new
+ object.
+ #. :c:member:`~PyTypeObject.tp_init` initializes the newly created object.
+ :c:member:`!tp_init` can be called again to re-initialize an object, if
+ desired. The :c:member:`!tp_init` call can also be skipped entirely,
+ for example by Python code calling :py:meth:`~object.__new__`.
+
+ * After :c:member:`!tp_init` completes, the object is ready to use.
+ * Some time after the last reference to an object is removed:
+
+ #. If an object is not marked as *finalized*, it might be finalized by
+ marking it as *finalized* and calling its
+ :c:member:`~PyTypeObject.tp_finalize` function. Python does
+ *not* finalize an object when the last reference to it is deleted; use
+ :c:func:`PyObject_CallFinalizerFromDealloc` to ensure that
+ :c:member:`~PyTypeObject.tp_finalize` is always called.
+ #. If the object is marked as finalized,
+ :c:member:`~PyTypeObject.tp_clear` might be called by the garbage collector
+ to clear references held by the object. It is *not* called when the
+ object's reference count reaches zero.
+ #. :c:member:`~PyTypeObject.tp_dealloc` is called to destroy the object.
+ To avoid code duplication, :c:member:`~PyTypeObject.tp_dealloc` typically
+ calls into :c:member:`~PyTypeObject.tp_clear` to free up the object's
+ references.
+ #. When :c:member:`~PyTypeObject.tp_dealloc` finishes object destruction,
+ it directly calls :c:member:`~PyTypeObject.tp_free` (usually set to
+ :c:func:`PyObject_Free` or :c:func:`PyObject_GC_Del` automatically as
+ appropriate for the type) to deallocate the memory.
+
+ * The :c:member:`~PyTypeObject.tp_finalize` function is permitted to add a
+ reference to the object if desired. If it does, the object is
+ *resurrected*, preventing its pending destruction. (Only
+ :c:member:`!tp_finalize` is allowed to resurrect an object;
+ :c:member:`~PyTypeObject.tp_clear` and
+ :c:member:`~PyTypeObject.tp_dealloc` cannot without calling into
+ :c:member:`!tp_finalize`.) Resurrecting an object may
+ or may not cause the object's *finalized* mark to be removed. Currently,
+ Python does not remove the *finalized* mark from a resurrected object if
+ it supports garbage collection (i.e., the :c:macro:`Py_TPFLAGS_HAVE_GC`
+ flag is set) but does remove the mark if the object does not support
+ garbage collection; either or both of these behaviors may change in the
+ future.
+ * :c:member:`~PyTypeObject.tp_dealloc` can optionally call
+ :c:member:`~PyTypeObject.tp_finalize` via
+ :c:func:`PyObject_CallFinalizerFromDealloc` if it wishes to reuse that
+ code to help with object destruction. This is recommended because it
+ guarantees that :c:member:`!tp_finalize` is always called before
+ destruction. See the :c:member:`~PyTypeObject.tp_dealloc` documentation
+ for example code.
+ * If the object is a member of a :term:`cyclic isolate` and either
+ :c:member:`~PyTypeObject.tp_clear` fails to break the reference cycle or
+ the cyclic isolate is not detected (perhaps :func:`gc.disable` was called,
+ or the :c:macro:`Py_TPFLAGS_HAVE_GC` flag was erroneously omitted in one
+ of the involved types), the objects remain indefinitely uncollectable
+ (they "leak"). See :data:`gc.garbage`.
+
+ If the object is marked as supporting garbage collection (the
+ :c:macro:`Py_TPFLAGS_HAVE_GC` flag is set in
+ :c:member:`~PyTypeObject.tp_flags`), the following events are also possible:
+
+ * The garbage collector occasionally calls
+ :c:member:`~PyTypeObject.tp_traverse` to identify :term:`cyclic isolates
+ `.
+ * When the garbage collector discovers a :term:`cyclic isolate`, it
+ finalizes one of the objects in the group by marking it as *finalized* and
+ calling its :c:member:`~PyTypeObject.tp_finalize` function, if it has one.
+ This repeats until the cyclic isolate doesn't exist or all of the objects
+ have been finalized.
+ * :c:member:`~PyTypeObject.tp_finalize` is permitted to resurrect the object
+ by adding a reference from outside the :term:`cyclic isolate`. The new
+ reference causes the group of objects to no longer form a cyclic isolate
+ (the reference cycle may still exist, but if it does the objects are no
+ longer isolated).
+ * When the garbage collector discovers a :term:`cyclic isolate` and all of
+ the objects in the group have already been marked as *finalized*, the
+ garbage collector clears one or more of the uncleared objects in the group
+ (possibly concurrently) by calling each's
+ :c:member:`~PyTypeObject.tp_clear` function. This repeats as long as the
+ cyclic isolate still exists and not all of the objects have been cleared.
+
+
+Cyclic Isolate Destruction
+--------------------------
+
+Listed below are the stages of life of a hypothetical :term:`cyclic isolate`
+that continues to exist after each member object is finalized or cleared. It
+is a memory leak if a cyclic isolate progresses through all of these stages; it should
+vanish once all objects are cleared, if not sooner. A cyclic isolate can
+vanish either because the reference cycle is broken or because the objects are
+no longer isolated due to finalizer resurrection (see
+:c:member:`~PyTypeObject.tp_finalize`).
+
+0. **Reachable** (not yet a cyclic isolate): All objects are in their normal,
+ reachable state. A reference cycle could exist, but an external reference
+ means the objects are not yet isolated.
+#. **Unreachable but consistent:** The final reference from outside the cyclic
+ group of objects has been removed, causing the objects to become isolated
+ (thus a cyclic isolate is born). None of the group's objects have been
+ finalized or cleared yet. The cyclic isolate remains at this stage until
+ some future run of the garbage collector (not necessarily the next run
+ because the next run might not scan every object).
+#. **Mix of finalized and not finalized:** Objects in a cyclic isolate are
+ finalized one at a time, which means that there is a period of time when the
+ cyclic isolate is composed of a mix of finalized and non-finalized objects.
+ Finalization order is unspecified, so it can appear random. A finalized
+ object must behave in a sane manner when non-finalized objects interact with
+ it, and a non-finalized object must be able to tolerate the finalization of
+ an arbitrary subset of its referents.
+#. **All finalized:** All objects in a cyclic isolate are finalized before any
+ of them are cleared.
+#. **Mix of finalized and cleared:** The objects can be cleared serially or
+ concurrently (but with the :term:`GIL` held); either way, some will finish
+ before others. A finalized object must be able to tolerate the clearing of
+ a subset of its referents. :pep:`442` calls this stage "cyclic trash".
+#. **Leaked:** If a cyclic isolate still exists after all objects in the group
+ have been finalized and cleared, then the objects remain indefinitely
+ uncollectable (see :data:`gc.garbage`). It is a bug if a cyclic isolate
+ reaches this stage---it means the :c:member:`~PyTypeObject.tp_clear` methods
+ of the participating objects have failed to break the reference cycle as
+ required.
+
+If :c:member:`~PyTypeObject.tp_clear` did not exist, then Python would have no
+way to safely break a reference cycle. Simply destroying an object in a cyclic
+isolate would result in a dangling pointer, triggering undefined behavior when
+an object referencing the destroyed object is itself destroyed. The clearing
+step makes object destruction a two-phase process: first
+:c:member:`~PyTypeObject.tp_clear` is called to partially destroy the objects
+enough to detangle them from each other, then
+:c:member:`~PyTypeObject.tp_dealloc` is called to complete the destruction.
+
+Unlike clearing, finalization is not a phase of destruction. A finalized
+object must still behave properly by continuing to fulfill its design
+contracts. An object's finalizer is allowed to execute arbitrary Python code,
+and is even allowed to prevent the impending destruction by adding a reference.
+The finalizer is only related to destruction by call order---if it runs, it runs
+before destruction, which starts with :c:member:`~PyTypeObject.tp_clear` (if
+called) and concludes with :c:member:`~PyTypeObject.tp_dealloc`.
+
+The finalization step is not necessary to safely reclaim the objects in a
+cyclic isolate, but its existence makes it easier to design types that behave
+in a sane manner when objects are cleared. Clearing an object might
+necessarily leave it in a broken, partially destroyed state---it might be
+unsafe to call any of the cleared object's methods or access any of its
+attributes. With finalization, only finalized objects can possibly interact
+with cleared objects; non-finalized objects are guaranteed to interact with
+only non-cleared (but potentially finalized) objects.
+
+To summarize the possible interactions:
+
+* A non-finalized object might have references to or from non-finalized and
+ finalized objects, but not to or from cleared objects.
+* A finalized object might have references to or from non-finalized, finalized,
+ and cleared objects.
+* A cleared object might have references to or from finalized and cleared
+ objects, but not to or from non-finalized objects.
+
+Without any reference cycles, an object can be simply destroyed once its last
+reference is deleted; the finalization and clearing steps are not necessary to
+safely reclaim unused objects. However, it can be useful to automatically call
+:c:member:`~PyTypeObject.tp_finalize` and :c:member:`~PyTypeObject.tp_clear`
+before destruction anyway because type design is simplified when all objects
+always experience the same series of events regardless of whether they
+participated in a cyclic isolate. Python currently only calls
+:c:member:`~PyTypeObject.tp_finalize` and :c:member:`~PyTypeObject.tp_clear` as
+needed to destroy a cyclic isolate; this may change in a future version.
+
+
+Functions
+---------
+
+To allocate and free memory, see :ref:`allocating-objects`.
+
+
+.. c:function:: void PyObject_CallFinalizer(PyObject *op)
+
+ Finalizes the object as described in :c:member:`~PyTypeObject.tp_finalize`.
+ Call this function (or :c:func:`PyObject_CallFinalizerFromDealloc`) instead
+ of calling :c:member:`~PyTypeObject.tp_finalize` directly because this
+ function may deduplicate multiple calls to :c:member:`!tp_finalize`.
+ Currently, calls are only deduplicated if the type supports garbage
+ collection (i.e., the :c:macro:`Py_TPFLAGS_HAVE_GC` flag is set); this may
+ change in the future.
+
+
+.. c:function:: int PyObject_CallFinalizerFromDealloc(PyObject *op)
+
+ Same as :c:func:`PyObject_CallFinalizer` but meant to be called at the
+ beginning of the object's destructor (:c:member:`~PyTypeObject.tp_dealloc`).
+ There must not be any references to the object. If the object's finalizer
+ resurrects the object, this function returns -1; no further destruction
+ should happen. Otherwise, this function returns 0 and destruction can
+ continue normally.
+
+ .. seealso::
+
+ :c:member:`~PyTypeObject.tp_dealloc` for example code.
diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst
index 64ae35daa703b8..61fa49f8681cce 100644
--- a/Doc/c-api/memory.rst
+++ b/Doc/c-api/memory.rst
@@ -376,6 +376,24 @@ The :ref:`default object allocator ` uses the
If *p* is ``NULL``, no operation is performed.
+ Do not call this directly to free an object's memory; call the type's
+ :c:member:`~PyTypeObject.tp_free` slot instead.
+
+ Do not use this for memory allocated by :c:macro:`PyObject_GC_New` or
+ :c:macro:`PyObject_GC_NewVar`; use :c:func:`PyObject_GC_Del` instead.
+
+ .. seealso::
+
+ * :c:func:`PyObject_GC_Del` is the equivalent of this function for memory
+ allocated by types that support garbage collection.
+ * :c:func:`PyObject_Malloc`
+ * :c:func:`PyObject_Realloc`
+ * :c:func:`PyObject_Calloc`
+ * :c:macro:`PyObject_New`
+ * :c:macro:`PyObject_NewVar`
+ * :c:func:`PyType_GenericAlloc`
+ * :c:member:`~PyTypeObject.tp_free`
+
.. _default-memory-allocators:
diff --git a/Doc/c-api/objimpl.rst b/Doc/c-api/objimpl.rst
index 8bd8c107c98bdf..83de4248039949 100644
--- a/Doc/c-api/objimpl.rst
+++ b/Doc/c-api/objimpl.rst
@@ -12,6 +12,7 @@ object types.
.. toctree::
allocation.rst
+ lifecycle.rst
structures.rst
typeobj.rst
gcsupport.rst
diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst
index ec2867b0ce09ba..2176b8e492f306 100644
--- a/Doc/c-api/type.rst
+++ b/Doc/c-api/type.rst
@@ -151,14 +151,29 @@ Type Objects
.. c:function:: PyObject* PyType_GenericAlloc(PyTypeObject *type, Py_ssize_t nitems)
- Generic handler for the :c:member:`~PyTypeObject.tp_alloc` slot of a type object. Use
- Python's default memory allocation mechanism to allocate a new instance and
- initialize all its contents to ``NULL``.
+ Generic handler for the :c:member:`~PyTypeObject.tp_alloc` slot of a type
+ object. Uses Python's default memory allocation mechanism to allocate memory
+ for a new instance, zeros the memory, then initializes the memory as if by
+ calling :c:func:`PyObject_Init` or :c:func:`PyObject_InitVar`.
+
+ Do not call this directly to allocate memory for an object; call the type's
+ :c:member:`~PyTypeObject.tp_alloc` slot instead.
+
+ For types that support garbage collection (i.e., the
+ :c:macro:`Py_TPFLAGS_HAVE_GC` flag is set), this function behaves like
+ :c:macro:`PyObject_GC_New` or :c:macro:`PyObject_GC_NewVar` (except the
+ memory is guaranteed to be zeroed before initialization), and should be
+ paired with :c:func:`PyObject_GC_Del` in :c:member:`~PyTypeObject.tp_free`.
+ Otherwise, it behaves like :c:macro:`PyObject_New` or
+ :c:macro:`PyObject_NewVar` (except the memory is guaranteed to be zeroed
+ before initialization) and should be paired with :c:func:`PyObject_Free` in
+ :c:member:`~PyTypeObject.tp_free`.
.. c:function:: PyObject* PyType_GenericNew(PyTypeObject *type, PyObject *args, PyObject *kwds)
- Generic handler for the :c:member:`~PyTypeObject.tp_new` slot of a type object. Create a
- new instance using the type's :c:member:`~PyTypeObject.tp_alloc` slot.
+ Generic handler for the :c:member:`~PyTypeObject.tp_new` slot of a type
+ object. Creates a new instance using the type's
+ :c:member:`~PyTypeObject.tp_alloc` slot and returns the resulting object.
.. c:function:: int PyType_Ready(PyTypeObject *type)
diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst
index 3b9f07778d5ace..64353a8daca5f6 100644
--- a/Doc/c-api/typeobj.rst
+++ b/Doc/c-api/typeobj.rst
@@ -676,77 +676,122 @@ and :c:data:`PyType_Type` effectively act as defaults.)
.. c:member:: destructor PyTypeObject.tp_dealloc
- A pointer to the instance destructor function. This function must be defined
- unless the type guarantees that its instances will never be deallocated (as is
- the case for the singletons ``None`` and ``Ellipsis``). The function signature is::
+ A pointer to the instance destructor function. The function signature is::
void tp_dealloc(PyObject *self);
- The destructor function is called by the :c:func:`Py_DECREF` and
- :c:func:`Py_XDECREF` macros when the new reference count is zero. At this point,
- the instance is still in existence, but there are no references to it. The
- destructor function should free all references which the instance owns, free all
- memory buffers owned by the instance (using the freeing function corresponding
- to the allocation function used to allocate the buffer), and call the type's
- :c:member:`~PyTypeObject.tp_free` function. If the type is not subtypable
- (doesn't have the :c:macro:`Py_TPFLAGS_BASETYPE` flag bit set), it is
- permissible to call the object deallocator directly instead of via
- :c:member:`~PyTypeObject.tp_free`. The object deallocator should be the one used to allocate the
- instance; this is normally :c:func:`PyObject_Free` if the instance was allocated
- using :c:macro:`PyObject_New` or :c:macro:`PyObject_NewVar`, or
- :c:func:`PyObject_GC_Del` if the instance was allocated using
- :c:macro:`PyObject_GC_New` or :c:macro:`PyObject_GC_NewVar`.
-
- If the type supports garbage collection (has the :c:macro:`Py_TPFLAGS_HAVE_GC`
- flag bit set), the destructor should call :c:func:`PyObject_GC_UnTrack`
+ The destructor function should remove all references which the instance owns
+ (e.g., call :c:func:`Py_CLEAR`), free all memory buffers owned by the
+ instance, and call the type's :c:member:`~PyTypeObject.tp_free` function to
+ free the object itself.
+
+ No guarantees are made about when an object is destroyed, except:
+
+ * Python will destroy an object immediately or some time after the final
+ reference to the object is deleted, unless its finalizer
+ (:c:member:`~PyTypeObject.tp_finalize`) subsequently resurrects the
+ object.
+ * An object will not be destroyed while it is being automatically finalized
+ (:c:member:`~PyTypeObject.tp_finalize`) or automatically cleared
+ (:c:member:`~PyTypeObject.tp_clear`).
+
+ CPython currently destroys an object immediately from :c:func:`Py_DECREF`
+ when the new reference count is zero, but this may change in a future
+ version.
+
+ It is recommended to call :c:func:`PyObject_CallFinalizerFromDealloc` at the
+ beginning of :c:member:`!tp_dealloc` to guarantee that the object is always
+ finalized before destruction.
+
+ If the type supports garbage collection (the :c:macro:`Py_TPFLAGS_HAVE_GC`
+ flag is set), the destructor should call :c:func:`PyObject_GC_UnTrack`
before clearing any member fields.
- .. code-block:: c
+ It is permissible to call :c:member:`~PyTypeObject.tp_clear` from
+ :c:member:`!tp_dealloc` to reduce code duplication and to guarantee that the
+ object is always cleared before destruction. Beware that
+ :c:member:`!tp_clear` might have already been called.
- static void
- foo_dealloc(PyObject *op)
- {
+ If the type is heap allocated (:c:macro:`Py_TPFLAGS_HEAPTYPE`), the
+ deallocator should release the owned reference to its type object (via
+ :c:func:`Py_DECREF`) after calling the type deallocator. See the example
+ code below.::
+
+ static void
+ foo_dealloc(PyObject *op)
+ {
foo_object *self = (foo_object *) op;
PyObject_GC_UnTrack(self);
Py_CLEAR(self->ref);
Py_TYPE(self)->tp_free(self);
- }
+ }
+
+ :c:member:`!tp_dealloc` must leave the exception status unchanged. If it
+ needs to call something that might raise an exception, the exception state
+ must be backed up first and restored later (after logging any exceptions
+ with :c:func:`PyErr_WriteUnraisable`).
+
+ Example::
- Finally, if the type is heap allocated (:c:macro:`Py_TPFLAGS_HEAPTYPE`), the
- deallocator should release the owned reference to its type object
- (via :c:func:`Py_DECREF`) after
- calling the type deallocator. In order to avoid dangling pointers, the
- recommended way to achieve this is:
+ static void
+ foo_dealloc(PyObject *self)
+ {
+ PyObject *exc = PyErr_GetRaisedException();
- .. code-block:: c
+ if (PyObject_CallFinalizerFromDealloc(self) < 0) {
+ // self was resurrected.
+ goto done;
+ }
- static void
- foo_dealloc(PyObject *op)
- {
- PyTypeObject *tp = Py_TYPE(op);
- // free references and buffers here
- tp->tp_free(op);
- Py_DECREF(tp);
- }
+ PyTypeObject *tp = Py_TYPE(self);
- .. warning::
+ if (tp->tp_flags & Py_TPFLAGS_HAVE_GC) {
+ PyObject_GC_UnTrack(self);
+ }
- In a garbage collected Python, :c:member:`!tp_dealloc` may be called from
- any Python thread, not just the thread which created the object (if the
- object becomes part of a refcount cycle, that cycle might be collected by
- a garbage collection on any thread). This is not a problem for Python
- API calls, since the thread on which :c:member:`!tp_dealloc` is called
- with an :term:`attached thread state`. However, if the object being
- destroyed in turn destroys objects from some other C or C++ library, care
- should be taken to ensure that destroying those objects on the thread
- which called :c:member:`!tp_dealloc` will not violate any assumptions of
- the library.
+ // Optional, but convenient to avoid code duplication.
+ if (tp->tp_clear && tp->tp_clear(self) < 0) {
+ PyErr_WriteUnraisable(self);
+ }
+
+ // Any additional destruction goes here.
+
+ tp->tp_free(self);
+ self = NULL; // In case PyErr_WriteUnraisable() is called below.
+
+ if (tp->tp_flags & Py_TPFLAGS_HEAPTYPE) {
+ Py_CLEAR(tp);
+ }
+
+ done:
+ // Optional, if something was called that might have raised an
+ // exception.
+ if (PyErr_Occurred()) {
+ PyErr_WriteUnraisable(self);
+ }
+ PyErr_SetRaisedException(exc);
+ }
+
+ :c:member:`!tp_dealloc` may be called from
+ any Python thread, not just the thread which created the object (if the
+ object becomes part of a refcount cycle, that cycle might be collected by
+ a garbage collection on any thread). This is not a problem for Python
+ API calls, since the thread on which :c:member:`!tp_dealloc` is called
+ with an :term:`attached thread state`. However, if the object being
+ destroyed in turn destroys objects from some other C library, care
+ should be taken to ensure that destroying those objects on the thread
+ which called :c:member:`!tp_dealloc` will not violate any assumptions of
+ the library.
**Inheritance:**
This field is inherited by subtypes.
+ .. seealso::
+
+ :ref:`life-cycle` for details about how this slot relates to other slots.
+
.. c:member:: Py_ssize_t PyTypeObject.tp_vectorcall_offset
@@ -1137,11 +1182,11 @@ and :c:data:`PyType_Type` effectively act as defaults.)
.. c:macro:: Py_TPFLAGS_HAVE_GC
This bit is set when the object supports garbage collection. If this bit
- is set, instances must be created using :c:macro:`PyObject_GC_New` and
- destroyed using :c:func:`PyObject_GC_Del`. More information in section
- :ref:`supporting-cycle-detection`. This bit also implies that the
- GC-related fields :c:member:`~PyTypeObject.tp_traverse` and :c:member:`~PyTypeObject.tp_clear` are present in
- the type object.
+ is set, memory for new instances (see :c:member:`~PyTypeObject.tp_alloc`)
+ must be allocated using :c:macro:`PyObject_GC_New` or
+ :c:func:`PyType_GenericAlloc` and deallocated (see
+ :c:member:`~PyTypeObject.tp_free`) using :c:func:`PyObject_GC_Del`. More
+ information in section :ref:`supporting-cycle-detection`.
**Inheritance:**
@@ -1478,6 +1523,11 @@ and :c:data:`PyType_Type` effectively act as defaults.)
heap-allocated superclass).
If they do not, the type object may not be garbage-collected.
+ .. note::
+
+ The :c:member:`~PyTypeObject.tp_traverse` function can be called from any
+ thread.
+
.. versionchanged:: 3.9
Heap-allocated types are expected to visit ``Py_TYPE(self)`` in
@@ -1497,20 +1547,101 @@ and :c:data:`PyType_Type` effectively act as defaults.)
.. c:member:: inquiry PyTypeObject.tp_clear
- An optional pointer to a clear function for the garbage collector. This is only
- used if the :c:macro:`Py_TPFLAGS_HAVE_GC` flag bit is set. The signature is::
+ An optional pointer to a clear function. The signature is::
int tp_clear(PyObject *);
- The :c:member:`~PyTypeObject.tp_clear` member function is used to break reference cycles in cyclic
- garbage detected by the garbage collector. Taken together, all :c:member:`~PyTypeObject.tp_clear`
- functions in the system must combine to break all reference cycles. This is
- subtle, and if in any doubt supply a :c:member:`~PyTypeObject.tp_clear` function. For example,
- the tuple type does not implement a :c:member:`~PyTypeObject.tp_clear` function, because it's
- possible to prove that no reference cycle can be composed entirely of tuples.
- Therefore the :c:member:`~PyTypeObject.tp_clear` functions of other types must be sufficient to
- break any cycle containing a tuple. This isn't immediately obvious, and there's
- rarely a good reason to avoid implementing :c:member:`~PyTypeObject.tp_clear`.
+ The purpose of this function is to break reference cycles that are causing a
+ :term:`cyclic isolate` so that the objects can be safely destroyed. A
+ cleared object is a partially destroyed object; the object is not obligated
+ to satisfy design invariants held during normal use.
+
+ :c:member:`!tp_clear` does not need to delete references to objects that
+ can't participate in reference cycles, such as Python strings or Python
+ integers. However, it may be convenient to clear all references, and write
+ the type's :c:member:`~PyTypeObject.tp_dealloc` function to invoke
+ :c:member:`!tp_clear` to avoid code duplication. (Beware that
+ :c:member:`!tp_clear` might have already been called. Prefer calling
+ idempotent functions like :c:func:`Py_CLEAR`.)
+
+ Any non-trivial cleanup should be performed in
+ :c:member:`~PyTypeObject.tp_finalize` instead of :c:member:`!tp_clear`.
+
+ .. note::
+
+ If :c:member:`!tp_clear` fails to break a reference cycle then the
+ objects in the :term:`cyclic isolate` may remain indefinitely
+ uncollectable ("leak"). See :data:`gc.garbage`.
+
+ .. note::
+
+ Referents (direct and indirect) might have already been cleared; they are
+ not guaranteed to be in a consistent state.
+
+ .. note::
+
+ The :c:member:`~PyTypeObject.tp_clear` function can be called from any
+ thread.
+
+ .. note::
+
+ An object is not guaranteed to be automatically cleared before its
+ destructor (:c:member:`~PyTypeObject.tp_dealloc`) is called.
+
+ This function differs from the destructor
+ (:c:member:`~PyTypeObject.tp_dealloc`) in the following ways:
+
+ * The purpose of clearing an object is to remove references to other objects
+ that might participate in a reference cycle. The purpose of the
+ destructor, on the other hand, is a superset: it must release *all*
+ resources it owns, including references to objects that cannot participate
+ in a reference cycle (e.g., integers) as well as the object's own memory
+ (by calling :c:member:`~PyTypeObject.tp_free`).
+ * When :c:member:`!tp_clear` is called, other objects might still hold
+ references to the object being cleared. Because of this,
+ :c:member:`!tp_clear` must not deallocate the object's own memory
+ (:c:member:`~PyTypeObject.tp_free`). The destructor, on the other hand,
+ is only called when no (strong) references exist, and as such, must
+ safely destroy the object itself by deallocating it.
+ * :c:member:`!tp_clear` might never be automatically called. An object's
+ destructor, on the other hand, will be automatically called some time
+ after the object becomes unreachable (i.e., either there are no references
+ to the object or the object is a member of a :term:`cyclic isolate`).
+
+ No guarantees are made about when, if, or how often Python automatically
+ clears an object, except:
+
+ * Python will not automatically clear an object if it is reachable, i.e.,
+ there is a reference to it and it is not a member of a :term:`cyclic
+ isolate`.
+ * Python will not automatically clear an object if it has not been
+ automatically finalized (see :c:member:`~PyTypeObject.tp_finalize`). (If
+ the finalizer resurrected the object, the object may or may not be
+ automatically finalized again before it is cleared.)
+ * If an object is a member of a :term:`cyclic isolate`, Python will not
+ automatically clear it if any member of the cyclic isolate has not yet
+ been automatically finalized (:c:member:`~PyTypeObject.tp_finalize`).
+ * Python will not destroy an object until after any automatic calls to its
+ :c:member:`!tp_clear` function have returned. This ensures that the act
+ of breaking a reference cycle does not invalidate the ``self`` pointer
+ while :c:member:`!tp_clear` is still executing.
+ * Python will not automatically call :c:member:`!tp_clear` multiple times
+ concurrently.
+
+ CPython currently only automatically clears objects as needed to break
+ reference cycles in a :term:`cyclic isolate`, but future versions might
+ clear objects regularly before their destruction.
+
+ Taken together, all :c:member:`~PyTypeObject.tp_clear` functions in the
+ system must combine to break all reference cycles. This is subtle, and if
+ in any doubt supply a :c:member:`~PyTypeObject.tp_clear` function. For
+ example, the tuple type does not implement a
+ :c:member:`~PyTypeObject.tp_clear` function, because it's possible to prove
+ that no reference cycle can be composed entirely of tuples. Therefore the
+ :c:member:`~PyTypeObject.tp_clear` functions of other types are responsible
+ for breaking any cycle containing a tuple. This isn't immediately obvious,
+ and there's rarely a good reason to avoid implementing
+ :c:member:`~PyTypeObject.tp_clear`.
Implementations of :c:member:`~PyTypeObject.tp_clear` should drop the instance's references to
those of its members that may be Python objects, and set its pointers to those
@@ -1545,18 +1676,6 @@ and :c:data:`PyType_Type` effectively act as defaults.)
PyObject_ClearManagedDict((PyObject*)self);
- Note that :c:member:`~PyTypeObject.tp_clear` is not *always* called
- before an instance is deallocated. For example, when reference counting
- is enough to determine that an object is no longer used, the cyclic garbage
- collector is not involved and :c:member:`~PyTypeObject.tp_dealloc` is
- called directly.
-
- Because the goal of :c:member:`~PyTypeObject.tp_clear` functions is to break reference cycles,
- it's not necessary to clear contained objects like Python strings or Python
- integers, which can't participate in reference cycles. On the other hand, it may
- be convenient to clear all contained Python objects, and write the type's
- :c:member:`~PyTypeObject.tp_dealloc` function to invoke :c:member:`~PyTypeObject.tp_clear`.
-
More information about Python's garbage collection scheme can be found in
section :ref:`supporting-cycle-detection`.
@@ -1569,6 +1688,10 @@ and :c:data:`PyType_Type` effectively act as defaults.)
:c:member:`~PyTypeObject.tp_clear` are all inherited from the base type if they are all zero in
the subtype.
+ .. seealso::
+
+ :ref:`life-cycle` for details about how this slot relates to other slots.
+
.. c:member:: richcmpfunc PyTypeObject.tp_richcompare
@@ -1945,18 +2068,17 @@ and :c:data:`PyType_Type` effectively act as defaults.)
**Inheritance:**
- This field is inherited by static subtypes, but not by dynamic
- subtypes (subtypes created by a class statement).
+ Static subtypes inherit this slot, which will be
+ :c:func:`PyType_GenericAlloc` if inherited from :class:`object`.
+
+ :ref:`Heap subtypes ` do not inherit this slot.
**Default:**
- For dynamic subtypes, this field is always set to
- :c:func:`PyType_GenericAlloc`, to force a standard heap
- allocation strategy.
+ For heap subtypes, this field is always set to
+ :c:func:`PyType_GenericAlloc`.
- For static subtypes, :c:data:`PyBaseObject_Type` uses
- :c:func:`PyType_GenericAlloc`. That is the recommended value
- for all statically defined types.
+ For static subtypes, this slot is inherited (see above).
.. c:member:: newfunc PyTypeObject.tp_new
@@ -2004,20 +2126,27 @@ and :c:data:`PyType_Type` effectively act as defaults.)
void tp_free(void *self);
- An initializer that is compatible with this signature is :c:func:`PyObject_Free`.
+ This function must free the memory allocated by
+ :c:member:`~PyTypeObject.tp_alloc`.
**Inheritance:**
- This field is inherited by static subtypes, but not by dynamic
- subtypes (subtypes created by a class statement)
+ Static subtypes inherit this slot, which will be :c:func:`PyObject_Free` if
+ inherited from :class:`object`. Exception: If the type supports garbage
+ collection (i.e., the :c:macro:`Py_TPFLAGS_HAVE_GC` flag is set in
+ :c:member:`~PyTypeObject.tp_flags`) and it would inherit
+ :c:func:`PyObject_Free`, then this slot is not inherited but instead defaults
+ to :c:func:`PyObject_GC_Del`.
+
+ :ref:`Heap subtypes ` do not inherit this slot.
**Default:**
- In dynamic subtypes, this field is set to a deallocator suitable to
- match :c:func:`PyType_GenericAlloc` and the value of the
- :c:macro:`Py_TPFLAGS_HAVE_GC` flag bit.
+ For :ref:`heap subtypes `, this slot defaults to a deallocator suitable to match
+ :c:func:`PyType_GenericAlloc` and the value of the
+ :c:macro:`Py_TPFLAGS_HAVE_GC` flag.
- For static subtypes, :c:data:`PyBaseObject_Type` uses :c:func:`PyObject_Free`.
+ For static subtypes, this slot is inherited (see above).
.. c:member:: inquiry PyTypeObject.tp_is_gc
@@ -2144,29 +2273,138 @@ and :c:data:`PyType_Type` effectively act as defaults.)
.. c:member:: destructor PyTypeObject.tp_finalize
- An optional pointer to an instance finalization function. Its signature is::
+ An optional pointer to an instance finalization function. This is the C
+ implementation of the :meth:`~object.__del__` special method. Its signature
+ is::
void tp_finalize(PyObject *self);
- If :c:member:`~PyTypeObject.tp_finalize` is set, the interpreter calls it once when
- finalizing an instance. It is called either from the garbage
- collector (if the instance is part of an isolated reference cycle) or
- just before the object is deallocated. Either way, it is guaranteed
- to be called before attempting to break reference cycles, ensuring
- that it finds the object in a sane state.
+ The primary purpose of finalization is to perform any non-trivial cleanup
+ that must be performed before the object is destroyed, while the object and
+ any other objects it directly or indirectly references are still in a
+ consistent state. The finalizer is allowed to execute
+ arbitrary Python code.
+
+ Before Python automatically finalizes an object, some of the object's direct
+ or indirect referents might have themselves been automatically finalized.
+ However, none of the referents will have been automatically cleared
+ (:c:member:`~PyTypeObject.tp_clear`) yet.
+
+ Other non-finalized objects might still be using a finalized object, so the
+ finalizer must leave the object in a sane state (e.g., invariants are still
+ met).
+
+ .. note::
+
+ After Python automatically finalizes an object, Python might start
+ automatically clearing (:c:member:`~PyTypeObject.tp_clear`) the object
+ and its referents (direct and indirect). Cleared objects are not
+ guaranteed to be in a consistent state; a finalized object must be able
+ to tolerate cleared referents.
+
+ .. note::
+
+ An object is not guaranteed to be automatically finalized before its
+ destructor (:c:member:`~PyTypeObject.tp_dealloc`) is called. It is
+ recommended to call :c:func:`PyObject_CallFinalizerFromDealloc` at the
+ beginning of :c:member:`!tp_dealloc` to guarantee that the object is
+ always finalized before destruction.
+
+ .. note::
+
+ The :c:member:`~PyTypeObject.tp_finalize` function can be called from any
+ thread, although the :term:`GIL` will be held.
- :c:member:`~PyTypeObject.tp_finalize` should not mutate the current exception status;
- therefore, a recommended way to write a non-trivial finalizer is::
+ .. note::
+
+ The :c:member:`!tp_finalize` function can be called during shutdown,
+ after some global variables have been deleted. See the documentation of
+ the :meth:`~object.__del__` method for details.
+
+ When Python finalizes an object, it behaves like the following algorithm:
+
+ #. Python might mark the object as *finalized*. Currently, Python always
+ marks objects whose type supports garbage collection (i.e., the
+ :c:macro:`Py_TPFLAGS_HAVE_GC` flag is set in
+ :c:member:`~PyTypeObject.tp_flags`) and never marks other types of
+ objects; this might change in a future version.
+ #. If the object is not marked as *finalized* and its
+ :c:member:`!tp_finalize` finalizer function is non-``NULL``, the
+ finalizer function is called.
+ #. If the finalizer function was called and the finalizer made the object
+ reachable (i.e., there is a reference to the object and it is not a
+ member of a :term:`cyclic isolate`), then the finalizer is said to have
+ *resurrected* the object. It is unspecified whether the finalizer can
+ also resurrect the object by adding a new reference to the object that
+ does not make it reachable, i.e., the object is (still) a member of a
+ cyclic isolate.
+ #. If the finalizer resurrected the object, the object's pending destruction
+ is canceled and the object's *finalized* mark might be removed if
+ present. Currently, Python never removes the *finalized* mark; this
+ might change in a future version.
+
+ *Automatic finalization* refers to any finalization performed by Python
+ except via calls to :c:func:`PyObject_CallFinalizer` or
+ :c:func:`PyObject_CallFinalizerFromDealloc`. No guarantees are made about
+ when, if, or how often an object is automatically finalized, except:
+
+ * Python will not automatically finalize an object if it is reachable, i.e.,
+ there is a reference to it and it is not a member of a :term:`cyclic
+ isolate`.
+ * Python will not automatically finalize an object if finalizing it would
+ not mark the object as *finalized*. Currently, this applies to objects
+ whose type does not support garbage collection, i.e., the
+ :c:macro:`Py_TPFLAGS_HAVE_GC` flag is not set. Such objects can still be
+ manually finalized by calling :c:func:`PyObject_CallFinalizer` or
+ :c:func:`PyObject_CallFinalizerFromDealloc`.
+ * Python will not automatically finalize any two members of a :term:`cyclic
+ isolate` concurrently.
+ * Python will not automatically finalize an object after it has
+ automatically cleared (:c:member:`~PyTypeObject.tp_clear`) the object.
+ * If an object is a member of a :term:`cyclic isolate`, Python will not
+ automatically finalize it after automatically clearing (see
+ :c:member:`~PyTypeObject.tp_clear`) any other member.
+ * Python will automatically finalize every member of a :term:`cyclic
+ isolate` before it automatically clears (see
+ :c:member:`~PyTypeObject.tp_clear`) any of them.
+ * If Python is going to automatically clear an object
+ (:c:member:`~PyTypeObject.tp_clear`), it will automatically finalize the
+ object first.
+
+ Python currently only automatically finalizes objects that are members of a
+ :term:`cyclic isolate`, but future versions might finalize objects regularly
+ before their destruction.
+
+ To manually finalize an object, do not call this function directly; call
+ :c:func:`PyObject_CallFinalizer` or
+ :c:func:`PyObject_CallFinalizerFromDealloc` instead.
+
+ :c:member:`~PyTypeObject.tp_finalize` should leave the current exception
+ status unchanged. The recommended way to write a non-trivial finalizer is
+ to back up the exception at the beginning by calling
+ :c:func:`PyErr_GetRaisedException` and restore the exception at the end by
+ calling :c:func:`PyErr_SetRaisedException`. If an exception is encountered
+ in the middle of the finalizer, log and clear it with
+ :c:func:`PyErr_WriteUnraisable` or :c:func:`PyErr_FormatUnraisable`. For
+ example::
static void
- local_finalize(PyObject *self)
+ foo_finalize(PyObject *self)
{
- /* Save the current exception, if any. */
+ // Save the current exception, if any.
PyObject *exc = PyErr_GetRaisedException();
- /* ... */
+ // ...
- /* Restore the saved exception. */
+ if (do_something_that_might_raise() != success_indicator) {
+ PyErr_WriteUnraisable(self);
+ goto done;
+ }
+
+ done:
+ // Restore the saved exception. This silently discards any exception
+ // raised above, so be sure to call PyErr_WriteUnraisable first if
+ // necessary.
PyErr_SetRaisedException(exc);
}
@@ -2182,7 +2420,13 @@ and :c:data:`PyType_Type` effectively act as defaults.)
:c:macro:`Py_TPFLAGS_HAVE_FINALIZE` flags bit in order for this field to be
used. This is no longer required.
- .. seealso:: "Safe object finalization" (:pep:`442`)
+ .. seealso::
+
+ * :pep:`442`: "Safe object finalization"
+ * :ref:`life-cycle` for details about how this slot relates to other
+ slots.
+ * :c:func:`PyObject_CallFinalizer`
+ * :c:func:`PyObject_CallFinalizerFromDealloc`
.. c:member:: vectorcallfunc PyTypeObject.tp_vectorcall
diff --git a/Doc/glossary.rst b/Doc/glossary.rst
index 0b26e18efd7f1b..c5c7994f1262a9 100644
--- a/Doc/glossary.rst
+++ b/Doc/glossary.rst
@@ -355,6 +355,12 @@ Glossary
tasks (see :mod:`asyncio`) associate each task with a context which
becomes the current context whenever the task starts or resumes execution.
+ cyclic isolate
+ A subgroup of one or more objects that reference each other in a reference
+ cycle, but are not referenced by objects outside the group. The goal of
+ the :term:`cyclic garbage collector ` is to identify these groups and break the reference
+ cycles so that the memory can be reclaimed.
+
decorator
A function returning another function, usually applied as a function
transformation using the ``@wrapper`` syntax. Common examples for