@@ -15,15 +15,50 @@ See also :ref:`Remove functions <remove-funcs>`.
1515Borrowed references
1616===================
1717
18- A borrowed reference is a pointer which doesn't "hold" a reference. If the
19- object is destroyed, the borrowed reference becomes a dangling pointer: pointer
20- pointing to freed memory which might be reused by a new object.
18+ Problem caused by borrowed references
19+ -------------------------------------
2120
22- Borrowed references can lead to bugs and crashes when misused. The
23- :ref: `Specialized list for small integers <specialized-list >` optimization is
21+ A borrowed reference is a pointer which doesn't "hold" a reference. If the
22+ object is destroyed, the borrowed reference becomes a `dangling pointer
23+ <https://en.wikipedia.org/wiki/Dangling_pointer> `_: point to freed memory which
24+ might be reused by a new object. Borrowed references can lead to bugs and
25+ crashes when misused. Recent example of CPython bug: `bpo-25750: crash in
26+ type_getattro() <https://bugs.python.org/issue25750> `_.
27+
28+ Borrowed references are a problem whenever there is no reference to borrow:
29+ they assume that a reified object already exists (and thus have a positive
30+ refcout), so that it is just borrowed.
31+
32+ :ref: `Tagged pointers <tagged-pointer >` are an example of this: since there is
33+ no concrete ``PyObject* `` to represent the integer, it cannot easily be
34+ manipulated.
35+
36+ PyPy has a similar problem with list strategies: if there is a list containing
37+ only integers, it is stored as a compact C array of longs, and the W_IntObject
38+ is only created when an item is accessed (most of the time the W_IntObject is
39+ optimized away by the JIT, but this is another story).
40+
41+ But for cpyext, this is a problem: ``PyList_GetItem() `` returns a borrowed
42+ reference, but there is no any concrete ``PyObject* `` to return! The current
43+ ``cpyext `` solution is very bad: basically, the first time ``PyList_GetItem() ``
44+ is called, the *whole * list is converted to a list of ``PyObject* ``, just to
45+ have something to return: see `cpyext get_list_storage()
46+ <https://bitbucket.org/pypy/pypy/src/b9bbd6c0933349cbdbfe2b884a68a16ad16c3a8a/pypy/module/cpyext/listobject.py#lines-28> `_.
47+
48+ See also the :ref: `Specialized list for small integers <specialized-list >`
49+ optimization: same optimization applied to CPython. This optimization is
2450incompatible with borrowed references, since the runtime cannot guess when the
2551temporary object should be destroyed.
2652
53+
54+ If ``PyList_GetItem() `` returned a strong reference, the ``PyObject* `` could
55+ just be allocated on the fly and destroy it when the user decref it. Basically,
56+ by putting borrowed references in the API, we are fixing in advance the data
57+ structure to use!
58+
59+ C API using borrowed references
60+ -------------------------------
61+
2762CPython 3.7 has many functions and macros which return or use borrowed
2863references. For example, ``PyTuple_GetItem() `` returns a borrowed reference,
2964whereas ``PyTuple_SetItem() `` stores a borrowed reference (store an item into a
@@ -33,7 +68,7 @@ CPython contains ``Doc/data/refcounts.dat`` (file is edited manually) which
3368documents how functions handle reference count.
3469
3570Functions
36- ---------
71+ ^^^^^^^^^
3772
3873* ``PyDict_GetItem() ``
3974* ``PyDict_GetItemWithError() ``
@@ -56,6 +91,7 @@ Functions
5691* ``PyMethod_Class() ``
5792* ``PyMethod_Function() ``
5893* ``PyMethod_Self() ``
94+ * ``PyModule_AddObject() ``: steal a reference on success, but doesn't on error
5995* ``PyModule_GetDict() ``
6096* ``PyNumber_Check() ``
6197* ``PyObject_Init() ``
@@ -80,7 +116,8 @@ Macros
80116* ``PyTuple_SET_ITEM() ``
81117* ``PyWeakref_GET_OBJECT() ``
82118
83- Border line:
119+ Border line
120+ ^^^^^^^^^^^
84121
85122* ``Py_SETREF() ``, ``Py_XSETREF() ``: the caller has to manually increment the
86123 reference counter of the new value
0 commit comments