From af7f78eb9c0593f6df5c1e2f3dfee91ab774b7c7 Mon Sep 17 00:00:00 2001 From: Adam Dangoor Date: Fri, 7 Sep 2018 17:56:59 +0100 Subject: [PATCH 001/136] Fix ref syntax --- doc/consumers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/consumers.rst b/doc/consumers.rst index 9658ea2..83eb972 100644 --- a/doc/consumers.rst +++ b/doc/consumers.rst @@ -9,7 +9,7 @@ Popular C extensions using the C API * numpy * pandas -* :`ref:`Cython ` +* :ref:`Cython ` * Pillow * lxml From 8a948a77cf368144cacf603f680da333ccc5bdcf Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 11 Sep 2018 02:33:48 +0200 Subject: [PATCH 002/136] Add link to http://github.com/pythoncapi/cpython/ --- doc/index.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/index.rst b/doc/index.rst index 3ef08f7..5950967 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -62,6 +62,8 @@ Links ===== * `Python C API `_ (this documentation) +* `CPython fork implementing Py_NEWCAPI + `_ * `pythoncapi GitHub project `_ (this documentation can be found in the ``doc/`` subdirectory). * `capi-sig mailing list From 01709b01eaf15eb8fae711508285b7727f6893d1 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 20 Sep 2018 14:47:13 +0200 Subject: [PATCH 003/136] add bad api; add gilectomy --- doc/bad_api.rst | 6 ++++++ doc/index.rst | 1 + 2 files changed, 7 insertions(+) diff --git a/doc/bad_api.rst b/doc/bad_api.rst index 003f5f7..9e71e6b 100644 --- a/doc/bad_api.rst +++ b/doc/bad_api.rst @@ -330,4 +330,10 @@ use only". Examples: Why ``PyImport_Cleanup()`` is still a public method? +Public but not documented and not part of Python.h: + +* ``PyFrame_FastToLocalsWithError()`` +* ``PyFrame_FastToLocals()`` +* ``PyFrame_LocalsToFast()`` + These functions should be made really private and removed from the C API. diff --git a/doc/index.rst b/doc/index.rst index 5950967..3268b71 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -54,6 +54,7 @@ Pages cpyext cython cffi + gilectomy remove_c_api performance split_include From f8bf98a7292fc45c92e832dcb31f1801389f0e2b Mon Sep 17 00:00:00 2001 From: Michael Scherer Date: Thu, 20 Sep 2018 17:35:58 +0200 Subject: [PATCH 004/136] Rework the FAQ to be a bit more neutral and detailed --- doc/remove_c_api.rst | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/doc/remove_c_api.rst b/doc/remove_c_api.rst index 3c29cf9..2524130 100644 --- a/doc/remove_c_api.rst +++ b/doc/remove_c_api.rst @@ -4,15 +4,19 @@ Remove the C API ++++++++++++++++ -One obvious alternative to a new better C API is no C API at all! Existing +One proposed alternative to a new better C API is no C API at all. The reasoning is that since existing solutions are already available, complete and reliable: * :ref:`Cython ` * :ref:`cffi ` -Open questions: What about the long tail of C extensions on PyPI which still -use the C extension? Would it mean a new Python without all these C extensions -on PyPI? +We do not need to have one for python itself. -The C API is part of Python success. There would be no numpy without the C API, -for example. See :ref:`Consumers of the Python C API `. +However, this approach has lots of open questions without satisfying answers: + +What about the long tail of C extensions on PyPI which still use the C extension? +Would it mean a new Python without all these C extensions on PyPI? + +Moreover, lots of project do not use those solution, and the C API is part of Python success. For example, there would be no numpy without the C API, and one can look at :ref:`Consumers of the Python C API ` to see others examples. + +Removing it would negatively impact those projects, so this doesn't sound like a workable solution. From b50f077a882e1816b09a08e5c1730e6b47448d64 Mon Sep 17 00:00:00 2001 From: Pradyun Gedam Date: Thu, 20 Sep 2018 23:25:53 +0530 Subject: [PATCH 005/136] Fix a few typos and change form of some words --- doc/bad_api.rst | 10 +++++----- doc/cpyext.rst | 2 +- doc/gilectomy.rst | 4 ++-- doc/new_api.rst | 2 +- doc/old_c_api.rst | 2 +- doc/optimization_ideas.rst | 12 ++++++------ doc/os_vendors.rst | 2 +- doc/performance.rst | 2 +- doc/roadmap.rst | 2 +- doc/runtimes.rst | 2 +- doc/stable_abi.rst | 4 ++-- 11 files changed, 22 insertions(+), 22 deletions(-) diff --git a/doc/bad_api.rst b/doc/bad_api.rst index 9e71e6b..b6775da 100644 --- a/doc/bad_api.rst +++ b/doc/bad_api.rst @@ -32,8 +32,8 @@ crashes when misused. Recent example of CPython bug: `bpo-25750: crash in type_getattro() `_. Borrowed references are a problem whenever there is no reference to borrow: -they assume that a reified object already exists (and thus have a positive -refcout), so that it is just borrowed. +they assume that a referenced object already exists (and thus have a positive +refcount), so that it is just borrowed. :ref:`Tagged pointers ` are an example of this: since there is no concrete ``PyObject*`` to represent the integer, it cannot easily be @@ -138,7 +138,7 @@ Py_TYPE() corner case --------------------- Technically, ``Py_TYPE()`` returns a borrowed reference to a ``PyTypeObject*``. -In pratice, for heap types, an instance holds already a strong reference +In practice, for heap types, an instance holds already a strong reference to the type in ``PyObject.ob_type``. For static types, instances use a borrowed reference, but static types are never destroyed. @@ -180,7 +180,7 @@ Bad? implementations for concrete types. Examples: Implementations for concrete types don't *have to* be part of the C API. Moreover, using directly them introduce bugs when the caller pass a subtype. For example, PyDict_GetItem() **must not** be used on a dict subtype, since -``__getitem__()`` be be overriden for good reasons. +``__getitem__()`` be be overridden for good reasons. Functions kept for backward compatibility @@ -265,7 +265,7 @@ Example of macros: PyType_Ready() and setting directly PyTypeObject fields ======================================================= -* ``PyTypeObject`` structure should become opaquet +* ``PyTypeObject`` structure should become opaque * ``PyType_Ready()`` should be removed See :ref:`Implement a PyTypeObject in C ` for the rationale. diff --git a/doc/cpyext.rst b/doc/cpyext.rst index 80cbc4a..0d1b25b 100644 --- a/doc/cpyext.rst +++ b/doc/cpyext.rst @@ -7,7 +7,7 @@ PyPy cpyext module cpyext is the implementation of the C API in `PyPy `__. See Ronan Lamy's talk `Adventures in compatibility emulating CPython's C API in -PyPy `_ (Youtube video) at +PyPy `_ (YouTube video) at EuroPython 2018. Source diff --git a/doc/gilectomy.rst b/doc/gilectomy.rst index ecf8873..6300228 100644 --- a/doc/gilectomy.rst +++ b/doc/gilectomy.rst @@ -20,9 +20,9 @@ Gilectomy has multiple issues, but the two main issues are: For "Gilectomy 2.0", Hastings will be looking at using a tracing garbage collector (GC), rather than the CPython GC that is based on reference counts. -Tracing GCs are more multicore friendly, but he doesn't know anything about +Tracing GCs are more multi-core friendly, but he doesn't know anything about them. He also would rather not write his own GC. * https://github.com/larryhastings/gilectomy * May 2018: https://lwn.net/Articles/754577/ -* Pycon US 2017: https://speakerdeck.com/pycon2017/larry-hastings-the-gilectomy-hows-it-going +* PyCon US 2017: https://speakerdeck.com/pycon2017/larry-hastings-the-gilectomy-hows-it-going diff --git a/doc/new_api.rst b/doc/new_api.rst index 488c1b5..1826e70 100644 --- a/doc/new_api.rst +++ b/doc/new_api.rst @@ -105,7 +105,7 @@ would become:: or maybe even:: - PyObject* PyList_GET_ITEM(PyObjcet *op, PyObject *i) { return PyList_GetItem(op, i); } + PyObject* PyList_GET_ITEM(PyObject *op, PyObject *i) { return PyList_GetItem(op, i); } Adding a **new** ``PyList_GET_ITEM()`` **function** would make the ABI larger, whereas the ABI should become smaller. diff --git a/doc/old_c_api.rst b/doc/old_c_api.rst index e2c0495..f56c438 100644 --- a/doc/old_c_api.rst +++ b/doc/old_c_api.rst @@ -13,7 +13,7 @@ use cases like :ref:`Cython ` (for best performances) and :ref:`debugging tools `, but also for the long tail of C extensions on PyPI. -See also :ref:`Calling conventensions `. +See also :ref:`Calling conventions `. What is the Python C API? ========================= diff --git a/doc/optimization_ideas.rst b/doc/optimization_ideas.rst index 1eff960..ac2bd4b 100644 --- a/doc/optimization_ideas.rst +++ b/doc/optimization_ideas.rst @@ -30,10 +30,10 @@ There are discussing to use a tracing garbage collector for CPython, but this idea remains highly hypothetical since it very likely require deep changes in the C API, which is out of the scope of the :ref:`new C API project `. The main risk is to break too many C extensions which would make -this idea unsuable in practice. +this idea unusable in practice. It may be possible to emulate reference counting for the C API. Py_INCREF() and -Py_DECREF() would be reimplemented using an hash table: object => reference +Py_DECREF() would be re-implemented using an hash table: object => reference counter. Larry Hastings consider to use a tracing garbage collector for :ref:`Gilectomy @@ -51,7 +51,7 @@ smaller locks, maybe one lock per Python object. Jython has not GIL. -Reference couting remains a good and convenient API for C extension. Maybe this +Reference counting remains a good and convenient API for C extension. Maybe this design can be kept for the public C API for external C extensions, but CPython internals can be modified to avoid reference counting, like using a tracing garbage collector for example. Once the C API stops leaking implementation @@ -60,11 +60,11 @@ details, many new options become possible. Gilectomy project is CPython 3.6 fork which tries to remove the GIL. In 2017, the project did not succeed yet to scale linearly performances with the number of threads. It seems like **reference counting is a performance killer** for -multithreading. +multi-threading. By the way, using atomic operations to access (increase in ``Py_INCREF()``, decrease and test in ``Py_DECREF()``) the reference count has been proposed, -but experiment showed a slowdown of 20% on single threaded microbenchmarks. +but experiment showed a slowdown of 20% on single threaded micro-benchmarks. .. _tagged-pointer: @@ -77,7 +77,7 @@ See `Wikipedia: Tagged pointer Tagged pointers are used by MicroPython to reduce the memory footprint. -Using tagged pointers is a common optimization technic to reduce the +Using tagged pointers is a common optimization technique to reduce the boxing/unboxing cost and to reduce the memory consumption. Currently, it's not possible to implement such optimization, since most of the diff --git a/doc/os_vendors.rst b/doc/os_vendors.rst index f20c071..37861d7 100644 --- a/doc/os_vendors.rst +++ b/doc/os_vendors.rst @@ -26,7 +26,7 @@ Time scale: * A Python release is supported upstream for 5 years. * A Fedora release is supported for less than one year. * Ubuntu LTS releases are supported for 5 years. -* Red Hat Entreprise Linux (RHEL) is supported for 10 years, and customers can +* Red Hat Enterprise Linux (RHEL) is supported for 10 years, and customers can subscribe to an extended support up to 15 years. In 2018, the latest macOS release still only provides Python 2.7 which will diff --git a/doc/performance.rst b/doc/performance.rst index 3866bdf..ae14949 100644 --- a/doc/performance.rst +++ b/doc/performance.rst @@ -16,7 +16,7 @@ Hiding implementation details is likely to make tiny loops slower, since it adds function calls instead of directly accessing the memory. The performance slowdown is expected to be negligible, but has to be measured -once a concrete implmenetation will be written. +once a concrete implementation will be written. Question: would it be acceptable to have a new better C API if the average slowdown is around 10%? What if the slowdown is up to 25%? Or even 50%? diff --git a/doc/roadmap.rst b/doc/roadmap.rst index 6532c59..9ee8a38 100644 --- a/doc/roadmap.rst +++ b/doc/roadmap.rst @@ -67,7 +67,7 @@ Timeline * 2017-07: Idea proposed on python-ideas. `[Python-ideas] PEP: Hide implementation details in the C API `_ -* 2017-05: Idea proposed at the Python Language Summit, during Pycon US 2017. +* 2017-05: Idea proposed at the Python Language Summit, during PyCon US 2017. My `"Python performance" slides (PDF) `_. LWN article: `Keeping Python competitive diff --git a/doc/runtimes.rst b/doc/runtimes.rst index 0062e4d..440766a 100644 --- a/doc/runtimes.rst +++ b/doc/runtimes.rst @@ -57,7 +57,7 @@ Debug runtime: /usr/bin/python3-dbg * Runtime compiled with C assertion: crash (kill itself with SIGABRT signal) if a C assertion fails (``assert(...);``). * Use the debug hooks on memory allocators by default, as ``PYTHONDEBUG=debug`` - environment variable: detect memory under- and overflow and misusage of + environment variable: detect memory under- and overflow and misuse of memory allocators. * Compiled without compiler optimizations (``-Og`` or even ``-O0``) to be usable with a debugger like ``gdb``: python-gdb.py should work perfectly. diff --git a/doc/stable_abi.rst b/doc/stable_abi.rst index 794ad6a..bbb391d 100644 --- a/doc/stable_abi.rst +++ b/doc/stable_abi.rst @@ -42,8 +42,8 @@ whereas a debug build uses an offset of **40** instead of **24**, because As a consequence, the compiled C extension is incompatible at the ABI level: a C extension has to be build twice, once in release mode and once in debug mode. -To reduce the maintaince burden, :ref:`Linux vendors ` only provide -C extensions compiled in release mode, making the :ref:`debug mode +To reduce the maintenance burden, :ref:`Linux vendors ` only +provide C extensions compiled in release mode, making the :ref:`debug mode ` mostly unusable on Linux in practice. From 3db48abe4e3acbcbdf5dbda74d7b777a812b51ae Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 17 Oct 2018 23:19:34 +0200 Subject: [PATCH 006/136] bytearray --- doc/optimization_ideas.rst | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/doc/optimization_ideas.rst b/doc/optimization_ideas.rst index ac2bd4b..3d516c6 100644 --- a/doc/optimization_ideas.rst +++ b/doc/optimization_ideas.rst @@ -156,6 +156,36 @@ it becomes possible to modify PyObject structures. The main benefit of the memory footprint, but the overall on performances is unknown at this point. +O(1) conversion of bytearray to bytes +===================================== + +TODO: find a better method name :-) + +Problem: memory copy, memory usage +---------------------------------- + +When a function produces a bytes string but the output length is enough, using +a temporary bytearray object is recommended to use the efficient ``bytearray += +bytes`` pattern (bytearray overallocates its internal buffer and so reduce the +number of reallocations). Problem: if the result type must be bytes, the +bytearray must be converted to bytes... and this operation currently requires +to copy the memory. For example, _pyio.FileIO.readall() copies the full content +of a binary file and doubles the memory usage. + +In Python 3.7, a bytes object always use a single memory block: content follows +the object header, whereas a bytearray uses two memory blocks. It's not +possible to transfer data from bytearray to bytes to implement a O(1) +conversion. + +Solution: support bytes using two memory blocks +----------------------------------------------- + +If the bytes type is modified to also support storing data in a second memory +block, it becomes possible to implement O(1) conversion of bytearray to bytes. +The bytearray would pass its memory block to the bytes object and then "loose +its content" (becomes an empty buffer). + + And more! ========= From 299644becf59ff96786795f94c630f53aada9c1c Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 21 Nov 2018 00:24:44 +0100 Subject: [PATCH 007/136] finalizer API --- doc/bad_api.rst | 25 ++++++++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/doc/bad_api.rst b/doc/bad_api.rst index b6775da..f4ae83f 100644 --- a/doc/bad_api.rst +++ b/doc/bad_api.rst @@ -302,9 +302,28 @@ Weird ``PyObject_CallFunction()`` API: `bpo-28977 PyPy requests ------------- -* Deprecate finalizer API. -* Deprecate Unicode API introduced by the PEP 393, compact strings, like - PyUnicode_4BYTE_DATA(str_obj). +Finalizer API +^^^^^^^^^^^^^ + +Deprecate finalizer API: PyTypeObject.tp_finalize of `PEP 442 +`_. Too specific to the CPython +garbage collector? Destructors (``__del__()``) are not deterministic in PyPy +because of their garbage collector: context manager must be used +(ex: ``with file:``), or resources must be explicitly released +(ex: ``file.close()``). + +Cython uses ``_PyGC_FINALIZED()``, see: + +* https://github.com/cython/cython/issues/2721 +* https://bugs.python.org/issue35081#msg330045 +* `Cython generate_dealloc_function() +`_. + +Compact Unicode API +^^^^^^^^^^^^^^^^^^^ + +Deprecate Unicode API introduced by the PEP 393, compact strings, like +``PyUnicode_4BYTE_DATA(str_obj)``. PyArg_ParseTuple ---------------- From 2d047052650de11ccab9dd76759053318dd494d5 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 22 Nov 2018 13:20:35 +0100 Subject: [PATCH 008/136] +py3c --- doc/index.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/index.rst b/doc/index.rst index 3268b71..dabcb32 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -69,6 +69,10 @@ Links (this documentation can be found in the ``doc/`` subdirectory). * `capi-sig mailing list `_ +* `py3c `_ + (`py3c on GitHub `_): + A Python 2/3 compatibility layer for C extensions. + Table of Contents ================= From b1ff234a1b4cd3234d729b3a28bea1ac184ad062 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 30 Nov 2018 23:32:27 +0100 Subject: [PATCH 009/136] steal ref --- doc/bad_api.rst | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/doc/bad_api.rst b/doc/bad_api.rst index f4ae83f..7fd9198 100644 --- a/doc/bad_api.rst +++ b/doc/bad_api.rst @@ -73,6 +73,8 @@ tuple without increasing the reference counter). CPython contains ``Doc/data/refcounts.dat`` (file is edited manually) which documents how functions handle reference count. +See also :ref:`functions steal references `. + Functions ^^^^^^^^^ @@ -99,7 +101,6 @@ Functions * ``PyMethod_Class()`` * ``PyMethod_Function()`` * ``PyMethod_Self()`` -* ``PyModule_AddObject()``: steal a reference on success, but doesn't on error * ``PyModule_GetDict()`` * ``PyNumber_Check()`` * ``PyObject_Init()`` @@ -279,6 +280,23 @@ Integer overflow See also ``PyLong_AsLongAndOverflow()``. + +.. _steal-ref: + +Functions stealing references +============================= + +* ``PyContext_Exit()``: *ctx* +* ``PyContextVar_Reset()``: *token* +* ``PyErr_Restore()``: *type*, *value*, *traceback* +* ``PySet_Discard()``: *key*, no effect if key not found +* ``PyString_ConcatAndDel()``: *newpart* +* ``Py_DECREF()``: *o* +* ``Py_XDECREF()``: *o*, if *o* is not NULL +* ``PyModule_AddObject()``: *o* on success, no change on error! + +See also :ref:`borrowed references `. + Open questions ============== @@ -317,7 +335,7 @@ Cython uses ``_PyGC_FINALIZED()``, see: * https://github.com/cython/cython/issues/2721 * https://bugs.python.org/issue35081#msg330045 * `Cython generate_dealloc_function() -`_. + `_. Compact Unicode API ^^^^^^^^^^^^^^^^^^^ From f97fed7391305d17236a7da2630f040bbc5eb738 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 17 Dec 2018 14:53:57 +0100 Subject: [PATCH 010/136] PyHandle --- doc/index.rst | 1 + doc/pyhandle.rst | 62 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+) create mode 100644 doc/pyhandle.rst diff --git a/doc/index.rst b/doc/index.rst index dabcb32..2bcf943 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -99,4 +99,5 @@ Table of Contents remove_c_api performance split_include + pyhandle diff --git a/doc/pyhandle.rst b/doc/pyhandle.rst new file mode 100644 index 0000000..80c71b7 --- /dev/null +++ b/doc/pyhandle.rst @@ -0,0 +1,62 @@ +++++++++ +PyHandle +++++++++ + +Email +===== + +Copy of Armin Rigo's email (Nov 2018): + +https://mail.python.org/pipermail/python-dev/2018-November/155806.html + +FWIW, a "handle" is typically something that users of an API store and +pass around, and which can be used to do all operations on some +object. It is whatever a specific implementation needs to describe +references to an object. In the CPython C API, this is ``PyObject*``. +I think that using "handle" for something more abstract is just going +to create confusion. + +Also FWIW, my own 2 cents on the topic of changing the C API: let's +entirely drop ``PyObject *`` and instead use more opaque +handles---like a ``PyHandle`` that is defined as a pointer-sized C +type but is not actually directly a pointer. The main difference this +would make is that the user of the API cannot dereference anything +from the opaque handle, nor directly compare handles with each other +to learn about object identity. They would work exactly like Windows +handles or POSIX file descriptors. These handles would be returned by +C API calls, and would need to be closed when no longer used. Several +different handles may refer to the same object, which stays alive for +at least as long as there are open handles to it. Doing it this way +would untangle the notion of objects from their actual implementation. +In CPython objects would internally use reference counting, a handle +is really just a PyObject pointer in disguise, and closing a handle +decreases the reference counter. In PyPy we'd have a global table of +"open objects", and a handle would be an index in that table; closing +a handle means writing NULL into that table entry. No emulated +reference counting needed: we simply use the existing GC to keep alive +objects that are referenced from one or more table entries. The cost +is limited to a single indirection. + +The C API would change a lot, so it's not reasonable to do that in the +CPython repo. But it could be a third-party project, attempting to +define an API like this and implement it well on top of both CPython +and PyPy. IMHO this might be a better idea than just changing the API +of functions defined long ago to make them more regular (e.g. stop +returning borrowed references); by now this would mostly mean creating +more work for the PyPy team to track and adapt to the changes, with no +real benefits. + +POSIX and Windows API +===================== + +POSIX uses file descriptors, ``int`` type: + +* open() creates a file descriptor +* dup() duplicates a file descriptor +* close() closes a file descriptor + +Windows uses an oquaque ``HANDLE`` type: + +* CreateFile() creates a handle +* DuplicateHandle() duplicates a handle +* CloseHandle() closes a handle From 0d809ded8a661139f5df720e8ca93765d4a0741d Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 18 Dec 2018 23:46:54 +0100 Subject: [PATCH 011/136] Add deprecate.rst --- doc/deprecate.rst | 23 +++++++++++++++++++++++ doc/index.rst | 1 + 2 files changed, 24 insertions(+) create mode 100644 doc/deprecate.rst diff --git a/doc/deprecate.rst b/doc/deprecate.rst new file mode 100644 index 0000000..1410771 --- /dev/null +++ b/doc/deprecate.rst @@ -0,0 +1,23 @@ +++++++++++++++++++ +Deprecate old APIs +++++++++++++++++++ + +CPython is old, the code evolved. Some functions became useless and so should +be removed. But backward compatibility matters in Python, so we need a +transition period with a deprecation process. + +The deprecation can be: + +* Emitted at runtime: ``DeprecationWarning`` +* At the compilation: ``Py_DEPRECATED()`` +* In the documentation + +Functions that should deprecated: + +* Unicode functions using ``Py_UNICODE`` type +* Py_VA_COPY: see `email `_ + +See: + +* https://bugs.python.org/issue19569 +* diff --git a/doc/index.rst b/doc/index.rst index 2bcf943..8c97654 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -100,4 +100,5 @@ Table of Contents performance split_include pyhandle + deprecate From 8cb40cfefefd522f46f7fd8eb89aeb6afa05e629 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 18 Dec 2018 23:49:03 +0100 Subject: [PATCH 012/136] deprecate --- doc/deprecate.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/deprecate.rst b/doc/deprecate.rst index 1410771..541694b 100644 --- a/doc/deprecate.rst +++ b/doc/deprecate.rst @@ -15,7 +15,10 @@ The deprecation can be: Functions that should deprecated: * Unicode functions using ``Py_UNICODE`` type -* Py_VA_COPY: see `email `_ +* ``Py_VA_COPY()``: use directly standard ``va_copy()``, + see `email `_ +* ``Py_MEMCPY()``: `kept for backwards compatibility + `_ See: From 3367cea7234d80cd117c2b3a1b16d7b22554c12a Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 20 Dec 2018 09:15:47 +0100 Subject: [PATCH 013/136] add Python IR --- doc/index.rst | 1 + doc/python_ir.rst | 11 +++++++++++ 2 files changed, 12 insertions(+) create mode 100644 doc/python_ir.rst diff --git a/doc/index.rst b/doc/index.rst index 8c97654..33ab34f 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -101,4 +101,5 @@ Table of Contents split_include pyhandle deprecate + python_ir diff --git a/doc/python_ir.rst b/doc/python_ir.rst new file mode 100644 index 0000000..0a5e981 --- /dev/null +++ b/doc/python_ir.rst @@ -0,0 +1,11 @@ ++++++++++++++++++++++++++++++++++++++++ +Python Intermediate representation (IR) ++++++++++++++++++++++++++++++++++++++++ + +Nathaniel Smith's (njs) IR idea for numpy: + +* video: `Inside NumPy: how it works and how we can make it better + `_ (starting at ~26:00). +* `slides + `__ + From 06931bda5757006992d0157d068a1f4ca9a5b317 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 20 Dec 2018 09:22:23 +0100 Subject: [PATCH 014/136] update index --- doc/index.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/index.rst b/doc/index.rst index 33ab34f..d295850 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -58,6 +58,10 @@ Pages remove_c_api performance split_include + pyhandle + deprecate + python_ir + Links ===== From a3059b985bf945f66b75325f1045b7a07f9a0f16 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 20 Dec 2018 09:35:40 +0100 Subject: [PATCH 015/136] python_ir --- doc/python_ir.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc/python_ir.rst b/doc/python_ir.rst index 0a5e981..70f6ca0 100644 --- a/doc/python_ir.rst +++ b/doc/python_ir.rst @@ -2,6 +2,8 @@ Python Intermediate representation (IR) +++++++++++++++++++++++++++++++++++++++ +Status: no one is working on the implementation, it's an idea. + Nathaniel Smith's (njs) IR idea for numpy: * video: `Inside NumPy: how it works and how we can make it better @@ -9,3 +11,6 @@ Nathaniel Smith's (njs) IR idea for numpy: * `slides `__ +`Python Compilers Workshop `_ +(2016) where the idea was proposed. The `notes from the meeting +`_. From e77a5a78ae11ce592d70d15fe293937454b10c15 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 11 Jan 2019 11:55:06 +0100 Subject: [PATCH 016/136] debug checks --- doc/optimization_ideas.rst | 2 ++ doc/runtimes.rst | 14 ++++++++++++++ 2 files changed, 16 insertions(+) diff --git a/doc/optimization_ideas.rst b/doc/optimization_ideas.rst index 3d516c6..1868ec2 100644 --- a/doc/optimization_ideas.rst +++ b/doc/optimization_ideas.rst @@ -10,6 +10,8 @@ implement new optimizations. See :ref:`Experimental runtime `. +.. _remove-debug-checks: + Remove debug checks =================== diff --git a/doc/runtimes.rst b/doc/runtimes.rst index 440766a..b96c160 100644 --- a/doc/runtimes.rst +++ b/doc/runtimes.rst @@ -67,6 +67,20 @@ Debug runtime: /usr/bin/python3-dbg For example, the debug runtime can check that the GIL is held by the caller. +See also + +* :ref:`Remove debug checks ` +* Rejected idea: `Check index in PyTuple_GET_ITEM/PyTuple_SET_ITEM in debug + mode `__ and `PyTuple_SET_ITEM could + check bounds in debug mode `__. Issues: + + * Serhiy Storchaka: "I think we can break this only after adding public API for accessing + internal storage of a tuple: PyTuple_ITEMS()." + * Stefan Krah: "I'm using &PyTuple_GET_ITEM(args, 0), so Serhiy's concern is + not theoretical." + * Stefan Behnel: "If this is really just about debugging, then I would + suggest to not break existing code at all." + .. _exp-runtime: New experimental runtime: python3-exp From 98e592436dacad9d21e1f59aed8882473afa99d8 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 10 May 2019 23:34:32 +0200 Subject: [PATCH 017/136] Add runtime.rst --- doc/index.rst | 2 ++ doc/runtime.rst | 12 ++++++++++++ 2 files changed, 14 insertions(+) create mode 100644 doc/runtime.rst diff --git a/doc/index.rst b/doc/index.rst index d295850..6d1e4d0 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -61,6 +61,7 @@ Pages pyhandle deprecate python_ir + runtime Links @@ -106,4 +107,5 @@ Table of Contents pyhandle deprecate python_ir + runtime diff --git a/doc/runtime.rst b/doc/runtime.rst new file mode 100644 index 0000000..694f697 --- /dev/null +++ b/doc/runtime.rst @@ -0,0 +1,12 @@ ++++++++++++++++++++++++++++ +Reorganize Python "runtime" ++++++++++++++++++++++++++++ + +Starter point: `PEP 554 -- Multiple Interpreters in the Stdlib +`_. + +Issues +====== + +* https://bugs.python.org/issue36876 +* https://bugs.python.org/issue36877 From 400b750e414a61c401da4159c467dfd70e5fcffc Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 24 May 2019 17:04:03 +0200 Subject: [PATCH 018/136] Runtime --- doc/runtime.rst | 85 +++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 83 insertions(+), 2 deletions(-) diff --git a/doc/runtime.rst b/doc/runtime.rst index 694f697..2e29e2d 100644 --- a/doc/runtime.rst +++ b/doc/runtime.rst @@ -5,8 +5,89 @@ Reorganize Python "runtime" Starter point: `PEP 554 -- Multiple Interpreters in the Stdlib `_. -Issues -====== +Goal +==== +The goal is to support running multiple Python interpreters in parallel with +one lock per interpreter (no more "Global Interpreter Lock", but one +"Interpreter Lock" per interpreter). An interpreter would only be able to run +one Python thread holding the interpreter lock at the same time, but multiple +Python threads which released the interpreter lock (ex: to call a system call +like ``read()``) can be run in parallel. + +What do we need? +================ + +To maximize performances, shared states between interpreters must be minimized. +Each share state must be carefully protected by a lock, which prevent to run +code in parallel. + +Current state of the code (2019-05-24) +====================================== + +During Python 3.7 and 3.8 dev cycle, Eric Snow moved scattered core global +variables into a _PyRuntimeState structure which has a single global and shared +instance: ``_PyRuntime``. + +Most functions access directly to ``_PyRuntime``, directly or indirectly: + +* ``PyThreadState *tstate = _PyThreadState_GET();`` access implicitly + ``_PyRuntime``. +* ``PyThreadState *tstate = _PyRuntimeState_GetThreadState(&_PyRuntime);`` gets + access explicitly ``_PyRuntime``. Get ``runtime->gilstate.tstate_current``. + +``_PyRuntimeState`` fields: + +* ``ceval`` +* ``exitfuncs``, ``nexitfuncs`` +* ``finalizing`` +* ``gc`` +* ``gilstate`` +* ``interpreters`` +* ``main_thread`` +* ``open_code_hook``, ``open_code_userdata``, ``audit_hook_head`` +* ``pre_initialized``, ``core_initialized``, ``initialized`` +* ``preconfig`` +* ``xidregistry`` + + +TODO +==== + +* Move ``_PyRuntimeState.gilstate`` to ``PyInterpreterState``: + + * Remove ``_PyRuntimeState_GetThreadState()`` + * Update ``_PyThreadState_GET()`` + +* Move most ``_PyRuntimeState`` fields into ``PyInterpreterState`` +* Pass the "context" to private C functions: the context can be ``_PyRuntime``, + a field of ``_PyRuntime``, the Python thread state (``tstate``), etc. + +Out of the scope +================ + +* Functions of public C API must not be modified at this stage to add + new "context" parameters. Only the internal C API can be modified. + +Roots +===== + +* Get the current Python thread: + ``_PyRuntimeState_GetThreadState(&_PyRuntime)``. WIP: ``gilstate`` must + move to ``PyInterpreterState`` +* Get the current interpreter: ``tstate->interp``. + +Status (2019-05-24) +=================== + +* ``PyInterpreterState`` moved to the internal C API +* ``_PyRuntimeState`` structure and ``_PyRuntime`` variable created + + +Links +===== + +* https://bugs.python.org/issue36710 * https://bugs.python.org/issue36876 * https://bugs.python.org/issue36877 +* https://mail.python.org/archives/list/capi-sig@python.org/thread/RBLU35OUT2KDFCABK32VNOH4UKSKEUWW/ From 0913b95dfbc492e9bf9b186e15546320e0308bb2 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 24 May 2019 17:05:27 +0200 Subject: [PATCH 019/136] runtime --- doc/runtime.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/runtime.rst b/doc/runtime.rst index 2e29e2d..b1d9026 100644 --- a/doc/runtime.rst +++ b/doc/runtime.rst @@ -91,3 +91,4 @@ Links * https://bugs.python.org/issue36876 * https://bugs.python.org/issue36877 * https://mail.python.org/archives/list/capi-sig@python.org/thread/RBLU35OUT2KDFCABK32VNOH4UKSKEUWW/ +* https://twitter.com/VictorStinner/status/1125887394220269568 From 47f04d453965578ff858186f883098c42acf7474 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 28 May 2019 23:43:55 +0200 Subject: [PATCH 020/136] Add opaque_pyobject.rst --- doc/index.rst | 4 +++- doc/opaque_pyobject.rst | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+), 1 deletion(-) create mode 100644 doc/opaque_pyobject.rst diff --git a/doc/index.rst b/doc/index.rst index 6d1e4d0..38b3279 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -62,6 +62,7 @@ Pages deprecate python_ir runtime + opaque_pyobject Links @@ -101,6 +102,7 @@ Table of Contents cpyext cython cffi + gilectomy remove_c_api performance split_include @@ -108,4 +110,4 @@ Table of Contents deprecate python_ir runtime - + opaque_pyobject diff --git a/doc/opaque_pyobject.rst b/doc/opaque_pyobject.rst new file mode 100644 index 0000000..5459ae6 --- /dev/null +++ b/doc/opaque_pyobject.rst @@ -0,0 +1,34 @@ +========================= +Opaque PyObject structure +========================= + +A blocker issue for many :ref:`optimization ideas ` is that the +``PyObject`` structure fields are exposed in the public C API. Example:: + + PyObject * + PyUnicode_FromObject(PyObject *obj) + { + ... + PyErr_Format(PyExc_TypeError, + "Can't convert '%.100s' object to str implicitly", + Py_TYPE(obj)->tp_name); + return NULL; + } + +with:: + + #define Py_TYPE(ob) (_PyObject_CAST(ob)->ob_type) + #define _PyObject_CAST(op) ((PyObject*)(op)) + +The issue is that ``obj->ob_type`` is accessed directly. It prevents to +implement :ref:`Tagged pointers ` for example. + +By the way, ``Py_TYPE()`` returns a :ref:`borrowed reference ` +which is another kind of problem. + +In the long term, ``PyObject`` structure should be opaque. Accessing +``ob_refcnt`` and ``ob_type`` fields should always go through functions. + +XXX which functions? + +XXX how to convert old code to these new functions? From 13ca0561f7728530888e41373762f6d49d9e1c2a Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 28 May 2019 23:46:23 +0200 Subject: [PATCH 021/136] Py_TYPE --- doc/bad_api.rst | 14 +++++++++++--- doc/opaque_pyobject.rst | 4 +++- 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/doc/bad_api.rst b/doc/bad_api.rst index 7fd9198..90b460a 100644 --- a/doc/bad_api.rst +++ b/doc/bad_api.rst @@ -149,9 +149,17 @@ Hugh Fisher summarized: and incur a performance hit, to eliminate a rare bug from badly written code. -See the discussion on capi-sig: `Open questions about borrowed reference. -`_ -(Sept 2018). +Discussions: + +* `[Python-Dev] bpo-34595: How to format a type name? + `_ + (Sept 2018) +* capi-sig: `Open questions about borrowed reference. + `_ + (Sept 2018). + + +See also :ref:`Opaque PyObject structure `. Duplicated functions diff --git a/doc/opaque_pyobject.rst b/doc/opaque_pyobject.rst index 5459ae6..2c003f5 100644 --- a/doc/opaque_pyobject.rst +++ b/doc/opaque_pyobject.rst @@ -1,3 +1,5 @@ +.. _opaque-pyobject: + ========================= Opaque PyObject structure ========================= @@ -24,7 +26,7 @@ The issue is that ``obj->ob_type`` is accessed directly. It prevents to implement :ref:`Tagged pointers ` for example. By the way, ``Py_TYPE()`` returns a :ref:`borrowed reference ` -which is another kind of problem. +which is another kind of problem. See :ref:`Py_TYPE() corner case `. In the long term, ``PyObject`` structure should be opaque. Accessing ``ob_refcnt`` and ``ob_type`` fields should always go through functions. From 603f42c99247f476cee2cee1d888fa7fbf3be18c Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 13 Jun 2019 18:18:31 +0200 Subject: [PATCH 022/136] update --- doc/roadmap.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/doc/roadmap.rst b/doc/roadmap.rst index 9ee8a38..43a4d40 100644 --- a/doc/roadmap.rst +++ b/doc/roadmap.rst @@ -30,6 +30,12 @@ Open questions Status ====== +* 2019-05-01: `Status of the stable API and ABI in Python 3.8 + `_, + slides of Victor Stinner's lightning talk at the Language Summit (during + Pycon US 2019) +* 2019-02-22: `[capi-sig] Update on CPython header files reorganization + `_ * 2018-07-29: `pythoncapi project `_ created on GitHub * 2017-12-21: It's an idea. There is an old PEP draft, but no implementation, From fed02b248ff7999c2b01fd86b4f766508918ddf6 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 19 Jun 2019 00:26:11 +0200 Subject: [PATCH 023/136] includes: issues --- doc/split_include.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/split_include.rst b/doc/split_include.rst index 2b5efe4..1603772 100644 --- a/doc/split_include.rst +++ b/doc/split_include.rst @@ -14,3 +14,6 @@ levels: In the past, many functions have been added to the wrong API level, just because everything is at the same place. To prevent such mistakes, headers files should be reorganized with clearly separated files. + +* https://bugs.python.org/issue35134 +* https://bugs.python.org/issue35081 From 19f80cefeb0c5376b7c42a68b181bd0e10b7d8b1 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 7 Feb 2020 09:58:48 +0100 Subject: [PATCH 024/136] PyImport functions moved to the internal C API --- doc/bad_api.rst | 9 --------- 1 file changed, 9 deletions(-) diff --git a/doc/bad_api.rst b/doc/bad_api.rst index 90b460a..73544ea 100644 --- a/doc/bad_api.rst +++ b/doc/bad_api.rst @@ -366,15 +366,6 @@ Is it an issue? Should we do something? For internal use only ===================== -The C API documentation contains a few functions with the note "For internal -use only". Examples: - -* ``_PyImport_Init()`` -* ``PyImport_Cleanup()`` -* ``_PyImport_Fini()`` - -Why ``PyImport_Cleanup()`` is still a public method? - Public but not documented and not part of Python.h: * ``PyFrame_FastToLocalsWithError()`` From b73f79feff4d7837d9b6eb99dff23814a0da7c3c Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 7 Mar 2020 01:43:45 +0100 Subject: [PATCH 025/136] PEP draft --- doc/roadmap.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/roadmap.rst b/doc/roadmap.rst index 43a4d40..2173961 100644 --- a/doc/roadmap.rst +++ b/doc/roadmap.rst @@ -40,6 +40,9 @@ Status created on GitHub * 2017-12-21: It's an idea. There is an old PEP draft, but no implementation, the PEP has no number and was not accepted yet (nor really proposed). +* 2017-07-11: + `[Python-ideas] PEP: Hide implementation details in the C API + `_ Players ======= From 740613e85e0380048918a235c4ea81e6df278942 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 22 Jun 2020 12:25:20 +0200 Subject: [PATCH 026/136] roadmap --- doc/roadmap.rst | 41 ++++++++++++++++++++--------------------- 1 file changed, 20 insertions(+), 21 deletions(-) diff --git a/doc/roadmap.rst b/doc/roadmap.rst index 2173961..47a190f 100644 --- a/doc/roadmap.rst +++ b/doc/roadmap.rst @@ -30,6 +30,9 @@ Open questions Status ====== +* 2020-04-10: `PEP: Modify the C API to hide implementation details + `_ + sent to python-dev. * 2019-05-01: `Status of the stable API and ABI in Python 3.8 `_, slides of Victor Stinner's lightning talk at the Language Summit (during @@ -38,11 +41,28 @@ Status `_ * 2018-07-29: `pythoncapi project `_ created on GitHub +* 2018-06: capi-sig mailing list migrated to Mailman 3 * 2017-12-21: It's an idea. There is an old PEP draft, but no implementation, the PEP has no number and was not accepted yet (nor really proposed). +* 2017-11: Idea proposed on python-dev, `[Python-Dev] Make the stable API-ABI + usable + `_ +* 2017-09: Blog post: `A New C API for CPython + `_ +* 2017-09: Idea discussed at the CPython sprint at Instagram (California). + Liked by all core developers. The expected performance slowdown is likely to + be accepted. * 2017-07-11: `[Python-ideas] PEP: Hide implementation details in the C API `_ +* 2017-07: Idea proposed on python-ideas. `[Python-ideas] PEP: Hide + implementation details in the C API + `_ +* 2017-05: Idea proposed at the Python Language Summit, during PyCon US 2017. + My `"Python performance" slides (PDF) + `_. + LWN article: `Keeping Python competitive + `_. Players ======= @@ -60,24 +80,3 @@ Unknown status: * Pyjion * Pyston * any other? - -Timeline -======== - -* 2018-06: capi-sig mailing list migrated to Mailman 3 -* 2017-11: Idea proposed on python-dev, `[Python-Dev] Make the stable API-ABI - usable - `_ -* 2017-09: Blog post: `A New C API for CPython - `_ -* 2017-09: Idea discussed at the CPython sprint at Instagram (California). - Liked by all core developers. The expected performance slowdown is likely to - be accepted. -* 2017-07: Idea proposed on python-ideas. `[Python-ideas] PEP: Hide - implementation details in the C API - `_ -* 2017-05: Idea proposed at the Python Language Summit, during PyCon US 2017. - My `"Python performance" slides (PDF) - `_. - LWN article: `Keeping Python competitive - `_. From 3630779809ffac9613d62ee04b64f53b86be2eee Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 9 Jul 2020 00:30:48 +0200 Subject: [PATCH 027/136] tagged pointer --- doc/optimization_ideas.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/optimization_ideas.rst b/doc/optimization_ideas.rst index 1868ec2..eaf3744 100644 --- a/doc/optimization_ideas.rst +++ b/doc/optimization_ideas.rst @@ -89,6 +89,9 @@ Note: ARM64 was recently extended its address space to 48 bits, causing issue in LuaJIT: `47 bit address space restriction on ARM64 `_. +Neil Schemenauer PoC (Sept 2018): +https://mail.python.org/archives/list/capi-sig@python.org/thread/EGAY55ZWMF2WSEMP7VAZSFZCZ4VARU7L/#EGAY55ZWMF2WSEMP7VAZSFZCZ4VARU7L + Copy-on-Write (CoW): doable =========================== From 81e7933702386a78e2fccbb9cf64b518f1969e07 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 5 Nov 2020 14:52:53 +0100 Subject: [PATCH 028/136] Stable ABI: add links --- doc/stable_abi.rst | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/doc/stable_abi.rst b/doc/stable_abi.rst index bbb391d..cc74391 100644 --- a/doc/stable_abi.rst +++ b/doc/stable_abi.rst @@ -4,6 +4,49 @@ Python stable ABI? ++++++++++++++++++ +Links +===== + +* `Petr Viktorin's Python Stable ABI improvement + `_ notes +* `PEP 489 -- Multi-phase extension module initialization + `_ + + * `bpo-1635741: Py_Finalize() doesn't clear all Python objects at exit + `_ + (convert extensions to multi-phase init PEP 489) + +* `PEP 620 -- Hide implementation details from the C API + `_ (Victor Stinner) + + * Move the default C API towards the limited C API + * Make structures opaque: + + * PyObject: https://bugs.python.org/issue39573 + * PyTypeObject: https://bugs.python.org/issue40170 + * PyFrameObject: https://bugs.python.org/issue40421 + * PyThreadState: https://bugs.python.org/issue39947 + * PyInterpreterState: DONE in Python 3.8! + * PyGC_Head: DONE in Python 3.9! + + * `bpo-40989: Remove _Py_NewReference() and _Py_ForgetReference() from the + public C API `_ + * `bpo-41078: Convert PyTuple_GET_ITEM() macro to a static inline function + `_ + * `bpo-40601: Hide static types from the limited C API + `_ + +* `PEP 630 -- Isolating Extension Modules + `_ (Petr Viktorin) + + * `bpo-40077: Convert static types to heap types: use PyType_FromSpec() + `_ + +* `bpo-41111: Convert a few stdlib extensions to the limited C API + `_ +* `HPy project `_ + + .. _from-api-to-api: Relationship between the C API and the ABI From 118c98fcb015377eea8d631f9bc65a490e9799b3 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 5 Nov 2020 14:53:56 +0100 Subject: [PATCH 029/136] Change theme --- doc/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/conf.py b/doc/conf.py index e2b28f6..ae6cf52 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -83,7 +83,7 @@ # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # -html_theme = 'alabaster' +html_theme = 'default' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the From d56ed3d781e55630ae1f5c5681150b52e9d8f971 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 5 Nov 2020 14:59:35 +0100 Subject: [PATCH 030/136] PEP 573 --- doc/stable_abi.rst | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/doc/stable_abi.rst b/doc/stable_abi.rst index cc74391..01b7bab 100644 --- a/doc/stable_abi.rst +++ b/doc/stable_abi.rst @@ -36,6 +36,18 @@ Links * `bpo-40601: Hide static types from the limited C API `_ +* `PEP 573 -- Module State Access from C Extension Methods + `_ + + * Python 3.10 has a new ``_PyType_GetModuleByDef()`` function + * Python 3.9 added: + + * New PyType_FromModuleAndSpec() + * New PyType_GetModuleState() + * New METH_METHOD calling convention flag + * New PyCMethod function signature + * New ``defining_class`` type in Argument Clinic + * `PEP 630 -- Isolating Extension Modules `_ (Petr Viktorin) From 6e8f8de9ac7f52796f120af1fd74ea6747772501 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 8 Dec 2020 11:15:58 +0100 Subject: [PATCH 031/136] Optimize Python --- doc/index.rst | 1 + doc/optimize_python.rst | 143 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 144 insertions(+) create mode 100644 doc/optimize_python.rst diff --git a/doc/index.rst b/doc/index.rst index 38b3279..5d3fefe 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -38,6 +38,7 @@ Pages .. toctree:: :maxdepth: 1 + optimize_python rationale roadmap bad_api diff --git a/doc/optimize_python.rst b/doc/optimize_python.rst new file mode 100644 index 0000000..96a7d4b --- /dev/null +++ b/doc/optimize_python.rst @@ -0,0 +1,143 @@ ++++++++++++++++++++++++++++++++++++++++ +Fix the Python C API to optimize Python ++++++++++++++++++++++++++++++++++++++++ + +CPython cannot be optimized and other Python implementations see their +performances limited by the C API. The relationship between the C API and +performance is not obvious, and even can be counterintuitive. + +Optimizations +============= + +Faster object allocation +------------------------ + +CPython requires to allocate all Python objects on the heap memory and objects +cannot be moved during their life cycle. + +It would be more efficient to allow to allocate temporary objects on the stack, +implement nurseries of young objects and compact memory to remove "holes" when +many objects are deallocated. + +Faster and incremental garbage collection +----------------------------------------- + +CPython relies on reference counting to collect garbage. Reference counting +does not scale for parallelism with multithreading. + +A tracing and moving garbage collector would be more efficient. The garbage +collection could be done in multiple steps in separated thread rather than +having long delays causing by the CPython blocking stop-the-world gabage +collector. + +The ability to deference pointers like ``PyObject*`` make the implementation +of a moving gabarge collector more complicated. Only using handles would make +the implementation simpler. + +Run Python threads in parallel +------------------------------ + +CPython uses a GIL for objects consistency and to ease the implementation +of the C API. The GIL has many convenient advantages to simplify the +implementation. But it basically limits CPython to a single thread to run +CPU-bound workload distributed in multiple threads. + +Per-object locks would allow to help to scale threads on multiple threads. + +More efficient data structures (boxing/unboxing) +------------------------------------------------ + +CPython requires builtin types like list and dict to only contain +``PyObject*``. + +PyPy implements a list strategy for integers: integers are stored directly as +integers, not as objects. Integers are only boxed on demand. + + +Reasons why the C API prevents to optimize Python +================================================= + +Structures are part of the public C API (make them opaque) +---------------------------------------------------------- + +Core C structures like ``PyObject`` are part of the public C API and so every +Python implementations must implement exactly this structure. + +The C API directly or indirectly access structure members. For example, the +``Py_INCREF()`` function modifies directly ``PyObject.ob_refcnt`` and so makes +the assumption that objects have a reference counter. Another example is +``PyTuple_GET_ITEM()`` which reads directly the ``PyTupleObject.ob_item`` +member and so requires ``PyTupleObject`` to only store ``PyObject*`` objects. + + +The C API should be modified to abstract accesses to objects through function +calls rather than using macros which access directly to structure members. + +Structures must be excluded from the public C API: become "opaque". + +PyObject* type can be dereferenced (use handles) +------------------------------------------------ + +Since structures a public, it is possible to deference pointers to access +structure members. For example, access directly to ``PyObject.ob_type`` member +from a ``PyObject*`` pointer, or access directly to +``PyTupleObject.ob_type[index]`` from a ``PyTupleObject*`` pointer. + +Using opaque **handles** like HPy what does would prevent that. + +Borrowed references (avoid them) +-------------------------------- + +Many C API functions like ``PyDict_GetItem()`` or ``PyTuple_GetItem()`` return +a borrowed references. They make the assumption that all objects are actual +objects. For example, if tagged pointers are implemented, a ``PyObject*`` does +not point to a concrete object: the value must be boxed to get a ``PyObject*``. +The problem with borrowed references is to decide when it is safe to destroy +the temporary ``PyObject*`` object. One heuristic is to consider that it must +remain alive as long as its container (ex: a list) remains alive. + +PyObject must be allocated on the stack +--------------------------------------- + +In CPython, all objects must be allocated on the stack. Using reference +counting, when an object is passed to a function, the function can store it in +another container and so the object remains alive after the function completes. +The caller cannot destroy the object, since it does not take care of the object +lifecycle. The object can only be destroyed when the last strong reference to +the object is deleted. + +Pseudo-code:: + + void func(void) + { + PyObject *x = PyLong_FromLong(1); + func(x); + Py_DECREF(x); + // if func() creates a new strong reference to x, + // x is still alive at this point. + } + +HPy uses a different strategy: if a function wants to create a new reference to +a handle, ``HPy_Dup()`` function must be called. ``HPy_Dup()`` can continue to +use the same object, but it can also duplicate an immutable object. + +PyObject cannot be moved in memory +---------------------------------- + +Since ``PyObject*`` is a direct memory address to a ``PyObject``, moving +a ``PyObject`` requires to change all ``PyObject*`` values pointing to it. +Using handles, there is not such issue. + +Other C API functions give a direct memory address into an object content +with no API to "release" the resource. For example, ``PyBytes_AsString()`` +gives a direct access into the bytes string, there is no way for the object +to know when the caller no longer needs this pointer. The string cannot be +moved in memory. + +Functions using ``PyObject**`` type (array of ``PyObject*`` pointers) have a +similar issue. Example: ``&PyTuple_GET_ITEM()`` is used to get +``&PyTupleObject.ob_item``. + +The ``PyObject_GetBuffer()`` is a sane API: it requires the caller to call +``PuBuffer_Release()`` to release the ``Py_buffer`` object. Memory can be +copied if needed to allow to move the object while the buffer is used. From b6f89ba51fe875787cb0cc17f40572a2b379f8ed Mon Sep 17 00:00:00 2001 From: "Paul m. p. P" Date: Tue, 8 Dec 2020 17:11:28 +0100 Subject: [PATCH 032/136] minor fixes --- doc/optimize_python.rst | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/doc/optimize_python.rst b/doc/optimize_python.rst index 96a7d4b..6ea4267 100644 --- a/doc/optimize_python.rst +++ b/doc/optimize_python.rst @@ -27,11 +27,11 @@ does not scale for parallelism with multithreading. A tracing and moving garbage collector would be more efficient. The garbage collection could be done in multiple steps in separated thread rather than -having long delays causing by the CPython blocking stop-the-world gabage +having long delays causing by the CPython blocking stop-the-world garbage collector. -The ability to deference pointers like ``PyObject*`` make the implementation -of a moving gabarge collector more complicated. Only using handles would make +The ability to deference pointers like ``PyObject*`` makes the implementation +of a moving garbarge collector more complicated. Only using handles would make the implementation simpler. Run Python threads in parallel @@ -42,7 +42,7 @@ of the C API. The GIL has many convenient advantages to simplify the implementation. But it basically limits CPython to a single thread to run CPU-bound workload distributed in multiple threads. -Per-object locks would allow to help to scale threads on multiple threads. +Per-object locks would allow to help scaling threads on multiple CPU. More efficient data structures (boxing/unboxing) ------------------------------------------------ @@ -71,28 +71,28 @@ member and so requires ``PyTupleObject`` to only store ``PyObject*`` objects. The C API should be modified to abstract accesses to objects through function -calls rather than using macros which access directly to structure members. +calls rather than using macros which access directly structure members. Structures must be excluded from the public C API: become "opaque". PyObject* type can be dereferenced (use handles) ------------------------------------------------ -Since structures a public, it is possible to deference pointers to access +Since structures are public, it is possible to deference pointers to access structure members. For example, access directly to ``PyObject.ob_type`` member from a ``PyObject*`` pointer, or access directly to ``PyTupleObject.ob_type[index]`` from a ``PyTupleObject*`` pointer. -Using opaque **handles** like HPy what does would prevent that. +Using opaque **handles** like HPy does would prevent that. Borrowed references (avoid them) -------------------------------- -Many C API functions like ``PyDict_GetItem()`` or ``PyTuple_GetItem()`` return -a borrowed references. They make the assumption that all objects are actual +Many C API functions like ``PyDict_GetItem()`` or ``PyTuple_GetItem()`` return +a borrowed reference. They make the assumption that all objects are actual objects. For example, if tagged pointers are implemented, a ``PyObject*`` does not point to a concrete object: the value must be boxed to get a ``PyObject*``. -The problem with borrowed references is to decide when it is safe to destroy +The problem with borrowed reference is to decide when it is safe to destroy the temporary ``PyObject*`` object. One heuristic is to consider that it must remain alive as long as its container (ex: a list) remains alive. @@ -140,4 +140,4 @@ similar issue. Example: ``&PyTuple_GET_ITEM()`` is used to get The ``PyObject_GetBuffer()`` is a sane API: it requires the caller to call ``PuBuffer_Release()`` to release the ``Py_buffer`` object. Memory can be -copied if needed to allow to move the object while the buffer is used. +copied if needed to allow moving the object while the buffer is used. From 077b6ba8f8ee5b6271d1867736d05704100b81a4 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 25 Mar 2021 17:55:22 +0100 Subject: [PATCH 033/136] Add stats --- doc/index.rst | 2 ++ doc/stats.rst | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) create mode 100644 doc/stats.rst diff --git a/doc/index.rst b/doc/index.rst index 5d3fefe..a6fbfa8 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -64,6 +64,7 @@ Pages python_ir runtime opaque_pyobject + stats Links @@ -112,3 +113,4 @@ Table of Contents python_ir runtime opaque_pyobject + stats diff --git a/doc/stats.rst b/doc/stats.rst new file mode 100644 index 0000000..8f43914 --- /dev/null +++ b/doc/stats.rst @@ -0,0 +1,52 @@ +============================== +Statistics on the Python C API +============================== + +Line numbers +============ + +Number of C API line numbers per Python version: + +======= ============= =========== ============ ======= +Version Public CPython Internal Total +======= ============= =========== ============ ======= +2.7 12686 (100%) 0 0 12686 +3.6 16011 (100%) 0 0 16011 +3.7 16517 (96%) 0 705 (4%) 17222 +3.8 13160 (70%) 3417 (18%) 2230 (12%) 18807 +3.9 12264 (62%) 4343 (22%) 3066 (16%) 19673 +3.10 10305 (52%) 4513 (23%) 5092 (26%) 19910 +======= ============= =========== ============ ======= + +Comands: + +* public: ``wc -l Include/*.h`` +* cpython: ``wc -l Include/cpython/*.h`` +* internal: ``wc -l Include/internal/*.h`` + +Symbols +======= + +Symbols exported with PyAPI_FUNC() and PyAPI_DATA(): + +======= =========== +Version Symbols +======= =========== +2.7 1098 +3.6 1460 +3.7 1547 (+87) +3.8 1561 (+14) +3.9 1552 (-9) +3.10 1495 (-57) +======= =========== + +Command:: + + grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|wc -l + +Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid +exporting symbols which are not **explicitly** exported. + +The ``make smelly`` command checks for public symbols of libpython and C +extension which are prefixed by ``Py`` or ``_Py``. See +``Tools/scripts/smelly.py`` script. From b012e62214b51ac7ef0d0638f5820ebd7c8df033 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 26 Mar 2021 12:03:16 +0100 Subject: [PATCH 034/136] roadmap --- doc/roadmap.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/roadmap.rst b/doc/roadmap.rst index 47a190f..fec302c 100644 --- a/doc/roadmap.rst +++ b/doc/roadmap.rst @@ -39,6 +39,9 @@ Status Pycon US 2019) * 2019-02-22: `[capi-sig] Update on CPython header files reorganization `_ +* 2018-09-04: Creation of CPython fork to experiment a new incompatible C + API excluding borrowed references and not access directly structure + members. * 2018-07-29: `pythoncapi project `_ created on GitHub * 2018-06: capi-sig mailing list migrated to Mailman 3 From 707f25a187bfa377f0f35f757cbfd6b00497af7c Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 2 Apr 2021 12:04:09 +0200 Subject: [PATCH 035/136] stats --- doc/stats.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 8f43914..2f4a9d1 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -8,7 +8,7 @@ Line numbers Number of C API line numbers per Python version: ======= ============= =========== ============ ======= -Version Public CPython Internal Total +Python Public CPython Internal Total ======= ============= =========== ============ ======= 2.7 12686 (100%) 0 0 12686 3.6 16011 (100%) 0 0 16011 @@ -30,7 +30,7 @@ Symbols Symbols exported with PyAPI_FUNC() and PyAPI_DATA(): ======= =========== -Version Symbols +Python Symbols ======= =========== 2.7 1098 3.6 1460 From 6175a76a6a3214353369181a12dab6223ebc3a92 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 14 Apr 2021 15:06:14 +0200 Subject: [PATCH 036/136] test Python next --- doc/remove_c_api.rst | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/doc/remove_c_api.rst b/doc/remove_c_api.rst index 2524130..49c4bc9 100644 --- a/doc/remove_c_api.rst +++ b/doc/remove_c_api.rst @@ -4,6 +4,11 @@ Remove the C API ++++++++++++++++ +See `Test the next Python +`_ to estimate how many +C extension modules would be broken by a C API change, like removing a +function. + One proposed alternative to a new better C API is no C API at all. The reasoning is that since existing solutions are already available, complete and reliable: @@ -14,9 +19,9 @@ We do not need to have one for python itself. However, this approach has lots of open questions without satisfying answers: -What about the long tail of C extensions on PyPI which still use the C extension? +What about the long tail of C extensions on PyPI which still use the C extension? Would it mean a new Python without all these C extensions on PyPI? -Moreover, lots of project do not use those solution, and the C API is part of Python success. For example, there would be no numpy without the C API, and one can look at :ref:`Consumers of the Python C API ` to see others examples. +Moreover, lots of project do not use those solution, and the C API is part of Python success. For example, there would be no numpy without the C API, and one can look at :ref:`Consumers of the Python C API ` to see others examples. Removing it would negatively impact those projects, so this doesn't sound like a workable solution. From 7edca7273e0775e0d3ac0bcec3d1de3b0941c416 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 28 Sep 2021 01:15:33 +0200 Subject: [PATCH 037/136] stats --- doc/stats.rst | 80 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 69 insertions(+), 11 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 2f4a9d1..d6bf983 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -29,24 +29,82 @@ Symbols Symbols exported with PyAPI_FUNC() and PyAPI_DATA(): -======= =========== -Python Symbols -======= =========== -2.7 1098 -3.6 1460 -3.7 1547 (+87) -3.8 1561 (+14) -3.9 1552 (-9) -3.10 1495 (-57) -======= =========== +======= ============== =============== =========== +Python Public Private Total +======= ============== =============== =========== +2.7 891 207 1098 +3.6 1041 (+150) 419 (+212) 1460 (+362) +3.7 1068 (+27) 479 (+60) 1547 (+87) +3.8 1105 (+37) 456 (-23) 1561 (+14) +3.9 1115 (+10) 437 (-19) 1552 (-9) +3.10 1080 (-35) 435 (-2) 1515 (-37) +3.11 1062 (-18) 437 (+2) 1499 (-16) +======= ============== =============== =========== -Command:: +Command (total):: grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|wc -l +Command (private):: + + grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|grep ' _Py'|wc -l + Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid exporting symbols which are not **explicitly** exported. The ``make smelly`` command checks for public symbols of libpython and C extension which are prefixed by ``Py`` or ``_Py``. See ``Tools/scripts/smelly.py`` script. + +Functions defined as macros and static inline functions +======================================================= + +Functions defined as macros (only public) and static inline functions (public +or private): + +====== ===== ============= ===== +Python Macro Static inline Total +====== ===== ============= ===== +2.7 396 0 396 +3.6 394 0 394 +3.7 403 0 403 +3.8 399 14 413 +3.9 406 30 436 +3.10 412 36 448 +3.11 412 40 452 +====== ===== ============= ===== + +If a function is defined as a private static inline function and exposed as a +public macro, it is counted twice in this table. For example, the public +Py_INCREF() macro and the private _Py_INCREF() static inline functions are +counted as 2 functions, whereas only the "Py_INCREF" name is public. + +Commands:: + + grep -E 'define P[Yy][A-Za-z_]+ *\(' Include/*.h Include/cpython/*.h|wc -l + grep 'static inline ' Include/*.h Include/cpython/*.h|grep -v pydtrace|grep -v 'define Py_LOCAL_INLINE'|wc -l + + +Structures +========== + +Public structures in the Python C API: + +====== ========== +Python Structures +====== ========== +2.7 97 +3.6 124 (+27) +3.7 137 (+13) +3.8 127 (-10) +3.9 125 (-2) +3.10 104 (-21) +3.11 104 (same) +====== ========== + +These numbers exclude opaque structures like PyInterpreterState (since Python +3.8). + +Command:: + + grep 'typedef struct' Include/*.h Include/cpython/*.h|wc -l From d356a6c92204da39bf6b61adcebcd32443532caa Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 28 Sep 2021 14:44:41 +0200 Subject: [PATCH 038/136] stats --- doc/stats.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index d6bf983..ab139b7 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,11 +18,11 @@ Python Public CPython Internal Total 3.10 10305 (52%) 4513 (23%) 5092 (26%) 19910 ======= ============= =========== ============ ======= -Comands: +Commands: -* public: ``wc -l Include/*.h`` -* cpython: ``wc -l Include/cpython/*.h`` -* internal: ``wc -l Include/internal/*.h`` +* Public: ``wc -l Include/*.h`` +* CPython: ``wc -l Include/cpython/*.h`` +* Internal: ``wc -l Include/internal/*.h`` Symbols ======= @@ -54,7 +54,7 @@ exporting symbols which are not **explicitly** exported. The ``make smelly`` command checks for public symbols of libpython and C extension which are prefixed by ``Py`` or ``_Py``. See -``Tools/scripts/smelly.py`` script. +the ``Tools/scripts/smelly.py`` script. Functions defined as macros and static inline functions ======================================================= From 58e405a4ed6225f3341aba8acbf525f454a5ebe7 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 28 Sep 2021 14:48:37 +0200 Subject: [PATCH 039/136] stats --- doc/stats.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/doc/stats.rst b/doc/stats.rst index ab139b7..e166d13 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -103,7 +103,12 @@ Python Structures ====== ========== These numbers exclude opaque structures like PyInterpreterState (since Python -3.8). +3.8). The grep command is not exact. For example, PyODictObject is seen as +public, whereas the structure is opaque:: + + typedef struct _odictobject PyODictObject; + +The _odictobject structure is only defined in Objects/odictobject.c. Command:: From 2181f9c96434cf0878ce1b6c0fded9caedb7a329 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 4 Oct 2021 17:54:14 +0200 Subject: [PATCH 040/136] stats --- doc/stats.rst | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index e166d13..7cf7746 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -7,20 +7,20 @@ Line numbers Number of C API line numbers per Python version: -======= ============= =========== ============ ======= -Python Public CPython Internal Total -======= ============= =========== ============ ======= -2.7 12686 (100%) 0 0 12686 -3.6 16011 (100%) 0 0 16011 -3.7 16517 (96%) 0 705 (4%) 17222 -3.8 13160 (70%) 3417 (18%) 2230 (12%) 18807 -3.9 12264 (62%) 4343 (22%) 3066 (16%) 19673 -3.10 10305 (52%) 4513 (23%) 5092 (26%) 19910 -======= ============= =========== ============ ======= +======= ============== =========== ============ ======= +Python Limited API CPython API Internal API Total +======= ============== =========== ============ ======= +2.7 12,686 (100%) 0 0 12,686 +3.6 16,011 (100%) 0 0 16,011 +3.7 16,517 (96%) 0 705 (4%) 17,222 +3.8 13,160 (70%) 3,417 (18%) 2,230 (12%) 18,807 +3.9 12,264 (62%) 4,343 (22%) 3,066 (16%) 19,673 +3.10 10,305 (52%) 4,513 (23%) 5,092 (26%) 19,910 +======= ============== =========== ============ ======= Commands: -* Public: ``wc -l Include/*.h`` +* Limited: ``wc -l Include/*.h`` * CPython: ``wc -l Include/cpython/*.h`` * Internal: ``wc -l Include/internal/*.h`` @@ -47,7 +47,7 @@ Command (total):: Command (private):: - grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|grep ' _Py'|wc -l + grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|grep -v ' _Py'|wc -l Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid exporting symbols which are not **explicitly** exported. From 50c2a847e67dcba521787a222b44dac00ec7e3ca Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 12 Oct 2021 12:11:50 +0200 Subject: [PATCH 041/136] Update 3.10 stats --- doc/stats.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 7cf7746..bbae370 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -15,7 +15,7 @@ Python Limited API CPython API Internal API Total 3.7 16,517 (96%) 0 705 (4%) 17,222 3.8 13,160 (70%) 3,417 (18%) 2,230 (12%) 18,807 3.9 12,264 (62%) 4,343 (22%) 3,066 (16%) 19,673 -3.10 10,305 (52%) 4,513 (23%) 5,092 (26%) 19,910 +3.10 10,395 (51%) 4,584 (22%) 5,520 (27%) 20,499 ======= ============== =========== ============ ======= Commands: @@ -37,8 +37,8 @@ Python Public Private Total 3.7 1068 (+27) 479 (+60) 1547 (+87) 3.8 1105 (+37) 456 (-23) 1561 (+14) 3.9 1115 (+10) 437 (-19) 1552 (-9) -3.10 1080 (-35) 435 (-2) 1515 (-37) -3.11 1062 (-18) 437 (+2) 1499 (-16) +3.10 1080 (-35) 436 (-1) 1516 (-36) +3.11 1062 (-18) 437 (+1) 1499 (-17) ======= ============== =============== =========== Command (total):: From d0b7b3401bc9593f42e1d4de0de60222f295d4f0 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 13 Oct 2021 21:54:34 +0200 Subject: [PATCH 042/136] stats: files --- doc/stats.rst | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/doc/stats.rst b/doc/stats.rst index bbae370..2673c0b 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -24,6 +24,29 @@ Commands: * CPython: ``wc -l Include/cpython/*.h`` * Internal: ``wc -l Include/internal/*.h`` +Header file numbers +=================== + +Number of header file numbers per Python version: + +======= ============== =========== ============ ========== +Python Limited API CPython API Internal API Total +======= ============== =========== ============ ========== +2.7 91 0 0 91 +3.6 99 (+8) 0 0 99 (+8) +3.7 99 (same) 0 11 (+11) 110 (+11) +3.8 97 (-2) 15 (+15) 21 (+10) 133 (+23) +3.9 98 (+1) 24 (+9) 33 (+12) 155 (+22) +3.10 81 (-17) 32 (+8) 48 (+15) 161 (+6) +======= ============== =========== ============ ========== + +Commands: + +* Limited: ``ls Include/*.h|wc -l`` +* CPython: ``ls Include/cpython/*.h|wc -l`` +* Internal: ``ls Include/internal/*.h|wc -l`` + + Symbols ======= From 801b1397a08ea547b0762524d6d1fab639f19894 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 7 Apr 2022 00:24:03 +0200 Subject: [PATCH 043/136] Update 3.11 stats --- doc/stats.rst | 38 ++++++++++++++++++++++++-------------- 1 file changed, 24 insertions(+), 14 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 2673c0b..cb65347 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -16,6 +16,7 @@ Python Limited API CPython API Internal API Total 3.8 13,160 (70%) 3,417 (18%) 2,230 (12%) 18,807 3.9 12,264 (62%) 4,343 (22%) 3,066 (16%) 19,673 3.10 10,395 (51%) 4,584 (22%) 5,520 (27%) 20,499 +3.11 9,620 (41%) 4,865 (21%) 9,227 (38%) 23,712 ======= ============== =========== ============ ======= Commands: @@ -38,6 +39,7 @@ Python Limited API CPython API Internal API Total 3.8 97 (-2) 15 (+15) 21 (+10) 133 (+23) 3.9 98 (+1) 24 (+9) 33 (+12) 155 (+22) 3.10 81 (-17) 32 (+8) 48 (+15) 161 (+6) +3.11 73 (-8) 42 (+10) 66 (+18) 181 (+20) ======= ============== =========== ============ ========== Commands: @@ -52,26 +54,34 @@ Symbols Symbols exported with PyAPI_FUNC() and PyAPI_DATA(): -======= ============== =============== =========== -Python Public Private Total -======= ============== =============== =========== -2.7 891 207 1098 -3.6 1041 (+150) 419 (+212) 1460 (+362) -3.7 1068 (+27) 479 (+60) 1547 (+87) -3.8 1105 (+37) 456 (-23) 1561 (+14) -3.9 1115 (+10) 437 (-19) 1552 (-9) -3.10 1080 (-35) 436 (-1) 1516 (-36) -3.11 1062 (-18) 437 (+1) 1499 (-17) -======= ============== =============== =========== +======= ============ =========== ========== =========== +Python Public Private Internal Total +======= ============ =========== ========== =========== +2.7 891 207 0 1098 +3.6 1041 (+150) 419 (+212) 0 1460 (+362) +3.7 1068 (+27) 479 (+60) 22 1569 (+109) +3.8 1105 (+37) 456 (-23) 91 (+69) 1652 (+83) +3.9 1115 (+10) 439 (-17) 124 (+33) 1678 (+26) +3.10 1080 (-35) 439 (same) 129 (+5) 1648 (-30) +3.11 1083 (+3) 379 (-60) 187 (+58) 1649 (+1) +======= ============ =========== ========== =========== Command (total):: - grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|wc -l + grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h Include/internal/*.h|wc -l -Command (private):: +Command (public):: grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|grep -v ' _Py'|wc -l +Command (public + private):: + + grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|wc -l + +Command (internal):: + + grep -E 'PyAPI_(FUNC|DATA)' Include/internal/*.h|wc -l + Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid exporting symbols which are not **explicitly** exported. @@ -94,7 +104,7 @@ Python Macro Static inline Total 3.8 399 14 413 3.9 406 30 436 3.10 412 36 448 -3.11 412 40 452 +3.11 399 29 428 ====== ===== ============= ===== If a function is defined as a private static inline function and exposed as a From 17469aaa54fcf02f6df097247c136f46c1507d95 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 21 Apr 2022 23:19:14 +0200 Subject: [PATCH 044/136] stats --- doc/stats.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index cb65347..dc87647 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -16,7 +16,7 @@ Python Limited API CPython API Internal API Total 3.8 13,160 (70%) 3,417 (18%) 2,230 (12%) 18,807 3.9 12,264 (62%) 4,343 (22%) 3,066 (16%) 19,673 3.10 10,395 (51%) 4,584 (22%) 5,520 (27%) 20,499 -3.11 9,620 (41%) 4,865 (21%) 9,227 (38%) 23,712 +3.11 9,721 (40%) 5,241 (22%) 9,293 (38%) 24,255 ======= ============== =========== ============ ======= Commands: @@ -39,7 +39,7 @@ Python Limited API CPython API Internal API Total 3.8 97 (-2) 15 (+15) 21 (+10) 133 (+23) 3.9 98 (+1) 24 (+9) 33 (+12) 155 (+22) 3.10 81 (-17) 32 (+8) 48 (+15) 161 (+6) -3.11 73 (-8) 42 (+10) 66 (+18) 181 (+20) +3.11 72 (-9) 46 (+14) 66 (+18) 184 (+23) ======= ============== =========== ============ ========== Commands: @@ -63,7 +63,7 @@ Python Public Private Internal Total 3.8 1105 (+37) 456 (-23) 91 (+69) 1652 (+83) 3.9 1115 (+10) 439 (-17) 124 (+33) 1678 (+26) 3.10 1080 (-35) 439 (same) 129 (+5) 1648 (-30) -3.11 1083 (+3) 379 (-60) 187 (+58) 1649 (+1) +3.11 1086 (+6) 381 (-58) 187 (+58) 1654 (+6) ======= ============ =========== ========== =========== Command (total):: @@ -104,7 +104,7 @@ Python Macro Static inline Total 3.8 399 14 413 3.9 406 30 436 3.10 412 36 448 -3.11 399 29 428 +3.11 399 54 453 ====== ===== ============= ===== If a function is defined as a private static inline function and exposed as a @@ -132,7 +132,7 @@ Python Structures 3.8 127 (-10) 3.9 125 (-2) 3.10 104 (-21) -3.11 104 (same) +3.11 106 (+2) ====== ========== These numbers exclude opaque structures like PyInterpreterState (since Python From 75b2436980c27d28d268a28c4239ad3bef7df820 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 26 Apr 2022 11:08:59 +0200 Subject: [PATCH 045/136] stats --- doc/stats.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index dc87647..bfd0567 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -25,8 +25,8 @@ Commands: * CPython: ``wc -l Include/cpython/*.h`` * Internal: ``wc -l Include/internal/*.h`` -Header file numbers -=================== +File numbers +============ Number of header file numbers per Python version: From 4bb83c5001d9f38f3d141989c39730fcdb83fc19 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 27 May 2022 15:55:36 +0200 Subject: [PATCH 046/136] Update Status --- doc/index.rst | 2 +- doc/roadmap.rst | 85 ----------------------------------- doc/status.rst | 116 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 117 insertions(+), 86 deletions(-) delete mode 100644 doc/roadmap.rst create mode 100644 doc/status.rst diff --git a/doc/index.rst b/doc/index.rst index a6fbfa8..4c7acfd 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -38,9 +38,9 @@ Pages .. toctree:: :maxdepth: 1 + status optimize_python rationale - roadmap bad_api new_api runtimes diff --git a/doc/roadmap.rst b/doc/roadmap.rst deleted file mode 100644 index fec302c..0000000 --- a/doc/roadmap.rst +++ /dev/null @@ -1,85 +0,0 @@ -++++++++++++++++++++++++++++++ -Roadmap for a new Python C API -++++++++++++++++++++++++++++++ - -Roadmap -======= - -* Step 1: Identify :ref:`Bad C API ` and list functions that should - be modified or even removed -* Step 2: Add an **opt-in** :ref:`new C API ` with these cleanups. Test :ref:`popular - C extensions ` to measure how much code is broken. Start to fix - these C extensions by making them **forward** compatible. Slowly involve more - and more players into the game. -* Step 3: :ref:`Remove more functions `. Maybe replace - :ref:`Py_INCREF() macro ` with a function call. Finish to hide all C - structures especially ``PyObject.ob_refcnt``. Measure the performance. - Decide what to do. -* Step 4: if step 3 gone fine and most people are still ok to continue, make - the :ref:`new C API ` as the default in CPython and add an option - for **opt-out** to stick with the :ref:`old C API `. - -Open questions -============== - -* Remove or deprecate APIs using borrowed references? If ``PyTuple_GetItem()`` - must be replaced with ``PyTuple_GetItemRef()``, how do we provide - ``PyTuple_GetItemRef()`` for Python 3.7 and older? See :ref:`Backward - compatibility `. - -Status -====== - -* 2020-04-10: `PEP: Modify the C API to hide implementation details - `_ - sent to python-dev. -* 2019-05-01: `Status of the stable API and ABI in Python 3.8 - `_, - slides of Victor Stinner's lightning talk at the Language Summit (during - Pycon US 2019) -* 2019-02-22: `[capi-sig] Update on CPython header files reorganization - `_ -* 2018-09-04: Creation of CPython fork to experiment a new incompatible C - API excluding borrowed references and not access directly structure - members. -* 2018-07-29: `pythoncapi project `_ - created on GitHub -* 2018-06: capi-sig mailing list migrated to Mailman 3 -* 2017-12-21: It's an idea. There is an old PEP draft, but no implementation, - the PEP has no number and was not accepted yet (nor really proposed). -* 2017-11: Idea proposed on python-dev, `[Python-Dev] Make the stable API-ABI - usable - `_ -* 2017-09: Blog post: `A New C API for CPython - `_ -* 2017-09: Idea discussed at the CPython sprint at Instagram (California). - Liked by all core developers. The expected performance slowdown is likely to - be accepted. -* 2017-07-11: - `[Python-ideas] PEP: Hide implementation details in the C API - `_ -* 2017-07: Idea proposed on python-ideas. `[Python-ideas] PEP: Hide - implementation details in the C API - `_ -* 2017-05: Idea proposed at the Python Language Summit, during PyCon US 2017. - My `"Python performance" slides (PDF) - `_. - LWN article: `Keeping Python competitive - `_. - -Players -======= - -* CPython: Victor Stinner -* PyPy :ref:`cpyext `: Ronan Lamy -* :ref:`Cython `: Stefan Behnel - -Unknown status: - -* `RustPython `_ -* MicroPython? -* IronPython? -* Jython? -* Pyjion -* Pyston -* any other? diff --git a/doc/status.rst b/doc/status.rst new file mode 100644 index 0000000..5c06044 --- /dev/null +++ b/doc/status.rst @@ -0,0 +1,116 @@ +++++++++++++++++++++++++++++++ +Status of the new Python C API +++++++++++++++++++++++++++++++ + +Status +====== + +2021 +---- + +* 2022-03-04: The `pythoncapi-compat project moves + `_ to the GitHub + Python organization. +* 2022-04-22: Petr Viktorin proposes `PEP 689 – Unstable C API tier + `_. +* 2022-04-12: the Python Steering Council accepts the `PEP 670 – Convert macros + to functions in the Python C API `_. It's + implemented in Python 3.11. +* 2022-04-04: Petr Viktorin `proposes + `_ + the document: `Draft: C API design strategy + `_ +* 2022-01-28: python-dev: `Slowly bend the C API towards the limited API to get + a stable ABI for everyone + `_ + +2021 +---- + +* 2021-11-30: Victor Stinner proposes `PEP 674 – Disallow using macros as l-values + `_. +* 2021-10-19: Victor Stinner proposes `PEP 670 – Convert macros to functions in the Python C API + `_. +* 2021-10-05: `Python C API: Add functions to access PyObject + `_ article. +* 2021-10-04: `C API changes between Python 3.5 to 3.10 + `_ article. +* 2021-09-28: Victor Stinner proposes `PEP: Taking the Python C API to the Next Level + `_. +* 2021-03-26: `Make structures opaque in the Python C API + `_ article. + +2020 +---- + +* 2020-12-04: On the capi-sig list, `New script: add Python 3.10 support to your C + extensions without losing Python 3.6 support + `_. +* 2020-10-16: Simon Cross writes `Taking the C API to the Next Level + `_ + for HPy. +* 2020-06-19: Victor Stinner proposes `PEP 620 – Hide implementation details from the C API + `_. +* 2020-06-04: Creation of the `pythoncapi-compat project + `_ +* 2020-04-10: `PEP: Modify the C API to hide implementation details + `_ + sent to python-dev. + +2019 +---- + +* 2019-07-12: Creation of the `HPy project `_ on + GitHub. +* 2019-06-19: `Split Include/ directory in Python 3.8 + `_ article. +* 2019-05-01: Pycon US 2019: `Status of the stable API and ABI in Python 3.8 + `_, + slides of Victor Stinner's lightning talk at the Language Summit. +* 2019-04-25: In Python 3.8, the Python debug build ABI becomes compatible with the release build ABI: + + * `commit `__) + * `What’s New In Python 3.8: Debug build uses the same ABI as release build + `_. + +* 2019-02-22: `[capi-sig] Update on CPython header files reorganization + `_ + +2018 +---- + +* 2018-09-21: Antonio Cuni wrote `Inside cpyext: Why emulating CPython C API is + so Hard + `_ + article about the PyPy cpyext module. +* 2018-09-04: Creation of CPython fork to experiment a new incompatible C + API excluding borrowed references and not access directly structure + members. +* 2018-07-29: Creation of the `pythoncapi project + `_ on GitHub. +* 2018-06: The Python capi-sig mailing list migrated to Mailman 3. + +2017 +---- + +* 2017-12-21: It's an idea. There is an old PEP draft, but no implementation, + the PEP has no number and was not accepted yet (nor really proposed). +* 2017-11: Idea proposed on python-dev, `[Python-Dev] Make the stable API-ABI + usable + `_ +* 2017-09: Blog post: `A New C API for CPython + `_ +* 2017-09: Idea discussed at the CPython sprint at Instagram (California). + Liked by all core developers. The expected performance slowdown is likely to + be accepted. +* 2017-07-11: + `[Python-ideas] PEP: Hide implementation details in the C API + `_ +* 2017-07: Idea proposed on python-ideas. `[Python-ideas] PEP: Hide + implementation details in the C API + `_ +* 2017-05: Idea proposed at the Python Language Summit, during PyCon US 2017. + My `"Python performance" slides (PDF) + `_. + LWN article: `Keeping Python competitive + `_. From 78b738c91f2f721ffcdac52d3395f4c449a8af14 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 27 May 2022 16:25:37 +0200 Subject: [PATCH 047/136] Status --- doc/status.rst | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/doc/status.rst b/doc/status.rst index 5c06044..ec39a39 100644 --- a/doc/status.rst +++ b/doc/status.rst @@ -67,12 +67,11 @@ Status * 2019-05-01: Pycon US 2019: `Status of the stable API and ABI in Python 3.8 `_, slides of Victor Stinner's lightning talk at the Language Summit. -* 2019-04-25: In Python 3.8, the Python debug build ABI becomes compatible with the release build ABI: - - * `commit `__) - * `What’s New In Python 3.8: Debug build uses the same ABI as release build - `_. - +* 2019-04-25: In Python 3.8, the Python debug build ABI becomes compatible with + the release build ABI (`commit + `__): + `What’s New In Python 3.8: Debug build uses the same ABI as release build + `_. * 2019-02-22: `[capi-sig] Update on CPython header files reorganization `_ From 945391918993b1c4a783c46f228db8ab2dfccc21 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 27 May 2022 18:00:05 +0200 Subject: [PATCH 048/136] Add stats.py --- doc/index.rst | 7 +- doc/stable_abi.rst | 6 +- doc/stats.py | 313 +++++++++++++++++++++++++++++++++++++++++++++ doc/stats.rst | 146 ++++++++------------- 4 files changed, 376 insertions(+), 96 deletions(-) create mode 100755 doc/stats.py diff --git a/doc/index.rst b/doc/index.rst index 4c7acfd..3fa84a6 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -39,6 +39,8 @@ Pages :maxdepth: 1 status + stats + stats2 optimize_python rationale bad_api @@ -64,7 +66,6 @@ Pages python_ir runtime opaque_pyobject - stats Links @@ -88,8 +89,9 @@ Table of Contents .. toctree:: :maxdepth: 2 + stats + status rationale - roadmap bad_api new_api runtimes @@ -113,4 +115,3 @@ Table of Contents python_ir runtime opaque_pyobject - stats diff --git a/doc/stable_abi.rst b/doc/stable_abi.rst index 01b7bab..1e47a9e 100644 --- a/doc/stable_abi.rst +++ b/doc/stable_abi.rst @@ -1,8 +1,8 @@ .. _stable-abi: -++++++++++++++++++ -Python stable ABI? -++++++++++++++++++ ++++++++++++++++++ +Python stable ABI ++++++++++++++++++ Links ===== diff --git a/doc/stats.py b/doc/stats.py new file mode 100755 index 0000000..ca0f05e --- /dev/null +++ b/doc/stats.py @@ -0,0 +1,313 @@ +#!/usr/bin/python +import builtins +import contextlib +import functools +import glob +import os +import subprocess +import sys + + +# Checkout of Python Git repository, one directory per branch: +# 2.7/ = Python 2.7 branch +# 3.6/ = Python 3.6 branch +# main/ = Python main branch +# etc. +PYTHON_ROOT = '/home/vstinner/python' + + +RST_FILENAME = 'stats.rst' +MAIN_BRANCH = '3.12' +BRANCHES = [ + '2.7', + '3.6', + '3.7', + '3.8', + '3.9', + '3.10', + '3.11', + 'main', +] +COLUMNS = ['Python', 'Limited API', 'CPython API', 'Internal API', 'Total'] +TABLE_SPACE = ' ' + + +PATH_LIMITED_API = 'Include' +PATH_CPYTHON_API = os.path.join('Include', 'cpython') +PATH_INTERNAL_API = os.path.join('Include', 'internal') + + +output = [] +def log(msg=''): + output.append(msg) + + +def files(path): + return glob.glob(os.path.join(path, '*.h')) + + +def get_output(cmd): + proc = subprocess.run(cmd, + shell=True, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL) + out = proc.stdout + + exitcode = proc.returncode + if exitcode: + print(f"Command failed with exit code {exitcode}") + print(f"cmd: {cmd}") + print(f"cwd: {os.getcwd()}") + sys.exit(exitcode) + + return out + + +def get_int(cmd): + out = get_output(cmd) + try: + return int(out) + except ValueError: + print("Command output is not an integer") + print(f"cmd: {cmd}") + print(f"cwd: {os.getcwd()}") + print("stdout:", out) + sys.exit(1) + + +@contextlib.contextmanager +def change_directory(path): + old_dir = os.getcwd() + try: + os.chdir(path) + yield + finally: + os.chdir(old_dir) + + +def iter_branches(): + for name in BRANCHES: + with change_directory(name): + if name == 'main': + name = MAIN_BRANCH + yield name + + +def display_title(title): + log(title) + log('=' * len(title)) + log() + + +def paragraph(text): + log(text.strip()) + log() + + +def main_title(): + title = 'Statistics on the Python C API' + log('+' * len(title)) + log(title) + log('+' * len(title)) + log() + + +def render_table_line(widths, line): + text = [] + for width, cell in zip(widths[:-1], line[:-1]): + cell = cell.ljust(width) + text.append(cell) + text.append(line[-1]) + log(TABLE_SPACE.join(text)) + + +def render_table(lines): + widths = [0] * len(lines[0]) + for line in lines: + for index, cell in enumerate(line): + widths[index] = max(widths[index], len(cell)) + + table_line = [] + for width in widths: + table_line.append('=' * width) + table_line = TABLE_SPACE.join(table_line) + + log(table_line) + for line_number, line in enumerate(lines, 1): + render_table_line(widths, line) + if line_number == 1: + log(table_line) + log(table_line) + log() + + +def format_number(number): + return format(number, ',d') + + +def format_diff(number): + if number != 0: + return format(number, '+,d') + else: + return 'same' + + +def table_compute_diff(lines): + previous = None + for index, line in enumerate(lines): + if index >= 2: + new_line = [line[0]] + for prev_value, value in zip(previous[1:], line[1:]): + cell = f'{format_number(value)} ({format_diff(value - prev_value)})' + new_line.append(cell) + previous = line + lines[index] = new_line + elif index == 1: + previous = line + new_line = [line[0]] + for value in line[1:]: + cell = f'{format_number(value)}' + new_line.append(cell) + lines[index] = new_line + else: + pass + + +def line_numbers(): + display_title('Line Numbers') + paragraph('Number of C API line numbers per Python version:') + + def get(cmd): + out = get_output(cmd) + value = out.split()[0] + return int(value) + + lines = [COLUMNS] + for name in iter_branches(): + limited = get('wc -l Include/*.h|grep total') + if name not in ['2.7', '3.6', '3.7']: + cpython = get('wc -l Include/cpython/*.h|grep total') + else: + cpython = 0 + if name not in ['2.7', '3.6']: + internal = get('wc -l Include/internal/*.h|grep total') + else: + internal = 0 + + total = limited + cpython + internal + line = [name] + for value in (limited, cpython, internal): + line.append(f'{format_number(value)} ({value * 100 / total:.0f}%)') + line.append(format_number(total)) + lines.append(line) + render_table(lines) + + +def file_numbers(): + display_title('File Numbers') + paragraph('Number of header file numbers per Python version:') + lines = [COLUMNS] + for name in iter_branches(): + limited = len(files(PATH_LIMITED_API)) + cpython = len(files(PATH_CPYTHON_API)) + internal = len(files(PATH_INTERNAL_API)) + line = [name, limited, cpython, internal, limited + cpython + internal] + lines.append(line) + table_compute_diff(lines) + render_table(lines) + + +def symbols(): + display_title('Symbols') + paragraph('Symbols exported with PyAPI_FUNC() and PyAPI_DATA():') + lines = [('Python', 'Public', 'Private', 'Internal', 'Total')] + for name in iter_branches(): + total = get_int("grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h Include/internal/*.h|wc -l") + public = get_int("grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|grep -v ' _Py'|wc -l") + public_private = get_int("grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|wc -l") + private = public_private - public + internal = get_int("grep -E 'PyAPI_(FUNC|DATA)' Include/internal/*.h|wc -l") + line = [name, public, private, internal, total] + lines.append(line) + table_compute_diff(lines) + render_table(lines) + + paragraph(""" +Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid +exporting symbols which are not **explicitly** exported. + +The ``make smelly`` command checks for public symbols of libpython and C +extension which are prefixed by ``Py`` or ``_Py``. See +the ``Tools/scripts/smelly.py`` script. + """) + + +def static_inline_func(): + display_title('Functions defined as macros and static inline functions') + paragraph('Functions defined as macros (only public) and static inline functions (public or private):') + + lines = [('Python', 'Macro', 'Static inline', 'Total')] + for name in iter_branches(): + macros = get_int("grep -E 'define P[Yy][A-Za-z_]+ *\(' Include/*.h Include/cpython/*.h|wc -l") + static_inline = get_int("grep 'static inline ' Include/*.h Include/cpython/*.h|grep -v pydtrace|grep -v 'define Py_LOCAL_INLINE'|wc -l") + + line = [name, macros, static_inline, macros + static_inline] + lines.append(line) + table_compute_diff(lines) + render_table(lines) + + paragraph(""" +If a function is defined as a private static inline function and exposed as a +public macro, it is counted twice in this table. For example, the public +Py_INCREF() macro and the private _Py_INCREF() static inline functions are +counted as 2 functions, whereas only the "Py_INCREF" name is public. + """) + + +def structures(): + display_title('Structures') + paragraph('Public structures in the Python C API:') + + lines = [('Python', 'Structures')] + for name in iter_branches(): + count = get_int("grep 'typedef struct' Include/*.h Include/cpython/*.h|wc -l") + line = [name, count] + lines.append(line) + table_compute_diff(lines) + render_table(lines) + + paragraph(""" +These numbers exclude opaque structures like PyInterpreterState (since Python +3.8). The grep command is not exact. For example, PyODictObject is seen as +public, whereas the structure is opaque:: + + typedef struct _odictobject PyODictObject; + +The _odictobject structure is only defined in Objects/odictobject.c. + """) + + +def render_page(): + main_title() + line_numbers() + file_numbers() + symbols() + static_inline_func() + structures() + + +def main(): + with change_directory(PYTHON_ROOT): + render_page() + + with open(RST_FILENAME, 'w') as fp: + for line in output: + print(line, file=fp) + + print(f"Write into {RST_FILENAME}") + + + + +if __name__ == "__main__": + main() diff --git a/doc/stats.rst b/doc/stats.rst index bfd0567..26aac8d 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -1,86 +1,60 @@ -============================== +++++++++++++++++++++++++++++++ Statistics on the Python C API -============================== +++++++++++++++++++++++++++++++ -Line numbers +Line Numbers ============ Number of C API line numbers per Python version: -======= ============== =========== ============ ======= -Python Limited API CPython API Internal API Total -======= ============== =========== ============ ======= -2.7 12,686 (100%) 0 0 12,686 -3.6 16,011 (100%) 0 0 16,011 -3.7 16,517 (96%) 0 705 (4%) 17,222 -3.8 13,160 (70%) 3,417 (18%) 2,230 (12%) 18,807 -3.9 12,264 (62%) 4,343 (22%) 3,066 (16%) 19,673 -3.10 10,395 (51%) 4,584 (22%) 5,520 (27%) 20,499 -3.11 9,721 (40%) 5,241 (22%) 9,293 (38%) 24,255 -======= ============== =========== ============ ======= - -Commands: - -* Limited: ``wc -l Include/*.h`` -* CPython: ``wc -l Include/cpython/*.h`` -* Internal: ``wc -l Include/internal/*.h`` - -File numbers +====== ============= =========== ============ ====== +Python Limited API CPython API Internal API Total +====== ============= =========== ============ ====== +2.7 12,686 (100%) 0 (0%) 0 (0%) 12,686 +3.6 15,984 (100%) 0 (0%) 0 (0%) 15,984 +3.7 16,517 (96%) 0 (0%) 717 (4%) 17,234 +3.8 13,166 (70%) 3,417 (18%) 2,242 (12%) 18,825 +3.9 12,265 (62%) 4,358 (22%) 3,093 (16%) 19,716 +3.10 10,396 (51%) 4,613 (22%) 5,533 (27%) 20,542 +3.11 9,255 (37%) 5,387 (22%) 10,107 (41%) 24,749 +3.12 9,189 (37%) 5,246 (21%) 10,176 (41%) 24,611 +====== ============= =========== ============ ====== + +File Numbers ============ Number of header file numbers per Python version: -======= ============== =========== ============ ========== -Python Limited API CPython API Internal API Total -======= ============== =========== ============ ========== -2.7 91 0 0 91 -3.6 99 (+8) 0 0 99 (+8) -3.7 99 (same) 0 11 (+11) 110 (+11) -3.8 97 (-2) 15 (+15) 21 (+10) 133 (+23) -3.9 98 (+1) 24 (+9) 33 (+12) 155 (+22) -3.10 81 (-17) 32 (+8) 48 (+15) 161 (+6) -3.11 72 (-9) 46 (+14) 66 (+18) 184 (+23) -======= ============== =========== ============ ========== - -Commands: - -* Limited: ``ls Include/*.h|wc -l`` -* CPython: ``ls Include/cpython/*.h|wc -l`` -* Internal: ``ls Include/internal/*.h|wc -l`` - +====== =========== =========== ============ ========= +Python Limited API CPython API Internal API Total +====== =========== =========== ============ ========= +2.7 91 0 0 91 +3.6 99 (+8) 0 (same) 0 (same) 99 (+8) +3.7 99 (same) 0 (same) 11 (+11) 110 (+11) +3.8 97 (-2) 15 (+15) 21 (+10) 133 (+23) +3.9 98 (+1) 24 (+9) 33 (+12) 155 (+22) +3.10 81 (-17) 32 (+8) 48 (+15) 161 (+6) +3.11 72 (-9) 46 (+14) 68 (+20) 186 (+25) +3.12 72 (same) 46 (same) 69 (+1) 187 (+1) +====== =========== =========== ============ ========= Symbols ======= Symbols exported with PyAPI_FUNC() and PyAPI_DATA(): -======= ============ =========== ========== =========== -Python Public Private Internal Total -======= ============ =========== ========== =========== -2.7 891 207 0 1098 -3.6 1041 (+150) 419 (+212) 0 1460 (+362) -3.7 1068 (+27) 479 (+60) 22 1569 (+109) -3.8 1105 (+37) 456 (-23) 91 (+69) 1652 (+83) -3.9 1115 (+10) 439 (-17) 124 (+33) 1678 (+26) -3.10 1080 (-35) 439 (same) 129 (+5) 1648 (-30) -3.11 1086 (+6) 381 (-58) 187 (+58) 1654 (+6) -======= ============ =========== ========== =========== - -Command (total):: - - grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h Include/internal/*.h|wc -l - -Command (public):: - - grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|grep -v ' _Py'|wc -l - -Command (public + private):: - - grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|wc -l - -Command (internal):: - - grep -E 'PyAPI_(FUNC|DATA)' Include/internal/*.h|wc -l +====== ============ ========== ========= ============ +Python Public Private Internal Total +====== ============ ========== ========= ============ +2.7 891 207 0 1,098 +3.6 1,041 (+150) 419 (+212) 0 (same) 1,460 (+362) +3.7 1,068 (+27) 479 (+60) 22 (+22) 1,569 (+109) +3.8 1,105 (+37) 456 (-23) 91 (+69) 1,652 (+83) +3.9 1,115 (+10) 439 (-17) 124 (+33) 1,678 (+26) +3.10 1,080 (-35) 439 (same) 129 (+5) 1,648 (-30) +3.11 1,087 (+7) 381 (-58) 188 (+59) 1,656 (+8) +3.12 1,081 (-6) 378 (-3) 192 (+4) 1,651 (-5) +====== ============ ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid exporting symbols which are not **explicitly** exported. @@ -92,32 +66,26 @@ the ``Tools/scripts/smelly.py`` script. Functions defined as macros and static inline functions ======================================================= -Functions defined as macros (only public) and static inline functions (public -or private): - -====== ===== ============= ===== -Python Macro Static inline Total -====== ===== ============= ===== -2.7 396 0 396 -3.6 394 0 394 -3.7 403 0 403 -3.8 399 14 413 -3.9 406 30 436 -3.10 412 36 448 -3.11 399 54 453 -====== ===== ============= ===== +Functions defined as macros (only public) and static inline functions (public or private): + +====== ========= ============= ========= +Python Macro Static inline Total +====== ========= ============= ========= +2.7 396 0 396 +3.6 394 (-2) 0 (same) 394 (-2) +3.7 403 (+9) 0 (same) 403 (+9) +3.8 399 (-4) 14 (+14) 413 (+10) +3.9 406 (+7) 30 (+16) 436 (+23) +3.10 412 (+6) 36 (+6) 448 (+12) +3.11 400 (-12) 66 (+30) 466 (+18) +3.12 385 (-15) 75 (+9) 460 (-6) +====== ========= ============= ========= If a function is defined as a private static inline function and exposed as a public macro, it is counted twice in this table. For example, the public Py_INCREF() macro and the private _Py_INCREF() static inline functions are counted as 2 functions, whereas only the "Py_INCREF" name is public. -Commands:: - - grep -E 'define P[Yy][A-Za-z_]+ *\(' Include/*.h Include/cpython/*.h|wc -l - grep 'static inline ' Include/*.h Include/cpython/*.h|grep -v pydtrace|grep -v 'define Py_LOCAL_INLINE'|wc -l - - Structures ========== @@ -133,6 +101,7 @@ Python Structures 3.9 125 (-2) 3.10 104 (-21) 3.11 106 (+2) +3.12 111 (+5) ====== ========== These numbers exclude opaque structures like PyInterpreterState (since Python @@ -143,6 +112,3 @@ public, whereas the structure is opaque:: The _odictobject structure is only defined in Objects/odictobject.c. -Command:: - - grep 'typedef struct' Include/*.h Include/cpython/*.h|wc -l From cc7c2e57b65b4c373e6aa4787adcf3a816378818 Mon Sep 17 00:00:00 2001 From: Erlend Egeberg Aasland Date: Fri, 27 May 2022 21:06:24 +0200 Subject: [PATCH 049/136] 2022 section is incorrectly named --- doc/status.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/status.rst b/doc/status.rst index ec39a39..5d305c5 100644 --- a/doc/status.rst +++ b/doc/status.rst @@ -5,7 +5,7 @@ Status of the new Python C API Status ====== -2021 +2022 ---- * 2022-03-04: The `pythoncapi-compat project moves From 7aa4163a5fec5e29c6a295cc2dcee22fe6bbe4ed Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 27 Sep 2022 00:43:48 +0200 Subject: [PATCH 050/136] Get types --- doc/list_structs.py | 22 +++++++++++++++ doc/pythoncapi.py | 52 +++++++++++++++++++++++++++++++++++ doc/stats.py | 27 +++++-------------- doc/stats.rst | 66 ++++++++++++++++++++++----------------------- 4 files changed, 114 insertions(+), 53 deletions(-) create mode 100755 doc/list_structs.py create mode 100644 doc/pythoncapi.py diff --git a/doc/list_structs.py b/doc/list_structs.py new file mode 100755 index 0000000..e7635d4 --- /dev/null +++ b/doc/list_structs.py @@ -0,0 +1,22 @@ +#!/usr/bin/python +import os.path +import re +import sys + +from pythoncapi import PATH_LIMITED_API, PATH_CPYTHON_API, get_types + + +def main(): + if not os.path.exists('pyconfig.h.in'): + print("program must be run in the Python source code directory") + sys.exit(1) + + limited = get_types(PATH_LIMITED_API) + cpython = get_types(PATH_CPYTHON_API) + names = sorted(set(limited) | set(cpython)) + for name in names: + print("*", name) + print() + print("Total:", len(names)) + +main() diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py new file mode 100644 index 0000000..3cf7e59 --- /dev/null +++ b/doc/pythoncapi.py @@ -0,0 +1,52 @@ +import glob +import os.path +import re + + +# Checkout of Python Git repository, one directory per branch: +# +# - 2.7/ = Python 2.7 branch +# - 3.6/ = Python 3.6 branch +# - main/ = Python main branch +# - etc. +PYTHON_ROOT = '/home/vstinner/python' + + +PATH_LIMITED_API = 'Include' +PATH_CPYTHON_API = os.path.join('Include', 'cpython') +PATH_INTERNAL_API = os.path.join('Include', 'internal') + +RE_STRUCT_START = re.compile(r'^(?:typedef +)?struct(?: +([A-Za-z0-9_]+))? *{', re.MULTILINE) +RE_STRUCT_END = re.compile(r'^}(?: +([A-Za-z0-9_]+))? *;', re.MULTILINE) + + +def files(path): + return glob.glob(os.path.join(path, '*.h')) + + +def _get_types(filename, names): + if 'pthread_stubs.h' in filename: + # skip special header file used by WASM + return + + with open(filename, encoding="utf-8") as fp: + content = fp.read() + + for match in RE_STRUCT_START.finditer(content): + struct_name = match.group(1) + match2 = RE_STRUCT_END.search(content, match.end()) + if not match2: + raise Exception(f"cannot find end of: {filename}: {match.group()}") + name = match2.group(1) + if not name: + name = struct_name + if not name: + raise Exception(f"structure has no name: {filename}: {match.group()})") + names.add(name) + + +def get_types(directory): + names = set() + for filename in files(directory): + _get_types(filename, names) + return sorted(names) diff --git a/doc/stats.py b/doc/stats.py index ca0f05e..759f327 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -7,13 +7,7 @@ import subprocess import sys - -# Checkout of Python Git repository, one directory per branch: -# 2.7/ = Python 2.7 branch -# 3.6/ = Python 3.6 branch -# main/ = Python main branch -# etc. -PYTHON_ROOT = '/home/vstinner/python' +from pythoncapi import files, PYTHON_ROOT, PATH_LIMITED_API, PATH_CPYTHON_API, PATH_INTERNAL_API, get_types RST_FILENAME = 'stats.rst' @@ -32,20 +26,11 @@ TABLE_SPACE = ' ' -PATH_LIMITED_API = 'Include' -PATH_CPYTHON_API = os.path.join('Include', 'cpython') -PATH_INTERNAL_API = os.path.join('Include', 'internal') - - output = [] def log(msg=''): output.append(msg) -def files(path): - return glob.glob(os.path.join(path, '*.h')) - - def get_output(cmd): proc = subprocess.run(cmd, shell=True, @@ -266,12 +251,14 @@ def static_inline_func(): def structures(): display_title('Structures') - paragraph('Public structures in the Python C API:') + paragraph('Structures in the Python C API:') - lines = [('Python', 'Structures')] + lines = [COLUMNS] for name in iter_branches(): - count = get_int("grep 'typedef struct' Include/*.h Include/cpython/*.h|wc -l") - line = [name, count] + limited = len(get_types(PATH_LIMITED_API)) + cpython = len(get_types(PATH_CPYTHON_API)) + internal = len(get_types(PATH_INTERNAL_API)) + line = [name, limited, cpython, internal, limited + cpython + internal] lines.append(line) table_compute_diff(lines) render_table(lines) diff --git a/doc/stats.rst b/doc/stats.rst index 26aac8d..b305ab6 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -11,13 +11,13 @@ Number of C API line numbers per Python version: Python Limited API CPython API Internal API Total ====== ============= =========== ============ ====== 2.7 12,686 (100%) 0 (0%) 0 (0%) 12,686 -3.6 15,984 (100%) 0 (0%) 0 (0%) 15,984 -3.7 16,517 (96%) 0 (0%) 717 (4%) 17,234 -3.8 13,166 (70%) 3,417 (18%) 2,242 (12%) 18,825 -3.9 12,265 (62%) 4,358 (22%) 3,093 (16%) 19,716 -3.10 10,396 (51%) 4,613 (22%) 5,533 (27%) 20,542 -3.11 9,255 (37%) 5,387 (22%) 10,107 (41%) 24,749 -3.12 9,189 (37%) 5,246 (21%) 10,176 (41%) 24,611 +3.6 16,051 (100%) 0 (0%) 0 (0%) 16,051 +3.7 16,517 (96%) 0 (0%) 772 (4%) 17,289 +3.8 13,166 (70%) 3,417 (18%) 2,295 (12%) 18,878 +3.9 12,265 (62%) 4,358 (22%) 3,146 (16%) 19,769 +3.10 10,396 (51%) 4,616 (22%) 5,572 (27%) 20,584 +3.11 9,232 (37%) 5,495 (22%) 10,046 (41%) 24,773 +3.12 9,227 (29%) 5,308 (17%) 17,633 (55%) 32,168 ====== ============= =========== ============ ====== File Numbers @@ -30,12 +30,12 @@ Python Limited API CPython API Internal API Total ====== =========== =========== ============ ========= 2.7 91 0 0 91 3.6 99 (+8) 0 (same) 0 (same) 99 (+8) -3.7 99 (same) 0 (same) 11 (+11) 110 (+11) -3.8 97 (-2) 15 (+15) 21 (+10) 133 (+23) -3.9 98 (+1) 24 (+9) 33 (+12) 155 (+22) -3.10 81 (-17) 32 (+8) 48 (+15) 161 (+6) -3.11 72 (-9) 46 (+14) 68 (+20) 186 (+25) -3.12 72 (same) 46 (same) 69 (+1) 187 (+1) +3.7 99 (same) 0 (same) 12 (+12) 111 (+12) +3.8 97 (-2) 15 (+15) 22 (+10) 134 (+23) +3.9 98 (+1) 24 (+9) 34 (+12) 156 (+22) +3.10 81 (-17) 32 (+8) 48 (+14) 161 (+5) +3.11 72 (-9) 48 (+16) 68 (+20) 188 (+27) +3.12 72 (same) 48 (same) 71 (+3) 191 (+3) ====== =========== =========== ============ ========= Symbols @@ -47,13 +47,13 @@ Symbols exported with PyAPI_FUNC() and PyAPI_DATA(): Python Public Private Internal Total ====== ============ ========== ========= ============ 2.7 891 207 0 1,098 -3.6 1,041 (+150) 419 (+212) 0 (same) 1,460 (+362) -3.7 1,068 (+27) 479 (+60) 22 (+22) 1,569 (+109) +3.6 1,041 (+150) 420 (+213) 0 (same) 1,461 (+363) +3.7 1,068 (+27) 479 (+59) 22 (+22) 1,569 (+108) 3.8 1,105 (+37) 456 (-23) 91 (+69) 1,652 (+83) 3.9 1,115 (+10) 439 (-17) 124 (+33) 1,678 (+26) -3.10 1,080 (-35) 439 (same) 129 (+5) 1,648 (-30) -3.11 1,087 (+7) 381 (-58) 188 (+59) 1,656 (+8) -3.12 1,081 (-6) 378 (-3) 192 (+4) 1,651 (-5) +3.10 1,080 (-35) 441 (+2) 129 (+5) 1,650 (-28) +3.11 1,113 (+33) 383 (-58) 187 (+58) 1,683 (+33) +3.12 1,116 (+3) 382 (-1) 186 (-1) 1,684 (+1) ====== ============ ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -78,7 +78,7 @@ Python Macro Static inline Total 3.9 406 (+7) 30 (+16) 436 (+23) 3.10 412 (+6) 36 (+6) 448 (+12) 3.11 400 (-12) 66 (+30) 466 (+18) -3.12 385 (-15) 75 (+9) 460 (-6) +3.12 390 (-10) 77 (+11) 467 (+1) ====== ========= ============= ========= If a function is defined as a private static inline function and exposed as a @@ -89,20 +89,20 @@ counted as 2 functions, whereas only the "Py_INCREF" name is public. Structures ========== -Public structures in the Python C API: - -====== ========== -Python Structures -====== ========== -2.7 97 -3.6 124 (+27) -3.7 137 (+13) -3.8 127 (-10) -3.9 125 (-2) -3.10 104 (-21) -3.11 106 (+2) -3.12 111 (+5) -====== ========== +Structures in the Python C API: + +====== =========== =========== ============ ========= +Python Limited API CPython API Internal API Total +====== =========== =========== ============ ========= +2.7 92 0 0 92 +3.6 110 (+18) 0 (same) 0 (same) 110 (+18) +3.7 114 (+4) 0 (same) 18 (+18) 132 (+22) +3.8 81 (-33) 34 (+34) 28 (+10) 143 (+11) +3.9 68 (-13) 46 (+12) 38 (+10) 152 (+9) +3.10 41 (-27) 53 (+7) 91 (+53) 185 (+33) +3.11 19 (-22) 75 (+22) 112 (+21) 206 (+21) +3.12 24 (+5) 75 (same) 113 (+1) 212 (+6) +====== =========== =========== ============ ========= These numbers exclude opaque structures like PyInterpreterState (since Python 3.8). The grep command is not exact. For example, PyODictObject is seen as From 336b09010c317646411cc95c3578056aab5203e2 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 27 Sep 2022 00:44:37 +0200 Subject: [PATCH 051/136] update --- doc/stats.py | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/doc/stats.py b/doc/stats.py index 759f327..b49a2d0 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -263,16 +263,6 @@ def structures(): table_compute_diff(lines) render_table(lines) - paragraph(""" -These numbers exclude opaque structures like PyInterpreterState (since Python -3.8). The grep command is not exact. For example, PyODictObject is seen as -public, whereas the structure is opaque:: - - typedef struct _odictobject PyODictObject; - -The _odictobject structure is only defined in Objects/odictobject.c. - """) - def render_page(): main_title() From 5f8337dce35e44be869d40b15e5c014ebe6daac4 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 27 Sep 2022 00:49:26 +0200 Subject: [PATCH 052/136] add TYPEDEFS --- doc/pythoncapi.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index 3cf7e59..609bf27 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -20,6 +20,19 @@ RE_STRUCT_END = re.compile(r'^}(?: +([A-Za-z0-9_]+))? *;', re.MULTILINE) +TYPEDEFS = { + '_object': 'PyObject', + '_longobject': 'PyLongObject', + '_typeobject': 'PyTypeObject', + 'PyCodeObject': 'PyCodeObject', + '_frame': 'PyFrameObject', + '_ts': 'PyThreadState', + '_is': 'PyInterpreterState', + '_xid': '_PyCrossInterpreterData', + '_traceback': 'PyTracebackObject', +} + + def files(path): return glob.glob(os.path.join(path, '*.h')) @@ -42,6 +55,8 @@ def _get_types(filename, names): name = struct_name if not name: raise Exception(f"structure has no name: {filename}: {match.group()})") + if name in TYPEDEFS: + name = TYPEDEFS[name] names.add(name) From 1b5b2697c33a0720e28968cf99f77495cc576810 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 27 Sep 2022 01:04:41 +0200 Subject: [PATCH 053/136] Add misc --- doc/index.rst | 53 +----------------------- doc/links.rst | 15 +++++++ doc/misc.rst | 110 ++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 127 insertions(+), 51 deletions(-) create mode 100644 doc/links.rst create mode 100644 doc/misc.rst diff --git a/doc/index.rst b/doc/index.rst index 3fa84a6..79969fb 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -32,57 +32,6 @@ Existing C extensions will still be supported and will not have to be modified. The :ref:`old C API ` is not deprecated and there is no plan penalize users of the old C API. -Pages -===== - -.. toctree:: - :maxdepth: 1 - - status - stats - stats2 - optimize_python - rationale - bad_api - new_api - runtimes - old_c_api - type_object - optimization_ideas - backward_compatibility - os_vendors - calling_conventions - stable_abi - consumers - cpyext - cython - cffi - gilectomy - remove_c_api - performance - split_include - pyhandle - deprecate - python_ir - runtime - opaque_pyobject - - -Links -===== - -* `Python C API `_ (this documentation) -* `CPython fork implementing Py_NEWCAPI - `_ -* `pythoncapi GitHub project `_ - (this documentation can be found in the ``doc/`` subdirectory). -* `capi-sig mailing list - `_ -* `py3c `_ - (`py3c on GitHub `_): - A Python 2/3 compatibility layer for C extensions. - - Table of Contents ================= @@ -115,3 +64,5 @@ Table of Contents python_ir runtime opaque_pyobject + links + misc diff --git a/doc/links.rst b/doc/links.rst new file mode 100644 index 0000000..33f39c2 --- /dev/null +++ b/doc/links.rst @@ -0,0 +1,15 @@ ++++++ +Links ++++++ + +* `Python C API `_ (this documentation) +* `CPython fork implementing Py_NEWCAPI + `_ +* `pythoncapi GitHub project `_ + (this documentation can be found in the ``doc/`` subdirectory). +* `capi-sig mailing list + `_ +* `py3c `_ + (`py3c on GitHub `_): + A Python 2/3 compatibility layer for C extensions. + diff --git a/doc/misc.rst b/doc/misc.rst new file mode 100644 index 0000000..63779c6 --- /dev/null +++ b/doc/misc.rst @@ -0,0 +1,110 @@ +++++++++++ +Misc notes +++++++++++ + +Public structures of Python types in Python 3.12 (49) +===================================================== + +* PyASCIIObject +* PyAsyncGenObject +* PyAttributeErrorObject +* PyBaseExceptionGroupObject +* PyBaseExceptionObject +* PyByteArrayObject +* PyBytesObject +* PyCFunctionObject +* PyCMethodObject +* PyCellObject +* PyCompactUnicodeObject +* PyComplexObject +* PyCoroObject +* PyCursesWindowObject +* PyDescrObject +* PyDictObject +* PyFloatObject +* PyFunctionObject +* PyGenObject +* PyGetSetDescrObject +* PyHeapTypeObject +* PyImportErrorObject +* PyInstanceMethodObject +* PyListObject +* PyLongObject +* PyMemberDescrObject +* PyMemoryViewObject +* PyMethodDescrObject +* PyMethodObject +* PyModuleDef +* PyModuleDef_Base +* PyNameErrorObject +* PyOSErrorObject +* PyObject +* PySetObject +* PySliceObject +* PyStopIterationObject +* PySyntaxErrorObject +* PySystemExitObject +* PyTracebackObject +* PyTupleObject +* PyTypeObject +* PyUnicodeErrorObject +* PyUnicodeObject +* PyVarObject +* PyWrapperDescrObject +* _PyDictViewObject +* _PyManagedBufferObject +* _PyWeakReference + +Structures which don't inherit from PyObject (50) +================================================= + +* CallStats +* ObjectStats +* OpcodeStats +* PyAsyncMethods +* PyBufferProcs +* PyCodeAddressRange +* PyCompilerFlags +* PyConfig +* PyDateTime_CAPI +* PyFrameConstructor +* PyFutureFeatures +* PyGetSetDef +* PyHash_FuncDef +* PyMappingMethods +* PyMemAllocatorEx +* PyMemberDef +* PyMethodDef +* PyModuleDef_Slot +* PyNumberMethods +* PyObjectArenaAllocator +* PyPreConfig +* PySequenceMethods +* PyStats +* PyStatus +* PyStructSequence_Desc +* PyStructSequence_Field +* PyThreadState +* PyTraceInfo +* PyType_Slot +* PyType_Spec +* PyWideStringList +* Py_buffer +* Py_complex +* SpecializationStats +* _PyArg_Parser +* _PyBytesWriter +* _PyCFrame +* _PyCrossInterpreterData +* _PyErr_StackItem +* _PyStackChunk +* _PyUnicodeWriter +* _Py_Identifier +* _Py_clock_info_t +* _Py_tss_t +* _frozen +* _inittab +* _opaque +* _specialization_cache +* setentry +* wrapperbase From 831f0656e0cdb426bb4ed17ea014392a78a61c6a Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 27 Sep 2022 10:52:08 +0200 Subject: [PATCH 054/136] gitignore: __pycache__ --- .gitignore | 1 + doc/list_structs.py | 7 +++++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/.gitignore b/.gitignore index 1edf253..d2161d4 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ doc/build/ +__pycache__ diff --git a/doc/list_structs.py b/doc/list_structs.py index e7635d4..1c083e1 100755 --- a/doc/list_structs.py +++ b/doc/list_structs.py @@ -12,8 +12,11 @@ def main(): sys.exit(1) limited = get_types(PATH_LIMITED_API) - cpython = get_types(PATH_CPYTHON_API) - names = sorted(set(limited) | set(cpython)) + if 1: + cpython = get_types(PATH_CPYTHON_API) + names = sorted(set(limited) | set(cpython)) + else: + names = limited for name in names: print("*", name) print() From 4e25890f03e5c75cabf98f174ff04e01ca937b79 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 28 Sep 2022 16:15:55 +0200 Subject: [PATCH 055/136] Resource --- doc/index.rst | 1 + doc/resource.rst | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) create mode 100644 doc/resource.rst diff --git a/doc/index.rst b/doc/index.rst index 79969fb..0b09d6c 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -64,5 +64,6 @@ Table of Contents python_ir runtime opaque_pyobject + resource links misc diff --git a/doc/resource.rst b/doc/resource.rst new file mode 100644 index 0000000..96e9ffb --- /dev/null +++ b/doc/resource.rst @@ -0,0 +1,35 @@ ++++++++++++++++++++++++++++++++++++++++++ +Implicit and explicit resource management ++++++++++++++++++++++++++++++++++++++++++ + +Pointer API +=========== + +The Python C API has multiple APIs which implicit resource managements: + +* ``PyBytes_AsString()`` +* ``PyUnicode_AsUTF8AndSize()`` + +These functions return a pointer to data. The pointer becomes a dangling +pointer once the object is deleted. + +This API makes the assumption that the memory address of Python objects are +fixed. Python implementations moving objects in memory have to pin these +objects in memory to implement the API, and pinning memory is inefficient in +this case. + +The ``Py_buffer`` API has a ``PyBuffer_Release()`` function which can execute +code when a buffer is no longer needed: + +* Call ``Py_DECREF(obj)`` +* Free memory if the buffer was a memory copy +* Close files +* etc. + +Array of objects +================ + +``array = &PyTuple_GET_ITEM(tuple, 0)`` is a common code pattern to access a +Python tuple as a C array of ``PyObject*``: array of ``PyObject**`` type. It +makes the assumption that a tuple stores ``PyObject*`` and ``PyObject**`` +becomes a dangling pointer once the tuple is destroyed. From 1d4ccf5712d3fc781213743c6820e86a4d0e4e4d Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 21 Nov 2022 17:35:13 +0100 Subject: [PATCH 056/136] Complete README --- README.rst | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 8146298..150ae23 100644 --- a/README.rst +++ b/README.rst @@ -10,8 +10,26 @@ Subtitle: "Make the stable API usable". * `capi-sig mailing list `_ -Content: +Content +======= * ``doc/`` subdirectory: the current documentation * ``refcounts_borrowed.py``: search for functions returning a borrowed reference from CPython ``Doc/data/refcounts.dat`` file. + +Build the documentation +======================= + +Work in the ``doc/`` sub-directory. + +Install dependencies:: + + python -m pip install --user --upgrade sphinx + +Build the documentation:: + + make html + +Regenerate statistics:: + + python stats.py From c7ccfaca7b1613d753811fac5d913f90cfb98b6f Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 21 Nov 2022 17:35:29 +0100 Subject: [PATCH 057/136] add optimize_python in the doc --- doc/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/index.rst b/doc/index.rst index 0b09d6c..c39722e 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -47,6 +47,7 @@ Table of Contents old_c_api type_object optimization_ideas + optimize_python backward_compatibility os_vendors calling_conventions From 7e5a9083b6074542f56b7fc1966005e78feae8f2 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 21 Nov 2022 17:35:39 +0100 Subject: [PATCH 058/136] stats: separate functions from variables --- doc/stats.py | 33 +++++++++++++++++++------ doc/stats.rst | 68 +++++++++++++++++++++++++++++---------------------- 2 files changed, 64 insertions(+), 37 deletions(-) diff --git a/doc/stats.py b/doc/stats.py index b49a2d0..d52f026 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -202,16 +202,16 @@ def file_numbers(): render_table(lines) -def symbols(): - display_title('Symbols') - paragraph('Symbols exported with PyAPI_FUNC() and PyAPI_DATA():') +def list_functions(): + display_title('Functions') + paragraph('Functions exported with PyAPI_FUNC():') lines = [('Python', 'Public', 'Private', 'Internal', 'Total')] for name in iter_branches(): - total = get_int("grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h Include/internal/*.h|wc -l") - public = get_int("grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|grep -v ' _Py'|wc -l") - public_private = get_int("grep -E 'PyAPI_(FUNC|DATA)' Include/*.h Include/cpython/*.h|wc -l") + total = get_int("grep 'PyAPI_FUNC' Include/*.h Include/cpython/*.h Include/internal/*.h|wc -l") + public = get_int("grep 'PyAPI_FUNC' Include/*.h Include/cpython/*.h|grep -v ' _Py'|wc -l") + public_private = get_int("grep 'PyAPI_FUNC' Include/*.h Include/cpython/*.h|wc -l") private = public_private - public - internal = get_int("grep -E 'PyAPI_(FUNC|DATA)' Include/internal/*.h|wc -l") + internal = get_int("grep 'PyAPI_FUNC' Include/internal/*.h|wc -l") line = [name, public, private, internal, total] lines.append(line) table_compute_diff(lines) @@ -227,6 +227,22 @@ def symbols(): """) +def list_variables(): + display_title('Variables') + paragraph('Symbols exported with PyAPI_DATA():') + lines = [('Python', 'Public', 'Private', 'Internal', 'Total')] + for name in iter_branches(): + total = get_int("grep 'PyAPI_DATA' Include/*.h Include/cpython/*.h Include/internal/*.h|wc -l") + public = get_int("grep 'PyAPI_DATA' Include/*.h Include/cpython/*.h|grep -v ' _Py'|wc -l") + public_private = get_int("grep 'PyAPI_DATA' Include/*.h Include/cpython/*.h|wc -l") + private = public_private - public + internal = get_int("grep 'PyAPI_DATA' Include/internal/*.h|wc -l") + line = [name, public, private, internal, total] + lines.append(line) + table_compute_diff(lines) + render_table(lines) + + def static_inline_func(): display_title('Functions defined as macros and static inline functions') paragraph('Functions defined as macros (only public) and static inline functions (public or private):') @@ -268,7 +284,8 @@ def render_page(): main_title() line_numbers() file_numbers() - symbols() + list_functions() + list_variables() static_inline_func() structures() diff --git a/doc/stats.rst b/doc/stats.rst index b305ab6..e645db3 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -11,13 +11,13 @@ Number of C API line numbers per Python version: Python Limited API CPython API Internal API Total ====== ============= =========== ============ ====== 2.7 12,686 (100%) 0 (0%) 0 (0%) 12,686 -3.6 16,051 (100%) 0 (0%) 0 (0%) 16,051 +3.6 16,011 (100%) 0 (0%) 0 (0%) 16,011 3.7 16,517 (96%) 0 (0%) 772 (4%) 17,289 3.8 13,166 (70%) 3,417 (18%) 2,295 (12%) 18,878 3.9 12,265 (62%) 4,358 (22%) 3,146 (16%) 19,769 3.10 10,396 (51%) 4,616 (22%) 5,572 (27%) 20,584 -3.11 9,232 (37%) 5,495 (22%) 10,046 (41%) 24,773 -3.12 9,227 (29%) 5,308 (17%) 17,633 (55%) 32,168 +3.11 9,237 (37%) 5,499 (22%) 10,046 (41%) 24,782 +3.12 9,296 (31%) 5,405 (18%) 15,017 (51%) 29,718 ====== ============= =========== ============ ====== File Numbers @@ -35,26 +35,26 @@ Python Limited API CPython API Internal API Total 3.9 98 (+1) 24 (+9) 34 (+12) 156 (+22) 3.10 81 (-17) 32 (+8) 48 (+14) 161 (+5) 3.11 72 (-9) 48 (+16) 68 (+20) 188 (+27) -3.12 72 (same) 48 (same) 71 (+3) 191 (+3) +3.12 72 (same) 48 (same) 77 (+9) 197 (+9) ====== =========== =========== ============ ========= -Symbols -======= +Functions +========= -Symbols exported with PyAPI_FUNC() and PyAPI_DATA(): +Functions exported with PyAPI_FUNC(): -====== ============ ========== ========= ============ -Python Public Private Internal Total -====== ============ ========== ========= ============ -2.7 891 207 0 1,098 -3.6 1,041 (+150) 420 (+213) 0 (same) 1,461 (+363) -3.7 1,068 (+27) 479 (+59) 22 (+22) 1,569 (+108) -3.8 1,105 (+37) 456 (-23) 91 (+69) 1,652 (+83) -3.9 1,115 (+10) 439 (-17) 124 (+33) 1,678 (+26) -3.10 1,080 (-35) 441 (+2) 129 (+5) 1,650 (-28) -3.11 1,113 (+33) 383 (-58) 187 (+58) 1,683 (+33) -3.12 1,116 (+3) 382 (-1) 186 (-1) 1,684 (+1) -====== ============ ========== ========= ============ +====== ========== ========== ========= ============ +Python Public Private Internal Total +====== ========== ========== ========= ============ +2.7 758 176 0 934 +3.6 870 (+112) 378 (+202) 0 (same) 1,248 (+314) +3.7 893 (+23) 445 (+67) 13 (+13) 1,351 (+103) +3.8 926 (+33) 420 (-25) 79 (+66) 1,425 (+74) +3.9 937 (+11) 405 (-15) 110 (+31) 1,452 (+27) +3.10 902 (-35) 406 (+1) 116 (+6) 1,424 (-28) +3.11 931 (+29) 351 (-55) 177 (+61) 1,459 (+35) +3.12 939 (+8) 354 (+3) 176 (-1) 1,469 (+10) +====== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid exporting symbols which are not **explicitly** exported. @@ -63,6 +63,24 @@ The ``make smelly`` command checks for public symbols of libpython and C extension which are prefixed by ``Py`` or ``_Py``. See the ``Tools/scripts/smelly.py`` script. +Variables +========= + +Symbols exported with PyAPI_DATA(): + +====== ========== ======== ======== ========== +Python Public Private Internal Total +====== ========== ======== ======== ========== +2.7 133 31 0 164 +3.6 171 (+38) 41 (+10) 0 (same) 212 (+48) +3.7 175 (+4) 34 (-7) 9 (+9) 218 (+6) +3.8 179 (+4) 36 (+2) 12 (+3) 227 (+9) +3.9 178 (-1) 34 (-2) 14 (+2) 226 (-1) +3.10 178 (same) 35 (+1) 13 (-1) 226 (same) +3.11 182 (+4) 32 (-3) 10 (-3) 224 (-2) +3.12 183 (+1) 30 (-2) 9 (-1) 222 (-2) +====== ========== ======== ======== ========== + Functions defined as macros and static inline functions ======================================================= @@ -78,7 +96,7 @@ Python Macro Static inline Total 3.9 406 (+7) 30 (+16) 436 (+23) 3.10 412 (+6) 36 (+6) 448 (+12) 3.11 400 (-12) 66 (+30) 466 (+18) -3.12 390 (-10) 77 (+11) 467 (+1) +3.12 394 (-6) 82 (+16) 476 (+10) ====== ========= ============= ========= If a function is defined as a private static inline function and exposed as a @@ -101,14 +119,6 @@ Python Limited API CPython API Internal API Total 3.9 68 (-13) 46 (+12) 38 (+10) 152 (+9) 3.10 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11 19 (-22) 75 (+22) 112 (+21) 206 (+21) -3.12 24 (+5) 75 (same) 113 (+1) 212 (+6) +3.12 24 (+5) 78 (+3) 140 (+28) 242 (+36) ====== =========== =========== ============ ========= -These numbers exclude opaque structures like PyInterpreterState (since Python -3.8). The grep command is not exact. For example, PyODictObject is seen as -public, whereas the structure is opaque:: - - typedef struct _odictobject PyODictObject; - -The _odictobject structure is only defined in Objects/odictobject.c. - From 1fad4d413d7cd9d555da109a431b69191fb352c3 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 1 Dec 2022 15:43:25 +0100 Subject: [PATCH 059/136] Exclude duplicated macros Fix also the regex search for macros. --- doc/stats.py | 69 ++++++++++++++++++++++++++++++++++++++++----------- doc/stats.rst | 39 +++++++++++++---------------- 2 files changed, 71 insertions(+), 37 deletions(-) diff --git a/doc/stats.py b/doc/stats.py index d52f026..8694ee2 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -3,6 +3,7 @@ import contextlib import functools import glob +import re import os import subprocess import sys @@ -24,6 +25,7 @@ ] COLUMNS = ['Python', 'Limited API', 'CPython API', 'Internal API', 'Total'] TABLE_SPACE = ' ' +RE_IDENTIFIER = r'[A-Za-z_][A-Za-z0-9_]*' output = [] @@ -31,15 +33,16 @@ def log(msg=''): output.append(msg) -def get_output(cmd): +def get_output(cmd, error=True): proc = subprocess.run(cmd, shell=True, stdout=subprocess.PIPE, - stderr=subprocess.DEVNULL) + stderr=subprocess.DEVNULL, + text=True) out = proc.stdout exitcode = proc.returncode - if exitcode: + if exitcode and error: print(f"Command failed with exit code {exitcode}") print(f"cmd: {cmd}") print(f"cwd: {os.getcwd()}") @@ -158,6 +161,10 @@ def table_compute_diff(lines): pass +def has_include_cpython(name): + return (name not in ('2.7', '3.6', '3.7')) + + def line_numbers(): display_title('Line Numbers') paragraph('Number of C API line numbers per Python version:') @@ -170,7 +177,7 @@ def get(cmd): lines = [COLUMNS] for name in iter_branches(): limited = get('wc -l Include/*.h|grep total') - if name not in ['2.7', '3.6', '3.7']: + if has_include_cpython(name): cpython = get('wc -l Include/cpython/*.h|grep total') else: cpython = 0 @@ -243,27 +250,59 @@ def list_variables(): render_table(lines) +def iter_header_filenames(name): + if has_include_cpython(name): + patterns = ['Include/*.h', 'Include/cpython/*.h'] + else: + patterns = ['Include/*.h'] + for pattern in patterns: + yield from glob.glob(pattern) + + +def cat_files(filenames): + for filename in filenames: + with open(filename, encoding='utf-8') as fp: + yield fp.read() + +def grep(regex, files, group=0): + for content in files: + for match in regex.finditer(content): + yield match.group(group) + + def static_inline_func(): display_title('Functions defined as macros and static inline functions') paragraph('Functions defined as macros (only public) and static inline functions (public or private):') lines = [('Python', 'Macro', 'Static inline', 'Total')] for name in iter_branches(): - macros = get_int("grep -E 'define P[Yy][A-Za-z_]+ *\(' Include/*.h Include/cpython/*.h|wc -l") - static_inline = get_int("grep 'static inline ' Include/*.h Include/cpython/*.h|grep -v pydtrace|grep -v 'define Py_LOCAL_INLINE'|wc -l") - - line = [name, macros, static_inline, macros + static_inline] + if has_include_cpython(name): + headers = 'Include/*.h Include/cpython/*.h' + else: + headers = 'Include/*.h' + + args = r'[a-zA-Z][a-zA-Z_, ]*' + regex = re.compile(fr'^ *# *define (P[Yy][A-Za-z_]+) *\( *{args}\)', re.MULTILINE) + files = cat_files(iter_header_filenames(name)) + macros = set(grep(regex, files, group=1)) + + regex = re.compile(fr'^static inline [^(\n]+ ({RE_IDENTIFIER}) *\(', re.MULTILINE) + files = cat_files(iter_header_filenames(name)) + static_inline = set(grep(regex, files, group=1)) + # FIXME: exclude 'pydtrace'? + + # Remove macros only used to cast arguments types. Like: + # "static inline void Py_INCREF(...) { ...}" + # "#define Py_INCREF(obj) Py_INCREF(_PyObject_CAST(obj))" + # Only count 1 static inline function, ignore the macro. + macros = macros - static_inline + + line = [name, len(macros), len(static_inline), + len(macros) + len(static_inline)] lines.append(line) table_compute_diff(lines) render_table(lines) - paragraph(""" -If a function is defined as a private static inline function and exposed as a -public macro, it is counted twice in this table. For example, the public -Py_INCREF() macro and the private _Py_INCREF() static inline functions are -counted as 2 functions, whereas only the "Py_INCREF" name is public. - """) - def structures(): display_title('Structures') diff --git a/doc/stats.rst b/doc/stats.rst index e645db3..c4a7f54 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -16,8 +16,8 @@ Python Limited API CPython API Internal API Total 3.8 13,166 (70%) 3,417 (18%) 2,295 (12%) 18,878 3.9 12,265 (62%) 4,358 (22%) 3,146 (16%) 19,769 3.10 10,396 (51%) 4,616 (22%) 5,572 (27%) 20,584 -3.11 9,237 (37%) 5,499 (22%) 10,046 (41%) 24,782 -3.12 9,296 (31%) 5,405 (18%) 15,017 (51%) 29,718 +3.11 9,232 (37%) 5,495 (22%) 10,046 (41%) 24,773 +3.12 9,392 (31%) 5,569 (19%) 15,065 (50%) 30,026 ====== ============= =========== ============ ====== File Numbers @@ -35,7 +35,7 @@ Python Limited API CPython API Internal API Total 3.9 98 (+1) 24 (+9) 34 (+12) 156 (+22) 3.10 81 (-17) 32 (+8) 48 (+14) 161 (+5) 3.11 72 (-9) 48 (+16) 68 (+20) 188 (+27) -3.12 72 (same) 48 (same) 77 (+9) 197 (+9) +3.12 73 (+1) 49 (+1) 77 (+9) 199 (+11) ====== =========== =========== ============ ========= Functions @@ -53,7 +53,7 @@ Python Public Private Internal Total 3.9 937 (+11) 405 (-15) 110 (+31) 1,452 (+27) 3.10 902 (-35) 406 (+1) 116 (+6) 1,424 (-28) 3.11 931 (+29) 351 (-55) 177 (+61) 1,459 (+35) -3.12 939 (+8) 354 (+3) 176 (-1) 1,469 (+10) +3.12 941 (+10) 354 (+3) 176 (-1) 1,471 (+12) ====== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -86,23 +86,18 @@ Functions defined as macros and static inline functions Functions defined as macros (only public) and static inline functions (public or private): -====== ========= ============= ========= -Python Macro Static inline Total -====== ========= ============= ========= -2.7 396 0 396 -3.6 394 (-2) 0 (same) 394 (-2) -3.7 403 (+9) 0 (same) 403 (+9) -3.8 399 (-4) 14 (+14) 413 (+10) -3.9 406 (+7) 30 (+16) 436 (+23) -3.10 412 (+6) 36 (+6) 448 (+12) -3.11 400 (-12) 66 (+30) 466 (+18) -3.12 394 (-6) 82 (+16) 476 (+10) -====== ========= ============= ========= - -If a function is defined as a private static inline function and exposed as a -public macro, it is counted twice in this table. For example, the public -Py_INCREF() macro and the private _Py_INCREF() static inline functions are -counted as 2 functions, whereas only the "Py_INCREF" name is public. +====== ========== ============= ========== +Python Macro Static inline Total +====== ========== ============= ========== +2.7 274 0 274 +3.6 305 (+31) 18 (+18) 323 (+49) +3.7 312 (+7) 22 (+4) 334 (+11) +3.8 315 (+3) 31 (+9) 346 (+12) +3.9 315 (same) 35 (+4) 350 (+4) +3.10 319 (+4) 42 (+7) 361 (+11) +3.11 269 (-50) 70 (+28) 339 (-22) +3.12 246 (-23) 93 (+23) 339 (same) +====== ========== ============= ========== Structures ========== @@ -119,6 +114,6 @@ Python Limited API CPython API Internal API Total 3.9 68 (-13) 46 (+12) 38 (+10) 152 (+9) 3.10 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11 19 (-22) 75 (+22) 112 (+21) 206 (+21) -3.12 24 (+5) 78 (+3) 140 (+28) 242 (+36) +3.12 22 (+3) 80 (+5) 140 (+28) 242 (+36) ====== =========== =========== ============ ========= From a9dcd1cd591435615c4a86b174660a7265dfbeae Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 1 Dec 2022 15:56:16 +0100 Subject: [PATCH 060/136] Python 3.12: ignore pystats.h in type stats --- doc/pythoncapi.py | 4 ++++ doc/stats.rst | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index 609bf27..5b75687 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -41,6 +41,10 @@ def _get_types(filename, names): if 'pthread_stubs.h' in filename: # skip special header file used by WASM return + if os.path.basename(filename) == 'pystats.h': + # skip Include/pystats.h which is code only used if Python is built + # with --enable-pystats (if the Py_STATS macro is defined) + return with open(filename, encoding="utf-8") as fp: content = fp.read() diff --git a/doc/stats.rst b/doc/stats.rst index c4a7f54..366ad82 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -114,6 +114,6 @@ Python Limited API CPython API Internal API Total 3.9 68 (-13) 46 (+12) 38 (+10) 152 (+9) 3.10 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11 19 (-22) 75 (+22) 112 (+21) 206 (+21) -3.12 22 (+3) 80 (+5) 140 (+28) 242 (+36) +3.12 17 (-2) 80 (+5) 140 (+28) 237 (+31) ====== =========== =========== ============ ========= From 95f44ab15edac2d78490e7aad2c7ef92725326d2 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 1 Dec 2022 16:04:17 +0100 Subject: [PATCH 061/136] Cleanup Python code --- doc/list_structs.py | 1 - doc/pythoncapi.py | 6 ++++-- doc/stats.py | 29 ++++++++++------------------- 3 files changed, 14 insertions(+), 22 deletions(-) diff --git a/doc/list_structs.py b/doc/list_structs.py index 1c083e1..66206b3 100755 --- a/doc/list_structs.py +++ b/doc/list_structs.py @@ -1,6 +1,5 @@ #!/usr/bin/python import os.path -import re import sys from pythoncapi import PATH_LIMITED_API, PATH_CPYTHON_API, get_types diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index 5b75687..c4c7dd5 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -33,7 +33,9 @@ } -def files(path): +def list_files(path): + if not os.path.exists(path): + return [] return glob.glob(os.path.join(path, '*.h')) @@ -66,6 +68,6 @@ def _get_types(filename, names): def get_types(directory): names = set() - for filename in files(directory): + for filename in list_files(directory): _get_types(filename, names) return sorted(names) diff --git a/doc/stats.py b/doc/stats.py index 8694ee2..f486206 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -1,14 +1,12 @@ #!/usr/bin/python -import builtins import contextlib -import functools import glob import re import os import subprocess import sys -from pythoncapi import files, PYTHON_ROOT, PATH_LIMITED_API, PATH_CPYTHON_API, PATH_INTERNAL_API, get_types +from pythoncapi import list_files, PYTHON_ROOT, PATH_LIMITED_API, PATH_CPYTHON_API, PATH_INTERNAL_API, get_types RST_FILENAME = 'stats.rst' @@ -200,9 +198,9 @@ def file_numbers(): paragraph('Number of header file numbers per Python version:') lines = [COLUMNS] for name in iter_branches(): - limited = len(files(PATH_LIMITED_API)) - cpython = len(files(PATH_CPYTHON_API)) - internal = len(files(PATH_INTERNAL_API)) + limited = len(list_files(PATH_LIMITED_API)) + cpython = len(list_files(PATH_CPYTHON_API)) + internal = len(list_files(PATH_INTERNAL_API)) line = [name, limited, cpython, internal, limited + cpython + internal] lines.append(line) table_compute_diff(lines) @@ -250,22 +248,20 @@ def list_variables(): render_table(lines) -def iter_header_filenames(name): - if has_include_cpython(name): +def iter_header_filenames(): + if os.path.exists('Include/cpython'): patterns = ['Include/*.h', 'Include/cpython/*.h'] else: - patterns = ['Include/*.h'] + patterns = [] for pattern in patterns: yield from glob.glob(pattern) -def cat_files(filenames): +def grep(regex, filenames, group=0): for filename in filenames: with open(filename, encoding='utf-8') as fp: - yield fp.read() + content = fp.read() -def grep(regex, files, group=0): - for content in files: for match in regex.finditer(content): yield match.group(group) @@ -276,18 +272,13 @@ def static_inline_func(): lines = [('Python', 'Macro', 'Static inline', 'Total')] for name in iter_branches(): - if has_include_cpython(name): - headers = 'Include/*.h Include/cpython/*.h' - else: - headers = 'Include/*.h' + files = list_files(PATH_LIMITED_API) + list_files(PATH_CPYTHON_API) args = r'[a-zA-Z][a-zA-Z_, ]*' regex = re.compile(fr'^ *# *define (P[Yy][A-Za-z_]+) *\( *{args}\)', re.MULTILINE) - files = cat_files(iter_header_filenames(name)) macros = set(grep(regex, files, group=1)) regex = re.compile(fr'^static inline [^(\n]+ ({RE_IDENTIFIER}) *\(', re.MULTILINE) - files = cat_files(iter_header_filenames(name)) static_inline = set(grep(regex, files, group=1)) # FIXME: exclude 'pydtrace'? From 3abcf546928f775f43c53a7d8a23d67c07669124 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 6 Dec 2022 18:03:10 +0100 Subject: [PATCH 062/136] move macro code to pythoncapi.py --- doc/pythoncapi.py | 30 ++++++++++++++++++++++++++++++ doc/stats.py | 28 ++-------------------------- 2 files changed, 32 insertions(+), 26 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index c4c7dd5..8cf2e99 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -16,6 +16,7 @@ PATH_CPYTHON_API = os.path.join('Include', 'cpython') PATH_INTERNAL_API = os.path.join('Include', 'internal') +RE_IDENTIFIER = r'[A-Za-z_][A-Za-z0-9_]*' RE_STRUCT_START = re.compile(r'^(?:typedef +)?struct(?: +([A-Za-z0-9_]+))? *{', re.MULTILINE) RE_STRUCT_END = re.compile(r'^}(?: +([A-Za-z0-9_]+))? *;', re.MULTILINE) @@ -71,3 +72,32 @@ def get_types(directory): for filename in list_files(directory): _get_types(filename, names) return sorted(names) + + +def grep(regex, filenames, group=0): + for filename in filenames: + with open(filename, encoding='utf-8') as fp: + content = fp.read() + + for match in regex.finditer(content): + yield match.group(group) + + +def get_macros_static_inline_funcs(): + files = list_files(PATH_LIMITED_API) + list_files(PATH_CPYTHON_API) + + args = r'[a-zA-Z][a-zA-Z_, ]*' + regex = re.compile(fr'^ *# *define (P[Yy][A-Za-z_]+) *\( *{args}\)', re.MULTILINE) + macros = set(grep(regex, files, group=1)) + + regex = re.compile(fr'^static inline [^(\n]+ ({RE_IDENTIFIER}) *\(', re.MULTILINE) + funcs = set(grep(regex, files, group=1)) + # FIXME: exclude 'pydtrace'? + + # Remove macros only used to cast arguments types. Like: + # "static inline void Py_INCREF(...) { ...}" + # "#define Py_INCREF(obj) Py_INCREF(_PyObject_CAST(obj))" + # Only count 1 static inline function, ignore the macro. + macros = macros - funcs + + return (macros, funcs) diff --git a/doc/stats.py b/doc/stats.py index f486206..5bf8623 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -6,7 +6,7 @@ import subprocess import sys -from pythoncapi import list_files, PYTHON_ROOT, PATH_LIMITED_API, PATH_CPYTHON_API, PATH_INTERNAL_API, get_types +from pythoncapi import list_files, PYTHON_ROOT, PATH_LIMITED_API, PATH_CPYTHON_API, PATH_INTERNAL_API, get_types, get_macros_static_inline_funcs RST_FILENAME = 'stats.rst' @@ -23,7 +23,6 @@ ] COLUMNS = ['Python', 'Limited API', 'CPython API', 'Internal API', 'Total'] TABLE_SPACE = ' ' -RE_IDENTIFIER = r'[A-Za-z_][A-Za-z0-9_]*' output = [] @@ -257,36 +256,13 @@ def iter_header_filenames(): yield from glob.glob(pattern) -def grep(regex, filenames, group=0): - for filename in filenames: - with open(filename, encoding='utf-8') as fp: - content = fp.read() - - for match in regex.finditer(content): - yield match.group(group) - - def static_inline_func(): display_title('Functions defined as macros and static inline functions') paragraph('Functions defined as macros (only public) and static inline functions (public or private):') lines = [('Python', 'Macro', 'Static inline', 'Total')] for name in iter_branches(): - files = list_files(PATH_LIMITED_API) + list_files(PATH_CPYTHON_API) - - args = r'[a-zA-Z][a-zA-Z_, ]*' - regex = re.compile(fr'^ *# *define (P[Yy][A-Za-z_]+) *\( *{args}\)', re.MULTILINE) - macros = set(grep(regex, files, group=1)) - - regex = re.compile(fr'^static inline [^(\n]+ ({RE_IDENTIFIER}) *\(', re.MULTILINE) - static_inline = set(grep(regex, files, group=1)) - # FIXME: exclude 'pydtrace'? - - # Remove macros only used to cast arguments types. Like: - # "static inline void Py_INCREF(...) { ...}" - # "#define Py_INCREF(obj) Py_INCREF(_PyObject_CAST(obj))" - # Only count 1 static inline function, ignore the macro. - macros = macros - static_inline + macros, static_inline = get_macros_static_inline_funcs() line = [name, len(macros), len(static_inline), len(macros) + len(static_inline)] From a41a2bc04ff196745fa3658d4c6dc2784fde7807 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 6 Dec 2022 18:03:40 +0100 Subject: [PATCH 063/136] Don't count Py_INCREF() macro wrapping _Py_INCREF() func --- doc/pythoncapi.py | 6 ++++++ doc/stats.rst | 16 ++++++++-------- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index 8cf2e99..65eec12 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -100,4 +100,10 @@ def get_macros_static_inline_funcs(): # Only count 1 static inline function, ignore the macro. macros = macros - funcs + for name in list(macros): + # In Python 3.10, the Py_INCREF() was wrapping the _Py_INCREF() static + # inline function. + if f"_{name}" in funcs: + macros.discard(name) + return (macros, funcs) diff --git a/doc/stats.rst b/doc/stats.rst index 366ad82..4fcee68 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -17,7 +17,7 @@ Python Limited API CPython API Internal API Total 3.9 12,265 (62%) 4,358 (22%) 3,146 (16%) 19,769 3.10 10,396 (51%) 4,616 (22%) 5,572 (27%) 20,584 3.11 9,232 (37%) 5,495 (22%) 10,046 (41%) 24,773 -3.12 9,392 (31%) 5,569 (19%) 15,065 (50%) 30,026 +3.12 9,249 (31%) 5,620 (19%) 15,071 (50%) 29,940 ====== ============= =========== ============ ====== File Numbers @@ -35,7 +35,7 @@ Python Limited API CPython API Internal API Total 3.9 98 (+1) 24 (+9) 34 (+12) 156 (+22) 3.10 81 (-17) 32 (+8) 48 (+14) 161 (+5) 3.11 72 (-9) 48 (+16) 68 (+20) 188 (+27) -3.12 73 (+1) 49 (+1) 77 (+9) 199 (+11) +3.12 72 (same) 49 (+1) 77 (+9) 198 (+10) ====== =========== =========== ============ ========= Functions @@ -53,7 +53,7 @@ Python Public Private Internal Total 3.9 937 (+11) 405 (-15) 110 (+31) 1,452 (+27) 3.10 902 (-35) 406 (+1) 116 (+6) 1,424 (-28) 3.11 931 (+29) 351 (-55) 177 (+61) 1,459 (+35) -3.12 941 (+10) 354 (+3) 176 (-1) 1,471 (+12) +3.12 943 (+12) 358 (+7) 176 (-1) 1,477 (+18) ====== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -92,11 +92,11 @@ Python Macro Static inline Total 2.7 274 0 274 3.6 305 (+31) 18 (+18) 323 (+49) 3.7 312 (+7) 22 (+4) 334 (+11) -3.8 315 (+3) 31 (+9) 346 (+12) -3.9 315 (same) 35 (+4) 350 (+4) -3.10 319 (+4) 42 (+7) 361 (+11) -3.11 269 (-50) 70 (+28) 339 (-22) -3.12 246 (-23) 93 (+23) 339 (same) +3.8 311 (-1) 31 (+9) 342 (+8) +3.9 305 (-6) 35 (+4) 340 (-2) +3.10 305 (same) 42 (+7) 347 (+7) +3.11 267 (-38) 70 (+28) 337 (-10) +3.12 244 (-23) 93 (+23) 337 (same) ====== ========== ============= ========== Structures From 1fbd866bc9b342f67bb0e2fb7a8f51d676f75df2 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 6 Dec 2022 23:45:10 +0100 Subject: [PATCH 064/136] Fix macro regex --- doc/pythoncapi.py | 5 +++-- doc/stats.rst | 16 ++++++++-------- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index 65eec12..b72c7f4 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -86,8 +86,9 @@ def grep(regex, filenames, group=0): def get_macros_static_inline_funcs(): files = list_files(PATH_LIMITED_API) + list_files(PATH_CPYTHON_API) - args = r'[a-zA-Z][a-zA-Z_, ]*' - regex = re.compile(fr'^ *# *define (P[Yy][A-Za-z_]+) *\( *{args}\)', re.MULTILINE) + # Match '#define func(' + # Don't match '#define constant (&obj)': space before '(' + regex = re.compile(fr'^ *# *define (P[Yy][A-Za-z_]+)\(', re.MULTILINE) macros = set(grep(regex, files, group=1)) regex = re.compile(fr'^static inline [^(\n]+ ({RE_IDENTIFIER}) *\(', re.MULTILINE) diff --git a/doc/stats.rst b/doc/stats.rst index 4fcee68..3a9a6ae 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -89,14 +89,14 @@ Functions defined as macros (only public) and static inline functions (public or ====== ========== ============= ========== Python Macro Static inline Total ====== ========== ============= ========== -2.7 274 0 274 -3.6 305 (+31) 18 (+18) 323 (+49) -3.7 312 (+7) 22 (+4) 334 (+11) -3.8 311 (-1) 31 (+9) 342 (+8) -3.9 305 (-6) 35 (+4) 340 (-2) -3.10 305 (same) 42 (+7) 347 (+7) -3.11 267 (-38) 70 (+28) 337 (-10) -3.12 244 (-23) 93 (+23) 337 (same) +2.7 276 0 276 +3.6 306 (+30) 18 (+18) 324 (+48) +3.7 315 (+9) 22 (+4) 337 (+13) +3.8 314 (-1) 31 (+9) 345 (+8) +3.9 307 (-7) 35 (+4) 342 (-3) +3.10 307 (same) 42 (+7) 349 (+7) +3.11 269 (-38) 70 (+28) 339 (-10) +3.12 246 (-23) 93 (+23) 339 (same) ====== ========== ============= ========== Structures From dee9201417a9bba1bd6f7c9708fd37588f33577d Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 7 Dec 2022 12:00:49 +0100 Subject: [PATCH 065/136] Remove PyDTrace macros and functions --- doc/pythoncapi.py | 6 +++++- doc/stats.rst | 16 ++++++++-------- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index b72c7f4..4ca908c 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -93,7 +93,6 @@ def get_macros_static_inline_funcs(): regex = re.compile(fr'^static inline [^(\n]+ ({RE_IDENTIFIER}) *\(', re.MULTILINE) funcs = set(grep(regex, files, group=1)) - # FIXME: exclude 'pydtrace'? # Remove macros only used to cast arguments types. Like: # "static inline void Py_INCREF(...) { ...}" @@ -107,4 +106,9 @@ def get_macros_static_inline_funcs(): if f"_{name}" in funcs: macros.discard(name) + # Remove PyDTrace_xxx functions + for name in list(funcs): + if name.startswith("PyDTrace_"): + funcs.discard(name) + return (macros, funcs) diff --git a/doc/stats.rst b/doc/stats.rst index 3a9a6ae..2245d05 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -17,7 +17,7 @@ Python Limited API CPython API Internal API Total 3.9 12,265 (62%) 4,358 (22%) 3,146 (16%) 19,769 3.10 10,396 (51%) 4,616 (22%) 5,572 (27%) 20,584 3.11 9,232 (37%) 5,495 (22%) 10,046 (41%) 24,773 -3.12 9,249 (31%) 5,620 (19%) 15,071 (50%) 29,940 +3.12 9,287 (31%) 5,651 (19%) 15,071 (50%) 30,009 ====== ============= =========== ============ ====== File Numbers @@ -90,13 +90,13 @@ Functions defined as macros (only public) and static inline functions (public or Python Macro Static inline Total ====== ========== ============= ========== 2.7 276 0 276 -3.6 306 (+30) 18 (+18) 324 (+48) -3.7 315 (+9) 22 (+4) 337 (+13) -3.8 314 (-1) 31 (+9) 345 (+8) -3.9 307 (-7) 35 (+4) 342 (-3) -3.10 307 (same) 42 (+7) 349 (+7) -3.11 269 (-38) 70 (+28) 339 (-10) -3.12 246 (-23) 93 (+23) 339 (same) +3.6 306 (+30) 0 (same) 306 (+30) +3.7 315 (+9) 0 (same) 315 (+9) +3.8 314 (-1) 7 (+7) 321 (+6) +3.9 307 (-7) 11 (+4) 318 (-3) +3.10 307 (same) 18 (+7) 325 (+7) +3.11 269 (-38) 46 (+28) 315 (-10) +3.12 246 (-23) 69 (+23) 315 (same) ====== ========== ============= ========== Structures From 427998b7dbe3c2e53558c9b91f48ec49dad658eb Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 20 Jun 2023 09:40:51 +0200 Subject: [PATCH 066/136] Bad API --- doc/bad_api.rst | 71 ++++++++++++++++++++++++++----------------------- 1 file changed, 38 insertions(+), 33 deletions(-) diff --git a/doc/bad_api.rst b/doc/bad_api.rst index 73544ea..bd0c25d 100644 --- a/doc/bad_api.rst +++ b/doc/bad_api.rst @@ -78,59 +78,54 @@ See also :ref:`functions steal references `. Functions ^^^^^^^^^ +* ``PyCell_GET()`` * ``PyDict_GetItem()`` -* ``PyDict_GetItemWithError()`` * ``PyDict_GetItemString()`` +* ``PyDict_GetItemWithError()`` * ``PyDict_SetDefault()`` * ``PyErr_Occurred()`` * ``PyEval_GetBuiltins()`` -* ``PyEval_GetFuncName()``: return the internal ``const char*`` inside a - borrowed reference to a function ``__name__``. -* ``PyFile_Name()`` +* ``PyEval_GetFrame()`` +* ``PyEval_GetGlobals()`` +* ``PyEval_GetLocals()`` +* ``PyFunction_GetAnnotations()`` * ``PyFunction_GetClosure()`` * ``PyFunction_GetCode()`` * ``PyFunction_GetDefaults()`` * ``PyFunction_GetGlobals()`` * ``PyFunction_GetModule()`` -* ``Py_InitModule()`` -* ``Py_InitModule3()`` -* ``Py_InitModule4()`` * ``PyImport_GetModuleDict()`` +* ``PyInstanceMethod_Function()`` +* ``PyInstanceMethod_GET_FUNCTION()`` +* ``PyList_GET_ITEM()`` * ``PyList_GetItem()`` -* ``PyList_SetItem()`` -* ``PyMethod_Class()`` * ``PyMethod_Function()`` +* ``PyMethod_GET_FUNCTION()`` +* ``PyMethod_GET_SELF()`` * ``PyMethod_Self()`` +* ``PyModuleDef_Init()`` * ``PyModule_GetDict()`` -* ``PyNumber_Check()`` * ``PyObject_Init()`` +* ``PySequence_Fast_GET_ITEM()`` +* ``PyState_FindModule()`` +* ``PyStructSequence_GET_ITEM()`` +* ``PyStructSequence_GetItem()`` * ``PySys_GetObject()`` * ``PySys_GetXOptions()`` * ``PyThreadState_GetDict()`` -* ``PyTuple_GetItem()`` -* ``PyTuple_SetItem()`` -* ``PyWeakref_GetObject()``: see https://mail.python.org/pipermail/python-dev/2016-October/146604.html - -Macros ------- - -* ``PyCell_GET()`` -* ``PyList_GET_ITEM()`` -* ``PyList_SET_ITEM()`` -* ``PyMethod_GET_CLASS()`` -* ``PyMethod_GET_FUNCTION()`` -* ``PyMethod_GET_SELF()`` -* ``PySequence_Fast_GET_ITEM()`` * ``PyTuple_GET_ITEM()`` -* ``PyTuple_SET_ITEM()`` +* ``PyTuple_GetItem()`` * ``PyWeakref_GET_OBJECT()`` +* ``PyWeakref_GetObject()``: see https://mail.python.org/pipermail/python-dev/2016-October/146604.html -Border line -^^^^^^^^^^^ +Raw pointer without relase function +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -* ``Py_SETREF()``, ``Py_XSETREF()``: the caller has to manually increment the - reference counter of the new value -* ``N`` format of ``Py_BuildValue()``? +* ``PyBytes_AS_STRING()`` +* ``PyBytes_AsString()`` +* ``PyEval_GetFuncName()`` +* ``PyUnicode_AsUTF8()`` +* ``PyUnicode_AsUTF8AndSize()`` .. _py-type: @@ -294,14 +289,24 @@ See also ``PyLong_AsLongAndOverflow()``. Functions stealing references ============================= -* ``PyContext_Exit()``: *ctx* * ``PyContextVar_Reset()``: *token* +* ``PyContext_Exit()``: *ctx* * ``PyErr_Restore()``: *type*, *value*, *traceback* +* ``PyList_SET_ITEM()`` +* ``PyList_SetItem()`` +* ``PyModule_AddObject()``: *o* on success, no change on error! * ``PySet_Discard()``: *key*, no effect if key not found * ``PyString_ConcatAndDel()``: *newpart* +* ``PyTuple_SET_ITEM()`` +* ``PyTuple_SetItem()`` * ``Py_DECREF()``: *o* -* ``Py_XDECREF()``: *o*, if *o* is not NULL -* ``PyModule_AddObject()``: *o* on success, no change on error! +* ``Py_XDECREF()``: *o* if *o* is not NULL + +Border line API: + +* ``Py_SETREF()``, ``Py_XSETREF()``: the caller has to manually increment the + reference counter of the new value +* ``N`` format of ``Py_BuildValue()``? See also :ref:`borrowed references `. From 3fdb63f6f40ce7a47f26fe105d418555aa095e32 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 20 Jun 2023 09:50:00 +0200 Subject: [PATCH 067/136] Set language to English --- doc/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/conf.py b/doc/conf.py index ae6cf52..a39aa3a 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -64,7 +64,7 @@ # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. -language = None +language = "en" # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. From 08e716d255bab90c90b7de84c3a896693e3dfb15 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 20 Jun 2023 16:08:15 +0200 Subject: [PATCH 068/136] Bad API --- doc/bad_api.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/bad_api.rst b/doc/bad_api.rst index bd0c25d..cdb5cbc 100644 --- a/doc/bad_api.rst +++ b/doc/bad_api.rst @@ -117,6 +117,7 @@ Functions * ``PyTuple_GetItem()`` * ``PyWeakref_GET_OBJECT()`` * ``PyWeakref_GetObject()``: see https://mail.python.org/pipermail/python-dev/2016-October/146604.html +* ``Py_TYPE()``: use ``PyObject_Type()`` Raw pointer without relase function ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From 39863953f16d76caef83408ec7badef080dbe75d Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 20 Jun 2023 23:58:50 +0200 Subject: [PATCH 069/136] Fix refcounts_borrowed.py --- refcounts_borrowed.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/refcounts_borrowed.py b/refcounts_borrowed.py index 20cade6..776670c 100755 --- a/refcounts_borrowed.py +++ b/refcounts_borrowed.py @@ -19,10 +19,12 @@ def parse_refcounts(filename): continue if not header: continue - if ':PyObject*:' in line and line.endswith((':0', ':0:')): - func = line.split(':')[0] - print("Borrowed reference: %s()" % func) - total += 1 + if ':PyObject*:' in line: + parts = line.split(':') + if parts[3] == '0': + func = parts[0] + print("Borrowed reference: %s()" % func) + total += 1 header = False print() From 21ab4caa78ba07bce0a35d6a5fc8ac3b14943890 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 20 Jun 2023 23:59:05 +0200 Subject: [PATCH 070/136] Bad API --- doc/bad_api.rst | 34 +++++++++++++++++++--------------- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/doc/bad_api.rst b/doc/bad_api.rst index cdb5cbc..1110264 100644 --- a/doc/bad_api.rst +++ b/doc/bad_api.rst @@ -78,12 +78,12 @@ See also :ref:`functions steal references `. Functions ^^^^^^^^^ -* ``PyCell_GET()`` -* ``PyDict_GetItem()`` -* ``PyDict_GetItemString()`` -* ``PyDict_GetItemWithError()`` +* ``PyCell_GET()``: use ``PyCell_Get()`` +* ``PyDict_GetItem()``: use ``PyObject_GetItem()`` +* ``PyDict_GetItemString()``: use ``PyObject_GetItem()`` +* ``PyDict_GetItemWithError()``: use ``PyObject_GetItem()`` * ``PyDict_SetDefault()`` -* ``PyErr_Occurred()`` +* ``PyErr_Occurred()``: use ``PyErr_GetRaisedException()`` * ``PyEval_GetBuiltins()`` * ``PyEval_GetFrame()`` * ``PyEval_GetGlobals()`` @@ -97,8 +97,8 @@ Functions * ``PyImport_GetModuleDict()`` * ``PyInstanceMethod_Function()`` * ``PyInstanceMethod_GET_FUNCTION()`` -* ``PyList_GET_ITEM()`` -* ``PyList_GetItem()`` +* ``PyList_GET_ITEM()``: use ``PySequence_GetItem()`` +* ``PyList_GetItem()``: use ``PySequence_GetItem()`` * ``PyMethod_Function()`` * ``PyMethod_GET_FUNCTION()`` * ``PyMethod_GET_SELF()`` @@ -113,8 +113,8 @@ Functions * ``PySys_GetObject()`` * ``PySys_GetXOptions()`` * ``PyThreadState_GetDict()`` -* ``PyTuple_GET_ITEM()`` -* ``PyTuple_GetItem()`` +* ``PyTuple_GET_ITEM()``: use ``PySequence_GetItem()`` +* ``PyTuple_GetItem()``: use ``PySequence_GetItem()`` * ``PyWeakref_GET_OBJECT()`` * ``PyWeakref_GetObject()``: see https://mail.python.org/pipermail/python-dev/2016-October/146604.html * ``Py_TYPE()``: use ``PyObject_Type()`` @@ -124,9 +124,12 @@ Raw pointer without relase function * ``PyBytes_AS_STRING()`` * ``PyBytes_AsString()`` +* ``PyByteArray_AS_STRING()`` +* ``PyByteArray_AsString()`` * ``PyEval_GetFuncName()`` * ``PyUnicode_AsUTF8()`` * ``PyUnicode_AsUTF8AndSize()`` +* ``PyCapsule_GetName()`` .. _py-type: @@ -290,18 +293,19 @@ See also ``PyLong_AsLongAndOverflow()``. Functions stealing references ============================= +* ``PyCell_SET(op, value)``: *value*, use ``PyCell_Set()`` * ``PyContextVar_Reset()``: *token* * ``PyContext_Exit()``: *ctx* * ``PyErr_Restore()``: *type*, *value*, *traceback* -* ``PyList_SET_ITEM()`` -* ``PyList_SetItem()`` +* ``PyList_SET_ITEM(op, index, value)``: *value* +* ``PyList_SetItem(op, i, newitem)``: *newitem* * ``PyModule_AddObject()``: *o* on success, no change on error! * ``PySet_Discard()``: *key*, no effect if key not found * ``PyString_ConcatAndDel()``: *newpart* -* ``PyTuple_SET_ITEM()`` -* ``PyTuple_SetItem()`` -* ``Py_DECREF()``: *o* -* ``Py_XDECREF()``: *o* if *o* is not NULL +* ``PyTuple_SET_ITEM(op, index, value)``: *value* +* ``PyTuple_SetItem(op, i, newitem)``: *newitem* +* ``Py_DECREF(op)``: *op* +* ``Py_XDECREF(op)``: *op* if *op* is not NULL Border line API: From 127949050ace30af8b337a35183264ee42542aa6 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 21 Jun 2023 00:02:18 +0200 Subject: [PATCH 071/136] Bad API --- doc/bad_api.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/bad_api.rst b/doc/bad_api.rst index 1110264..a857209 100644 --- a/doc/bad_api.rst +++ b/doc/bad_api.rst @@ -94,6 +94,8 @@ Functions * ``PyFunction_GetDefaults()`` * ``PyFunction_GetGlobals()`` * ``PyFunction_GetModule()`` +* ``PyImport_AddModule()``: use ``PyImport_AddModuleRef()`` +* ``PyImport_AddModuleObject()``: use ``PyImport_AddModuleRef()`` * ``PyImport_GetModuleDict()`` * ``PyInstanceMethod_Function()`` * ``PyInstanceMethod_GET_FUNCTION()`` @@ -105,7 +107,6 @@ Functions * ``PyMethod_Self()`` * ``PyModuleDef_Init()`` * ``PyModule_GetDict()`` -* ``PyObject_Init()`` * ``PySequence_Fast_GET_ITEM()`` * ``PyState_FindModule()`` * ``PyStructSequence_GET_ITEM()`` @@ -119,6 +120,11 @@ Functions * ``PyWeakref_GetObject()``: see https://mail.python.org/pipermail/python-dev/2016-October/146604.html * ``Py_TYPE()``: use ``PyObject_Type()`` +Misc: + +* ``PyObject_Init(op)`` returns *op* which is technically a borrrow reference, + see `GH-105944 issue `_ + Raw pointer without relase function ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From 9e41a20709526bfce8d20235f9c2c2854547f92b Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 23 Jun 2023 03:36:44 +0200 Subject: [PATCH 072/136] Bad API --- doc/bad_api.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/bad_api.rst b/doc/bad_api.rst index a857209..6a7f3df 100644 --- a/doc/bad_api.rst +++ b/doc/bad_api.rst @@ -116,8 +116,8 @@ Functions * ``PyThreadState_GetDict()`` * ``PyTuple_GET_ITEM()``: use ``PySequence_GetItem()`` * ``PyTuple_GetItem()``: use ``PySequence_GetItem()`` -* ``PyWeakref_GET_OBJECT()`` -* ``PyWeakref_GetObject()``: see https://mail.python.org/pipermail/python-dev/2016-October/146604.html +* ``PyWeakref_GET_OBJECT()``: use ``PyWeakref_GetRef()`` +* ``PyWeakref_GetObject()``: see ``PyWeakref_GetRef()`` * ``Py_TYPE()``: use ``PyObject_Type()`` Misc: From c13f7732722c656ae7c9ea81724d2fead9e88122 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 23 Jun 2023 03:37:28 +0200 Subject: [PATCH 073/136] Bad API --- doc/bad_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/bad_api.rst b/doc/bad_api.rst index 6a7f3df..065065d 100644 --- a/doc/bad_api.rst +++ b/doc/bad_api.rst @@ -117,7 +117,7 @@ Functions * ``PyTuple_GET_ITEM()``: use ``PySequence_GetItem()`` * ``PyTuple_GetItem()``: use ``PySequence_GetItem()`` * ``PyWeakref_GET_OBJECT()``: use ``PyWeakref_GetRef()`` -* ``PyWeakref_GetObject()``: see ``PyWeakref_GetRef()`` +* ``PyWeakref_GetObject()``: use ``PyWeakref_GetRef()`` * ``Py_TYPE()``: use ``PyObject_Type()`` Misc: From edfaeede495b515bcfe0bd9534fe956da1eee46e Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 23 Jun 2023 04:18:09 +0200 Subject: [PATCH 074/136] Update stats for 3.13 --- doc/stats.py | 3 ++- doc/stats.rst | 50 ++++++++++++++++++++++++++++---------------------- 2 files changed, 30 insertions(+), 23 deletions(-) diff --git a/doc/stats.py b/doc/stats.py index 5bf8623..f55b3b3 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -10,7 +10,7 @@ RST_FILENAME = 'stats.rst' -MAIN_BRANCH = '3.12' +MAIN_BRANCH = '3.13' BRANCHES = [ '2.7', '3.6', @@ -19,6 +19,7 @@ '3.9', '3.10', '3.11', + '3.12', 'main', ] COLUMNS = ['Python', 'Limited API', 'CPython API', 'Internal API', 'Total'] diff --git a/doc/stats.rst b/doc/stats.rst index 2245d05..dd05279 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -16,8 +16,9 @@ Python Limited API CPython API Internal API Total 3.8 13,166 (70%) 3,417 (18%) 2,295 (12%) 18,878 3.9 12,265 (62%) 4,358 (22%) 3,146 (16%) 19,769 3.10 10,396 (51%) 4,616 (22%) 5,572 (27%) 20,584 -3.11 9,232 (37%) 5,495 (22%) 10,046 (41%) 24,773 -3.12 9,287 (31%) 5,651 (19%) 15,071 (50%) 30,009 +3.11 9,235 (37%) 5,499 (22%) 10,047 (41%) 24,781 +3.12 9,598 (29%) 5,903 (18%) 17,618 (53%) 33,119 +3.13 9,482 (29%) 5,916 (18%) 17,660 (53%) 33,058 ====== ============= =========== ============ ====== File Numbers @@ -35,7 +36,8 @@ Python Limited API CPython API Internal API Total 3.9 98 (+1) 24 (+9) 34 (+12) 156 (+22) 3.10 81 (-17) 32 (+8) 48 (+14) 161 (+5) 3.11 72 (-9) 48 (+16) 68 (+20) 188 (+27) -3.12 72 (same) 49 (+1) 77 (+9) 198 (+10) +3.12 72 (same) 49 (+1) 91 (+23) 212 (+24) +3.13 73 (+1) 51 (+2) 91 (same) 215 (+3) ====== =========== =========== ============ ========= Functions @@ -52,8 +54,9 @@ Python Public Private Internal Total 3.8 926 (+33) 420 (-25) 79 (+66) 1,425 (+74) 3.9 937 (+11) 405 (-15) 110 (+31) 1,452 (+27) 3.10 902 (-35) 406 (+1) 116 (+6) 1,424 (-28) -3.11 931 (+29) 351 (-55) 177 (+61) 1,459 (+35) -3.12 943 (+12) 358 (+7) 176 (-1) 1,477 (+18) +3.11 931 (+29) 352 (-54) 177 (+61) 1,460 (+36) +3.12 966 (+35) 378 (+26) 178 (+1) 1,522 (+62) +3.13 948 (-18) 375 (-3) 174 (-4) 1,497 (-25) ====== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -68,27 +71,28 @@ Variables Symbols exported with PyAPI_DATA(): -====== ========== ======== ======== ========== -Python Public Private Internal Total -====== ========== ======== ======== ========== -2.7 133 31 0 164 -3.6 171 (+38) 41 (+10) 0 (same) 212 (+48) -3.7 175 (+4) 34 (-7) 9 (+9) 218 (+6) -3.8 179 (+4) 36 (+2) 12 (+3) 227 (+9) -3.9 178 (-1) 34 (-2) 14 (+2) 226 (-1) -3.10 178 (same) 35 (+1) 13 (-1) 226 (same) -3.11 182 (+4) 32 (-3) 10 (-3) 224 (-2) -3.12 183 (+1) 30 (-2) 9 (-1) 222 (-2) -====== ========== ======== ======== ========== +====== ========== ======== ========= ========== +Python Public Private Internal Total +====== ========== ======== ========= ========== +2.7 133 31 0 164 +3.6 171 (+38) 41 (+10) 0 (same) 212 (+48) +3.7 175 (+4) 34 (-7) 9 (+9) 218 (+6) +3.8 179 (+4) 36 (+2) 12 (+3) 227 (+9) +3.9 178 (-1) 34 (-2) 14 (+2) 226 (-1) +3.10 178 (same) 35 (+1) 13 (-1) 226 (same) +3.11 182 (+4) 32 (-3) 10 (-3) 224 (-2) +3.12 185 (+3) 29 (-3) 12 (+2) 226 (+2) +3.13 185 (same) 30 (+1) 12 (same) 227 (+1) +====== ========== ======== ========= ========== Functions defined as macros and static inline functions ======================================================= Functions defined as macros (only public) and static inline functions (public or private): -====== ========== ============= ========== +====== ========== ============= ========= Python Macro Static inline Total -====== ========== ============= ========== +====== ========== ============= ========= 2.7 276 0 276 3.6 306 (+30) 0 (same) 306 (+30) 3.7 315 (+9) 0 (same) 315 (+9) @@ -96,8 +100,9 @@ Python Macro Static inline Total 3.9 307 (-7) 11 (+4) 318 (-3) 3.10 307 (same) 18 (+7) 325 (+7) 3.11 269 (-38) 46 (+28) 315 (-10) -3.12 246 (-23) 69 (+23) 315 (same) -====== ========== ============= ========== +3.12 251 (-18) 71 (+25) 322 (+7) +3.13 248 (-3) 71 (same) 319 (-3) +====== ========== ============= ========= Structures ========== @@ -114,6 +119,7 @@ Python Limited API CPython API Internal API Total 3.9 68 (-13) 46 (+12) 38 (+10) 152 (+9) 3.10 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11 19 (-22) 75 (+22) 112 (+21) 206 (+21) -3.12 17 (-2) 80 (+5) 140 (+28) 237 (+31) +3.12 18 (-1) 84 (+9) 170 (+58) 272 (+66) +3.13 18 (same) 88 (+4) 168 (-2) 274 (+2) ====== =========== =========== ============ ========= From d6f1e2bc2e42a103443951f1a64fe6cdec9c4f02 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 23 Jun 2023 15:03:52 +0200 Subject: [PATCH 075/136] stats: struct --- doc/stats.py | 2 ++ doc/stats.rst | 6 ++++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/doc/stats.py b/doc/stats.py index f55b3b3..c36e093 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -285,6 +285,8 @@ def structures(): lines.append(line) table_compute_diff(lines) render_table(lines) + paragraph('Count also private structures like "_PyCFrame" ' + 'and structures with names not starting with Py like "_frozen".') def render_page(): diff --git a/doc/stats.rst b/doc/stats.rst index dd05279..dd16664 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10 10,396 (51%) 4,616 (22%) 5,572 (27%) 20,584 3.11 9,235 (37%) 5,499 (22%) 10,047 (41%) 24,781 3.12 9,598 (29%) 5,903 (18%) 17,618 (53%) 33,119 -3.13 9,482 (29%) 5,916 (18%) 17,660 (53%) 33,058 +3.13 9,484 (29%) 5,912 (18%) 17,664 (53%) 33,060 ====== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10 902 (-35) 406 (+1) 116 (+6) 1,424 (-28) 3.11 931 (+29) 352 (-54) 177 (+61) 1,460 (+36) 3.12 966 (+35) 378 (+26) 178 (+1) 1,522 (+62) -3.13 948 (-18) 375 (-3) 174 (-4) 1,497 (-25) +3.13 950 (-16) 373 (-5) 174 (-4) 1,497 (-25) ====== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -123,3 +123,5 @@ Python Limited API CPython API Internal API Total 3.13 18 (same) 88 (+4) 168 (-2) 274 (+2) ====== =========== =========== ============ ========= +Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". + From f10d41f082354569c6410effa84f115394d2d1ae Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 28 Jun 2023 03:19:01 +0200 Subject: [PATCH 076/136] steal ref --- doc/bad_api.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/bad_api.rst b/doc/bad_api.rst index 065065d..5a26cab 100644 --- a/doc/bad_api.rst +++ b/doc/bad_api.rst @@ -303,11 +303,17 @@ Functions stealing references * ``PyContextVar_Reset()``: *token* * ``PyContext_Exit()``: *ctx* * ``PyErr_Restore()``: *type*, *value*, *traceback* +* ``PyErr_SetExcInfo(exc_type, exc_value, exc_tb)``: the 3 arguments +* ``PyErr_SetRaisedException(exc)``: *exc* +* ``PyException_SetCause(exc, cause)``: *cause* +* ``PyException_SetContext(exc, ctx)``: *ctx* * ``PyList_SET_ITEM(op, index, value)``: *value* * ``PyList_SetItem(op, i, newitem)``: *newitem* -* ``PyModule_AddObject()``: *o* on success, no change on error! +* ``PyModule_AddObject()``: use ``PyModule_AddObjectRef()`` * ``PySet_Discard()``: *key*, no effect if key not found * ``PyString_ConcatAndDel()``: *newpart* +* ``PyStructSequence_SET_ITEM(op, index, value)``: *value* +* ``PyStructSequence_SetItem(op, index, value)``: *value* * ``PyTuple_SET_ITEM(op, index, value)``: *value* * ``PyTuple_SetItem(op, i, newitem)``: *newitem* * ``Py_DECREF(op)``: *op* From f0d509fdde434662b0cd0618edeba33b66d06294 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 30 Jun 2023 11:07:05 +0200 Subject: [PATCH 077/136] Update stats --- doc/stats.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index dd16664..bf10066 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -17,8 +17,8 @@ Python Limited API CPython API Internal API Total 3.9 12,265 (62%) 4,358 (22%) 3,146 (16%) 19,769 3.10 10,396 (51%) 4,616 (22%) 5,572 (27%) 20,584 3.11 9,235 (37%) 5,499 (22%) 10,047 (41%) 24,781 -3.12 9,598 (29%) 5,903 (18%) 17,618 (53%) 33,119 -3.13 9,484 (29%) 5,912 (18%) 17,664 (53%) 33,060 +3.12 9,598 (29%) 5,903 (18%) 17,630 (53%) 33,131 +3.13 9,489 (29%) 5,792 (17%) 17,845 (54%) 33,126 ====== ============= =========== ============ ====== File Numbers @@ -37,7 +37,7 @@ Python Limited API CPython API Internal API Total 3.10 81 (-17) 32 (+8) 48 (+14) 161 (+5) 3.11 72 (-9) 48 (+16) 68 (+20) 188 (+27) 3.12 72 (same) 49 (+1) 91 (+23) 212 (+24) -3.13 73 (+1) 51 (+2) 91 (same) 215 (+3) +3.13 73 (+1) 51 (+2) 92 (+1) 216 (+4) ====== =========== =========== ============ ========= Functions @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10 902 (-35) 406 (+1) 116 (+6) 1,424 (-28) 3.11 931 (+29) 352 (-54) 177 (+61) 1,460 (+36) 3.12 966 (+35) 378 (+26) 178 (+1) 1,522 (+62) -3.13 950 (-16) 373 (-5) 174 (-4) 1,497 (-25) +3.13 949 (-17) 359 (-19) 174 (-4) 1,482 (-40) ====== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -101,7 +101,7 @@ Python Macro Static inline Total 3.10 307 (same) 18 (+7) 325 (+7) 3.11 269 (-38) 46 (+28) 315 (-10) 3.12 251 (-18) 71 (+25) 322 (+7) -3.13 248 (-3) 71 (same) 319 (-3) +3.13 247 (-4) 70 (-1) 317 (-5) ====== ========== ============= ========= Structures @@ -120,7 +120,7 @@ Python Limited API CPython API Internal API Total 3.10 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12 18 (-1) 84 (+9) 170 (+58) 272 (+66) -3.13 18 (same) 88 (+4) 168 (-2) 274 (+2) +3.13 18 (same) 88 (+4) 171 (+1) 277 (+5) ====== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From b0ccc9c153e6adf265e867495abd959e58a3d1e3 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 1 Jul 2023 20:07:51 +0200 Subject: [PATCH 078/136] Fix regex searching for functions and variables --- doc/pythoncapi.py | 92 +++++++++++++++++++++++++++++++++++++++++++++++ doc/stats.py | 26 +++++++------- doc/stats.rst | 48 ++++++++++++------------- 3 files changed, 128 insertions(+), 38 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index 4ca908c..ec8c7a5 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -112,3 +112,95 @@ def get_macros_static_inline_funcs(): funcs.discard(name) return (macros, funcs) + + +def get_functions(): + regex = re.compile( + # 'PyAPI_FUNC(int) ' + fr'PyAPI_FUNC\([^)]+\)[ |\n]*' + # '_Py_NO_RETURN ' + fr'(?:{RE_IDENTIFIER}+[ |\n]+)*' + # 'PyLong_FromLong(' + fr'({RE_IDENTIFIER})[ |\n]*\(', + re.MULTILINE | re.DOTALL) + + def get(path): + files = list_files(path) + return set(grep(regex, files, group=1)) + + limited = get(PATH_LIMITED_API) + cpython = get(PATH_CPYTHON_API) + internal = get(PATH_INTERNAL_API) + + public = set() + private = set() + for name in limited | cpython: + if name.startswith('_Py'): + private.add(name) + else: + public.add(name) + + return (public, private, internal) + + +def get_variables(): + regex = re.compile( + # 'Py_DEPRECATED(3.13) ' + fr'^ *(?:Py_DEPRECATED\([^)]+\) *)?' + # 'PyAPI_DATA' ... ';' + fr'PyAPI_DATA.*;', + re.MULTILINE) + + RE_VARIABLE = ( + # 'name' + # 'name, name2' + fr'(?:const *)?' + fr'({RE_IDENTIFIER}(?:, *{RE_IDENTIFIER})*)' + # '[]', '[256]', '[PY_EXECUTABLE_KINDS+1]' + fr'(?:\[[^]]*\])?' + ) + + RE_FUNC = ( + # '(*name) (' ... ')' + # '*(*name) (' ... ')' + # 'name (' ... ')' + fr'(?:\*? *\( *\* *({RE_IDENTIFIER}) *\)|({RE_IDENTIFIER})) *\([^)]*\)' + ) + + regex2 = re.compile( + # 'PyAPI_FUNC(int) ' + fr'PyAPI_DATA\([^)]+\) +' + # 'Py_VerboseFlag;' + fr'(?:{RE_VARIABLE}|{RE_FUNC}) *;', + re.MULTILINE) + + def get(path): + files = list_files(path) + names = set() + for line in grep(regex, files): + match = regex2.search(line) + if match is None: + raise ValueError(f'fail to parse PyAPI_DATA: {line!r}') + parts = match.group(1) # variable name + if not parts: + parts = match.group(2) # func 1 + if not parts: + parts = match.group(3) # func 2 + for part in parts.split(','): + part = part.strip() + names.add(part) + return names + + limited = get(PATH_LIMITED_API) + cpython = get(PATH_CPYTHON_API) + internal = get(PATH_INTERNAL_API) + + public = set() + private = set() + for name in limited | cpython: + if name.startswith('_Py'): + private.add(name) + else: + public.add(name) + + return (public, private, internal) diff --git a/doc/stats.py b/doc/stats.py index c36e093..aa9db59 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -6,7 +6,10 @@ import subprocess import sys -from pythoncapi import list_files, PYTHON_ROOT, PATH_LIMITED_API, PATH_CPYTHON_API, PATH_INTERNAL_API, get_types, get_macros_static_inline_funcs +from pythoncapi import ( + PYTHON_ROOT, PATH_LIMITED_API, PATH_CPYTHON_API, PATH_INTERNAL_API, + list_files, + get_types, get_macros_static_inline_funcs, get_functions, get_variables) RST_FILENAME = 'stats.rst' @@ -211,14 +214,12 @@ def list_functions(): display_title('Functions') paragraph('Functions exported with PyAPI_FUNC():') lines = [('Python', 'Public', 'Private', 'Internal', 'Total')] - for name in iter_branches(): - total = get_int("grep 'PyAPI_FUNC' Include/*.h Include/cpython/*.h Include/internal/*.h|wc -l") - public = get_int("grep 'PyAPI_FUNC' Include/*.h Include/cpython/*.h|grep -v ' _Py'|wc -l") - public_private = get_int("grep 'PyAPI_FUNC' Include/*.h Include/cpython/*.h|wc -l") - private = public_private - public - internal = get_int("grep 'PyAPI_FUNC' Include/internal/*.h|wc -l") - line = [name, public, private, internal, total] + for branch_name in iter_branches(): + public, private, internal = get_functions() + total = len(public) + len(private) + len(internal) + line = [branch_name, len(public), len(private), len(internal), total] lines.append(line) + table_compute_diff(lines) render_table(lines) @@ -237,12 +238,9 @@ def list_variables(): paragraph('Symbols exported with PyAPI_DATA():') lines = [('Python', 'Public', 'Private', 'Internal', 'Total')] for name in iter_branches(): - total = get_int("grep 'PyAPI_DATA' Include/*.h Include/cpython/*.h Include/internal/*.h|wc -l") - public = get_int("grep 'PyAPI_DATA' Include/*.h Include/cpython/*.h|grep -v ' _Py'|wc -l") - public_private = get_int("grep 'PyAPI_DATA' Include/*.h Include/cpython/*.h|wc -l") - private = public_private - public - internal = get_int("grep 'PyAPI_DATA' Include/internal/*.h|wc -l") - line = [name, public, private, internal, total] + public, private, internal = get_variables() + total = len(public) + len(private) + len(internal) + line = [name, len(public), len(private), len(internal), total] lines.append(line) table_compute_diff(lines) render_table(lines) diff --git a/doc/stats.rst b/doc/stats.rst index bf10066..fc3df58 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10 10,396 (51%) 4,616 (22%) 5,572 (27%) 20,584 3.11 9,235 (37%) 5,499 (22%) 10,047 (41%) 24,781 3.12 9,598 (29%) 5,903 (18%) 17,630 (53%) 33,131 -3.13 9,489 (29%) 5,792 (17%) 17,845 (54%) 33,126 +3.13 9,417 (28%) 5,786 (17%) 17,889 (54%) 33,092 ====== ============= =========== ============ ====== File Numbers @@ -37,7 +37,7 @@ Python Limited API CPython API Internal API Total 3.10 81 (-17) 32 (+8) 48 (+14) 161 (+5) 3.11 72 (-9) 48 (+16) 68 (+20) 188 (+27) 3.12 72 (same) 49 (+1) 91 (+23) 212 (+24) -3.13 73 (+1) 51 (+2) 92 (+1) 216 (+4) +3.13 73 (+1) 51 (+2) 93 (+2) 217 (+5) ====== =========== =========== ============ ========= Functions @@ -48,15 +48,15 @@ Functions exported with PyAPI_FUNC(): ====== ========== ========== ========= ============ Python Public Private Internal Total ====== ========== ========== ========= ============ -2.7 758 176 0 934 -3.6 870 (+112) 378 (+202) 0 (same) 1,248 (+314) -3.7 893 (+23) 445 (+67) 13 (+13) 1,351 (+103) -3.8 926 (+33) 420 (-25) 79 (+66) 1,425 (+74) -3.9 937 (+11) 405 (-15) 110 (+31) 1,452 (+27) -3.10 902 (-35) 406 (+1) 116 (+6) 1,424 (-28) -3.11 931 (+29) 352 (-54) 177 (+61) 1,460 (+36) -3.12 966 (+35) 378 (+26) 178 (+1) 1,522 (+62) -3.13 949 (-17) 359 (-19) 174 (-4) 1,482 (-40) +2.7 748 178 0 926 +3.6 848 (+100) 392 (+214) 0 (same) 1,240 (+314) +3.7 872 (+24) 458 (+66) 13 (+13) 1,343 (+103) +3.8 910 (+38) 429 (-29) 79 (+66) 1,418 (+75) +3.9 920 (+10) 414 (-15) 109 (+30) 1,443 (+25) +3.10 885 (-35) 415 (+1) 115 (+6) 1,415 (-28) +3.11 914 (+29) 361 (-54) 176 (+61) 1,451 (+36) +3.12 951 (+37) 385 (+24) 177 (+1) 1,513 (+62) +3.13 935 (-16) 357 (-28) 173 (-4) 1,465 (-48) ====== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -71,19 +71,19 @@ Variables Symbols exported with PyAPI_DATA(): -====== ========== ======== ========= ========== -Python Public Private Internal Total -====== ========== ======== ========= ========== -2.7 133 31 0 164 -3.6 171 (+38) 41 (+10) 0 (same) 212 (+48) -3.7 175 (+4) 34 (-7) 9 (+9) 218 (+6) -3.8 179 (+4) 36 (+2) 12 (+3) 227 (+9) -3.9 178 (-1) 34 (-2) 14 (+2) 226 (-1) -3.10 178 (same) 35 (+1) 13 (-1) 226 (same) -3.11 182 (+4) 32 (-3) 10 (-3) 224 (-2) -3.12 185 (+3) 29 (-3) 12 (+2) 226 (+2) -3.13 185 (same) 30 (+1) 12 (same) 227 (+1) -====== ========== ======== ========= ========== +====== ========== ========= ========= ========== +Python Public Private Internal Total +====== ========== ========= ========= ========== +2.7 130 32 0 162 +3.6 166 (+36) 42 (+10) 0 (same) 208 (+46) +3.7 170 (+4) 35 (-7) 9 (+9) 214 (+6) +3.8 174 (+4) 37 (+2) 12 (+3) 223 (+9) +3.9 173 (-1) 35 (-2) 14 (+2) 222 (-1) +3.10 173 (same) 35 (same) 13 (-1) 221 (-1) +3.11 177 (+4) 32 (-3) 10 (-3) 219 (-2) +3.12 178 (+1) 29 (-3) 12 (+2) 219 (same) +3.13 178 (same) 30 (+1) 12 (same) 220 (+1) +====== ========== ========= ========= ========== Functions defined as macros and static inline functions ======================================================= From 357dcb921c0a7f9579ae0d793a4afe1e97a2d70a Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sun, 2 Jul 2023 01:18:17 +0200 Subject: [PATCH 079/136] Compute line numbers in Python --- doc/pythoncapi.py | 21 ++++++++++++++++ doc/stats.py | 63 +++-------------------------------------------- doc/stats.rst | 8 +++--- 3 files changed, 29 insertions(+), 63 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index ec8c7a5..5a00b84 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -204,3 +204,24 @@ def get(path): public.add(name) return (public, private, internal) + + +def get_line_number(filename): + with open(filename, encoding="utf-8") as fp: + line_number = 0 + for _ in fp: + line_number += 1 + return line_number + + +def get_line_numbers(): + def get(path): + line_number = 0 + for filename in list_files(path): + line_number += get_line_number(filename) + return line_number + + limited = get(PATH_LIMITED_API) + cpython = get(PATH_CPYTHON_API) + internal = get(PATH_INTERNAL_API) + return (limited, cpython, internal) diff --git a/doc/stats.py b/doc/stats.py index aa9db59..1cd27ed 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -1,15 +1,12 @@ #!/usr/bin/python import contextlib -import glob -import re import os -import subprocess -import sys from pythoncapi import ( PYTHON_ROOT, PATH_LIMITED_API, PATH_CPYTHON_API, PATH_INTERNAL_API, list_files, - get_types, get_macros_static_inline_funcs, get_functions, get_variables) + get_types, get_macros_static_inline_funcs, + get_functions, get_variables, get_line_numbers) RST_FILENAME = 'stats.rst' @@ -34,36 +31,6 @@ def log(msg=''): output.append(msg) -def get_output(cmd, error=True): - proc = subprocess.run(cmd, - shell=True, - stdout=subprocess.PIPE, - stderr=subprocess.DEVNULL, - text=True) - out = proc.stdout - - exitcode = proc.returncode - if exitcode and error: - print(f"Command failed with exit code {exitcode}") - print(f"cmd: {cmd}") - print(f"cwd: {os.getcwd()}") - sys.exit(exitcode) - - return out - - -def get_int(cmd): - out = get_output(cmd) - try: - return int(out) - except ValueError: - print("Command output is not an integer") - print(f"cmd: {cmd}") - print(f"cwd: {os.getcwd()}") - print("stdout:", out) - sys.exit(1) - - @contextlib.contextmanager def change_directory(path): old_dir = os.getcwd() @@ -170,24 +137,11 @@ def line_numbers(): display_title('Line Numbers') paragraph('Number of C API line numbers per Python version:') - def get(cmd): - out = get_output(cmd) - value = out.split()[0] - return int(value) - lines = [COLUMNS] for name in iter_branches(): - limited = get('wc -l Include/*.h|grep total') - if has_include_cpython(name): - cpython = get('wc -l Include/cpython/*.h|grep total') - else: - cpython = 0 - if name not in ['2.7', '3.6']: - internal = get('wc -l Include/internal/*.h|grep total') - else: - internal = 0 - + limited, cpython, internal = get_line_numbers() total = limited + cpython + internal + line = [name] for value in (limited, cpython, internal): line.append(f'{format_number(value)} ({value * 100 / total:.0f}%)') @@ -246,15 +200,6 @@ def list_variables(): render_table(lines) -def iter_header_filenames(): - if os.path.exists('Include/cpython'): - patterns = ['Include/*.h', 'Include/cpython/*.h'] - else: - patterns = [] - for pattern in patterns: - yield from glob.glob(pattern) - - def static_inline_func(): display_title('Functions defined as macros and static inline functions') paragraph('Functions defined as macros (only public) and static inline functions (public or private):') diff --git a/doc/stats.rst b/doc/stats.rst index fc3df58..ee7bb76 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -15,10 +15,10 @@ Python Limited API CPython API Internal API Total 3.7 16,517 (96%) 0 (0%) 772 (4%) 17,289 3.8 13,166 (70%) 3,417 (18%) 2,295 (12%) 18,878 3.9 12,265 (62%) 4,358 (22%) 3,146 (16%) 19,769 -3.10 10,396 (51%) 4,616 (22%) 5,572 (27%) 20,584 -3.11 9,235 (37%) 5,499 (22%) 10,047 (41%) 24,781 -3.12 9,598 (29%) 5,903 (18%) 17,630 (53%) 33,131 -3.13 9,417 (28%) 5,786 (17%) 17,889 (54%) 33,092 +3.10 10,396 (51%) 4,616 (22%) 5,573 (27%) 20,585 +3.11 9,235 (37%) 5,499 (22%) 10,048 (41%) 24,782 +3.12 9,598 (29%) 5,903 (18%) 17,631 (53%) 33,132 +3.13 9,417 (28%) 5,786 (17%) 17,890 (54%) 33,093 ====== ============= =========== ============ ====== File Numbers From 45f4ddbba035d97b85b5b1316fa14f828b66017d Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sun, 2 Jul 2023 01:45:35 +0200 Subject: [PATCH 080/136] Use Git tags, not the development versions --- doc/pythoncapi.py | 80 +++++++++++++++++++++--- doc/stats.py | 65 ++++++++----------- doc/stats.rst | 156 +++++++++++++++++++++++----------------------- 3 files changed, 175 insertions(+), 126 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index 5a00b84..c169baf 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -1,15 +1,12 @@ import glob import os.path import re +import subprocess -# Checkout of Python Git repository, one directory per branch: -# -# - 2.7/ = Python 2.7 branch -# - 3.6/ = Python 3.6 branch -# - main/ = Python main branch -# - etc. -PYTHON_ROOT = '/home/vstinner/python' +# Checkout of Python Git repository +CPYTHON_URL = 'https://github.com/python/cpython' +GIT_DIR = os.path.normpath(os.path.join(os.path.dirname(__file__), '..', 'cpython_git')) PATH_LIMITED_API = 'Include' @@ -34,6 +31,56 @@ } +def run_command(cmd, cwd): + subprocess.run(cmd, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + check=True, + cwd=cwd) + + +def git_clone(): + if os.path.exists(GIT_DIR): + return + + dst_name = os.path.basename(GIT_DIR) + cmd = ['git', 'clone', CPYTHON_URL, dst_name] + run_command(cwd=os.path.dirname(GIT_DIR)) + + +_CLEANED = False +_FETCHED = False + +def git_switch_branch(branch): + git_clone() + + global _CLEANED + if not _CLEANED: + cmd = ['git', 'clean', '-fdx'] + run_command(cmd, cwd=GIT_DIR) + + cmd = ['git', 'checkout', '.'] + run_command(cmd, cwd=GIT_DIR) + + _CLEANED = True + + if branch == 'main': + cmd = ['git', 'switch', branch] + run_command(cmd, cwd=GIT_DIR) + + global _FETCHED + if not _FETCHED: + cmd = ['git', 'fetch'] + run_command(cmd, cwd=GIT_DIR) + _FETCHED = True + + cmd = ['git', 'merge', '--ff'] + run_command(cmd, cwd=GIT_DIR) + else: + cmd = ['git', 'checkout', branch] + run_command(cmd, cwd=GIT_DIR) + + def list_files(path): if not os.path.exists(path): return [] @@ -67,13 +114,20 @@ def _get_types(filename, names): names.add(name) -def get_types(directory): +def get_types_path(directory): names = set() for filename in list_files(directory): _get_types(filename, names) return sorted(names) +def get_types(): + limited = len(get_types_path(PATH_LIMITED_API)) + cpython = len(get_types_path(PATH_CPYTHON_API)) + internal = len(get_types_path(PATH_INTERNAL_API)) + return (limited, cpython, internal) + + def grep(regex, filenames, group=0): for filename in filenames: with open(filename, encoding='utf-8') as fp: @@ -154,8 +208,9 @@ def get_variables(): RE_VARIABLE = ( # 'name' # 'name, name2' + # 'name, *name2' fr'(?:const *)?' - fr'({RE_IDENTIFIER}(?:, *{RE_IDENTIFIER})*)' + fr'({RE_IDENTIFIER}(?:, *\*? *{RE_IDENTIFIER})*)' # '[]', '[256]', '[PY_EXECUTABLE_KINDS+1]' fr'(?:\[[^]]*\])?' ) @@ -225,3 +280,10 @@ def get(path): cpython = get(PATH_CPYTHON_API) internal = get(PATH_INTERNAL_API) return (limited, cpython, internal) + + +def get_file_numbers(): + limited = len(list_files(PATH_LIMITED_API)) + cpython = len(list_files(PATH_CPYTHON_API)) + internal = len(list_files(PATH_INTERNAL_API)) + return (limited, cpython, internal) diff --git a/doc/stats.py b/doc/stats.py index 1cd27ed..b945931 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -3,25 +3,26 @@ import os from pythoncapi import ( - PYTHON_ROOT, PATH_LIMITED_API, PATH_CPYTHON_API, PATH_INTERNAL_API, + GIT_DIR, git_switch_branch, + PATH_LIMITED_API, PATH_CPYTHON_API, PATH_INTERNAL_API, list_files, get_types, get_macros_static_inline_funcs, - get_functions, get_variables, get_line_numbers) + get_functions, get_variables, get_line_numbers, get_file_numbers) -RST_FILENAME = 'stats.rst' -MAIN_BRANCH = '3.13' BRANCHES = [ - '2.7', - '3.6', - '3.7', - '3.8', - '3.9', - '3.10', - '3.11', - '3.12', - 'main', + ('v2.7', '2.7.0'), + ('v3.6.0', '3.6.0'), + ('v3.7.0', '3.7.0'), + ('v3.8.0', '3.8.0'), + ('v3.9.0', '3.9.0'), + ('v3.10.0', '3.10.0'), + ('v3.11.0', '3.11.0'), + #('v3.12.0', '3.12'), + ('3.12', '3.12 (dev)'), + ('main', 'main (3.13)'), ] +RST_FILENAME = os.path.normpath(os.path.join(os.path.dirname(__file__), 'stats.rst')) COLUMNS = ['Python', 'Limited API', 'CPython API', 'Internal API', 'Total'] TABLE_SPACE = ' ' @@ -31,22 +32,11 @@ def log(msg=''): output.append(msg) -@contextlib.contextmanager -def change_directory(path): - old_dir = os.getcwd() - try: - os.chdir(path) - yield - finally: - os.chdir(old_dir) - - def iter_branches(): - for name in BRANCHES: - with change_directory(name): - if name == 'main': - name = MAIN_BRANCH - yield name + for branch, version in BRANCHES: + git_switch_branch(branch) + os.chdir(GIT_DIR) + yield version def display_title(title): @@ -155,9 +145,7 @@ def file_numbers(): paragraph('Number of header file numbers per Python version:') lines = [COLUMNS] for name in iter_branches(): - limited = len(list_files(PATH_LIMITED_API)) - cpython = len(list_files(PATH_CPYTHON_API)) - internal = len(list_files(PATH_INTERNAL_API)) + limited, cpython, internal = get_file_numbers() line = [name, limited, cpython, internal, limited + cpython + internal] lines.append(line) table_compute_diff(lines) @@ -221,10 +209,9 @@ def structures(): lines = [COLUMNS] for name in iter_branches(): - limited = len(get_types(PATH_LIMITED_API)) - cpython = len(get_types(PATH_CPYTHON_API)) - internal = len(get_types(PATH_INTERNAL_API)) - line = [name, limited, cpython, internal, limited + cpython + internal] + limited, cpython, internal = get_types() + total = limited + cpython + internal + line = [name, limited, cpython, internal, total] lines.append(line) table_compute_diff(lines) render_table(lines) @@ -243,14 +230,14 @@ def render_page(): def main(): - with change_directory(PYTHON_ROOT): - render_page() + render_page() - with open(RST_FILENAME, 'w') as fp: + filename = RST_FILENAME + with open(filename, 'w') as fp: for line in output: print(line, file=fp) - print(f"Write into {RST_FILENAME}") + print(f"Write into {filename}") diff --git a/doc/stats.rst b/doc/stats.rst index ee7bb76..67ad7aa 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -7,57 +7,57 @@ Line Numbers Number of C API line numbers per Python version: -====== ============= =========== ============ ====== -Python Limited API CPython API Internal API Total -====== ============= =========== ============ ====== -2.7 12,686 (100%) 0 (0%) 0 (0%) 12,686 -3.6 16,011 (100%) 0 (0%) 0 (0%) 16,011 -3.7 16,517 (96%) 0 (0%) 772 (4%) 17,289 -3.8 13,166 (70%) 3,417 (18%) 2,295 (12%) 18,878 -3.9 12,265 (62%) 4,358 (22%) 3,146 (16%) 19,769 -3.10 10,396 (51%) 4,616 (22%) 5,573 (27%) 20,585 -3.11 9,235 (37%) 5,499 (22%) 10,048 (41%) 24,782 -3.12 9,598 (29%) 5,903 (18%) 17,631 (53%) 33,132 -3.13 9,417 (28%) 5,786 (17%) 17,890 (54%) 33,093 -====== ============= =========== ============ ====== +=========== ============= =========== ============ ====== +Python Limited API CPython API Internal API Total +=========== ============= =========== ============ ====== +2.7.0 12,392 (100%) 0 (0%) 0 (0%) 12,392 +3.6.0 15,829 (100%) 0 (0%) 0 (0%) 15,829 +3.7.0 16,442 (96%) 0 (0%) 678 (4%) 17,120 +3.8.0 13,147 (70%) 3,418 (18%) 2,207 (12%) 18,772 +3.9.0 12,243 (62%) 4,359 (22%) 3,084 (16%) 19,686 +3.10.0 10,408 (51%) 4,584 (22%) 5,519 (27%) 20,511 +3.11.0 9,232 (37%) 5,495 (22%) 10,047 (41%) 24,774 +3.12 (dev) 9,598 (29%) 5,903 (18%) 17,631 (53%) 33,132 +main (3.13) 9,416 (28%) 5,455 (16%) 18,204 (55%) 33,075 +=========== ============= =========== ============ ====== File Numbers ============ Number of header file numbers per Python version: -====== =========== =========== ============ ========= -Python Limited API CPython API Internal API Total -====== =========== =========== ============ ========= -2.7 91 0 0 91 -3.6 99 (+8) 0 (same) 0 (same) 99 (+8) -3.7 99 (same) 0 (same) 12 (+12) 111 (+12) -3.8 97 (-2) 15 (+15) 22 (+10) 134 (+23) -3.9 98 (+1) 24 (+9) 34 (+12) 156 (+22) -3.10 81 (-17) 32 (+8) 48 (+14) 161 (+5) -3.11 72 (-9) 48 (+16) 68 (+20) 188 (+27) -3.12 72 (same) 49 (+1) 91 (+23) 212 (+24) -3.13 73 (+1) 51 (+2) 93 (+2) 217 (+5) -====== =========== =========== ============ ========= +=========== =========== =========== ============ ========= +Python Limited API CPython API Internal API Total +=========== =========== =========== ============ ========= +2.7.0 91 0 0 91 +3.6.0 99 (+8) 0 (same) 0 (same) 99 (+8) +3.7.0 99 (same) 0 (same) 11 (+11) 110 (+11) +3.8.0 97 (-2) 15 (+15) 21 (+10) 133 (+23) +3.9.0 98 (+1) 24 (+9) 33 (+12) 155 (+22) +3.10.0 81 (-17) 32 (+8) 48 (+15) 161 (+6) +3.11.0 72 (-9) 48 (+16) 68 (+20) 188 (+27) +3.12 (dev) 72 (same) 49 (+1) 91 (+23) 212 (+24) +main (3.13) 73 (+1) 50 (+1) 93 (+2) 216 (+4) +=========== =========== =========== ============ ========= Functions ========= Functions exported with PyAPI_FUNC(): -====== ========== ========== ========= ============ -Python Public Private Internal Total -====== ========== ========== ========= ============ -2.7 748 178 0 926 -3.6 848 (+100) 392 (+214) 0 (same) 1,240 (+314) -3.7 872 (+24) 458 (+66) 13 (+13) 1,343 (+103) -3.8 910 (+38) 429 (-29) 79 (+66) 1,418 (+75) -3.9 920 (+10) 414 (-15) 109 (+30) 1,443 (+25) -3.10 885 (-35) 415 (+1) 115 (+6) 1,415 (-28) -3.11 914 (+29) 361 (-54) 176 (+61) 1,451 (+36) -3.12 951 (+37) 385 (+24) 177 (+1) 1,513 (+62) -3.13 935 (-16) 357 (-28) 173 (-4) 1,465 (-48) -====== ========== ========== ========= ============ +=========== ========= ========== ========= ============ +Python Public Private Internal Total +=========== ========= ========== ========= ============ +2.7.0 748 150 0 898 +3.6.0 847 (+99) 380 (+230) 0 (same) 1,227 (+329) +3.7.0 872 (+25) 448 (+68) 12 (+12) 1,332 (+105) +3.8.0 910 (+38) 429 (-19) 78 (+66) 1,417 (+85) +3.9.0 919 (+9) 412 (-17) 109 (+31) 1,440 (+23) +3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) +3.11.0 914 (+29) 360 (-52) 176 (+61) 1,450 (+38) +3.12 (dev) 951 (+37) 385 (+25) 177 (+1) 1,513 (+63) +main (3.13) 935 (-16) 321 (-64) 202 (+25) 1,458 (-55) +=========== ========= ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid exporting symbols which are not **explicitly** exported. @@ -71,57 +71,57 @@ Variables Symbols exported with PyAPI_DATA(): -====== ========== ========= ========= ========== -Python Public Private Internal Total -====== ========== ========= ========= ========== -2.7 130 32 0 162 -3.6 166 (+36) 42 (+10) 0 (same) 208 (+46) -3.7 170 (+4) 35 (-7) 9 (+9) 214 (+6) -3.8 174 (+4) 37 (+2) 12 (+3) 223 (+9) -3.9 173 (-1) 35 (-2) 14 (+2) 222 (-1) -3.10 173 (same) 35 (same) 13 (-1) 221 (-1) -3.11 177 (+4) 32 (-3) 10 (-3) 219 (-2) -3.12 178 (+1) 29 (-3) 12 (+2) 219 (same) -3.13 178 (same) 30 (+1) 12 (same) 220 (+1) -====== ========== ========= ========= ========== +=========== ========== ======== ========= ========== +Python Public Private Internal Total +=========== ========== ======== ========= ========== +2.7.0 130 29 0 159 +3.6.0 167 (+37) 42 (+13) 0 (same) 209 (+50) +3.7.0 170 (+3) 35 (-7) 9 (+9) 214 (+5) +3.8.0 174 (+4) 37 (+2) 12 (+3) 223 (+9) +3.9.0 173 (-1) 35 (-2) 14 (+2) 222 (-1) +3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) +3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) +3.12 (dev) 178 (+1) 29 (-3) 12 (+2) 219 (same) +main (3.13) 178 (same) 30 (+1) 12 (same) 220 (+1) +=========== ========== ======== ========= ========== Functions defined as macros and static inline functions ======================================================= Functions defined as macros (only public) and static inline functions (public or private): -====== ========== ============= ========= -Python Macro Static inline Total -====== ========== ============= ========= -2.7 276 0 276 -3.6 306 (+30) 0 (same) 306 (+30) -3.7 315 (+9) 0 (same) 315 (+9) -3.8 314 (-1) 7 (+7) 321 (+6) -3.9 307 (-7) 11 (+4) 318 (-3) -3.10 307 (same) 18 (+7) 325 (+7) -3.11 269 (-38) 46 (+28) 315 (-10) -3.12 251 (-18) 71 (+25) 322 (+7) -3.13 247 (-4) 70 (-1) 317 (-5) -====== ========== ============= ========= +=========== ========== ============= ========= +Python Macro Static inline Total +=========== ========== ============= ========= +2.7.0 274 0 274 +3.6.0 306 (+32) 0 (same) 306 (+32) +3.7.0 315 (+9) 0 (same) 315 (+9) +3.8.0 314 (-1) 7 (+7) 321 (+6) +3.9.0 307 (-7) 11 (+4) 318 (-3) +3.10.0 307 (same) 18 (+7) 325 (+7) +3.11.0 269 (-38) 46 (+28) 315 (-10) +3.12 (dev) 251 (-18) 71 (+25) 322 (+7) +main (3.13) 247 (-4) 70 (-1) 317 (-5) +=========== ========== ============= ========= Structures ========== Structures in the Python C API: -====== =========== =========== ============ ========= -Python Limited API CPython API Internal API Total -====== =========== =========== ============ ========= -2.7 92 0 0 92 -3.6 110 (+18) 0 (same) 0 (same) 110 (+18) -3.7 114 (+4) 0 (same) 18 (+18) 132 (+22) -3.8 81 (-33) 34 (+34) 28 (+10) 143 (+11) -3.9 68 (-13) 46 (+12) 38 (+10) 152 (+9) -3.10 41 (-27) 53 (+7) 91 (+53) 185 (+33) -3.11 19 (-22) 75 (+22) 112 (+21) 206 (+21) -3.12 18 (-1) 84 (+9) 170 (+58) 272 (+66) -3.13 18 (same) 88 (+4) 171 (+1) 277 (+5) -====== =========== =========== ============ ========= +=========== =========== =========== ============ ========= +Python Limited API CPython API Internal API Total +=========== =========== =========== ============ ========= +2.7.0 91 0 0 91 +3.6.0 109 (+18) 0 (same) 0 (same) 109 (+18) +3.7.0 114 (+5) 0 (same) 18 (+18) 132 (+23) +3.8.0 81 (-33) 34 (+34) 28 (+10) 143 (+11) +3.9.0 68 (-13) 46 (+12) 38 (+10) 152 (+9) +3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) +3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) +3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) +main (3.13) 18 (same) 87 (+3) 172 (+2) 277 (+5) +=========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From 460e01b591bb57589549a36c778be7f8676989d0 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sun, 2 Jul 2023 02:03:17 +0200 Subject: [PATCH 081/136] Change git branches less frequently --- doc/stats.py | 85 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 53 insertions(+), 32 deletions(-) diff --git a/doc/stats.py b/doc/stats.py index b945931..3988426 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -32,13 +32,6 @@ def log(msg=''): output.append(msg) -def iter_branches(): - for branch, version in BRANCHES: - git_switch_branch(branch) - os.chdir(GIT_DIR) - yield version - - def display_title(title): log(title) log('=' * len(title)) @@ -50,6 +43,32 @@ def paragraph(text): log() +class Data: + pass + + +def _get_data(): + data = Data() + data.line_numbers = get_line_numbers() + data.file_numbers = get_file_numbers() + data.functions = get_functions() + data.variables = get_variables() + data.macro_static_inline_funcs = get_macros_static_inline_funcs() + data.types = get_types() + return data + + +def get_data(): + result = [] + for branch, version in BRANCHES: + print(f"Parse {branch} header files") + git_switch_branch(branch) + os.chdir(GIT_DIR) + data = _get_data() + result.append((version, data)) + return result + + def main_title(): title = 'Statistics on the Python C API' log('+' * len(title)) @@ -123,13 +142,13 @@ def has_include_cpython(name): return (name not in ('2.7', '3.6', '3.7')) -def line_numbers(): +def line_numbers(results): display_title('Line Numbers') paragraph('Number of C API line numbers per Python version:') lines = [COLUMNS] - for name in iter_branches(): - limited, cpython, internal = get_line_numbers() + for name, data in results: + limited, cpython, internal = data.line_numbers total = limited + cpython + internal line = [name] @@ -140,26 +159,26 @@ def line_numbers(): render_table(lines) -def file_numbers(): +def file_numbers(results): display_title('File Numbers') paragraph('Number of header file numbers per Python version:') lines = [COLUMNS] - for name in iter_branches(): - limited, cpython, internal = get_file_numbers() + for name, data in results: + limited, cpython, internal = data.file_numbers line = [name, limited, cpython, internal, limited + cpython + internal] lines.append(line) table_compute_diff(lines) render_table(lines) -def list_functions(): +def list_functions(results): display_title('Functions') paragraph('Functions exported with PyAPI_FUNC():') lines = [('Python', 'Public', 'Private', 'Internal', 'Total')] - for branch_name in iter_branches(): - public, private, internal = get_functions() + for name, data in results: + public, private, internal = data.functions total = len(public) + len(private) + len(internal) - line = [branch_name, len(public), len(private), len(internal), total] + line = [name, len(public), len(private), len(internal), total] lines.append(line) table_compute_diff(lines) @@ -175,12 +194,12 @@ def list_functions(): """) -def list_variables(): +def list_variables(results): display_title('Variables') paragraph('Symbols exported with PyAPI_DATA():') lines = [('Python', 'Public', 'Private', 'Internal', 'Total')] - for name in iter_branches(): - public, private, internal = get_variables() + for name, data in results: + public, private, internal = data.variables total = len(public) + len(private) + len(internal) line = [name, len(public), len(private), len(internal), total] lines.append(line) @@ -188,13 +207,13 @@ def list_variables(): render_table(lines) -def static_inline_func(): +def static_inline_func(results): display_title('Functions defined as macros and static inline functions') paragraph('Functions defined as macros (only public) and static inline functions (public or private):') lines = [('Python', 'Macro', 'Static inline', 'Total')] - for name in iter_branches(): - macros, static_inline = get_macros_static_inline_funcs() + for name, data in results: + macros, static_inline = data.macro_static_inline_funcs line = [name, len(macros), len(static_inline), len(macros) + len(static_inline)] @@ -203,13 +222,13 @@ def static_inline_func(): render_table(lines) -def structures(): +def structures(results): display_title('Structures') paragraph('Structures in the Python C API:') lines = [COLUMNS] - for name in iter_branches(): - limited, cpython, internal = get_types() + for name, data in results: + limited, cpython, internal = data.types total = limited + cpython + internal line = [name, limited, cpython, internal, total] lines.append(line) @@ -220,13 +239,15 @@ def structures(): def render_page(): + results = get_data() + main_title() - line_numbers() - file_numbers() - list_functions() - list_variables() - static_inline_func() - structures() + line_numbers(results) + file_numbers(results) + list_functions(results) + list_variables(results) + static_inline_func(results) + structures(results) def main(): From 377871baa5bae6c77142f1dd936c5f333eedff6e Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sun, 2 Jul 2023 18:07:16 +0200 Subject: [PATCH 082/136] git clone is more verbose --- doc/pythoncapi.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index c169baf..89cc856 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -43,9 +43,10 @@ def git_clone(): if os.path.exists(GIT_DIR): return + print(f"Clone CPython Git repository: {CPYTHON_URL}") dst_name = os.path.basename(GIT_DIR) cmd = ['git', 'clone', CPYTHON_URL, dst_name] - run_command(cwd=os.path.dirname(GIT_DIR)) + run_command(cmd, cwd=os.path.dirname(GIT_DIR)) _CLEANED = False From ec6d9f6beb5c9ceca8decf8c8fd2ee0f8f2e17bc Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sun, 2 Jul 2023 18:09:34 +0200 Subject: [PATCH 083/136] Update stats --- doc/stats.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 67ad7aa..a4fbc4c 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 10,408 (51%) 4,584 (22%) 5,519 (27%) 20,511 3.11.0 9,232 (37%) 5,495 (22%) 10,047 (41%) 24,774 3.12 (dev) 9,598 (29%) 5,903 (18%) 17,631 (53%) 33,132 -main (3.13) 9,416 (28%) 5,455 (16%) 18,204 (55%) 33,075 +main (3.13) 9,344 (28%) 5,421 (16%) 18,299 (55%) 33,064 =========== ============= =========== ============ ====== File Numbers @@ -37,7 +37,7 @@ Python Limited API CPython API Internal API Total 3.10.0 81 (-17) 32 (+8) 48 (+15) 161 (+6) 3.11.0 72 (-9) 48 (+16) 68 (+20) 188 (+27) 3.12 (dev) 72 (same) 49 (+1) 91 (+23) 212 (+24) -main (3.13) 73 (+1) 50 (+1) 93 (+2) 216 (+4) +main (3.13) 72 (same) 51 (+2) 93 (+2) 216 (+4) =========== =========== =========== ============ ========= Functions @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 914 (+29) 360 (-52) 176 (+61) 1,450 (+38) 3.12 (dev) 951 (+37) 385 (+25) 177 (+1) 1,513 (+63) -main (3.13) 935 (-16) 321 (-64) 202 (+25) 1,458 (-55) +main (3.13) 935 (-16) 300 (-85) 207 (+30) 1,442 (-71) =========== ========= ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From 8e13abf520059470ba4d16679bc5b65de563a88b Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sun, 2 Jul 2023 18:09:51 +0200 Subject: [PATCH 084/136] .gitignore: ignore cpython_git/ directory --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index d2161d4..2eddfd9 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ doc/build/ __pycache__ +cpython_git/ From 2b6dbf81fbc7b5b67d2126b19a394902227fa9ce Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sun, 2 Jul 2023 21:54:28 +0200 Subject: [PATCH 085/136] Enhance stats code --- doc/pythoncapi.py | 5 +++-- doc/stats.py | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index 89cc856..0908e76 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -33,8 +33,8 @@ def run_command(cmd, cwd): subprocess.run(cmd, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, check=True, cwd=cwd) @@ -71,6 +71,7 @@ def git_switch_branch(branch): global _FETCHED if not _FETCHED: + print(f"Update the CPython Git repository (git fetch)") cmd = ['git', 'fetch'] run_command(cmd, cwd=GIT_DIR) _FETCHED = True diff --git a/doc/stats.py b/doc/stats.py index 3988426..ce604dc 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -61,8 +61,8 @@ def _get_data(): def get_data(): result = [] for branch, version in BRANCHES: - print(f"Parse {branch} header files") git_switch_branch(branch) + print(f"Parse {branch} header files") os.chdir(GIT_DIR) data = _get_data() result.append((version, data)) From abc7500a2933426beb9ae7125b6c2b7582d0a464 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 3 Jul 2023 12:19:33 +0200 Subject: [PATCH 086/136] Update stats --- doc/stats.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index a4fbc4c..626e7ab 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 10,408 (51%) 4,584 (22%) 5,519 (27%) 20,511 3.11.0 9,232 (37%) 5,495 (22%) 10,047 (41%) 24,774 3.12 (dev) 9,598 (29%) 5,903 (18%) 17,631 (53%) 33,132 -main (3.13) 9,344 (28%) 5,421 (16%) 18,299 (55%) 33,064 +main (3.13) 9,344 (28%) 5,249 (16%) 18,506 (56%) 33,099 =========== ============= =========== ============ ====== File Numbers @@ -37,7 +37,7 @@ Python Limited API CPython API Internal API Total 3.10.0 81 (-17) 32 (+8) 48 (+15) 161 (+6) 3.11.0 72 (-9) 48 (+16) 68 (+20) 188 (+27) 3.12 (dev) 72 (same) 49 (+1) 91 (+23) 212 (+24) -main (3.13) 72 (same) 51 (+2) 93 (+2) 216 (+4) +main (3.13) 72 (same) 51 (+2) 95 (+4) 218 (+6) =========== =========== =========== ============ ========= Functions @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 914 (+29) 360 (-52) 176 (+61) 1,450 (+38) 3.12 (dev) 951 (+37) 385 (+25) 177 (+1) 1,513 (+63) -main (3.13) 935 (-16) 300 (-85) 207 (+30) 1,442 (-71) +main (3.13) 935 (-16) 278 (-107) 226 (+49) 1,439 (-74) =========== ========= ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -120,7 +120,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) -main (3.13) 18 (same) 87 (+3) 172 (+2) 277 (+5) +main (3.13) 18 (same) 86 (+2) 173 (+3) 277 (+5) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From fe97e22b4e0ac61503ddd013f7aa774b2739b6ad Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 4 Jul 2023 00:15:30 +0200 Subject: [PATCH 087/136] Update stats --- doc/stats.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 626e7ab..7afc551 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 10,408 (51%) 4,584 (22%) 5,519 (27%) 20,511 3.11.0 9,232 (37%) 5,495 (22%) 10,047 (41%) 24,774 3.12 (dev) 9,598 (29%) 5,903 (18%) 17,631 (53%) 33,132 -main (3.13) 9,344 (28%) 5,249 (16%) 18,506 (56%) 33,099 +main (3.13) 9,344 (28%) 5,207 (16%) 18,556 (56%) 33,107 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 914 (+29) 360 (-52) 176 (+61) 1,450 (+38) 3.12 (dev) 951 (+37) 385 (+25) 177 (+1) 1,513 (+63) -main (3.13) 935 (-16) 278 (-107) 226 (+49) 1,439 (-74) +main (3.13) 935 (-16) 268 (-117) 236 (+59) 1,439 (-74) =========== ========= ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From 1e8d6c47471a93ccde200cde035e70fca9fa28b2 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 4 Jul 2023 00:42:25 +0200 Subject: [PATCH 088/136] Exclude pthread_stubs.h --- doc/pythoncapi.py | 38 ++++++++++++++++++++++++++------------ doc/stats.rst | 46 +++++++++++++++++++++++----------------------- 2 files changed, 49 insertions(+), 35 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index 0908e76..eb8338e 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -86,13 +86,16 @@ def git_switch_branch(branch): def list_files(path): if not os.path.exists(path): return [] - return glob.glob(os.path.join(path, '*.h')) + files = glob.glob(os.path.join(path, '*.h')) + # Don't parse pthread_stubs.h: special header file used by WASM + for index, name in enumerate(files): + if os.path.basename(name) == 'pthread_stubs.h': + del files[index] + break + return files def _get_types(filename, names): - if 'pthread_stubs.h' in filename: - # skip special header file used by WASM - return if os.path.basename(filename) == 'pystats.h': # skip Include/pystats.h which is code only used if Python is built # with --enable-pystats (if the Py_STATS macro is defined) @@ -105,16 +108,19 @@ def _get_types(filename, names): struct_name = match.group(1) match2 = RE_STRUCT_END.search(content, match.end()) if not match2: - raise Exception(f"cannot find end of: {filename}: {match.group()}") + raise Exception(f"{filename}: cannot find end of: {match.group()}") name = match2.group(1) if not name: name = struct_name if not name: - raise Exception(f"structure has no name: {filename}: {match.group()})") + raise Exception(f"{filename}: structure has no name: {match.group()})") if name in TYPEDEFS: name = TYPEDEFS[name] names.add(name) + if 'pthread_mutex_t' in names: + raise Exception('pthread_stubs.h was parsed') + def get_types_path(directory): names = set() @@ -170,6 +176,10 @@ def get_macros_static_inline_funcs(): return (macros, funcs) +def is_function_public(name): + return name.startswith(('Py', 'PY')) + + def get_functions(): regex = re.compile( # 'PyAPI_FUNC(int) ' @@ -188,13 +198,17 @@ def get(path): cpython = get(PATH_CPYTHON_API) internal = get(PATH_INTERNAL_API) + for names in (limited, cpython, internal): + if 'pthread_create' in names: + raise Exception('pthread_stubs.h was parsed') + public = set() private = set() for name in limited | cpython: - if name.startswith('_Py'): - private.add(name) - else: + if is_function_public(name): public.add(name) + else: + private.add(name) return (public, private, internal) @@ -255,10 +269,10 @@ def get(path): public = set() private = set() for name in limited | cpython: - if name.startswith('_Py'): - private.add(name) - else: + if is_function_public(name): public.add(name) + else: + private.add(name) return (public, private, internal) diff --git a/doc/stats.rst b/doc/stats.rst index 7afc551..9760f52 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -16,9 +16,9 @@ Python Limited API CPython API Internal API Total 3.8.0 13,147 (70%) 3,418 (18%) 2,207 (12%) 18,772 3.9.0 12,243 (62%) 4,359 (22%) 3,084 (16%) 19,686 3.10.0 10,408 (51%) 4,584 (22%) 5,519 (27%) 20,511 -3.11.0 9,232 (37%) 5,495 (22%) 10,047 (41%) 24,774 -3.12 (dev) 9,598 (29%) 5,903 (18%) 17,631 (53%) 33,132 -main (3.13) 9,344 (28%) 5,207 (16%) 18,556 (56%) 33,107 +3.11.0 9,232 (37%) 5,407 (22%) 10,047 (41%) 24,686 +3.12 (dev) 9,598 (29%) 5,815 (18%) 17,631 (53%) 33,044 +main (3.13) 9,344 (28%) 5,050 (15%) 18,624 (56%) 33,018 =========== ============= =========== ============ ====== File Numbers @@ -35,9 +35,9 @@ Python Limited API CPython API Internal API Total 3.8.0 97 (-2) 15 (+15) 21 (+10) 133 (+23) 3.9.0 98 (+1) 24 (+9) 33 (+12) 155 (+22) 3.10.0 81 (-17) 32 (+8) 48 (+15) 161 (+6) -3.11.0 72 (-9) 48 (+16) 68 (+20) 188 (+27) -3.12 (dev) 72 (same) 49 (+1) 91 (+23) 212 (+24) -main (3.13) 72 (same) 51 (+2) 95 (+4) 218 (+6) +3.11.0 72 (-9) 47 (+15) 68 (+20) 187 (+26) +3.12 (dev) 72 (same) 48 (+1) 91 (+23) 211 (+24) +main (3.13) 72 (same) 50 (+2) 95 (+4) 217 (+6) =========== =========== =========== ============ ========= Functions @@ -45,19 +45,19 @@ Functions Functions exported with PyAPI_FUNC(): -=========== ========= ========== ========= ============ -Python Public Private Internal Total -=========== ========= ========== ========= ============ -2.7.0 748 150 0 898 -3.6.0 847 (+99) 380 (+230) 0 (same) 1,227 (+329) -3.7.0 872 (+25) 448 (+68) 12 (+12) 1,332 (+105) -3.8.0 910 (+38) 429 (-19) 78 (+66) 1,417 (+85) -3.9.0 919 (+9) 412 (-17) 109 (+31) 1,440 (+23) -3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) -3.11.0 914 (+29) 360 (-52) 176 (+61) 1,450 (+38) -3.12 (dev) 951 (+37) 385 (+25) 177 (+1) 1,513 (+63) -main (3.13) 935 (-16) 268 (-117) 236 (+59) 1,439 (-74) -=========== ========= ========== ========= ============ +=========== ========== ========== ========= ============ +Python Public Private Internal Total +=========== ========== ========== ========= ============ +2.7.0 738 160 0 898 +3.6.0 844 (+106) 383 (+223) 0 (same) 1,227 (+329) +3.7.0 869 (+25) 451 (+68) 12 (+12) 1,332 (+105) +3.8.0 909 (+40) 430 (-21) 78 (+66) 1,417 (+85) +3.9.0 919 (+10) 412 (-18) 109 (+31) 1,440 (+23) +3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) +3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) +3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) +main (3.13) 912 (-16) 259 (-126) 237 (+60) 1,408 (-82) +=========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid exporting symbols which are not **explicitly** exported. @@ -74,15 +74,15 @@ Symbols exported with PyAPI_DATA(): =========== ========== ======== ========= ========== Python Public Private Internal Total =========== ========== ======== ========= ========== -2.7.0 130 29 0 159 -3.6.0 167 (+37) 42 (+13) 0 (same) 209 (+50) +2.7.0 129 30 0 159 +3.6.0 167 (+38) 42 (+12) 0 (same) 209 (+50) 3.7.0 170 (+3) 35 (-7) 9 (+9) 214 (+5) 3.8.0 174 (+4) 37 (+2) 12 (+3) 223 (+9) 3.9.0 173 (-1) 35 (-2) 14 (+2) 222 (-1) 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) -3.12 (dev) 178 (+1) 29 (-3) 12 (+2) 219 (same) -main (3.13) 178 (same) 30 (+1) 12 (same) 220 (+1) +3.12 (dev) 176 (-1) 31 (-1) 12 (+2) 219 (same) +main (3.13) 176 (same) 32 (+1) 12 (same) 220 (+1) =========== ========== ======== ========= ========== Functions defined as macros and static inline functions From fb7434f3f9b3d22a1b3bb5cca0f1054600812dbe Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 4 Jul 2023 00:48:45 +0200 Subject: [PATCH 089/136] Ignore "#define PyAPI_FUNC" --- doc/pythoncapi.py | 2 ++ doc/stats.rst | 10 +++++----- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index eb8338e..76dcc00 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -182,6 +182,8 @@ def is_function_public(name): def get_functions(): regex = re.compile( + # Ignore "#define PyAPI_FUNC(RTYPE) ..." (pyport.h) + fr'(? Date: Tue, 4 Jul 2023 01:15:01 +0200 Subject: [PATCH 090/136] get_types() return names, not int --- doc/pythoncapi.py | 6 +++--- doc/stats.py | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index 76dcc00..020c0dd 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -130,9 +130,9 @@ def get_types_path(directory): def get_types(): - limited = len(get_types_path(PATH_LIMITED_API)) - cpython = len(get_types_path(PATH_CPYTHON_API)) - internal = len(get_types_path(PATH_INTERNAL_API)) + limited = get_types_path(PATH_LIMITED_API) + cpython = get_types_path(PATH_CPYTHON_API) + internal = get_types_path(PATH_INTERNAL_API) return (limited, cpython, internal) diff --git a/doc/stats.py b/doc/stats.py index ce604dc..c6653ae 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -229,8 +229,8 @@ def structures(results): lines = [COLUMNS] for name, data in results: limited, cpython, internal = data.types - total = limited + cpython + internal - line = [name, limited, cpython, internal, total] + total = len(limited) + len(cpython) + len(internal) + line = [name, len(limited), len(cpython), len(internal), total] lines.append(line) table_compute_diff(lines) render_table(lines) From f34eab1912f4e6837de57090d81069a5e304734b Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 4 Jul 2023 09:47:12 +0200 Subject: [PATCH 091/136] Update stats --- doc/stats.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 2b247f3..b012aff 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 10,408 (51%) 4,584 (22%) 5,519 (27%) 20,511 3.11.0 9,232 (37%) 5,407 (22%) 10,047 (41%) 24,686 3.12 (dev) 9,598 (29%) 5,815 (18%) 17,631 (53%) 33,044 -main (3.13) 9,344 (28%) 5,050 (15%) 18,624 (56%) 33,018 +main (3.13) 9,344 (28%) 4,924 (15%) 18,742 (57%) 33,010 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 912 (-16) 259 (-126) 237 (+60) 1,408 (-82) +main (3.13) 912 (-16) 241 (-144) 246 (+69) 1,399 (-91) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -82,7 +82,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12 (dev) 176 (-1) 31 (-1) 12 (+2) 219 (same) -main (3.13) 176 (same) 32 (+1) 12 (same) 220 (+1) +main (3.13) 176 (same) 30 (-1) 12 (same) 218 (-1) =========== ========== ======== ========= ========== Functions defined as macros and static inline functions From bb4317a80cdedb1b750e6f457facf8988c2cfd80 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 4 Jul 2023 10:38:53 +0200 Subject: [PATCH 092/136] Update stats --- doc/stats.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index b012aff..8f24c17 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 10,408 (51%) 4,584 (22%) 5,519 (27%) 20,511 3.11.0 9,232 (37%) 5,407 (22%) 10,047 (41%) 24,686 3.12 (dev) 9,598 (29%) 5,815 (18%) 17,631 (53%) 33,044 -main (3.13) 9,344 (28%) 4,924 (15%) 18,742 (57%) 33,010 +main (3.13) 9,344 (28%) 4,844 (15%) 18,824 (57%) 33,012 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 912 (-16) 241 (-144) 246 (+69) 1,399 (-91) +main (3.13) 912 (-16) 234 (-151) 253 (+76) 1,399 (-91) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -120,7 +120,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) -main (3.13) 18 (same) 86 (+2) 173 (+3) 277 (+5) +main (3.13) 18 (same) 85 (+1) 174 (+4) 277 (+5) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From a736accd67d4acf615ee5a1d432b2eb819603551 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 4 Jul 2023 11:13:31 +0200 Subject: [PATCH 093/136] Update stats --- doc/stats.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 8f24c17..f4cae3e 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 10,408 (51%) 4,584 (22%) 5,519 (27%) 20,511 3.11.0 9,232 (37%) 5,407 (22%) 10,047 (41%) 24,686 3.12 (dev) 9,598 (29%) 5,815 (18%) 17,631 (53%) 33,044 -main (3.13) 9,344 (28%) 4,844 (15%) 18,824 (57%) 33,012 +main (3.13) 9,344 (28%) 4,807 (15%) 18,859 (57%) 33,010 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 912 (-16) 234 (-151) 253 (+76) 1,399 (-91) +main (3.13) 912 (-16) 225 (-160) 258 (+81) 1,395 (-95) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From 26a03d44e74fbedfa4adb5ade02523bc8e87f0c3 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 4 Jul 2023 13:09:31 +0200 Subject: [PATCH 094/136] Update stats --- doc/stats.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index f4cae3e..3817cf2 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 10,408 (51%) 4,584 (22%) 5,519 (27%) 20,511 3.11.0 9,232 (37%) 5,407 (22%) 10,047 (41%) 24,686 3.12 (dev) 9,598 (29%) 5,815 (18%) 17,631 (53%) 33,044 -main (3.13) 9,344 (28%) 4,807 (15%) 18,859 (57%) 33,010 +main (3.13) 9,344 (28%) 4,784 (14%) 18,885 (57%) 33,013 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 912 (-16) 225 (-160) 258 (+81) 1,395 (-95) +main (3.13) 912 (-16) 212 (-173) 252 (+75) 1,376 (-114) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From 9b1bd4dbfd6113b6c68ebb42f8b11b405e1d940a Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 4 Jul 2023 18:56:20 +0200 Subject: [PATCH 095/136] Update stats --- doc/stats.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 3817cf2..7c30ef3 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 10,408 (51%) 4,584 (22%) 5,519 (27%) 20,511 3.11.0 9,232 (37%) 5,407 (22%) 10,047 (41%) 24,686 3.12 (dev) 9,598 (29%) 5,815 (18%) 17,631 (53%) 33,044 -main (3.13) 9,344 (28%) 4,784 (14%) 18,885 (57%) 33,013 +main (3.13) 9,344 (28%) 4,754 (14%) 18,920 (57%) 33,018 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 912 (-16) 212 (-173) 252 (+75) 1,376 (-114) +main (3.13) 913 (-15) 211 (-174) 252 (+75) 1,376 (-114) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From e4092a6525e0f1c826af3f80653ecc31090c9c00 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 12 Jul 2023 18:16:41 +0200 Subject: [PATCH 096/136] Add test scripts --- doc/funcs.py | 11 +++++++++++ doc/types.py | 11 +++++++++++ 2 files changed, 22 insertions(+) create mode 100755 doc/funcs.py create mode 100755 doc/types.py diff --git a/doc/funcs.py b/doc/funcs.py new file mode 100755 index 0000000..df66e5a --- /dev/null +++ b/doc/funcs.py @@ -0,0 +1,11 @@ +#!/usr/bin/python3 +import os +import subprocess +from pythoncapi import get_functions + +os.chdir('/home/vstinner/python/3.12') +#os.chdir('/home/vstinner/python/2.7') +#os.chdir('/home/vstinner/python/main') +public, private, internal = get_functions() +for name in sorted(private): + print(name) diff --git a/doc/types.py b/doc/types.py new file mode 100755 index 0000000..3bff6ad --- /dev/null +++ b/doc/types.py @@ -0,0 +1,11 @@ +#!/usr/bin/python3 +import os +import subprocess +from pythoncapi import get_types + +#os.chdir('/home/vstinner/python/3.12') +#os.chdir('/home/vstinner/python/2.7') +os.chdir('/home/vstinner/python/main') +public, private, internal = get_types() +for name in sorted(private): + print(name) From b452d3d30fca45e9604d00229d589e2c32efd225 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 17 Jul 2023 23:52:58 +0200 Subject: [PATCH 097/136] Ignore private static inline functions --- doc/pythoncapi.py | 19 +++++++++++++++---- doc/stats.py | 5 ++++- doc/stats.rst | 30 ++++++++++++++++-------------- 3 files changed, 35 insertions(+), 19 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index 020c0dd..40b7c49 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -31,6 +31,9 @@ } +PUBLIC_NAME_PREFIX = ("Py", "PY") + + def run_command(cmd, cwd): subprocess.run(cmd, stdout=subprocess.PIPE, @@ -145,6 +148,10 @@ def grep(regex, filenames, group=0): yield match.group(group) +def is_function_public(name): + return name.startswith(PUBLIC_NAME_PREFIX) + + def get_macros_static_inline_funcs(): files = list_files(PATH_LIMITED_API) + list_files(PATH_CPYTHON_API) @@ -173,11 +180,15 @@ def get_macros_static_inline_funcs(): if name.startswith("PyDTrace_"): funcs.discard(name) - return (macros, funcs) - + # Remove private static inline functions + for name in list(macros): + if not is_function_public(name): + macros.discard(name) + for name in list(funcs): + if not is_function_public(name): + funcs.discard(name) -def is_function_public(name): - return name.startswith(('Py', 'PY')) + return (macros, funcs) def get_functions(): diff --git a/doc/stats.py b/doc/stats.py index c6653ae..cbc0dd0 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -209,7 +209,7 @@ def list_variables(results): def static_inline_func(results): display_title('Functions defined as macros and static inline functions') - paragraph('Functions defined as macros (only public) and static inline functions (public or private):') + paragraph('Functions defined as macros and static inline functions:') lines = [('Python', 'Macro', 'Static inline', 'Total')] for name, data in results: @@ -221,6 +221,9 @@ def static_inline_func(results): table_compute_diff(lines) render_table(lines) + paragraph('Only count public macros and public static inline functions ' + '(name starting with "Py" or "PY").') + def structures(results): display_title('Structures') diff --git a/doc/stats.rst b/doc/stats.rst index 7c30ef3..d2d12ea 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 10,408 (51%) 4,584 (22%) 5,519 (27%) 20,511 3.11.0 9,232 (37%) 5,407 (22%) 10,047 (41%) 24,686 3.12 (dev) 9,598 (29%) 5,815 (18%) 17,631 (53%) 33,044 -main (3.13) 9,344 (28%) 4,754 (14%) 18,920 (57%) 33,018 +main (3.13) 9,397 (27%) 4,750 (14%) 20,455 (59%) 34,602 =========== ============= =========== ============ ====== File Numbers @@ -37,7 +37,7 @@ Python Limited API CPython API Internal API Total 3.10.0 81 (-17) 32 (+8) 48 (+15) 161 (+6) 3.11.0 72 (-9) 47 (+15) 68 (+20) 187 (+26) 3.12 (dev) 72 (same) 48 (+1) 91 (+23) 211 (+24) -main (3.13) 72 (same) 50 (+2) 95 (+4) 217 (+6) +main (3.13) 72 (same) 50 (+2) 96 (+5) 218 (+7) =========== =========== =========== ============ ========= Functions @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 913 (-15) 211 (-174) 252 (+75) 1,376 (-114) +main (3.13) 925 (-3) 209 (-176) 253 (+76) 1,387 (-103) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -88,21 +88,23 @@ main (3.13) 176 (same) 30 (-1) 12 (same) 218 (-1) Functions defined as macros and static inline functions ======================================================= -Functions defined as macros (only public) and static inline functions (public or private): +Functions defined as macros and static inline functions: -=========== ========== ============= ========= +=========== ========== ============= ========== Python Macro Static inline Total -=========== ========== ============= ========= +=========== ========== ============= ========== 2.7.0 274 0 274 3.6.0 306 (+32) 0 (same) 306 (+32) 3.7.0 315 (+9) 0 (same) 315 (+9) -3.8.0 314 (-1) 7 (+7) 321 (+6) -3.9.0 307 (-7) 11 (+4) 318 (-3) -3.10.0 307 (same) 18 (+7) 325 (+7) -3.11.0 269 (-38) 46 (+28) 315 (-10) -3.12 (dev) 251 (-18) 71 (+25) 322 (+7) -main (3.13) 247 (-4) 70 (-1) 317 (-5) -=========== ========== ============= ========= +3.8.0 314 (-1) 0 (same) 314 (-1) +3.9.0 307 (-7) 0 (same) 307 (-7) +3.10.0 307 (same) 0 (same) 307 (same) +3.11.0 269 (-38) 42 (+42) 311 (+4) +3.12 (dev) 251 (-18) 64 (+22) 315 (+4) +main (3.13) 246 (-5) 63 (-1) 309 (-6) +=========== ========== ============= ========== + +Only count public macros and public static inline functions (name starting with "Py" or "PY"). Structures ========== @@ -120,7 +122,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) -main (3.13) 18 (same) 85 (+1) 174 (+4) 277 (+5) +main (3.13) 18 (same) 85 (+1) 177 (+7) 280 (+8) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From 3d7810e05dbc4bc4b371b84557fdddc859738a58 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 19 Jul 2023 01:13:32 +0200 Subject: [PATCH 098/136] Count private macros --- doc/pythoncapi.py | 27 ++++++++++++++------ doc/stats.py | 27 +++++++++++++++----- doc/stats.rst | 63 +++++++++++++++++++++++++++++++---------------- 3 files changed, 83 insertions(+), 34 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index 40b7c49..b7ed065 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -4,6 +4,14 @@ import subprocess +EXCLUDE_HEADERS = frozenset(( + # Don't parse pthread_stubs.h: special header file used by WASM + 'pthread_stubs.h', + # Don't parse dynamic_annotations.h: not included by Python.h. + 'dynamic_annotations.h', +)) + + # Checkout of Python Git repository CPYTHON_URL = 'https://github.com/python/cpython' GIT_DIR = os.path.normpath(os.path.join(os.path.dirname(__file__), '..', 'cpython_git')) @@ -90,9 +98,8 @@ def list_files(path): if not os.path.exists(path): return [] files = glob.glob(os.path.join(path, '*.h')) - # Don't parse pthread_stubs.h: special header file used by WASM for index, name in enumerate(files): - if os.path.basename(name) == 'pthread_stubs.h': + if os.path.basename(name) in EXCLUDE_HEADERS: del files[index] break return files @@ -157,7 +164,7 @@ def get_macros_static_inline_funcs(): # Match '#define func(' # Don't match '#define constant (&obj)': space before '(' - regex = re.compile(fr'^ *# *define (P[Yy][A-Za-z_]+)\(', re.MULTILINE) + regex = re.compile(fr'^ *# *define (_?P[Yy][A-Za-z_]+)\(', re.MULTILINE) macros = set(grep(regex, files, group=1)) regex = re.compile(fr'^static inline [^(\n]+ ({RE_IDENTIFIER}) *\(', re.MULTILINE) @@ -166,12 +173,14 @@ def get_macros_static_inline_funcs(): # Remove macros only used to cast arguments types. Like: # "static inline void Py_INCREF(...) { ...}" # "#define Py_INCREF(obj) Py_INCREF(_PyObject_CAST(obj))" - # Only count 1 static inline function, ignore the macro. + # Only count the static inline function, ignore the macro. macros = macros - funcs + # In Python 3.10, the Py_INCREF() was wrapping the _Py_INCREF() static + # inline function. + # In Python 3.11, Py_NewRef() macro just calls _Py_NewRef() static inline + # function. for name in list(macros): - # In Python 3.10, the Py_INCREF() was wrapping the _Py_INCREF() static - # inline function. if f"_{name}" in funcs: macros.discard(name) @@ -181,14 +190,18 @@ def get_macros_static_inline_funcs(): funcs.discard(name) # Remove private static inline functions + private_macros = set() + private_funcs = set() for name in list(macros): if not is_function_public(name): macros.discard(name) + private_macros.add(name) for name in list(funcs): if not is_function_public(name): funcs.discard(name) + private_funcs.add(name) - return (macros, funcs) + return (macros, funcs, private_macros, private_funcs) def get_functions(): diff --git a/doc/stats.py b/doc/stats.py index cbc0dd0..3808521 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -207,13 +207,20 @@ def list_variables(results): render_table(lines) -def static_inline_func(results): - display_title('Functions defined as macros and static inline functions') - paragraph('Functions defined as macros and static inline functions:') +def _static_inline_func(results, public): + if public: + display_title('Public functions defined as macros and static inline functions') + paragraph('Public functions defined as macros and static inline functions:') + else: + display_title('Private functions defined as macros and static inline functions') + paragraph('Private functions defined as macros and static inline functions:') lines = [('Python', 'Macro', 'Static inline', 'Total')] for name, data in results: - macros, static_inline = data.macro_static_inline_funcs + if public: + macros, static_inline = data.macro_static_inline_funcs[:2] + else: + macros, static_inline = data.macro_static_inline_funcs[2:] line = [name, len(macros), len(static_inline), len(macros) + len(static_inline)] @@ -221,8 +228,16 @@ def static_inline_func(results): table_compute_diff(lines) render_table(lines) - paragraph('Only count public macros and public static inline functions ' - '(name starting with "Py" or "PY").') + if public: + paragraph('Only count public macros and public static inline functions ' + '(names starting with "Py" or "PY").') + else: + paragraph('Only count private macros and public static inline functions ' + '(ignore names starting with "Py" or "PY").') + +def static_inline_func(results): + _static_inline_func(results, True) + _static_inline_func(results, False) def structures(results): diff --git a/doc/stats.rst b/doc/stats.rst index d2d12ea..d350bc6 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -11,14 +11,14 @@ Number of C API line numbers per Python version: Python Limited API CPython API Internal API Total =========== ============= =========== ============ ====== 2.7.0 12,392 (100%) 0 (0%) 0 (0%) 12,392 -3.6.0 15,829 (100%) 0 (0%) 0 (0%) 15,829 -3.7.0 16,442 (96%) 0 (0%) 678 (4%) 17,120 -3.8.0 13,147 (70%) 3,418 (18%) 2,207 (12%) 18,772 -3.9.0 12,243 (62%) 4,359 (22%) 3,084 (16%) 19,686 -3.10.0 10,408 (51%) 4,584 (22%) 5,519 (27%) 20,511 -3.11.0 9,232 (37%) 5,407 (22%) 10,047 (41%) 24,686 -3.12 (dev) 9,598 (29%) 5,815 (18%) 17,631 (53%) 33,044 -main (3.13) 9,397 (27%) 4,750 (14%) 20,455 (59%) 34,602 +3.6.0 15,330 (100%) 0 (0%) 0 (0%) 15,330 +3.7.0 15,943 (96%) 0 (0%) 678 (4%) 16,621 +3.8.0 12,648 (69%) 3,418 (19%) 2,207 (12%) 18,273 +3.9.0 11,744 (61%) 4,359 (23%) 3,084 (16%) 19,187 +3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 +3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 +3.12 (dev) 9,099 (28%) 5,815 (18%) 17,631 (54%) 32,545 +main (3.13) 8,904 (26%) 4,754 (14%) 20,459 (60%) 34,117 =========== ============= =========== ============ ====== File Numbers @@ -30,14 +30,14 @@ Number of header file numbers per Python version: Python Limited API CPython API Internal API Total =========== =========== =========== ============ ========= 2.7.0 91 0 0 91 -3.6.0 99 (+8) 0 (same) 0 (same) 99 (+8) -3.7.0 99 (same) 0 (same) 11 (+11) 110 (+11) -3.8.0 97 (-2) 15 (+15) 21 (+10) 133 (+23) -3.9.0 98 (+1) 24 (+9) 33 (+12) 155 (+22) -3.10.0 81 (-17) 32 (+8) 48 (+15) 161 (+6) -3.11.0 72 (-9) 47 (+15) 68 (+20) 187 (+26) -3.12 (dev) 72 (same) 48 (+1) 91 (+23) 211 (+24) -main (3.13) 72 (same) 50 (+2) 96 (+5) 218 (+7) +3.6.0 98 (+7) 0 (same) 0 (same) 98 (+7) +3.7.0 98 (same) 0 (same) 11 (+11) 109 (+11) +3.8.0 96 (-2) 15 (+15) 21 (+10) 132 (+23) +3.9.0 97 (+1) 24 (+9) 33 (+12) 154 (+22) +3.10.0 80 (-17) 32 (+8) 48 (+15) 160 (+6) +3.11.0 71 (-9) 47 (+15) 68 (+20) 186 (+26) +3.12 (dev) 71 (same) 48 (+1) 91 (+23) 210 (+24) +main (3.13) 71 (same) 50 (+2) 96 (+5) 217 (+7) =========== =========== =========== ============ ========= Functions @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 925 (-3) 209 (-176) 253 (+76) 1,387 (-103) +main (3.13) 929 (+1) 209 (-176) 253 (+76) 1,391 (-99) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -85,10 +85,10 @@ Python Public Private Internal Total main (3.13) 176 (same) 30 (-1) 12 (same) 218 (-1) =========== ========== ======== ========= ========== -Functions defined as macros and static inline functions -======================================================= +Public functions defined as macros and static inline functions +============================================================== -Functions defined as macros and static inline functions: +Public functions defined as macros and static inline functions: =========== ========== ============= ========== Python Macro Static inline Total @@ -104,7 +104,28 @@ Python Macro Static inline Total main (3.13) 246 (-5) 63 (-1) 309 (-6) =========== ========== ============= ========== -Only count public macros and public static inline functions (name starting with "Py" or "PY"). +Only count public macros and public static inline functions (names starting with "Py" or "PY"). + +Private functions defined as macros and static inline functions +=============================================================== + +Private functions defined as macros and static inline functions: + +=========== ======== ============= ======== +Python Macro Static inline Total +=========== ======== ============= ======== +2.7.0 21 0 21 +3.6.0 60 (+39) 0 (same) 60 (+39) +3.7.0 70 (+10) 0 (same) 70 (+10) +3.8.0 57 (-13) 7 (+7) 64 (-6) +3.9.0 43 (-14) 11 (+4) 54 (-10) +3.10.0 44 (+1) 18 (+7) 62 (+8) +3.11.0 53 (+9) 4 (-14) 57 (-5) +3.12 (dev) 66 (+13) 7 (+3) 73 (+16) +main (3.13) 61 (-5) 7 (same) 68 (-5) +=========== ======== ============= ======== + +Only count private macros and public static inline functions (ignore names starting with "Py" or "PY"). Structures ========== From be8c4439be752962a44c64c770c0de786d8f90d3 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 21 Jul 2023 23:03:34 +0200 Subject: [PATCH 099/136] macros --- doc/macros.py | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100755 doc/macros.py diff --git a/doc/macros.py b/doc/macros.py new file mode 100755 index 0000000..03f44e9 --- /dev/null +++ b/doc/macros.py @@ -0,0 +1,11 @@ +#!/usr/bin/python3 +import os +import subprocess +from pythoncapi import get_macros_static_inline_funcs + +#os.chdir('/home/vstinner/python/3.11') +#os.chdir('/home/vstinner/python/2.7') +os.chdir('/home/vstinner/python/main') +macros, funcs, private_macros, private_funcs = get_macros_static_inline_funcs() +for name in sorted(private_macros): + print(name) From 5051c32637dc54fa349c17582f4739729d60c79a Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 22 Jul 2023 17:56:17 +0200 Subject: [PATCH 100/136] Update stats --- doc/stats.rst | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index d350bc6..639ac02 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 9,099 (28%) 5,815 (18%) 17,631 (54%) 32,545 -main (3.13) 8,904 (26%) 4,754 (14%) 20,459 (60%) 34,117 +main (3.13) 8,835 (26%) 4,736 (14%) 20,596 (60%) 34,167 =========== ============= =========== ============ ====== File Numbers @@ -37,7 +37,7 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 48 (+15) 160 (+6) 3.11.0 71 (-9) 47 (+15) 68 (+20) 186 (+26) 3.12 (dev) 71 (same) 48 (+1) 91 (+23) 210 (+24) -main (3.13) 71 (same) 50 (+2) 96 (+5) 217 (+7) +main (3.13) 71 (same) 50 (+2) 97 (+6) 218 (+8) =========== =========== =========== ============ ========= Functions @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 929 (+1) 209 (-176) 253 (+76) 1,391 (-99) +main (3.13) 931 (+3) 203 (-182) 259 (+82) 1,393 (-97) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -71,19 +71,19 @@ Variables Symbols exported with PyAPI_DATA(): -=========== ========== ======== ========= ========== -Python Public Private Internal Total -=========== ========== ======== ========= ========== -2.7.0 129 30 0 159 -3.6.0 167 (+38) 42 (+12) 0 (same) 209 (+50) -3.7.0 170 (+3) 35 (-7) 9 (+9) 214 (+5) -3.8.0 174 (+4) 37 (+2) 12 (+3) 223 (+9) -3.9.0 173 (-1) 35 (-2) 14 (+2) 222 (-1) -3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) -3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) -3.12 (dev) 176 (-1) 31 (-1) 12 (+2) 219 (same) -main (3.13) 176 (same) 30 (-1) 12 (same) 218 (-1) -=========== ========== ======== ========= ========== +=========== ========== ======== ======== ========== +Python Public Private Internal Total +=========== ========== ======== ======== ========== +2.7.0 129 30 0 159 +3.6.0 167 (+38) 42 (+12) 0 (same) 209 (+50) +3.7.0 170 (+3) 35 (-7) 9 (+9) 214 (+5) +3.8.0 174 (+4) 37 (+2) 12 (+3) 223 (+9) +3.9.0 173 (-1) 35 (-2) 14 (+2) 222 (-1) +3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) +3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) +3.12 (dev) 176 (-1) 31 (-1) 12 (+2) 219 (same) +main (3.13) 176 (same) 20 (-11) 16 (+4) 212 (-7) +=========== ========== ======== ======== ========== Public functions defined as macros and static inline functions ============================================================== @@ -143,7 +143,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) -main (3.13) 18 (same) 85 (+1) 177 (+7) 280 (+8) +main (3.13) 18 (same) 85 (+1) 179 (+9) 282 (+10) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From f875cacab6bda9f605c512513164dd360808aa1a Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 22 Jul 2023 18:20:16 +0200 Subject: [PATCH 101/136] Skip pystats.h --- doc/pythoncapi.py | 8 +++----- doc/stats.rst | 8 ++++---- 2 files changed, 7 insertions(+), 9 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index b7ed065..ca217c9 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -9,6 +9,9 @@ 'pthread_stubs.h', # Don't parse dynamic_annotations.h: not included by Python.h. 'dynamic_annotations.h', + # Skip Include/pystats.h: the code is skipped unless if Python + # is built with --enable-pystats (if the Py_STATS macro is defined) + 'pystats.h', )) @@ -106,11 +109,6 @@ def list_files(path): def _get_types(filename, names): - if os.path.basename(filename) == 'pystats.h': - # skip Include/pystats.h which is code only used if Python is built - # with --enable-pystats (if the Py_STATS macro is defined) - return - with open(filename, encoding="utf-8") as fp: content = fp.read() diff --git a/doc/stats.rst b/doc/stats.rst index 639ac02..cd26578 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 9,099 (28%) 5,815 (18%) 17,631 (54%) 32,545 -main (3.13) 8,835 (26%) 4,736 (14%) 20,596 (60%) 34,167 +main (3.13) 8,835 (26%) 4,731 (14%) 20,599 (60%) 34,165 =========== ============= =========== ============ ====== File Numbers @@ -82,7 +82,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12 (dev) 176 (-1) 31 (-1) 12 (+2) 219 (same) -main (3.13) 176 (same) 20 (-11) 16 (+4) 212 (-7) +main (3.13) 176 (same) 19 (-12) 17 (+5) 212 (-7) =========== ========== ======== ======== ========== Public functions defined as macros and static inline functions @@ -142,8 +142,8 @@ Python Limited API CPython API Internal API Total 3.9.0 68 (-13) 46 (+12) 38 (+10) 152 (+9) 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) -3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) -main (3.13) 18 (same) 85 (+1) 179 (+9) 282 (+10) +3.12 (dev) 23 (+4) 84 (+9) 170 (+58) 277 (+71) +main (3.13) 23 (same) 85 (+1) 179 (+9) 287 (+10) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From 57898c58aeff65f4a95eaac6c55be6c25df658f2 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 22 Jul 2023 18:20:31 +0200 Subject: [PATCH 102/136] Add variables.py --- doc/variables.py | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100755 doc/variables.py diff --git a/doc/variables.py b/doc/variables.py new file mode 100755 index 0000000..85009bd --- /dev/null +++ b/doc/variables.py @@ -0,0 +1,9 @@ +#!/usr/bin/python3 +import os +import subprocess +from pythoncapi import get_variables + +os.chdir('/home/vstinner/python/main') +(public, private, internal) = get_variables() +for name in sorted(private): + print(name) From 084e7c1265d424d3f1ef7beb39913bae9575470c Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 22 Jul 2023 18:25:07 +0200 Subject: [PATCH 103/136] Fix exclusion list --- doc/pythoncapi.py | 7 ++----- doc/stats.rst | 20 ++++++++++---------- 2 files changed, 12 insertions(+), 15 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index ca217c9..e43de18 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -101,11 +101,8 @@ def list_files(path): if not os.path.exists(path): return [] files = glob.glob(os.path.join(path, '*.h')) - for index, name in enumerate(files): - if os.path.basename(name) in EXCLUDE_HEADERS: - del files[index] - break - return files + return [name for name in files + if os.path.basename(name) not in EXCLUDE_HEADERS] def _get_types(filename, names): diff --git a/doc/stats.rst b/doc/stats.rst index cd26578..c6deb35 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -17,8 +17,8 @@ Python Limited API CPython API Internal API Total 3.9.0 11,744 (61%) 4,359 (23%) 3,084 (16%) 19,187 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 -3.12 (dev) 9,099 (28%) 5,815 (18%) 17,631 (54%) 32,545 -main (3.13) 8,835 (26%) 4,731 (14%) 20,599 (60%) 34,165 +3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 +main (3.13) 8,721 (26%) 4,731 (14%) 20,599 (60%) 34,051 =========== ============= =========== ============ ====== File Numbers @@ -36,8 +36,8 @@ Python Limited API CPython API Internal API Total 3.9.0 97 (+1) 24 (+9) 33 (+12) 154 (+22) 3.10.0 80 (-17) 32 (+8) 48 (+15) 160 (+6) 3.11.0 71 (-9) 47 (+15) 68 (+20) 186 (+26) -3.12 (dev) 71 (same) 48 (+1) 91 (+23) 210 (+24) -main (3.13) 71 (same) 50 (+2) 97 (+6) 218 (+8) +3.12 (dev) 70 (-1) 48 (+1) 91 (+23) 209 (+23) +main (3.13) 70 (same) 50 (+2) 97 (+6) 217 (+8) =========== =========== =========== ============ ========= Functions @@ -81,8 +81,8 @@ Python Public Private Internal Total 3.9.0 173 (-1) 35 (-2) 14 (+2) 222 (-1) 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) -3.12 (dev) 176 (-1) 31 (-1) 12 (+2) 219 (same) -main (3.13) 176 (same) 19 (-12) 17 (+5) 212 (-7) +3.12 (dev) 176 (-1) 29 (-3) 12 (+2) 217 (-2) +main (3.13) 176 (same) 17 (-12) 17 (+5) 210 (-7) =========== ========== ======== ======== ========== Public functions defined as macros and static inline functions @@ -121,8 +121,8 @@ Python Macro Static inline Total 3.9.0 43 (-14) 11 (+4) 54 (-10) 3.10.0 44 (+1) 18 (+7) 62 (+8) 3.11.0 53 (+9) 4 (-14) 57 (-5) -3.12 (dev) 66 (+13) 7 (+3) 73 (+16) -main (3.13) 61 (-5) 7 (same) 68 (-5) +3.12 (dev) 64 (+11) 7 (+3) 71 (+14) +main (3.13) 59 (-5) 7 (same) 66 (-5) =========== ======== ============= ======== Only count private macros and public static inline functions (ignore names starting with "Py" or "PY"). @@ -142,8 +142,8 @@ Python Limited API CPython API Internal API Total 3.9.0 68 (-13) 46 (+12) 38 (+10) 152 (+9) 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) -3.12 (dev) 23 (+4) 84 (+9) 170 (+58) 277 (+71) -main (3.13) 23 (same) 85 (+1) 179 (+9) 287 (+10) +3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) +main (3.13) 18 (same) 85 (+1) 179 (+9) 282 (+10) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From 9d58efa3d04a1c8f65ca3017ad0a49b294a721fe Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 22 Jul 2023 23:45:38 +0200 Subject: [PATCH 104/136] Update stats --- doc/funcs.py | 4 ++-- doc/stats.rst | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/funcs.py b/doc/funcs.py index df66e5a..861ae57 100755 --- a/doc/funcs.py +++ b/doc/funcs.py @@ -3,9 +3,9 @@ import subprocess from pythoncapi import get_functions -os.chdir('/home/vstinner/python/3.12') +#os.chdir('/home/vstinner/python/3.12') #os.chdir('/home/vstinner/python/2.7') -#os.chdir('/home/vstinner/python/main') +os.chdir('/home/vstinner/python/main') public, private, internal = get_functions() for name in sorted(private): print(name) diff --git a/doc/stats.rst b/doc/stats.rst index c6deb35..03c6aac 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,721 (26%) 4,731 (14%) 20,599 (60%) 34,051 +main (3.13) 8,693 (26%) 4,712 (14%) 20,641 (61%) 34,046 =========== ============= =========== ============ ====== File Numbers @@ -37,7 +37,7 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 48 (+15) 160 (+6) 3.11.0 71 (-9) 47 (+15) 68 (+20) 186 (+26) 3.12 (dev) 70 (-1) 48 (+1) 91 (+23) 209 (+23) -main (3.13) 70 (same) 50 (+2) 97 (+6) 217 (+8) +main (3.13) 69 (-1) 49 (+1) 98 (+7) 216 (+7) =========== =========== =========== ============ ========= Functions @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 931 (+3) 203 (-182) 259 (+82) 1,393 (-97) +main (3.13) 931 (+3) 196 (-189) 262 (+85) 1,389 (-101) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -82,7 +82,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12 (dev) 176 (-1) 29 (-3) 12 (+2) 217 (-2) -main (3.13) 176 (same) 17 (-12) 17 (+5) 210 (-7) +main (3.13) 176 (same) 15 (-14) 18 (+6) 209 (-8) =========== ========== ======== ======== ========== Public functions defined as macros and static inline functions From 8c0baaa81911b6fcea9c1ec810732af811d3a03d Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sun, 23 Jul 2023 21:36:10 +0200 Subject: [PATCH 105/136] Update stats --- doc/stats.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 03c6aac..64eb1be 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,693 (26%) 4,712 (14%) 20,641 (61%) 34,046 +main (3.13) 8,685 (26%) 4,709 (14%) 20,648 (61%) 34,042 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 931 (+3) 196 (-189) 262 (+85) 1,389 (-101) +main (3.13) 931 (+3) 194 (-191) 262 (+85) 1,387 (-103) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -122,7 +122,7 @@ Python Macro Static inline Total 3.10.0 44 (+1) 18 (+7) 62 (+8) 3.11.0 53 (+9) 4 (-14) 57 (-5) 3.12 (dev) 64 (+11) 7 (+3) 71 (+14) -main (3.13) 59 (-5) 7 (same) 66 (-5) +main (3.13) 58 (-6) 7 (same) 65 (-6) =========== ======== ============= ======== Only count private macros and public static inline functions (ignore names starting with "Py" or "PY"). From dc974be461874d6c18a729c0f53b8f8b683b6bd4 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 24 Jul 2023 00:38:32 +0200 Subject: [PATCH 106/136] Update stats --- doc/stats.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 64eb1be..df348f4 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,685 (26%) 4,709 (14%) 20,648 (61%) 34,042 +main (3.13) 8,685 (26%) 4,659 (14%) 20,697 (61%) 34,041 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 931 (+3) 194 (-191) 262 (+85) 1,387 (-103) +main (3.13) 931 (+3) 180 (-205) 263 (+86) 1,374 (-116) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -82,7 +82,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12 (dev) 176 (-1) 29 (-3) 12 (+2) 217 (-2) -main (3.13) 176 (same) 15 (-14) 18 (+6) 209 (-8) +main (3.13) 176 (same) 15 (-14) 19 (+7) 210 (-7) =========== ========== ======== ======== ========== Public functions defined as macros and static inline functions From bf78e162ad6573d23571c483d43db93c91a73d99 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 24 Jul 2023 19:31:30 +0200 Subject: [PATCH 107/136] Update stats --- doc/stats.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index df348f4..22bb1bc 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,685 (26%) 4,659 (14%) 20,697 (61%) 34,041 +main (3.13) 8,691 (26%) 4,648 (14%) 20,717 (61%) 34,056 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 931 (+3) 180 (-205) 263 (+86) 1,374 (-116) +main (3.13) 931 (+3) 173 (-212) 263 (+86) 1,367 (-123) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From 84065e89f66b533d907f9a7d97cbc9235b4d9348 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 24 Jul 2023 19:42:12 +0200 Subject: [PATCH 108/136] Add .readthedocs.yaml --- .readthedocs.yaml | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 .readthedocs.yaml diff --git a/.readthedocs.yaml b/.readthedocs.yaml new file mode 100644 index 0000000..8ff5ab7 --- /dev/null +++ b/.readthedocs.yaml @@ -0,0 +1,12 @@ +# Read the Docs configuration file +# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details + +version: 2 + +build: + os: ubuntu-22.04 + tools: + python: "3.11" + +sphinx: + configuration: doc/conf.py From 4f568641231c771853e68ee748ae81d6a6c0c615 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 24 Jul 2023 21:04:40 +0200 Subject: [PATCH 109/136] Update stats --- doc/stats.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 22bb1bc..a77acf3 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,691 (26%) 4,648 (14%) 20,717 (61%) 34,056 +main (3.13) 8,691 (26%) 4,600 (14%) 20,742 (61%) 34,033 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 931 (+3) 173 (-212) 263 (+86) 1,367 (-123) +main (3.13) 931 (+3) 161 (-224) 265 (+88) 1,357 (-133) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From 791d9d2cd315a281658cc65edc9ccb0d97f4052a Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 25 Jul 2023 04:31:55 +0200 Subject: [PATCH 110/136] Update stats --- doc/stats.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index a77acf3..cc97ebd 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,691 (26%) 4,600 (14%) 20,742 (61%) 34,033 +main (3.13) 8,686 (26%) 4,600 (14%) 20,744 (61%) 34,030 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 931 (+3) 161 (-224) 265 (+88) 1,357 (-133) +main (3.13) 931 (+3) 161 (-224) 159 (-18) 1,251 (-239) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -82,7 +82,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12 (dev) 176 (-1) 29 (-3) 12 (+2) 217 (-2) -main (3.13) 176 (same) 15 (-14) 19 (+7) 210 (-7) +main (3.13) 176 (same) 15 (-14) 18 (+6) 209 (-8) =========== ========== ======== ======== ========== Public functions defined as macros and static inline functions @@ -122,7 +122,7 @@ Python Macro Static inline Total 3.10.0 44 (+1) 18 (+7) 62 (+8) 3.11.0 53 (+9) 4 (-14) 57 (-5) 3.12 (dev) 64 (+11) 7 (+3) 71 (+14) -main (3.13) 58 (-6) 7 (same) 65 (-6) +main (3.13) 56 (-8) 7 (same) 63 (-8) =========== ======== ============= ======== Only count private macros and public static inline functions (ignore names starting with "Py" or "PY"). From a02ae765de2969eb06198a24a5d897b6441879f2 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 25 Jul 2023 14:50:15 +0200 Subject: [PATCH 111/136] Update stats --- doc/stats.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index cc97ebd..7170b90 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,686 (26%) 4,600 (14%) 20,744 (61%) 34,030 +main (3.13) 8,691 (26%) 4,600 (14%) 20,764 (61%) 34,055 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 931 (+3) 161 (-224) 159 (-18) 1,251 (-239) +main (3.13) 931 (+3) 161 (-224) 160 (-17) 1,252 (-238) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -82,7 +82,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12 (dev) 176 (-1) 29 (-3) 12 (+2) 217 (-2) -main (3.13) 176 (same) 15 (-14) 18 (+6) 209 (-8) +main (3.13) 176 (same) 15 (-14) 14 (+2) 205 (-12) =========== ========== ======== ======== ========== Public functions defined as macros and static inline functions From 395ca9c47d3da97f87de6181721f3be07f6a4012 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 21 Aug 2023 19:35:50 +0200 Subject: [PATCH 112/136] Update stats --- doc/stats.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 7170b90..10d3b8a 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,691 (26%) 4,600 (14%) 20,764 (61%) 34,055 +main (3.13) 8,744 (26%) 4,599 (13%) 20,940 (61%) 34,283 =========== ============= =========== ============ ====== File Numbers @@ -37,7 +37,7 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 48 (+15) 160 (+6) 3.11.0 71 (-9) 47 (+15) 68 (+20) 186 (+26) 3.12 (dev) 70 (-1) 48 (+1) 91 (+23) 209 (+23) -main (3.13) 69 (-1) 49 (+1) 98 (+7) 216 (+7) +main (3.13) 71 (+1) 50 (+2) 98 (+7) 219 (+10) =========== =========== =========== ============ ========= Functions @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 931 (+3) 161 (-224) 160 (-17) 1,252 (-238) +main (3.13) 937 (+9) 160 (-225) 159 (-18) 1,256 (-234) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -82,7 +82,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12 (dev) 176 (-1) 29 (-3) 12 (+2) 217 (-2) -main (3.13) 176 (same) 15 (-14) 14 (+2) 205 (-12) +main (3.13) 177 (+1) 15 (-14) 13 (+1) 205 (-12) =========== ========== ======== ======== ========== Public functions defined as macros and static inline functions @@ -143,7 +143,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) -main (3.13) 18 (same) 85 (+1) 179 (+9) 282 (+10) +main (3.13) 18 (same) 84 (same) 176 (+6) 278 (+6) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From e2b770e648389dc95a53d9f76987ffe19de4a74b Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 24 Aug 2023 14:41:12 +0200 Subject: [PATCH 113/136] Update stats --- doc/stats.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 10d3b8a..2236f01 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,744 (26%) 4,599 (13%) 20,940 (61%) 34,283 +main (3.13) 8,749 (26%) 4,598 (13%) 20,930 (61%) 34,277 =========== ============= =========== ============ ====== File Numbers @@ -37,7 +37,7 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 48 (+15) 160 (+6) 3.11.0 71 (-9) 47 (+15) 68 (+20) 186 (+26) 3.12 (dev) 70 (-1) 48 (+1) 91 (+23) 209 (+23) -main (3.13) 71 (+1) 50 (+2) 98 (+7) 219 (+10) +main (3.13) 71 (+1) 50 (+2) 97 (+6) 218 (+9) =========== =========== =========== ============ ========= Functions @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 937 (+9) 160 (-225) 159 (-18) 1,256 (-234) +main (3.13) 937 (+9) 160 (-225) 158 (-19) 1,255 (-235) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From 97329ebbd01645a6bac0dc2131db752aa37b223b Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 24 Aug 2023 18:23:00 +0200 Subject: [PATCH 114/136] Update stats --- doc/funcs.py | 2 +- doc/stats.rst | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/funcs.py b/doc/funcs.py index 861ae57..0ee130f 100755 --- a/doc/funcs.py +++ b/doc/funcs.py @@ -7,5 +7,5 @@ #os.chdir('/home/vstinner/python/2.7') os.chdir('/home/vstinner/python/main') public, private, internal = get_functions() -for name in sorted(private): +for name in sorted(internal): print(name) diff --git a/doc/stats.rst b/doc/stats.rst index 2236f01..985e54f 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,749 (26%) 4,598 (13%) 20,930 (61%) 34,277 +main (3.13) 8,749 (25%) 4,599 (13%) 21,016 (61%) 34,364 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 937 (+9) 160 (-225) 158 (-19) 1,255 (-235) +main (3.13) 938 (+10) 160 (-225) 150 (-27) 1,248 (-242) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From f907ac5ba7f22deac2b0c881040f5450b3177ca1 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 25 Aug 2023 01:14:49 +0200 Subject: [PATCH 115/136] Update stats --- doc/stats.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 985e54f..4807b43 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,749 (25%) 4,599 (13%) 21,016 (61%) 34,364 +main (3.13) 8,746 (25%) 4,478 (13%) 21,177 (62%) 34,401 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 938 (+10) 160 (-225) 150 (-27) 1,248 (-242) +main (3.13) 939 (+11) 123 (-262) 165 (-12) 1,227 (-263) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -82,7 +82,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12 (dev) 176 (-1) 29 (-3) 12 (+2) 217 (-2) -main (3.13) 177 (+1) 15 (-14) 13 (+1) 205 (-12) +main (3.13) 177 (+1) 14 (-15) 19 (+7) 210 (-7) =========== ========== ======== ======== ========== Public functions defined as macros and static inline functions @@ -122,7 +122,7 @@ Python Macro Static inline Total 3.10.0 44 (+1) 18 (+7) 62 (+8) 3.11.0 53 (+9) 4 (-14) 57 (-5) 3.12 (dev) 64 (+11) 7 (+3) 71 (+14) -main (3.13) 56 (-8) 7 (same) 63 (-8) +main (3.13) 55 (-9) 7 (same) 62 (-9) =========== ======== ============= ======== Only count private macros and public static inline functions (ignore names starting with "Py" or "PY"). @@ -143,7 +143,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) -main (3.13) 18 (same) 84 (same) 176 (+6) 278 (+6) +main (3.13) 18 (same) 83 (-1) 177 (+7) 278 (+6) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From 2d1a9802ba50aa1de1b5fa87685c9f59aee1103d Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 26 Aug 2023 02:35:32 +0200 Subject: [PATCH 116/136] Update stats --- doc/stats.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 4807b43..425e586 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,746 (25%) 4,478 (13%) 21,177 (62%) 34,401 +main (3.13) 8,746 (25%) 4,468 (13%) 21,205 (62%) 34,419 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 939 (+11) 123 (-262) 165 (-12) 1,227 (-263) +main (3.13) 939 (+11) 117 (-268) 171 (-6) 1,227 (-263) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From bf40090bd364f12168fa0b3cd2363cc436a6974b Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 26 Aug 2023 03:52:06 +0200 Subject: [PATCH 117/136] Update stats --- doc/stats.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 425e586..b69a2e2 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,746 (25%) 4,468 (13%) 21,205 (62%) 34,419 +main (3.13) 8,746 (25%) 4,464 (13%) 21,218 (62%) 34,428 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 939 (+11) 117 (-268) 171 (-6) 1,227 (-263) +main (3.13) 939 (+11) 115 (-270) 173 (-4) 1,227 (-263) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From da2ea8f7d1d44c17b6d1704d2688a6899e678a32 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 26 Aug 2023 04:43:11 +0200 Subject: [PATCH 118/136] Update stats --- doc/stats.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index b69a2e2..6d62e5b 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,746 (25%) 4,464 (13%) 21,218 (62%) 34,428 +main (3.13) 8,746 (25%) 4,454 (13%) 21,225 (62%) 34,425 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 939 (+11) 115 (-270) 173 (-4) 1,227 (-263) +main (3.13) 939 (+11) 113 (-272) 175 (-2) 1,227 (-263) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From a5e0eaf6e19be681678beb3e751c09e526e39481 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 29 Aug 2023 01:43:16 +0200 Subject: [PATCH 119/136] Update stats --- doc/stats.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/stats.rst b/doc/stats.rst index 6d62e5b..12ab297 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 939 (+11) 113 (-272) 175 (-2) 1,227 (-263) +main (3.13) 941 (+13) 111 (-274) 175 (-2) 1,227 (-263) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From 7fca01b006d11f6f44840929480595441e7d6e44 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 29 Aug 2023 02:43:23 +0200 Subject: [PATCH 120/136] Update stats --- doc/stats.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 12ab297..ca97d0e 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,746 (25%) 4,454 (13%) 21,225 (62%) 34,425 +main (3.13) 8,746 (25%) 4,412 (13%) 21,282 (62%) 34,440 =========== ============= =========== ============ ====== File Numbers @@ -37,7 +37,7 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 48 (+15) 160 (+6) 3.11.0 71 (-9) 47 (+15) 68 (+20) 186 (+26) 3.12 (dev) 70 (-1) 48 (+1) 91 (+23) 209 (+23) -main (3.13) 71 (+1) 50 (+2) 97 (+6) 218 (+9) +main (3.13) 71 (+1) 50 (+2) 98 (+7) 219 (+10) =========== =========== =========== ============ ========= Functions @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 941 (+13) 111 (-274) 175 (-2) 1,227 (-263) +main (3.13) 941 (+13) 107 (-278) 175 (-2) 1,223 (-267) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -122,7 +122,7 @@ Python Macro Static inline Total 3.10.0 44 (+1) 18 (+7) 62 (+8) 3.11.0 53 (+9) 4 (-14) 57 (-5) 3.12 (dev) 64 (+11) 7 (+3) 71 (+14) -main (3.13) 55 (-9) 7 (same) 62 (-9) +main (3.13) 52 (-12) 7 (same) 59 (-12) =========== ======== ============= ======== Only count private macros and public static inline functions (ignore names starting with "Py" or "PY"). @@ -143,7 +143,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) -main (3.13) 18 (same) 83 (-1) 177 (+7) 278 (+6) +main (3.13) 18 (same) 82 (-2) 178 (+8) 278 (+6) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From 0aee621becd64173e2cc114dff46d6025bf63e13 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 29 Aug 2023 05:02:46 +0200 Subject: [PATCH 121/136] Update stats --- doc/stats.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index ca97d0e..650e21c 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,746 (25%) 4,412 (13%) 21,282 (62%) 34,440 +main (3.13) 8,725 (25%) 4,359 (13%) 21,385 (62%) 34,469 =========== ============= =========== ============ ====== File Numbers @@ -37,7 +37,7 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 48 (+15) 160 (+6) 3.11.0 71 (-9) 47 (+15) 68 (+20) 186 (+26) 3.12 (dev) 70 (-1) 48 (+1) 91 (+23) 209 (+23) -main (3.13) 71 (+1) 50 (+2) 98 (+7) 219 (+10) +main (3.13) 71 (+1) 50 (+2) 100 (+9) 221 (+12) =========== =========== =========== ============ ========= Functions @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 941 (+13) 107 (-278) 175 (-2) 1,223 (-267) +main (3.13) 941 (+13) 92 (-293) 182 (+5) 1,215 (-275) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From 836252a4563af3ffc76278ad2d71d9515a27dc69 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 30 Aug 2023 05:36:00 +0200 Subject: [PATCH 122/136] Update stats --- doc/stats.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 650e21c..6cf5b34 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,725 (25%) 4,359 (13%) 21,385 (62%) 34,469 +main (3.13) 8,725 (25%) 4,354 (13%) 21,444 (62%) 34,523 =========== ============= =========== ============ ====== File Numbers @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) 3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) 3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 941 (+13) 92 (-293) 182 (+5) 1,215 (-275) +main (3.13) 941 (+13) 91 (-294) 183 (+6) 1,215 (-275) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid From 22ae4a699358d689c53fa34bf10405b5cbbaf85d Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 2 Sep 2023 19:46:13 +0200 Subject: [PATCH 123/136] Parse also Modules/posixmodule.h internal header --- doc/pythoncapi.py | 8 ++++++-- doc/stats.rst | 46 +++++++++++++++++++++++----------------------- 2 files changed, 29 insertions(+), 25 deletions(-) diff --git a/doc/pythoncapi.py b/doc/pythoncapi.py index e43de18..9c92336 100644 --- a/doc/pythoncapi.py +++ b/doc/pythoncapi.py @@ -23,6 +23,7 @@ PATH_LIMITED_API = 'Include' PATH_CPYTHON_API = os.path.join('Include', 'cpython') PATH_INTERNAL_API = os.path.join('Include', 'internal') +POSIXMODULE_H = 'Modules/posixmodule.h' RE_IDENTIFIER = r'[A-Za-z_][A-Za-z0-9_]*' RE_STRUCT_START = re.compile(r'^(?:typedef +)?struct(?: +([A-Za-z0-9_]+))? *{', re.MULTILINE) @@ -101,8 +102,11 @@ def list_files(path): if not os.path.exists(path): return [] files = glob.glob(os.path.join(path, '*.h')) - return [name for name in files - if os.path.basename(name) not in EXCLUDE_HEADERS] + if path == PATH_INTERNAL_API: + files.append(POSIXMODULE_H) + files = [name for name in files + if os.path.basename(name) not in EXCLUDE_HEADERS] + return files def _get_types(filename, names): diff --git a/doc/stats.rst b/doc/stats.rst index 6cf5b34..12853de 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -12,13 +12,13 @@ Python Limited API CPython API Internal API Total =========== ============= =========== ============ ====== 2.7.0 12,392 (100%) 0 (0%) 0 (0%) 12,392 3.6.0 15,330 (100%) 0 (0%) 0 (0%) 15,330 -3.7.0 15,943 (96%) 0 (0%) 678 (4%) 16,621 -3.8.0 12,648 (69%) 3,418 (19%) 2,207 (12%) 18,273 -3.9.0 11,744 (61%) 4,359 (23%) 3,084 (16%) 19,187 -3.10.0 9,909 (50%) 4,584 (23%) 5,519 (28%) 20,012 -3.11.0 8,733 (36%) 5,407 (22%) 10,047 (42%) 24,187 -3.12 (dev) 8,989 (28%) 5,815 (18%) 17,631 (54%) 32,435 -main (3.13) 8,725 (25%) 4,354 (13%) 21,444 (62%) 34,523 +3.7.0 15,943 (96%) 0 (0%) 703 (4%) 16,646 +3.8.0 12,648 (69%) 3,418 (19%) 2,241 (12%) 18,307 +3.9.0 11,744 (61%) 4,359 (23%) 3,118 (16%) 19,221 +3.10.0 9,909 (49%) 4,584 (23%) 5,553 (28%) 20,046 +3.11.0 8,733 (36%) 5,407 (22%) 10,079 (42%) 24,219 +3.12 (dev) 8,989 (28%) 5,815 (18%) 17,663 (54%) 32,467 +main (3.13) 8,617 (23%) 7,162 (19%) 21,483 (58%) 37,262 =========== ============= =========== ============ ====== File Numbers @@ -31,13 +31,13 @@ Python Limited API CPython API Internal API Total =========== =========== =========== ============ ========= 2.7.0 91 0 0 91 3.6.0 98 (+7) 0 (same) 0 (same) 98 (+7) -3.7.0 98 (same) 0 (same) 11 (+11) 109 (+11) -3.8.0 96 (-2) 15 (+15) 21 (+10) 132 (+23) -3.9.0 97 (+1) 24 (+9) 33 (+12) 154 (+22) -3.10.0 80 (-17) 32 (+8) 48 (+15) 160 (+6) -3.11.0 71 (-9) 47 (+15) 68 (+20) 186 (+26) -3.12 (dev) 70 (-1) 48 (+1) 91 (+23) 209 (+23) -main (3.13) 71 (+1) 50 (+2) 100 (+9) 221 (+12) +3.7.0 98 (same) 0 (same) 12 (+12) 110 (+12) +3.8.0 96 (-2) 15 (+15) 22 (+10) 133 (+23) +3.9.0 97 (+1) 24 (+9) 34 (+12) 155 (+22) +3.10.0 80 (-17) 32 (+8) 49 (+15) 161 (+6) +3.11.0 71 (-9) 47 (+15) 69 (+20) 187 (+26) +3.12 (dev) 70 (-1) 48 (+1) 92 (+23) 210 (+23) +main (3.13) 71 (+1) 54 (+6) 101 (+9) 226 (+16) =========== =========== =========== ============ ========= Functions @@ -50,13 +50,13 @@ Python Public Private Internal Total =========== ========== ========== ========= ============ 2.7.0 738 159 0 897 3.6.0 844 (+106) 382 (+223) 0 (same) 1,226 (+329) -3.7.0 869 (+25) 450 (+68) 12 (+12) 1,331 (+105) -3.8.0 909 (+40) 429 (-21) 78 (+66) 1,416 (+85) -3.9.0 919 (+10) 412 (-17) 109 (+31) 1,440 (+24) -3.10.0 885 (-34) 412 (same) 115 (+6) 1,412 (-28) -3.11.0 891 (+6) 360 (-52) 176 (+61) 1,427 (+15) -3.12 (dev) 928 (+37) 385 (+25) 177 (+1) 1,490 (+63) -main (3.13) 941 (+13) 91 (-294) 183 (+6) 1,215 (-275) +3.7.0 869 (+25) 450 (+68) 16 (+16) 1,335 (+109) +3.8.0 909 (+40) 429 (-21) 83 (+67) 1,421 (+86) +3.9.0 919 (+10) 412 (-17) 114 (+31) 1,445 (+24) +3.10.0 885 (-34) 412 (same) 120 (+6) 1,417 (-28) +3.11.0 891 (+6) 360 (-52) 181 (+61) 1,432 (+15) +3.12 (dev) 928 (+37) 385 (+25) 182 (+1) 1,495 (+63) +main (3.13) 941 (+13) 86 (-299) 192 (+10) 1,219 (-276) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -82,7 +82,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12 (dev) 176 (-1) 29 (-3) 12 (+2) 217 (-2) -main (3.13) 177 (+1) 14 (-15) 19 (+7) 210 (-7) +main (3.13) 178 (+2) 14 (-15) 18 (+6) 210 (-7) =========== ========== ======== ======== ========== Public functions defined as macros and static inline functions @@ -122,7 +122,7 @@ Python Macro Static inline Total 3.10.0 44 (+1) 18 (+7) 62 (+8) 3.11.0 53 (+9) 4 (-14) 57 (-5) 3.12 (dev) 64 (+11) 7 (+3) 71 (+14) -main (3.13) 52 (-12) 7 (same) 59 (-12) +main (3.13) 53 (-11) 9 (+2) 62 (-9) =========== ======== ============= ======== Only count private macros and public static inline functions (ignore names starting with "Py" or "PY"). From d1a71d90ca609aabf5daa53d9e9c22c33fab8914 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 7 Sep 2023 02:09:29 +0200 Subject: [PATCH 124/136] Update stats --- doc/stats.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 12853de..c4c2d6c 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (49%) 4,584 (23%) 5,553 (28%) 20,046 3.11.0 8,733 (36%) 5,407 (22%) 10,079 (42%) 24,219 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,663 (54%) 32,467 -main (3.13) 8,617 (23%) 7,162 (19%) 21,483 (58%) 37,262 +main (3.13) 8,578 (23%) 7,168 (19%) 21,517 (58%) 37,263 =========== ============= =========== ============ ====== File Numbers @@ -37,7 +37,7 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 49 (+15) 161 (+6) 3.11.0 71 (-9) 47 (+15) 69 (+20) 187 (+26) 3.12 (dev) 70 (-1) 48 (+1) 92 (+23) 210 (+23) -main (3.13) 71 (+1) 54 (+6) 101 (+9) 226 (+16) +main (3.13) 71 (+1) 54 (+6) 102 (+10) 227 (+17) =========== =========== =========== ============ ========= Functions @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 120 (+6) 1,417 (-28) 3.11.0 891 (+6) 360 (-52) 181 (+61) 1,432 (+15) 3.12 (dev) 928 (+37) 385 (+25) 182 (+1) 1,495 (+63) -main (3.13) 941 (+13) 86 (-299) 192 (+10) 1,219 (-276) +main (3.13) 941 (+13) 85 (-300) 191 (+9) 1,217 (-278) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -143,7 +143,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) -main (3.13) 18 (same) 82 (-2) 178 (+8) 278 (+6) +main (3.13) 18 (same) 83 (-1) 178 (+8) 279 (+7) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From 2fee8359e499b3b3ac423e2caf8c02f139124d4e Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 31 Oct 2023 18:33:04 +0100 Subject: [PATCH 125/136] Update stats --- doc/stats.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index c4c2d6c..01e39c5 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (49%) 4,584 (23%) 5,553 (28%) 20,046 3.11.0 8,733 (36%) 5,407 (22%) 10,079 (42%) 24,219 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,663 (54%) 32,467 -main (3.13) 8,578 (23%) 7,168 (19%) 21,517 (58%) 37,263 +main (3.13) 8,819 (23%) 7,196 (19%) 22,059 (58%) 38,074 =========== ============= =========== ============ ====== File Numbers @@ -37,7 +37,7 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 49 (+15) 161 (+6) 3.11.0 71 (-9) 47 (+15) 69 (+20) 187 (+26) 3.12 (dev) 70 (-1) 48 (+1) 92 (+23) 210 (+23) -main (3.13) 71 (+1) 54 (+6) 102 (+10) 227 (+17) +main (3.13) 72 (+2) 53 (+5) 109 (+17) 234 (+24) =========== =========== =========== ============ ========= Functions @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 120 (+6) 1,417 (-28) 3.11.0 891 (+6) 360 (-52) 181 (+61) 1,432 (+15) 3.12 (dev) 928 (+37) 385 (+25) 182 (+1) 1,495 (+63) -main (3.13) 941 (+13) 85 (-300) 191 (+9) 1,217 (-278) +main (3.13) 954 (+26) 71 (-314) 235 (+53) 1,260 (-235) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -82,7 +82,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12 (dev) 176 (-1) 29 (-3) 12 (+2) 217 (-2) -main (3.13) 178 (+2) 14 (-15) 18 (+6) 210 (-7) +main (3.13) 179 (+3) 14 (-15) 17 (+5) 210 (-7) =========== ========== ======== ======== ========== Public functions defined as macros and static inline functions @@ -143,7 +143,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) -main (3.13) 18 (same) 83 (-1) 178 (+8) 279 (+7) +main (3.13) 17 (-1) 84 (same) 184 (+14) 285 (+13) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From 9de176136ca520f3fd4b7cb35494a34d826a1801 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 16 Nov 2023 23:49:03 +0100 Subject: [PATCH 126/136] Update stats --- doc/stats.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 01e39c5..fdce357 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (49%) 4,584 (23%) 5,553 (28%) 20,046 3.11.0 8,733 (36%) 5,407 (22%) 10,079 (42%) 24,219 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,663 (54%) 32,467 -main (3.13) 8,819 (23%) 7,196 (19%) 22,059 (58%) 38,074 +main (3.13) 8,837 (23%) 7,498 (19%) 22,560 (58%) 38,895 =========== ============= =========== ============ ====== File Numbers @@ -37,7 +37,7 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 49 (+15) 161 (+6) 3.11.0 71 (-9) 47 (+15) 69 (+20) 187 (+26) 3.12 (dev) 70 (-1) 48 (+1) 92 (+23) 210 (+23) -main (3.13) 72 (+2) 53 (+5) 109 (+17) 234 (+24) +main (3.13) 72 (+2) 54 (+6) 110 (+18) 236 (+26) =========== =========== =========== ============ ========= Functions @@ -56,7 +56,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 120 (+6) 1,417 (-28) 3.11.0 891 (+6) 360 (-52) 181 (+61) 1,432 (+15) 3.12 (dev) 928 (+37) 385 (+25) 182 (+1) 1,495 (+63) -main (3.13) 954 (+26) 71 (-314) 235 (+53) 1,260 (-235) +main (3.13) 949 (+21) 103 (-282) 242 (+60) 1,294 (-201) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -122,7 +122,7 @@ Python Macro Static inline Total 3.10.0 44 (+1) 18 (+7) 62 (+8) 3.11.0 53 (+9) 4 (-14) 57 (-5) 3.12 (dev) 64 (+11) 7 (+3) 71 (+14) -main (3.13) 53 (-11) 9 (+2) 62 (-9) +main (3.13) 58 (-6) 9 (+2) 67 (-4) =========== ======== ============= ======== Only count private macros and public static inline functions (ignore names starting with "Py" or "PY"). @@ -143,7 +143,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) -main (3.13) 17 (-1) 84 (same) 184 (+14) 285 (+13) +main (3.13) 17 (-1) 87 (+3) 190 (+20) 294 (+22) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From 9067ee709c6dbaa7717adcba67bdbb7bcc65bda8 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Thu, 16 Nov 2023 23:50:21 +0100 Subject: [PATCH 127/136] stats: list functions first --- doc/stats.py | 4 +-- doc/stats.rst | 76 +++++++++++++++++++++++++-------------------------- 2 files changed, 40 insertions(+), 40 deletions(-) diff --git a/doc/stats.py b/doc/stats.py index 3808521..d5e9943 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -260,10 +260,10 @@ def render_page(): results = get_data() main_title() - line_numbers(results) - file_numbers(results) list_functions(results) list_variables(results) + line_numbers(results) + file_numbers(results) static_inline_func(results) structures(results) diff --git a/doc/stats.rst b/doc/stats.rst index fdce357..1602dc9 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -2,44 +2,6 @@ Statistics on the Python C API ++++++++++++++++++++++++++++++ -Line Numbers -============ - -Number of C API line numbers per Python version: - -=========== ============= =========== ============ ====== -Python Limited API CPython API Internal API Total -=========== ============= =========== ============ ====== -2.7.0 12,392 (100%) 0 (0%) 0 (0%) 12,392 -3.6.0 15,330 (100%) 0 (0%) 0 (0%) 15,330 -3.7.0 15,943 (96%) 0 (0%) 703 (4%) 16,646 -3.8.0 12,648 (69%) 3,418 (19%) 2,241 (12%) 18,307 -3.9.0 11,744 (61%) 4,359 (23%) 3,118 (16%) 19,221 -3.10.0 9,909 (49%) 4,584 (23%) 5,553 (28%) 20,046 -3.11.0 8,733 (36%) 5,407 (22%) 10,079 (42%) 24,219 -3.12 (dev) 8,989 (28%) 5,815 (18%) 17,663 (54%) 32,467 -main (3.13) 8,837 (23%) 7,498 (19%) 22,560 (58%) 38,895 -=========== ============= =========== ============ ====== - -File Numbers -============ - -Number of header file numbers per Python version: - -=========== =========== =========== ============ ========= -Python Limited API CPython API Internal API Total -=========== =========== =========== ============ ========= -2.7.0 91 0 0 91 -3.6.0 98 (+7) 0 (same) 0 (same) 98 (+7) -3.7.0 98 (same) 0 (same) 12 (+12) 110 (+12) -3.8.0 96 (-2) 15 (+15) 22 (+10) 133 (+23) -3.9.0 97 (+1) 24 (+9) 34 (+12) 155 (+22) -3.10.0 80 (-17) 32 (+8) 49 (+15) 161 (+6) -3.11.0 71 (-9) 47 (+15) 69 (+20) 187 (+26) -3.12 (dev) 70 (-1) 48 (+1) 92 (+23) 210 (+23) -main (3.13) 72 (+2) 54 (+6) 110 (+18) 236 (+26) -=========== =========== =========== ============ ========= - Functions ========= @@ -85,6 +47,44 @@ Python Public Private Internal Total main (3.13) 179 (+3) 14 (-15) 17 (+5) 210 (-7) =========== ========== ======== ======== ========== +Line Numbers +============ + +Number of C API line numbers per Python version: + +=========== ============= =========== ============ ====== +Python Limited API CPython API Internal API Total +=========== ============= =========== ============ ====== +2.7.0 12,392 (100%) 0 (0%) 0 (0%) 12,392 +3.6.0 15,330 (100%) 0 (0%) 0 (0%) 15,330 +3.7.0 15,943 (96%) 0 (0%) 703 (4%) 16,646 +3.8.0 12,648 (69%) 3,418 (19%) 2,241 (12%) 18,307 +3.9.0 11,744 (61%) 4,359 (23%) 3,118 (16%) 19,221 +3.10.0 9,909 (49%) 4,584 (23%) 5,553 (28%) 20,046 +3.11.0 8,733 (36%) 5,407 (22%) 10,079 (42%) 24,219 +3.12 (dev) 8,989 (28%) 5,815 (18%) 17,663 (54%) 32,467 +main (3.13) 8,837 (23%) 7,498 (19%) 22,560 (58%) 38,895 +=========== ============= =========== ============ ====== + +File Numbers +============ + +Number of header file numbers per Python version: + +=========== =========== =========== ============ ========= +Python Limited API CPython API Internal API Total +=========== =========== =========== ============ ========= +2.7.0 91 0 0 91 +3.6.0 98 (+7) 0 (same) 0 (same) 98 (+7) +3.7.0 98 (same) 0 (same) 12 (+12) 110 (+12) +3.8.0 96 (-2) 15 (+15) 22 (+10) 133 (+23) +3.9.0 97 (+1) 24 (+9) 34 (+12) 155 (+22) +3.10.0 80 (-17) 32 (+8) 49 (+15) 161 (+6) +3.11.0 71 (-9) 47 (+15) 69 (+20) 187 (+26) +3.12 (dev) 70 (-1) 48 (+1) 92 (+23) 210 (+23) +main (3.13) 72 (+2) 54 (+6) 110 (+18) 236 (+26) +=========== =========== =========== ============ ========= + Public functions defined as macros and static inline functions ============================================================== From 1be847f0d160ac2e13bb6e426cbb45b996ada574 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 15 Dec 2023 03:28:57 +0100 Subject: [PATCH 128/136] Update stats --- doc/stats.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 1602dc9..d09d944 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 120 (+6) 1,417 (-28) 3.11.0 891 (+6) 360 (-52) 181 (+61) 1,432 (+15) 3.12 (dev) 928 (+37) 385 (+25) 182 (+1) 1,495 (+63) -main (3.13) 949 (+21) 103 (-282) 242 (+60) 1,294 (-201) +main (3.13) 950 (+22) 102 (-283) 246 (+64) 1,298 (-197) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -44,7 +44,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12 (dev) 176 (-1) 29 (-3) 12 (+2) 217 (-2) -main (3.13) 179 (+3) 14 (-15) 17 (+5) 210 (-7) +main (3.13) 179 (+3) 14 (-15) 19 (+7) 212 (-5) =========== ========== ======== ======== ========== Line Numbers @@ -63,7 +63,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (49%) 4,584 (23%) 5,553 (28%) 20,046 3.11.0 8,733 (36%) 5,407 (22%) 10,079 (42%) 24,219 3.12 (dev) 8,989 (28%) 5,815 (18%) 17,663 (54%) 32,467 -main (3.13) 8,837 (23%) 7,498 (19%) 22,560 (58%) 38,895 +main (3.13) 8,895 (23%) 7,498 (19%) 22,886 (58%) 39,279 =========== ============= =========== ============ ====== File Numbers @@ -82,7 +82,7 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 49 (+15) 161 (+6) 3.11.0 71 (-9) 47 (+15) 69 (+20) 187 (+26) 3.12 (dev) 70 (-1) 48 (+1) 92 (+23) 210 (+23) -main (3.13) 72 (+2) 54 (+6) 110 (+18) 236 (+26) +main (3.13) 72 (+2) 54 (+6) 112 (+20) 238 (+28) =========== =========== =========== ============ ========= Public functions defined as macros and static inline functions @@ -143,7 +143,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) -main (3.13) 17 (-1) 87 (+3) 190 (+20) 294 (+22) +main (3.13) 17 (-1) 87 (+3) 191 (+21) 295 (+23) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From b16fc7ab503342e3ba814e033932a79b8d53f2ca Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 15 Dec 2023 04:13:09 +0100 Subject: [PATCH 129/136] stats: add PYthon 3.12.0 --- doc/stats.py | 3 +-- doc/stats.rst | 24 ++++++++++++------------ 2 files changed, 13 insertions(+), 14 deletions(-) diff --git a/doc/stats.py b/doc/stats.py index d5e9943..66ffd00 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -18,8 +18,7 @@ ('v3.9.0', '3.9.0'), ('v3.10.0', '3.10.0'), ('v3.11.0', '3.11.0'), - #('v3.12.0', '3.12'), - ('3.12', '3.12 (dev)'), + ('v3.12.0', '3.12.0'), ('main', 'main (3.13)'), ] RST_FILENAME = os.path.normpath(os.path.join(os.path.dirname(__file__), 'stats.rst')) diff --git a/doc/stats.rst b/doc/stats.rst index d09d944..d241f94 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -17,8 +17,8 @@ Python Public Private Internal Total 3.9.0 919 (+10) 412 (-17) 114 (+31) 1,445 (+24) 3.10.0 885 (-34) 412 (same) 120 (+6) 1,417 (-28) 3.11.0 891 (+6) 360 (-52) 181 (+61) 1,432 (+15) -3.12 (dev) 928 (+37) 385 (+25) 182 (+1) 1,495 (+63) -main (3.13) 950 (+22) 102 (-283) 246 (+64) 1,298 (-197) +3.12.0 929 (+38) 389 (+29) 179 (-2) 1,497 (+65) +main (3.13) 950 (+21) 102 (-287) 246 (+67) 1,298 (-199) =========== ========== ========== ========= ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -43,8 +43,8 @@ Python Public Private Internal Total 3.9.0 173 (-1) 35 (-2) 14 (+2) 222 (-1) 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) -3.12 (dev) 176 (-1) 29 (-3) 12 (+2) 217 (-2) -main (3.13) 179 (+3) 14 (-15) 19 (+7) 212 (-5) +3.12.0 176 (-1) 30 (-2) 11 (+1) 217 (-2) +main (3.13) 179 (+3) 14 (-16) 19 (+8) 212 (-5) =========== ========== ======== ======== ========== Line Numbers @@ -62,7 +62,7 @@ Python Limited API CPython API Internal API Total 3.9.0 11,744 (61%) 4,359 (23%) 3,118 (16%) 19,221 3.10.0 9,909 (49%) 4,584 (23%) 5,553 (28%) 20,046 3.11.0 8,733 (36%) 5,407 (22%) 10,079 (42%) 24,219 -3.12 (dev) 8,989 (28%) 5,815 (18%) 17,663 (54%) 32,467 +3.12.0 9,012 (28%) 5,836 (18%) 17,650 (54%) 32,498 main (3.13) 8,895 (23%) 7,498 (19%) 22,886 (58%) 39,279 =========== ============= =========== ============ ====== @@ -81,8 +81,8 @@ Python Limited API CPython API Internal API Total 3.9.0 97 (+1) 24 (+9) 34 (+12) 155 (+22) 3.10.0 80 (-17) 32 (+8) 49 (+15) 161 (+6) 3.11.0 71 (-9) 47 (+15) 69 (+20) 187 (+26) -3.12 (dev) 70 (-1) 48 (+1) 92 (+23) 210 (+23) -main (3.13) 72 (+2) 54 (+6) 112 (+20) 238 (+28) +3.12.0 71 (same) 49 (+2) 91 (+22) 211 (+24) +main (3.13) 72 (+1) 54 (+5) 112 (+21) 238 (+27) =========== =========== =========== ============ ========= Public functions defined as macros and static inline functions @@ -100,7 +100,7 @@ Python Macro Static inline Total 3.9.0 307 (-7) 0 (same) 307 (-7) 3.10.0 307 (same) 0 (same) 307 (same) 3.11.0 269 (-38) 42 (+42) 311 (+4) -3.12 (dev) 251 (-18) 64 (+22) 315 (+4) +3.12.0 251 (-18) 64 (+22) 315 (+4) main (3.13) 246 (-5) 63 (-1) 309 (-6) =========== ========== ============= ========== @@ -121,8 +121,8 @@ Python Macro Static inline Total 3.9.0 43 (-14) 11 (+4) 54 (-10) 3.10.0 44 (+1) 18 (+7) 62 (+8) 3.11.0 53 (+9) 4 (-14) 57 (-5) -3.12 (dev) 64 (+11) 7 (+3) 71 (+14) -main (3.13) 58 (-6) 9 (+2) 67 (-4) +3.12.0 62 (+9) 7 (+3) 69 (+12) +main (3.13) 58 (-4) 9 (+2) 67 (-2) =========== ======== ============= ======== Only count private macros and public static inline functions (ignore names starting with "Py" or "PY"). @@ -142,8 +142,8 @@ Python Limited API CPython API Internal API Total 3.9.0 68 (-13) 46 (+12) 38 (+10) 152 (+9) 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) -3.12 (dev) 18 (-1) 84 (+9) 170 (+58) 272 (+66) -main (3.13) 17 (-1) 87 (+3) 191 (+21) 295 (+23) +3.12.0 18 (-1) 85 (+10) 170 (+58) 273 (+67) +main (3.13) 17 (-1) 87 (+2) 191 (+21) 295 (+22) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From 8249eeedd30838997776a4efdf22912cb526c278 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 6 Mar 2024 12:12:18 +0100 Subject: [PATCH 130/136] Update stats --- doc/stats.rst | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index d241f94..774f43d 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -7,19 +7,19 @@ Functions Functions exported with PyAPI_FUNC(): -=========== ========== ========== ========= ============ -Python Public Private Internal Total -=========== ========== ========== ========= ============ -2.7.0 738 159 0 897 -3.6.0 844 (+106) 382 (+223) 0 (same) 1,226 (+329) -3.7.0 869 (+25) 450 (+68) 16 (+16) 1,335 (+109) -3.8.0 909 (+40) 429 (-21) 83 (+67) 1,421 (+86) -3.9.0 919 (+10) 412 (-17) 114 (+31) 1,445 (+24) -3.10.0 885 (-34) 412 (same) 120 (+6) 1,417 (-28) -3.11.0 891 (+6) 360 (-52) 181 (+61) 1,432 (+15) -3.12.0 929 (+38) 389 (+29) 179 (-2) 1,497 (+65) -main (3.13) 950 (+21) 102 (-287) 246 (+67) 1,298 (-199) -=========== ========== ========== ========= ============ +=========== ========== ========== ========== ============ +Python Public Private Internal Total +=========== ========== ========== ========== ============ +2.7.0 738 159 0 897 +3.6.0 844 (+106) 382 (+223) 0 (same) 1,226 (+329) +3.7.0 869 (+25) 450 (+68) 16 (+16) 1,335 (+109) +3.8.0 909 (+40) 429 (-21) 83 (+67) 1,421 (+86) +3.9.0 919 (+10) 412 (-17) 114 (+31) 1,445 (+24) +3.10.0 885 (-34) 412 (same) 120 (+6) 1,417 (-28) +3.11.0 891 (+6) 360 (-52) 181 (+61) 1,432 (+15) +3.12.0 929 (+38) 389 (+29) 179 (-2) 1,497 (+65) +main (3.13) 959 (+30) 104 (-285) 295 (+116) 1,358 (-139) +=========== ========== ========== ========== ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid exporting symbols which are not **explicitly** exported. @@ -44,7 +44,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12.0 176 (-1) 30 (-2) 11 (+1) 217 (-2) -main (3.13) 179 (+3) 14 (-16) 19 (+8) 212 (-5) +main (3.13) 181 (+5) 14 (-16) 24 (+13) 219 (+2) =========== ========== ======== ======== ========== Line Numbers @@ -63,7 +63,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (49%) 4,584 (23%) 5,553 (28%) 20,046 3.11.0 8,733 (36%) 5,407 (22%) 10,079 (42%) 24,219 3.12.0 9,012 (28%) 5,836 (18%) 17,650 (54%) 32,498 -main (3.13) 8,895 (23%) 7,498 (19%) 22,886 (58%) 39,279 +main (3.13) 8,907 (22%) 7,838 (19%) 24,129 (59%) 40,874 =========== ============= =========== ============ ====== File Numbers @@ -82,7 +82,7 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 49 (+15) 161 (+6) 3.11.0 71 (-9) 47 (+15) 69 (+20) 187 (+26) 3.12.0 71 (same) 49 (+2) 91 (+22) 211 (+24) -main (3.13) 72 (+1) 54 (+5) 112 (+21) 238 (+27) +main (3.13) 72 (+1) 55 (+6) 119 (+28) 246 (+35) =========== =========== =========== ============ ========= Public functions defined as macros and static inline functions @@ -122,7 +122,7 @@ Python Macro Static inline Total 3.10.0 44 (+1) 18 (+7) 62 (+8) 3.11.0 53 (+9) 4 (-14) 57 (-5) 3.12.0 62 (+9) 7 (+3) 69 (+12) -main (3.13) 58 (-4) 9 (+2) 67 (-2) +main (3.13) 59 (-3) 9 (+2) 68 (-1) =========== ======== ============= ======== Only count private macros and public static inline functions (ignore names starting with "Py" or "PY"). @@ -143,7 +143,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12.0 18 (-1) 85 (+10) 170 (+58) 273 (+67) -main (3.13) 17 (-1) 87 (+2) 191 (+21) 295 (+22) +main (3.13) 17 (-1) 90 (+5) 217 (+47) 324 (+51) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From b2805432b8356dbb381b0bffe72cd03f19e4e227 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Sat, 23 Mar 2024 14:08:54 +0100 Subject: [PATCH 131/136] Stats --- doc/stats.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index 774f43d..c983375 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 120 (+6) 1,417 (-28) 3.11.0 891 (+6) 360 (-52) 181 (+61) 1,432 (+15) 3.12.0 929 (+38) 389 (+29) 179 (-2) 1,497 (+65) -main (3.13) 959 (+30) 104 (-285) 295 (+116) 1,358 (-139) +main (3.13) 961 (+32) 105 (-284) 295 (+116) 1,361 (-136) =========== ========== ========== ========== ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -44,7 +44,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12.0 176 (-1) 30 (-2) 11 (+1) 217 (-2) -main (3.13) 181 (+5) 14 (-16) 24 (+13) 219 (+2) +main (3.13) 180 (+4) 14 (-16) 24 (+13) 218 (+1) =========== ========== ======== ======== ========== Line Numbers @@ -63,7 +63,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (49%) 4,584 (23%) 5,553 (28%) 20,046 3.11.0 8,733 (36%) 5,407 (22%) 10,079 (42%) 24,219 3.12.0 9,012 (28%) 5,836 (18%) 17,650 (54%) 32,498 -main (3.13) 8,907 (22%) 7,838 (19%) 24,129 (59%) 40,874 +main (3.13) 8,951 (22%) 7,828 (19%) 24,230 (59%) 41,009 =========== ============= =========== ============ ====== File Numbers @@ -82,7 +82,7 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 49 (+15) 161 (+6) 3.11.0 71 (-9) 47 (+15) 69 (+20) 187 (+26) 3.12.0 71 (same) 49 (+2) 91 (+22) 211 (+24) -main (3.13) 72 (+1) 55 (+6) 119 (+28) 246 (+35) +main (3.13) 71 (same) 54 (+5) 119 (+28) 244 (+33) =========== =========== =========== ============ ========= Public functions defined as macros and static inline functions @@ -143,7 +143,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12.0 18 (-1) 85 (+10) 170 (+58) 273 (+67) -main (3.13) 17 (-1) 90 (+5) 217 (+47) 324 (+51) +main (3.13) 17 (-1) 90 (+5) 219 (+49) 326 (+53) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From 89e1bc110622edfd350a0126efd3d21444d57675 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Fri, 5 Apr 2024 02:28:49 +0200 Subject: [PATCH 132/136] Bad API --- doc/bad_api.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/bad_api.rst b/doc/bad_api.rst index 5a26cab..f221bb4 100644 --- a/doc/bad_api.rst +++ b/doc/bad_api.rst @@ -79,10 +79,10 @@ Functions ^^^^^^^^^ * ``PyCell_GET()``: use ``PyCell_Get()`` -* ``PyDict_GetItem()``: use ``PyObject_GetItem()`` -* ``PyDict_GetItemString()``: use ``PyObject_GetItem()`` -* ``PyDict_GetItemWithError()``: use ``PyObject_GetItem()`` -* ``PyDict_SetDefault()`` +* ``PyDict_GetItem()``: use ``PyDict_GetItemRef()`` +* ``PyDict_GetItemString()``: use ``PyDict_GetItemRef()`` +* ``PyDict_GetItemWithError()``: use ``PyDict_GetItemRef()`` +* ``PyDict_SetDefault()``: use ``PyDict_SetDefaultRef()`` * ``PyErr_Occurred()``: use ``PyErr_GetRaisedException()`` * ``PyEval_GetBuiltins()`` * ``PyEval_GetFrame()`` @@ -99,8 +99,8 @@ Functions * ``PyImport_GetModuleDict()`` * ``PyInstanceMethod_Function()`` * ``PyInstanceMethod_GET_FUNCTION()`` -* ``PyList_GET_ITEM()``: use ``PySequence_GetItem()`` -* ``PyList_GetItem()``: use ``PySequence_GetItem()`` +* ``PyList_GET_ITEM()``: use ``PyList_GetItemRef()`` +* ``PyList_GetItem()``: use ``PyList_GetItemRef()`` * ``PyMethod_Function()`` * ``PyMethod_GET_FUNCTION()`` * ``PyMethod_GET_SELF()`` From 6efd3302c74ceac1f6333ee708fcce0c4cfa1385 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 8 May 2024 14:49:51 +0200 Subject: [PATCH 133/136] Update stats --- doc/stats.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index c983375..c48a5f5 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,7 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 120 (+6) 1,417 (-28) 3.11.0 891 (+6) 360 (-52) 181 (+61) 1,432 (+15) 3.12.0 929 (+38) 389 (+29) 179 (-2) 1,497 (+65) -main (3.13) 961 (+32) 105 (-284) 295 (+116) 1,361 (-136) +main (3.13) 977 (+48) 125 (-264) 312 (+133) 1,414 (-83) =========== ========== ========== ========== ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -44,7 +44,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12.0 176 (-1) 30 (-2) 11 (+1) 217 (-2) -main (3.13) 180 (+4) 14 (-16) 24 (+13) 218 (+1) +main (3.13) 181 (+5) 14 (-16) 24 (+13) 219 (+2) =========== ========== ======== ======== ========== Line Numbers @@ -63,7 +63,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (49%) 4,584 (23%) 5,553 (28%) 20,046 3.11.0 8,733 (36%) 5,407 (22%) 10,079 (42%) 24,219 3.12.0 9,012 (28%) 5,836 (18%) 17,650 (54%) 32,498 -main (3.13) 8,951 (22%) 7,828 (19%) 24,230 (59%) 41,009 +main (3.13) 9,046 (21%) 8,254 (19%) 25,860 (60%) 43,160 =========== ============= =========== ============ ====== File Numbers @@ -82,7 +82,7 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 49 (+15) 161 (+6) 3.11.0 71 (-9) 47 (+15) 69 (+20) 187 (+26) 3.12.0 71 (same) 49 (+2) 91 (+22) 211 (+24) -main (3.13) 71 (same) 54 (+5) 119 (+28) 244 (+33) +main (3.13) 72 (+1) 55 (+6) 123 (+32) 250 (+39) =========== =========== =========== ============ ========= Public functions defined as macros and static inline functions @@ -101,7 +101,7 @@ Python Macro Static inline Total 3.10.0 307 (same) 0 (same) 307 (same) 3.11.0 269 (-38) 42 (+42) 311 (+4) 3.12.0 251 (-18) 64 (+22) 315 (+4) -main (3.13) 246 (-5) 63 (-1) 309 (-6) +main (3.13) 247 (-4) 63 (-1) 310 (-5) =========== ========== ============= ========== Only count public macros and public static inline functions (names starting with "Py" or "PY"). @@ -122,7 +122,7 @@ Python Macro Static inline Total 3.10.0 44 (+1) 18 (+7) 62 (+8) 3.11.0 53 (+9) 4 (-14) 57 (-5) 3.12.0 62 (+9) 7 (+3) 69 (+12) -main (3.13) 59 (-3) 9 (+2) 68 (-1) +main (3.13) 63 (+1) 13 (+6) 76 (+7) =========== ======== ============= ======== Only count private macros and public static inline functions (ignore names starting with "Py" or "PY"). @@ -143,7 +143,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12.0 18 (-1) 85 (+10) 170 (+58) 273 (+67) -main (3.13) 17 (-1) 90 (+5) 219 (+49) 326 (+53) +main (3.13) 17 (-1) 90 (+5) 227 (+57) 334 (+61) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From 3cb9c708aeae7dc4119a3109cafa44809b87875e Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 10 Sep 2024 12:43:11 +0200 Subject: [PATCH 134/136] Update stats --- doc/stats.rst | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/doc/stats.rst b/doc/stats.rst index c48a5f5..1206324 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -7,19 +7,19 @@ Functions Functions exported with PyAPI_FUNC(): -=========== ========== ========== ========== ============ -Python Public Private Internal Total -=========== ========== ========== ========== ============ -2.7.0 738 159 0 897 -3.6.0 844 (+106) 382 (+223) 0 (same) 1,226 (+329) -3.7.0 869 (+25) 450 (+68) 16 (+16) 1,335 (+109) -3.8.0 909 (+40) 429 (-21) 83 (+67) 1,421 (+86) -3.9.0 919 (+10) 412 (-17) 114 (+31) 1,445 (+24) -3.10.0 885 (-34) 412 (same) 120 (+6) 1,417 (-28) -3.11.0 891 (+6) 360 (-52) 181 (+61) 1,432 (+15) -3.12.0 929 (+38) 389 (+29) 179 (-2) 1,497 (+65) -main (3.13) 977 (+48) 125 (-264) 312 (+133) 1,414 (-83) -=========== ========== ========== ========== ============ +=========== =========== ========== ========== ============ +Python Public Private Internal Total +=========== =========== ========== ========== ============ +2.7.0 738 159 0 897 +3.6.0 844 (+106) 382 (+223) 0 (same) 1,226 (+329) +3.7.0 869 (+25) 450 (+68) 16 (+16) 1,335 (+109) +3.8.0 909 (+40) 429 (-21) 83 (+67) 1,421 (+86) +3.9.0 919 (+10) 412 (-17) 114 (+31) 1,445 (+24) +3.10.0 885 (-34) 412 (same) 120 (+6) 1,417 (-28) +3.11.0 891 (+6) 360 (-52) 181 (+61) 1,432 (+15) +3.12.0 929 (+38) 389 (+29) 179 (-2) 1,497 (+65) +main (3.13) 1,022 (+93) 125 (-264) 339 (+160) 1,486 (-11) +=========== =========== ========== ========== ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid exporting symbols which are not **explicitly** exported. @@ -44,7 +44,7 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12.0 176 (-1) 30 (-2) 11 (+1) 217 (-2) -main (3.13) 181 (+5) 14 (-16) 24 (+13) 219 (+2) +main (3.13) 180 (+4) 14 (-16) 27 (+16) 221 (+4) =========== ========== ======== ======== ========== Line Numbers @@ -63,7 +63,7 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (49%) 4,584 (23%) 5,553 (28%) 20,046 3.11.0 8,733 (36%) 5,407 (22%) 10,079 (42%) 24,219 3.12.0 9,012 (28%) 5,836 (18%) 17,650 (54%) 32,498 -main (3.13) 9,046 (21%) 8,254 (19%) 25,860 (60%) 43,160 +main (3.13) 9,181 (20%) 8,401 (18%) 27,860 (61%) 45,442 =========== ============= =========== ============ ====== File Numbers @@ -82,7 +82,7 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 49 (+15) 161 (+6) 3.11.0 71 (-9) 47 (+15) 69 (+20) 187 (+26) 3.12.0 71 (same) 49 (+2) 91 (+22) 211 (+24) -main (3.13) 72 (+1) 55 (+6) 123 (+32) 250 (+39) +main (3.13) 75 (+4) 57 (+8) 126 (+35) 258 (+47) =========== =========== =========== ============ ========= Public functions defined as macros and static inline functions @@ -101,7 +101,7 @@ Python Macro Static inline Total 3.10.0 307 (same) 0 (same) 307 (same) 3.11.0 269 (-38) 42 (+42) 311 (+4) 3.12.0 251 (-18) 64 (+22) 315 (+4) -main (3.13) 247 (-4) 63 (-1) 310 (-5) +main (3.13) 250 (-1) 62 (-2) 312 (-3) =========== ========== ============= ========== Only count public macros and public static inline functions (names starting with "Py" or "PY"). @@ -122,7 +122,7 @@ Python Macro Static inline Total 3.10.0 44 (+1) 18 (+7) 62 (+8) 3.11.0 53 (+9) 4 (-14) 57 (-5) 3.12.0 62 (+9) 7 (+3) 69 (+12) -main (3.13) 63 (+1) 13 (+6) 76 (+7) +main (3.13) 55 (-7) 10 (+3) 65 (-4) =========== ======== ============= ======== Only count private macros and public static inline functions (ignore names starting with "Py" or "PY"). @@ -143,7 +143,7 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12.0 18 (-1) 85 (+10) 170 (+58) 273 (+67) -main (3.13) 17 (-1) 90 (+5) 227 (+57) 334 (+61) +main (3.13) 16 (-2) 83 (-2) 230 (+60) 329 (+56) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From 63e4d79d9bab5f7db0cd6e56e523ca9873658ceb Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Tue, 10 Sep 2024 12:45:09 +0200 Subject: [PATCH 135/136] stats: add 3.13 --- doc/stats.py | 3 ++- doc/stats.rst | 69 ++++++++++++++++++++++++++++----------------------- 2 files changed, 40 insertions(+), 32 deletions(-) diff --git a/doc/stats.py b/doc/stats.py index 66ffd00..43318e0 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -19,7 +19,8 @@ ('v3.10.0', '3.10.0'), ('v3.11.0', '3.11.0'), ('v3.12.0', '3.12.0'), - ('main', 'main (3.13)'), + ('3.13', '3.13 (dev)'), + ('main', 'main (3.14)'), ] RST_FILENAME = os.path.normpath(os.path.join(os.path.dirname(__file__), 'stats.rst')) COLUMNS = ['Python', 'Limited API', 'CPython API', 'Internal API', 'Total'] diff --git a/doc/stats.rst b/doc/stats.rst index 1206324..a962695 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,7 +18,8 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 120 (+6) 1,417 (-28) 3.11.0 891 (+6) 360 (-52) 181 (+61) 1,432 (+15) 3.12.0 929 (+38) 389 (+29) 179 (-2) 1,497 (+65) -main (3.13) 1,022 (+93) 125 (-264) 339 (+160) 1,486 (-11) +3.13 (dev) 979 (+50) 126 (-263) 329 (+150) 1,434 (-63) +main (3.14) 1,022 (+43) 125 (-1) 339 (+10) 1,486 (+52) =========== =========== ========== ========== ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -33,19 +34,20 @@ Variables Symbols exported with PyAPI_DATA(): -=========== ========== ======== ======== ========== -Python Public Private Internal Total -=========== ========== ======== ======== ========== -2.7.0 129 30 0 159 -3.6.0 167 (+38) 42 (+12) 0 (same) 209 (+50) -3.7.0 170 (+3) 35 (-7) 9 (+9) 214 (+5) -3.8.0 174 (+4) 37 (+2) 12 (+3) 223 (+9) -3.9.0 173 (-1) 35 (-2) 14 (+2) 222 (-1) -3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) -3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) -3.12.0 176 (-1) 30 (-2) 11 (+1) 217 (-2) -main (3.13) 180 (+4) 14 (-16) 27 (+16) 221 (+4) -=========== ========== ======== ======== ========== +=========== ========== ========= ======== ========== +Python Public Private Internal Total +=========== ========== ========= ======== ========== +2.7.0 129 30 0 159 +3.6.0 167 (+38) 42 (+12) 0 (same) 209 (+50) +3.7.0 170 (+3) 35 (-7) 9 (+9) 214 (+5) +3.8.0 174 (+4) 37 (+2) 12 (+3) 223 (+9) +3.9.0 173 (-1) 35 (-2) 14 (+2) 222 (-1) +3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) +3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) +3.12.0 176 (-1) 30 (-2) 11 (+1) 217 (-2) +3.13 (dev) 180 (+4) 14 (-16) 25 (+14) 219 (+2) +main (3.14) 180 (same) 14 (same) 27 (+2) 221 (+2) +=========== ========== ========= ======== ========== Line Numbers ============ @@ -63,7 +65,8 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (49%) 4,584 (23%) 5,553 (28%) 20,046 3.11.0 8,733 (36%) 5,407 (22%) 10,079 (42%) 24,219 3.12.0 9,012 (28%) 5,836 (18%) 17,650 (54%) 32,498 -main (3.13) 9,181 (20%) 8,401 (18%) 27,860 (61%) 45,442 +3.13 (dev) 9,097 (21%) 8,316 (19%) 26,897 (61%) 44,310 +main (3.14) 9,181 (20%) 8,401 (18%) 27,860 (61%) 45,442 =========== ============= =========== ============ ====== File Numbers @@ -82,7 +85,8 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 49 (+15) 161 (+6) 3.11.0 71 (-9) 47 (+15) 69 (+20) 187 (+26) 3.12.0 71 (same) 49 (+2) 91 (+22) 211 (+24) -main (3.13) 75 (+4) 57 (+8) 126 (+35) 258 (+47) +3.13 (dev) 74 (+3) 57 (+8) 123 (+32) 254 (+43) +main (3.14) 75 (+1) 57 (same) 126 (+3) 258 (+4) =========== =========== =========== ============ ========= Public functions defined as macros and static inline functions @@ -101,7 +105,8 @@ Python Macro Static inline Total 3.10.0 307 (same) 0 (same) 307 (same) 3.11.0 269 (-38) 42 (+42) 311 (+4) 3.12.0 251 (-18) 64 (+22) 315 (+4) -main (3.13) 250 (-1) 62 (-2) 312 (-3) +3.13 (dev) 249 (-2) 63 (-1) 312 (-3) +main (3.14) 250 (+1) 62 (-1) 312 (same) =========== ========== ============= ========== Only count public macros and public static inline functions (names starting with "Py" or "PY"). @@ -111,19 +116,20 @@ Private functions defined as macros and static inline functions Private functions defined as macros and static inline functions: -=========== ======== ============= ======== -Python Macro Static inline Total -=========== ======== ============= ======== -2.7.0 21 0 21 -3.6.0 60 (+39) 0 (same) 60 (+39) -3.7.0 70 (+10) 0 (same) 70 (+10) -3.8.0 57 (-13) 7 (+7) 64 (-6) -3.9.0 43 (-14) 11 (+4) 54 (-10) -3.10.0 44 (+1) 18 (+7) 62 (+8) -3.11.0 53 (+9) 4 (-14) 57 (-5) -3.12.0 62 (+9) 7 (+3) 69 (+12) -main (3.13) 55 (-7) 10 (+3) 65 (-4) -=========== ======== ============= ======== +=========== ========= ============= ========= +Python Macro Static inline Total +=========== ========= ============= ========= +2.7.0 21 0 21 +3.6.0 60 (+39) 0 (same) 60 (+39) +3.7.0 70 (+10) 0 (same) 70 (+10) +3.8.0 57 (-13) 7 (+7) 64 (-6) +3.9.0 43 (-14) 11 (+4) 54 (-10) +3.10.0 44 (+1) 18 (+7) 62 (+8) +3.11.0 53 (+9) 4 (-14) 57 (-5) +3.12.0 62 (+9) 7 (+3) 69 (+12) +3.13 (dev) 55 (-7) 10 (+3) 65 (-4) +main (3.14) 55 (same) 10 (same) 65 (same) +=========== ========= ============= ========= Only count private macros and public static inline functions (ignore names starting with "Py" or "PY"). @@ -143,7 +149,8 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12.0 18 (-1) 85 (+10) 170 (+58) 273 (+67) -main (3.13) 16 (-2) 83 (-2) 230 (+60) 329 (+56) +3.13 (dev) 16 (-2) 86 (+1) 232 (+62) 334 (+61) +main (3.14) 16 (same) 83 (-3) 230 (-2) 329 (-5) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen". From be54aad0157624260c1b4a0a235d36e499573072 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 27 Nov 2024 12:35:07 +0100 Subject: [PATCH 136/136] stats uses 3.13.0 release --- doc/stats.py | 2 +- doc/stats.rst | 52 +++++++++++++++++++++++++-------------------------- 2 files changed, 27 insertions(+), 27 deletions(-) diff --git a/doc/stats.py b/doc/stats.py index 43318e0..63cd837 100755 --- a/doc/stats.py +++ b/doc/stats.py @@ -19,7 +19,7 @@ ('v3.10.0', '3.10.0'), ('v3.11.0', '3.11.0'), ('v3.12.0', '3.12.0'), - ('3.13', '3.13 (dev)'), + ('v3.13.0', '3.13.0'), ('main', 'main (3.14)'), ] RST_FILENAME = os.path.normpath(os.path.join(os.path.dirname(__file__), 'stats.rst')) diff --git a/doc/stats.rst b/doc/stats.rst index a962695..ed6c4d8 100644 --- a/doc/stats.rst +++ b/doc/stats.rst @@ -18,8 +18,8 @@ Python Public Private Internal Total 3.10.0 885 (-34) 412 (same) 120 (+6) 1,417 (-28) 3.11.0 891 (+6) 360 (-52) 181 (+61) 1,432 (+15) 3.12.0 929 (+38) 389 (+29) 179 (-2) 1,497 (+65) -3.13 (dev) 979 (+50) 126 (-263) 329 (+150) 1,434 (-63) -main (3.14) 1,022 (+43) 125 (-1) 339 (+10) 1,486 (+52) +3.13.0 979 (+50) 126 (-263) 329 (+150) 1,434 (-63) +main (3.14) 1,032 (+53) 131 (+5) 342 (+13) 1,505 (+71) =========== =========== ========== ========== ============ Since Python 3.9, Python is now built with ``-fvisibility=hidden`` to avoid @@ -45,8 +45,8 @@ Python Public Private Internal Total 3.10.0 173 (same) 33 (-2) 13 (-1) 219 (-3) 3.11.0 177 (+4) 32 (-1) 10 (-3) 219 (same) 3.12.0 176 (-1) 30 (-2) 11 (+1) 217 (-2) -3.13 (dev) 180 (+4) 14 (-16) 25 (+14) 219 (+2) -main (3.14) 180 (same) 14 (same) 27 (+2) 221 (+2) +3.13.0 180 (+4) 14 (-16) 25 (+14) 219 (+2) +main (3.14) 180 (same) 14 (same) 28 (+3) 222 (+3) =========== ========== ========= ======== ========== Line Numbers @@ -65,8 +65,8 @@ Python Limited API CPython API Internal API Total 3.10.0 9,909 (49%) 4,584 (23%) 5,553 (28%) 20,046 3.11.0 8,733 (36%) 5,407 (22%) 10,079 (42%) 24,219 3.12.0 9,012 (28%) 5,836 (18%) 17,650 (54%) 32,498 -3.13 (dev) 9,097 (21%) 8,316 (19%) 26,897 (61%) 44,310 -main (3.14) 9,181 (20%) 8,401 (18%) 27,860 (61%) 45,442 +3.13.0 9,107 (21%) 8,316 (19%) 26,867 (61%) 44,290 +main (3.14) 9,262 (20%) 8,506 (18%) 29,359 (62%) 47,127 =========== ============= =========== ============ ====== File Numbers @@ -85,8 +85,8 @@ Python Limited API CPython API Internal API Total 3.10.0 80 (-17) 32 (+8) 49 (+15) 161 (+6) 3.11.0 71 (-9) 47 (+15) 69 (+20) 187 (+26) 3.12.0 71 (same) 49 (+2) 91 (+22) 211 (+24) -3.13 (dev) 74 (+3) 57 (+8) 123 (+32) 254 (+43) -main (3.14) 75 (+1) 57 (same) 126 (+3) 258 (+4) +3.13.0 74 (+3) 57 (+8) 123 (+32) 254 (+43) +main (3.14) 76 (+2) 57 (same) 130 (+7) 263 (+9) =========== =========== =========== ============ ========= Public functions defined as macros and static inline functions @@ -105,8 +105,8 @@ Python Macro Static inline Total 3.10.0 307 (same) 0 (same) 307 (same) 3.11.0 269 (-38) 42 (+42) 311 (+4) 3.12.0 251 (-18) 64 (+22) 315 (+4) -3.13 (dev) 249 (-2) 63 (-1) 312 (-3) -main (3.14) 250 (+1) 62 (-1) 312 (same) +3.13.0 249 (-2) 63 (-1) 312 (-3) +main (3.14) 251 (+2) 61 (-2) 312 (same) =========== ========== ============= ========== Only count public macros and public static inline functions (names starting with "Py" or "PY"). @@ -116,20 +116,20 @@ Private functions defined as macros and static inline functions Private functions defined as macros and static inline functions: -=========== ========= ============= ========= -Python Macro Static inline Total -=========== ========= ============= ========= -2.7.0 21 0 21 -3.6.0 60 (+39) 0 (same) 60 (+39) -3.7.0 70 (+10) 0 (same) 70 (+10) -3.8.0 57 (-13) 7 (+7) 64 (-6) -3.9.0 43 (-14) 11 (+4) 54 (-10) -3.10.0 44 (+1) 18 (+7) 62 (+8) -3.11.0 53 (+9) 4 (-14) 57 (-5) -3.12.0 62 (+9) 7 (+3) 69 (+12) -3.13 (dev) 55 (-7) 10 (+3) 65 (-4) -main (3.14) 55 (same) 10 (same) 65 (same) -=========== ========= ============= ========= +=========== ======== ============= ======== +Python Macro Static inline Total +=========== ======== ============= ======== +2.7.0 21 0 21 +3.6.0 60 (+39) 0 (same) 60 (+39) +3.7.0 70 (+10) 0 (same) 70 (+10) +3.8.0 57 (-13) 7 (+7) 64 (-6) +3.9.0 43 (-14) 11 (+4) 54 (-10) +3.10.0 44 (+1) 18 (+7) 62 (+8) +3.11.0 53 (+9) 4 (-14) 57 (-5) +3.12.0 62 (+9) 7 (+3) 69 (+12) +3.13.0 55 (-7) 10 (+3) 65 (-4) +main (3.14) 58 (+3) 10 (same) 68 (+3) +=========== ======== ============= ======== Only count private macros and public static inline functions (ignore names starting with "Py" or "PY"). @@ -149,8 +149,8 @@ Python Limited API CPython API Internal API Total 3.10.0 41 (-27) 53 (+7) 91 (+53) 185 (+33) 3.11.0 19 (-22) 75 (+22) 112 (+21) 206 (+21) 3.12.0 18 (-1) 85 (+10) 170 (+58) 273 (+67) -3.13 (dev) 16 (-2) 86 (+1) 232 (+62) 334 (+61) -main (3.14) 16 (same) 83 (-3) 230 (-2) 329 (-5) +3.13.0 16 (-2) 86 (+1) 231 (+61) 333 (+60) +main (3.14) 16 (same) 84 (-2) 232 (+1) 332 (-1) =========== =========== =========== ============ ========= Count also private structures like "_PyCFrame" and structures with names not starting with Py like "_frozen".