8000 Create more subpages · mscherer/pythoncapi@343fd88 · GitHub
[go: up one dir, main page]
Skip to content

Commit 343fd88

Browse files
vstinnervstinner
co 8000 mmitted
Create more subpages
1 parent bd0e4cb commit 343fd88

14 files changed

+186
-114
lines changed

‎doc/bad_api.rst

Lines changed: 27 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ only integers, it is stored as a compact C array of longs, and the W_IntObject
3838
is only created when an item is accessed (most of the time the W_IntObject is
3939
optimized away by the JIT, but this is another story).
4040

41-
But for cpyext, this is a problem: ``PyList_GetItem()`` returns a borrowed
41+
But for :ref:`cpyext <cpyext>`, this is a problem: ``PyList_GetItem()`` returns a borrowed
4242
reference, but there is no any concrete ``PyObject*`` to return! The current
4343
``cpyext`` solution is very bad: basically, the first time ``PyList_GetItem()``
4444
is called, the *whole* list is converted to a list of ``PyObject*``, just to
@@ -76,6 +76,8 @@ Functions
7676
* ``PyDict_SetDefault()``
7777
* ``PyErr_Occurred()``
7878
* ``PyEval_GetBuiltins()``
79+
* ``PyEval_GetFuncName()``: return the internal ``const char*`` inside a
80+
borrowed reference to a function ``__name__``.
7981
* ``PyFile_Name()``
8082
* ``PyFunction_GetClosure()``
8183
* ``PyFunction_GetCode()``
@@ -127,7 +129,7 @@ Border line
127129
.. _py-type:
128130

129131
Py_TYPE() corner case
130-
=====================
132+
---------------------
131133

132134
Technically, ``Py_TYPE()`` returns a borrowed reference to a ``PyTypeObject*``.
133135
In pratice, for heap types, an instance holds already a strong reference
@@ -144,11 +146,15 @@ See the discussion on capi-sig: `Open questions about borrowed reference.
144146
<https://mail.python.org/mm3/archives/list/capi-sig@python.org/thread/V5EMBIIJFJGJGBQPLCFFXCHAUFNTA45H/>`_
145147
(Sept 2018).
146148

147-
Borrowed references: PyEval_GetFuncName()
148-
=========================================
149149

150-
* ``PyEval_GetFuncName()`` returns the internal ``const char*`` inside a
151-
borrowed reference to a function ``__name__``.
150+
No public C functions if it can't be done in Python
151+
===================================================
152+
153+
There should not be C APIs that do something that you can't do in Python.
154+
155+
Example: the C buffer protocol, the Python ``memoryview`` type only expose a
156+
subset of ``buffer`` features.
157+
152158

153159
Array of pointers to Python objects (``PyObject**``)
154160
====================================================
@@ -265,3 +271,18 @@ a wide range of argument formats, but some of them leak implementation details:
265271
* ``s``: returns a pointer to internal storage
266272

267273
Is it an issue? Should we do something?
274+
275+
276+
For internal use only
277+
=====================
278+
279+
The C API documentation contains a few functions with the note "For internal
280+
use only". Examples:
281+
282+
* ``_PyImport_Init()``
283+
* ``PyImport_Cleanup()``
284+
* ``_PyImport_Fini()``
285+
286+
Why ``PyImport_Cleanup()`` is still a public method?
287+
288+
These functions should be made really private and removed from the C API.

‎doc/calling_conventions.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -136,5 +136,5 @@ Use case: call "C function" from a "C function".
136136
Two entry points? Regular ``PyObject*`` entry point, but efficient "C" entry
137137
point as well?
138138

139-
PyPy wants this, Cython would benefit as well.
139+
PyPy wants this, :ref:`Cython <cython>` would benefit as well.
140140

‎doc/cffi.rst

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
.. _cffi:
2+
3+
++++
4+
cffi
5+
++++
6+
7+
An alternative to the C API is to rewrite a C extension using `cffi
8+
<http://cffi.readthedocs.io/>`__.
9+
10+
XXX write a better rationale why migrating to cffi!
11+
12+
Questions:
13+
14+
* Is it easy to distribute binaries generated by cffi to avoid to require C
15+
headers and a C compiler? (Windows, macOS, Linux?)
16+
* How many popular Python modules use the C API? See :ref:`Consumers of the
17+
Python C API <consumers>`.
18+
* How long would it take to rewrite a C extension with ``cffi``?
19+
* What is the long-term transition plan to reach the "no C API" goal? At least,
20+
CPython will continue to use its own C API internally.
21+
* How to deal with :ref:`backward compatibility <back-compat>`?
22+
23+
Small practical issue: ``cffi`` is not part of the Python 3.7 standard library
24+
yet. Previous attempt to add it, in 2013: `[Python-Dev] cffi in stdlib
25+
<https://mail.python.org/pipermail/python-dev/2013-February/124337.html>`_.
26+
27+
See also :ref:`Cython <cython>` and :ref:`Remove the C API <remove-c-api>`.

‎doc/consumers.rst

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ Popular C extensions using the C API
99

1010
* numpy
1111
* pandas
12-
* cython
12+
* :`ref:`Cython <cython>`
1313
* Pillow
1414
* lxml
1515

@@ -18,6 +18,8 @@ Popular modules using Cython
1818

1919
* uvloop
2020

21+
See :ref:`Cython <cython>`.
22+
2123
.. _debug-tools:
2224

2325
Debugging tools

‎doc/cpyext.rst

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
.. _cpyext:
2+
3+
++++++++++++++++++
4+
PyPy cpyext module
5+
++++++++++++++++++
6+
7+
cpyext is the implementation of the C API in PyPy.
8+
9+
See Ronan Lamy's talk `Adventures in compatibility emulating CPython's C API in
10+
PyPy <https://www.youtube.com/watch?v=qH0eeh-4XE8>`_ (Youtube video) at
11+
EuroPython 2018.
12+
13+
Source
14+
======
15+
16+
See `pypy/module/cpyext/
17+
<https://bitbucket.org/pypy/pypy/src/default/pypy/module/cpyext/>`_ and
18+
`cpyext/stubs.py
19+
<https://bitbucket.org/pypy/pypy/src/default/pypy/module/cpyext/stubs.py>`_.
20+
21+
cpyext has unit tests written in Python.
22+
23+
Performance issue
24+
=================
25+
26+
PyPy with cpyext remains slower than CPython.
27+
28+
XXX how much?
29+
30+
Issue with borrowed references
31+
==============================
32+
33+
See :ref:`Borrowed references <borrowed-ref>`.
34+
35+
Replace macros with functions
36+
=============================
37+
38+
Already done in cpyext.

‎doc/cython.rst

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
.. _cython:
2+
3+
++++++++++++++++++++++++++++++++++
4+
The Cython C-Extensions for Python
5+
++++++++++++++++++++++++++++++++++
6+
7+
An alternative to using the C API directly is to rewrite a C extension using
8+
`Cython <http://cython.org/>`__ which generates C code using the C API.
9+
10+
XXX write a better rationale why migrating to Cython!
11+
12+
Questions:
13+
14+
* How many popular Python modules use Cython? See :ref:`Consumers of the
15+
Python C API <consumers>`.
16+
* How long would it take to rewrite a C extension with ``Cython``?
17+
* What is the long-term transition plan to reach the "no C API" goal? At least,
18+
CPython will continue to use its own C API internally.
19+
* How to deal with :ref:`backward compatibility <back-compat>`?
20+
21+
Small practical issue: ``Cython`` is not part of the Python 3.7 standard
22+
library yet.
23+
24+
See also :ref:`cffi <cffi>` and :ref:`Remove the C API <remove-c-api>`.

‎doc/index.rst

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ Pages
3333
.. toctree::
3434
:maxdepth: 2
3535

36+
rationale
3637
roadmap
3738
bad_api
3839
new_api
@@ -46,7 +47,11 @@ Pages
4647
calling_conventions
4748
stable_abi
4849
consumers
49-
rationale
50+
cpyext
51+
cython
52+
cffi
53+
remove_c_api
54+
performance
5055
split_include
5156

5257
Links

‎doc/new_api.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ Design goals
1515
question remains if it will be possible to replace ``Py_INCREF()`` and
1616
``Py_DECREF()`` with function calls without killing performances.
1717
* Reduce the size of the C API to reduce the maintenance burden of :ref:`Python
18-
implementations other than CPython <other-python-impl>`. See :ref:`Remove
18+
implementations other than CPython <other-python-impl>`: :ref:`Remove
1919
functions <remove-funcs>`.
2020
* Reduce the size of the ABI, especially export less symbols.
2121

@@ -56,7 +56,7 @@ functions/features are needed:
5656
Non-goal
5757
========
5858

59-
* Cython and cffi must be preferred to write new C extensions: there is no
59+
* :ref:`Cython <cython>` and cffi must be preferred to write new C extensions: there is no
6060
intent to replace Cython. Moreover, there is no intent to make Cython
6161
development harder. Cython will still be able to access directly the full C
6262
API which includes implementation details and low-level "private" APIs.

‎doc/old_c_api.rst

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ Old C API
77
The "Old C API" is the Python 3.7 API which "leaks" implementation details like
88
``PyObject.ob_refcnt`` through :ref:`Py_INCREF() <incref>`. This API will
99
remain available for CPython internals but also for specific use cases like
10-
Cython (for best performances) and :ref:`debugging tools <debug-tools>`.
10+
:ref:`Cython <cython>` (for best performances) and :ref:`debugging tools
11+
<debug-tools>`.
1112

1213
See also :ref:`Calling conventensions <calling-conventions>`.
1314

@@ -32,7 +33,7 @@ Current Python C API
3233
* CPython:
3334
`headers of the Include/ directory
3435
<https://github.com/python/cpython/tree/master/Include>`_
35-
* PyPy:
36+
* PyPy :ref:`cpyext <cpyext>`:
3637
`pypy/module/cpyext/
3738
<https://bitbucket.org/pypy/pypy/src/default/pypy/module/cpyext/>`_
3839
(`cpyext/stubs.py

‎doc/performance.rst

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
+++++++++++
2+
Performance
3+
+++++++++++
4+
5+
The C API exposes implementation details for historical reasons (there was no
6+
design for the public C API, the public C API is just the private API made
7+
public), but also for performance. Macros are designed for best performances,
8+
but should be reserved to developers who have a good understanding of CPython
9+
internals.
10+
11+
12+
Performance slowdown
13+
====================
14+
15+
Hiding implementation details is likely to make tiny loops slower, since it
16+
adds function calls instead of directly accessing the memory.
17+
18+
The performance slowdown is expected to be negligible, but has to be measured
19+
once a concrete implmenetation will be written.
20+
21+
Question: would it be acceptable to have a new better C API if the average
22+
slowdown is around 10%? What if the slowdown is up to 25%? Or even 50%?
23+
24+
Right now, the project is too young to guess anything or to bet. Performances
25+
will be carefully measured using the Python benchmark suite pyperformance,
26+
but only once the design of the new C API is complete.
27+

0 commit comments

Comments
 (0)
0