10000 Create more subpages · adamtheturtle/pythoncapi@343fd88 · GitHub
[go: up one dir, main page]
Skip to content

Commit 343fd88

Browse files
vstinnervstinner
committed
Create more subpages
1 parent bd0e4cb commit 343fd88

14 files changed

+186
-114
lines changed

‎doc/bad_api.rst

Lines changed: 27 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ only integers, it is stored as a compact C array of longs, and the W_IntObject
3838
is only created when an item is accessed (most of the time the W_IntObject is
3939
optimized away by the JIT, but this is another story).
4040

41-
But for cpyext, this is a problem: ``PyList_GetItem()`` returns a borrowed
41+
But for :ref:`cpyext <cpyext>`, this is a problem: ``PyList_GetItem()`` returns a borrowed
4242
reference, but there is no any concrete ``PyObject*`` to return! The current
4343
``cpyext`` solution is very bad: basically, the first time ``PyList_GetItem()``
4444
is called, the *whole* list is converted to a list of ``PyObject*``, just to
@@ -76,6 +76,8 @@ Functions
7676
* ``PyDict_SetDefault()``
7777
* ``PyErr_Occurred()``
7878
* ``PyEval_GetBuiltins()``
79+
* ``PyEval_GetFuncName()``: return the internal ``const char*`` inside a
80+
borrowed reference to a function ``__name__``.
7981
* ``PyFile_Name()``
8082
* ``PyFunction_GetClosure()``
8183
* ``PyFunction_GetCode()``
@@ -127,7 +129,7 @@ Border line
127129
.. _py-type:
128130

129131
Py_TYPE() corner case
130-
=====================
132+
---------------------
131133

132134
Technically, ``Py_TYPE()`` returns a borrowed reference to a ``PyTypeObject*``.
133135
In pratice, for heap types, an instance holds already a strong reference
@@ -144,11 +146,15 @@ See the discussion on capi-sig: `Open questions about borrowed reference.
144146
<https://mail.python.org/mm3/archives/list/capi-sig@python.org/thread/V5EMBIIJFJGJGBQPLCFFXCHAUFNTA45H/>`_
145147
(Sept 2018).
146148

147-
Borrowed references: PyEval_GetFuncName()
148-
=========================================
149149

150-
* ``PyEval_GetFuncName()`` returns the internal ``const char*`` inside a
151-
borrowed reference to a function ``__name__``.
150+
No public C functions if it can't be done in Python
151+
===================================================
152+
153+
There should not be C APIs that do something that you can't do in Python.
154+
155+
Example: the C buffer protocol, the Python ``memoryview`` type only expose a
156+
subset of ``buffer`` features.
157+
152158

153159
Array of pointers to Python objects (``PyObject**``)
154160
====================================================
@@ -265,3 +271,18 @@ a wide range of argument formats, but some of them leak implementation details:
265271
* ``s``: returns a pointer to internal storage
266272

267273
Is it an issue? Should we do something?
274+
275+
276+
For internal use only
277+
=====================
278+
279+
The C API documentation contains a few functions with the note "For internal
280+
use only". Examples:
281+
282+
* ``_PyImport_Init()``
283+
* ``PyImport_Cleanup()``
284+
* ``_PyImport_Fini()``
285+
286+
Why ``PyImport_Cleanup()`` is still a public method?
287+
288+
These functions should be made really private and removed from the C API.

‎doc/calling_conventions.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -136,5 +136,5 @@ Use case: call "C function" from a "C function".
136136
Two entry points? Regular ``PyObject*`` entry point, but efficient "C" entry
137137
point as well?
138138

139-
PyPy wants this, Cython would benefit as well.
139+
PyPy wants this, :ref:`Cython <cython>` would benefit as well.
140140

‎doc/cffi.rst

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
.. _cffi:
2+
3+
++++
4+
cffi
5+
++++
6+
7+
An alternative to the C API is to rewrite a C extension using `cffi
8+
<http://cffi.readthedocs.io/>`__.
9+
10+
XXX write a better rationale why migrating to cffi!
11+
12+
Questions:
13+
14+
* Is it easy to distribute binaries generated by cffi to avoid to require C
15+
headers and a C compiler? (Windows, macOS, Linux?)
16+
* How many popular Python modules use the C API? See :ref:`Consumers of the
17+
Python C API <consumers>`.
18+
* How long would it take to rewrite a C extension with ``cffi``?
19+
* What is the long-term transition plan to reach the "no C API" goal? At least,
20+
CPython will continue to use its own C API internally.
21+
* How to deal with :ref:`backward compatibility <back-compat>`?
22+
23+
Small practical issue: ``cffi`` is not part of the Python 3.7 standard library
24+
yet. Previous attempt to add it, in 2013: `[Python-Dev] cffi in stdlib
25+
<https://mail.python.org/pipermail/python-dev/2013-February/124337.html>`_.
26+
27+
See also :ref:`Cython <cython>` and :ref:`Remove the C API <remove-c-api>`.

‎doc/consumers.rst

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ Popular C extensions using the C API
99

1010
* numpy
1111
* pandas
12-
* cython
12+
* :`ref:`Cython <cython>`
1313
* Pillow
1414
* lxml
1515

@@ -18,6 +18,8 @@ Popular modules using Cython
1818

1919
* uvloop
2020

21+
See :ref:`Cython <cython>`.
22+
2123
.. _debug-tools:
2224

2325
Debugging tools

‎doc/cpyext.rst

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
.. _cpyext:
2+
3+
++++++++++++++++++
4+
PyPy cpyext module
5+
++++++++++++++++++
6+
7+
cpyext is the implementation of the C API in PyPy.
8+
9+
See Ronan Lamy's talk `Adventures in compatibility emulating CPython's C API in
10+
PyPy <https://www.youtube.com/watch?v=qH0eeh-4XE8>`_ (Youtube video) at
11+
EuroPython 2018.
12+
13+
Source
14+
======
15+
16+
See `pypy/module/cpyext/
17+
<https://bitbucket.org/pypy/pypy/src/default/pypy/module/cpyext/>`_ and
18+
`cpyext/stubs.py
19+
<https://bitbucket.org/pypy/pypy/src/default/pypy/module/cpyext/stubs.py>`_.
20+
21+
cpyext has unit tests written in Python.
22+
23+
Performance issue
24+
=================
25+
26+
PyPy with cpyext remains slower than CPython.
27+
28+
XXX how much?
29+
30+
Issue with borrowed references
31+
==============================
32+
33+
See :ref:`Borrowed references <borrowed-ref>`.
34+
35+
Replace macros with functions
36+
=============================
37+
38+
Already done in cpyext.

‎doc/cython.rst

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
.. _cython:
2+
3+
++++++++++++++++++++++++++++++++++
4+
The Cython C-Extensions for Python
5+
++++++++++++++++++++++++++++++++++
6+
7+
An alternative to using the C API directly is to rewrite a C extension using
8+
`Cython <http://cython.org/>`__ which generates C code using the C API.
9+
10+
XXX write a better rationale why migrating to Cython!
11+
12+
Questions:
13+
14+
* How many popular Python modules use Cython? See :ref:`Consumers of the
15+
Python C API <consumers>`.
16+
* How long would it take to rewrite a C extension with ``Cython``?
17+
* What is the long-term transition plan to reach the "no C API" goal? At least,
18+
CPython will continue to use its own C API internally.
19+
* How to deal with :ref:`backward compatibility <back-compat>`?
20+
21+
Small practical issue: ``Cython`` is not part of the Python 3.7 standard
22+
library yet.
23+
24+
See also :ref:`cffi <cffi>` and :ref:`Remove the C API <remove-c-api>`.

‎doc/index.rst

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ Pages
3333
.. toctree::
3434
:maxdepth: 2
3535

36+
rationale
3637
roadmap
3738
bad_api
3839
new_api
@@ -46,7 +47,11 @@ Pages
4647
calling_conventions
4748
stable_abi
4849
consumers
49-
rationale
50+
cpyext
51+
cython
52+
cffi
53+
remove_c_api
54+
performance
5055
split_include
5156

5257
Links

‎doc/new_api.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ Design goals
1515
question remains if it will be possible to replace ``Py_INCREF()`` and
1616
``Py_DECREF()`` with function calls without killing performances.
1717
* Reduce the size of the C API to reduce the maintenance burden of :ref:`Python
18-
implementations other than CPython <other-python-impl>`. See :ref:`Remove
18+
implementations other than CPython <other-python-impl>`: :ref:`Remove
1919
functions <remove-funcs>`.
2020
* Reduce the size of the ABI, especially export less symbols.
2121

@@ -56,7 +56,7 @@ functions/features are needed:
5656
Non-goal
5757
========
5858

59-
* Cython and cffi must be preferred to write new C extensions: there is no
59+
* :ref:`Cython <cython>` and cffi must be preferred to write new C extensions: there is no
6060
intent to replace Cython. Moreover, there is no intent to make Cython
6161
development harder. Cython will still be able to access directly the full C
6262
API which includes implementation details and low-level "private" APIs.

‎doc/old_c_api.rst

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ Old C API
77
The "Old C API" is the Python 3.7 API which "leaks" implementation details like
88
``PyObject.ob_refcnt`` through :ref:`Py_INCREF() <incref>`. This API will
99
remain available for CPython internals but also for specific use cases like
10-
Cython (for best performances) and :ref:`debugging tools <debug-tools>`.
10+
:ref:`Cython <cython>` (for best performances) and :ref:`debugging tools
11+
<debug-tools>`.
1112

1213
See also :ref:`Calling conventensions <calling-conventions>`.
1314

@@ -32,7 +33,7 @@ Current Python C API
3233
* CPython:
3334
`headers of the Include/ directory
3435
<https://github.com/python/cpython/tree/master/Include>`_
35-
* PyPy:
36+
* PyPy :ref:`cpyext <cpyext>`:
3637
`pypy/module/cpyext/
3738
<https://bitbucket.org/pypy/pypy/src/default/pypy/module/cpyext/>`_
3839
(`cpyext/stubs.py

‎doc/performance.rst

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
+++++++++++
2+
Performance
3+
+++++++++++
4+
5+
The C API exposes implementation details for historical reasons (there was no
6+
design for the public C API, the public C API is just the private API made
7+
public), but also for performance. Macros are designed for best performances,
8+
but should be reserved to developers who have a good understanding of CPython
9+
internals.
10+
11+
12+
Performance slowdown
13+
====================
14+
15+
Hiding implementation details is likely to make tiny loops slower, since it
16+
adds function calls instead of directly accessing the memory.
17+
18+
The performance slowdown is expected to be negligible, but has to be measured
19+
once a concrete implmenetation will be written.
20+
21+
Question: would it be acceptable to have a new better C API if the average
22+
slowdown is around 10%? What if the slowdown is up to 25%? Or even 50%?
23+
24+
Right now, the project is too young to guess anything or to bet. Performances
25+
will be carefully measured using the Python benchmark suite pyperformance,
26+
but only once the design of the new C API is complete.
27+

‎doc/rationale.rst

Lines changed: 1 addition & 95 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ The Big Carrot
1818
Changing the C API means that authors of C extensions have to do something. To
1919
justify these changes, we need a big carrot. Examples:
2020

21-
* faster Python if you pick new API? faster PyPy cpyext?
21+
* faster Python if you pick new API? faster PyPy :ref:`cpyext <cpyext>`?
2222
* less bugs? no more surprising borrow references causing "funny" crashes
2323
* new features?
2424

@@ -47,97 +47,3 @@ Issues with an unstable ABI
4747
debug builds would ease debugging since the debug builds add many debug
4848
checks which are too expensive (CPU/memory) to be enabled by default in a
4949
release build.
50-
51-
52-
Advantages of a stable ABI
53-
==========================
54-
55-
See :ref:`Optimization ideas <optim-ideas>`.
56-
57-
58-
59-
The C API is too big
60-
====================
61-
62-
**Goal:** Smaller C API.
63-
64 10000 -
Common complain from PyPy developers. Writing a new Python implementation with
65-
implements the full C API is a huge work.
66-
67-
Slowly **remove** functions from the future stable ABI? It should be done
68-
gradually and update most famous Python C extensions in parrallel to not "break
69-
Python".
70-
71-
Some C functions can easily be replaced by a function call, these functions are
72-
mostly written for **internal** usage, to make the CPython code base simpler.
73-
But they should not be exposed (they should be private).
74-
75-
76-
The C API must not leak implementation details anymore
77-
======================================================
78-
79-
**Goal:** Hide implementation details.
80-
81-
See :ref:`Bad C API <bad-c-api>`.
82-
83-
84-
Performance slowdown
85-
====================
86-
87-
Hiding implementation details is likely to make tiny loops slower, since it
88-
adds function calls instead of directly accessing the memory.
89-
90-
The performance slowdown is expected to be negligible, but has to be measured
91-
once a concrete implmenetation will be written.
92-
93-
94-
Alternative: Stop using the C API, use cffi
95-
===========================================
96-
97-
**Goal:** Remove the public C API. Or at least, stop using it.
98-
99-
Practical issue: ``cffi`` is not part of Python 3.7 standard library yet.
100-
Previous attempt to add it, in 2013: `[Python-Dev] cffi in stdlib
101-
<https://mail.python.org/pipermail/python-dev/2013-February/124337.html>`_.
102-
103-
Questions:
104-
105-
* How many popular Python modules use the C API?
106-
* How long would it take to rewrite a big famous Python module with ``cffi``?
107-
* What is the long-term transition plan to reach the "no C API" goal?
108-
109-
110-
Fix Python headers
111-
==================
112-
113-
**Goal**: Make private APIs private again: Py_BUILD_CORE vs Py_LIMITED_API.
114-
115-
Currently, the stable API (Py_LIMITED_API), the private functions (``_Py``
116-
prefix), functions that must only be used in CPython core (``Py_BUILD_CORE``)
117-
and other functions (regular C API) are all defined in the same file. It is
118-
easy to add a function to the wrong API by mistake.
119-
120-
121-
No public C functions if it can't be done in Python
122-
===================================================
123-
124-
**Goal**: Remove public functions which do things which are not doable in pure
125-
Python.
126-
127-
There shouldn't be C APIs that do something that you can't do in Python.
128-
129-
Example: the C buffer protocol, the Python ``memoryview`` type only expose a
130-
subset of ``buffer`` features.
131-
132-
133-
For internal use only
134-
=====================
135-
136-
The C API documentation contains a few functions with the note "For internal
137-
use only". Examples:
138-
139-
* _PyImport_Init()
140-
* PyImport_Cleanup()
141-
* _PyImport_Fini()
142-
143-
Why PyImport_Cleanup() is still a public method?

‎doc/remove_c_api.rst

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
.. _remove-c-api:
2+
3+
++++++++++++++++
4+
Remove the C API
5+
++++++++++++++++
6+
7+
One obvious alternative to a new better C API is no C API at all! Existing
8+
solutions are already available, complete and reliable:
9+
10+
* :ref:`Cython <cython>`
11+
* :ref:`cffi <cffi>`
12+
13+
Open questions: What about the long tail of C extensions on PyPI which still
14+
use the C extension? Would it mean a new Python without all these C extensions
15+
on PyPI?
16+
17+
The C API is part of Python success. There would be no numpy without the C API,
18+
for example. See :ref:`Consumers of the Python C API <consumers>`.

0 commit comments

Comments
 (0)
0