Description
The use of undocumented CPython internals in Cython causes subtle bugs and makes in unnecessarily difficult for Cython based modules to be used across different versions of (C)Python.
The latest example of a breakage (amongst many) is https://bugs.python.org/issue43760
We[1] are happy to consider API extensions to support Cython. We regard Cython as an important part of the Python ecosystem, and would love to cooperate, rather than conflict with Cython.
This cooperation has proved fruitful in the past.
For example, PEP 590 was designed to suuport Cython generated callables. Because it had to be designed to be both fast and general it has proved beneficial to CPython performance as well.
Cython has many __Pyx_Blah functions which access CPython internals. Moving these to the CPython C-API has a lot of advantages for both CPython and Cython:
- Improved reliability of Cython generated code. Subtle changes in the implicit semantics of CPython internals will not break Cython code in non-obvious ways.
- Improved portability of Cython modules across Python versions.
- Reduced maintenance for Cython: Just use the CPython API; no need to worry about CPython internals
- Reduced maintenance for CPython: We can treat C structs as opaque and not worry about breakage, as long as we maintain the C API functions
Performance
I assume that there are two reasons that Cython uses the direct access to CPython internals:
- There is no suitable API. As stated above, if there is no suitable API for a given purpose then please propose one[2].
- For performance.
Regarding performance, a well designed API may well be faster than probing and modifying CPython internals. It allows the CPython VM to make decisions as when to modify state lazily and when to cache values.
There will be a few things where the overhead of a call would be too high, but they should be very few, and understood by both CPython developers and Cython developers. The obvious example is Py_INCREF and Py_DECREF.
[1] The CPython developers. I hope the other CPython developers don't mind me speaking for them.
[2] Please propose APIs at as high and as abstract a level as possible. E.g don't propose an API for inserting Python frames in the frame stack, and for creating code objects. Propose an API for inserting a frame into the frame stack with writable location information.