8000 bpo-32124: Document C functions safe before init by vstinner · Pull Request #4540 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

bpo-32124: Document C functions safe before init #4540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Nov 24, 2017
Merged

bpo-32124: Document C functions safe before init #4540

Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b1375be
bpo-32124: Document C functions safe before init
vstinner Nov 24, 2017
938684e
Add Py_DecodeLocale()
vstinner Nov 24, 2017
12b3b5f
Complete and fix the Preinit doc
vstinner Nov 24, 2017
b58aba4
Address new round of comments
vstinner Nov 24, 2017
b2fc166
Fix typo
vstinner Nov 24, 2017
e01c316
Only keep PyMem_Raw memory allocators
vstinner Nov 24, 2017
f1c305b
man page: wizards => expert
vstinner Nov 24, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 125 additions & 3 deletions Doc/c-api/init.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,126 @@
Initialization, Finalization, and Threads
*****************************************

.. _pre-init-safe:

Before Python Initialization
============================

In an application embedding Python, the :c:func:`Py_Initialize` function must
be called before using any other Python/C API functions; with the exception of
a few functions and flags listed below.

The following functions can be safetely called before Python is initialized:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyImport_AppendInittab(), PyImport_ExtendInittab(), PyInitFrozenExtensions().

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo :) safetely -> safely

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

Crap. "safetely" looked weird, so I used my spell check: "Google safetely", but I found results. Maybe there is an issue with my spell checker? :-)


* Functions:

* :c:func:`PyMem_GetAllocator`
* :c:func:`PyMem_SetAllocator`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How much of the PyMem_*() API do we really need to promise to embedders? AFAICS, we've only required them to use PyMem_RawFree() before initialization, due to Py_DecodeLocale(). The less we promise pre-init, the better.

Ideally we would provide equivalent pre-init functions, along with pre-init allocator state, that embedders could use if they needed.
If we do end up needing more, could we instead provide equivalent pre-init functions that take explicit allocator state that

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting the allocators pre-initialization is required, as that allows the embedding application to control exactly how CPython allocates memory during startup.

* :c:func:`PyMem_SetupDebugHooks`
* :c:func:`PyObject_GetArenaAllocator`
* :c:func:`PyObject_SetArenaAllocator`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we promising that PyObject_SetArenaAllocator() (and PyObject_GetArenaAllocator) is available before initialization? What do embedders need it for before init? Can we hold off adding it to this list until necessary?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same reason as for PyMem_SetAllocators: the embedding application needs to be able to specify how memory allocation during Py_Initialize() itself should be handled.

* :c:func:`Py_DecodeLocale`
* :c:func:`Py_GetProgramName`
* :c:func:`Py_SetPath`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If Py_GetProgramName is included, shouldn't other Py_Get* functions be included? Py_GetPath(), Py_GetVersion(), and many others.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Py_GetPath() must not be called before Py_Initialize().

I added other Py_GetXXX() functions.

* :c:func:`Py_SetProgramName`
* :c:func:`Py_SetPythonHome`
* :c:func:`Py_SetStandardStreamEncoding`

* Memory allocators:

* :c:func:`PyMem_RawMalloc`
* :c:func:`PyMem_RawRealloc`
:c:func:`PyMem_RawCalloc`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there be a * in the beginning, like the rest?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo, fixed

* :c:func:`PyMem_RawFree`
* :c:func:`PyMem_Malloc`
* :c:func:`PyMem_Realloc`
* :c:func:`PyMem_Calloc`
* :c:func:`PyMem_Free`
* :c:func:`PyObject_Malloc`
* :c:func:`PyObject_Realloc`
* :c:func:`PyObject_Calloc`
* :c:func:`PyObject_Free`

The following flags can also be set:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is not this the all flags?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the text.


* :c:data:`Py_BytesWarningFlag`
* :c:data:`Py_DebugFlag`
* :c:data:`Py_InspectFlag`
* :c:data:`Py_InteractiveFlag`
* :c:data:`Py_IsolatedFlag`
* :c:data:`Py_OptimizeFlag`
* :c:data:`Py_DontWriteBytecodeFlag`
* :c:data:`Py_NoUserSiteDirectory`
* :c:data:`Py_NoSiteFlag`
* :c:data:`Py_UnbufferedStdioFlag`
* :c:data:`Py_VerboseFlag`
* :c:data:`Py_QuietFlag`
* :c:data:`Py_IgnoreEnvironmentFlag`


Global configuration variables
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In some ways, I'd rather we not advertise these flags as global variables, and particularly not as stable pre-init state. If the goal is PEP 432 then further publicizing this state helps to entrench and persist the API from which we're trying to move away.

On the other hand, PEP 432 and its API is not accepted (yet) and the status quo suggests that the documentation you've added here for existing state is useful.

From my biased position, I'd favor leaving this part out for now, but only if we're serious about PEP 432. :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For PEP 432, I want us to consider "Don't break backwards compatibility with the current embedding API" as a hard constraint. Part of that process is going to be reverse engineering what the current embedding API actually is in practice, since we don't have it fully documented, and we definitely don't have it fully tested.

So in a post-PEP-432 world, we'll end up with a situation where we have a "Legacy Embedding API" that we promise not to break (at least, not prior to Python 4.0 at the very earliest), and then a more fine-grained configuration-struct-based one.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In some ways, I'd rather we not advertise these flags as global variables, and particularly not as stable pre-init state.

What do you mean by "not stable"? The inital state of these flags is well defined.

These flags are already actively used in the wild, it's lilely to only way to tune some Python options.

Not documenting them only makes Python harder to embed, it doesn't prevent users to use it.

Moreover, variables are exported as public symbols (prefixed with "Py", not "_Py").

==============================

Python has variables for the global configuration to control different features
and options. By default, these flags are controlled by :ref:`command line
options <using-on-interface-options>`.

.. :c:data: Py_BytesWarningFlag
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

.. c:var:: Py_BytesWarningFlag

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


The :option:`-b` option.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong indentation (should be 3 spaces).

Document what is the value if use -bb. And the same for -OO below.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


.. :c:data: Py_DebugFlag

The :option:`-d` option.

.. :c:data: Py_InspectFlag

The :option:`-i` option.

.. :c:data: Py_InteractiveFlag

The :option:`-i` option.

.. :c:data: Py_IsolatedFlag

The :option:`-I` option.

.. versionadded:: 3.4

.. :c:data: Py_OptimizeFlag

The :option:`-O` option.

.. :c:data: Py_DontWriteBytecodeFlag

The :option:`-B` option.

.. :c:data: Py_NoUserSiteDirectory

The :option:`-s` option.

.. :c:data: Py_NoSiteFlag

The :option:`-S` option.

.. :c:data: Py_UnbufferedStdioFlag

The :option:`-u` option.

.. :c:data: Py_VerboseFlag

The :option:`-v` option.

.. :c:data: Py_QuietFlag

The :option:`-q` option.

.. versionadded:: 3.2

.. :c:data: Py_IgnoreEnvironmentFlag

The :option:`-E` and :option:`-I` options.


Initializing and finalizing the interpreter
===========================================
Expand All @@ -27,9 +147,11 @@ Initializing and finalizing the interpreter
single: PySys_SetArgvEx()
single: Py_FinalizeEx()

Initialize the Python interpreter. In an application embedding Python, this
should be called before using any other Python/C API functions; with the
exception of :c:func:`Py_SetProgramName`, :c:func:`Py_SetPythonHome` and :c:func:`Py_SetPath`. This initializes
Initialize the Python interpreter. In an application embedding Python,
this should be called before using any other Python/C API functions; see
:ref:`Before Python Initialization <pre-init-safe>` for the few exceptions.

This initializes
the table of loaded modules (``sys.modules``), and creates the fundamental
modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`. It also initializes
the module search path (``sys.path``). It does not set ``sys.argv``; use
Expand Down
0