10000 bpo-32124: Document C functions safe before init by vstinner · Pull Request #4540 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

bpo-32124: Document C functions safe before init #4540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Nov 24, 2017
Merged
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
243 changes: 233 additions & 10 deletions Doc/c-api/init.rst
10000
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,227 @@
Initialization, Finalization, and Threads
*****************************************

.. _pre-init-safe:

Before Python Initialization
============================

In an application embedding Python, the :c:func:`Py_Initialize` function must
be called before using any other Python/C API functions; with the exception of
a few functions and the :ref:`global configuration variables <global-conf-vars>`:

The following functions can be safetely called before Python is initialized:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyImport_AppendInittab(), PyImport_ExtendInittab(), PyInitFrozenExtensions().

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo :) safetely -> safely

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

Crap. "safetely" looked weird, so I used my spell check: "Google safetely", but I found results. Maybe there is an issue with my spell checker? :-)


* Configuration functions:

* :c:func:`PyImport_AppendInittab`
* :c:func:`PyImport_ExtendInittab`
* :c:func:`PyInitFrozenExtensions`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is PyInitFrozenExtensions() in this list?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was a request from @serhiy-storchaka. I don't know the rationale :-)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is called before Py_Initialize() in Python/frozenmain.c on Windows.

* :c:func:`PyMem_GetAllocator`
* :c:func:`PyMem_SetAllocator`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How much of the PyMem_*() API do we really need to promise to embedders? AFAICS, we've only required them to use PyMem_RawFree() before initialization, due to Py_DecodeLocale(). The less we promise pre-init, the better.

Ideally we would provide equivalent pre-init functions, along with pre-init allocator state, that embedders could use if they needed.
If we do end up needing more, could we instead provide equivalent pre-init functions that take explicit allocator state that

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting the allocators pre-initialization is required, as that allows the embedding application to control exactly how CPython allocates memory during startup.

* :c:func:`PyMem_SetupDebugHooks`
* :c:func:`PyObject_GetArenaAllocator`
* :c:func:`PyObject_SetArenaAllocator`
* :c:func:`Py_GetBuildInfo`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would group Py_GetBuildInfo() etc as "informative functions".

* :c:func:`Py_GetCompiler`
* :c:func:`Py_GetCopyright`
* :c:func:`Py_GetPlatform`
* :c:func:`Py_GetProgramName`
* :c:func:`Py_GetVersion`
* :c:func:`Py_SetPath`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If Py_GetProgramName is included, shouldn't other Py_Get* functions be included? Py_GetPath(), Py_GetVersion(), and many others.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Py_GetPath() must not be called before Py_Initialize().

I added other Py_GetXXX() functions.

* :c:func:`Py_SetProgramName`
* :c:func:`Py_SetPythonHome`
* :c:func:`Py_SetStandardStreamEncoding`

* Utilities:

* :c:func:`Py_DecodeLocale`

* Memory allocators:

* :c:func:`PyMem_RawMalloc`
* :c:func:`PyMem_RawRealloc`
* :c:func:`PyMem_RawCalloc`
* :c:func:`PyMem_RawFree`
* :c:func:`PyMem_Malloc`
* :c:func:`PyMem_Realloc`
* :c:func:`PyMem_Calloc`
* :c:func:`PyMem_Free`
* :c:func:`PyObject_Malloc`
* :c:func:`PyObject_Realloc`
* :c:func:`PyObject_Calloc`
* :c:func:`PyObject_Free`


.. note::

:c:func:`Py_GetPath`, :c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix` and
:c:func:`Py_GetProgramFullPath` and :c:func:`Py_GetPythonHome` **should not
be called** before :c:func:`Py_Initialize`.


.. _global-conf-vars:

Global configuration variables
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In some ways, I'd rather we not advertise these flags as global variables, and particularly not as stable pre-init state. If the goal is PEP 432 then further publicizing this state helps to entrench and persist the API from which we're trying to move away.

On the other hand, PEP 432 and its API is not accepted (yet) and the status quo suggests that the documentation you've added here for existing state is useful.

From my biased position, I'd favor leaving this part out for now, but only if we're serious about PEP 432. :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For PEP 432, I want us to consider "Don't break backwards compatibility with the current embedding API" as a hard constraint. Part of that process is going to be reverse engineering what the current embedding API actually is in practice, since we don't have it fully documented, and we definitely don't have it fully tested.

So in a post-PEP-432 world, we'll end up with a situation where we have a "Legacy Embedding API" that we promise not to break (at least, not prior to Python 4.0 at the very earliest), and then a more fine-grained configuration-struct-based one.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In some ways, I'd rather we not advertise these flags as global variables, and particularly not as stable pre-init state.

What do you mean by "not stable"? The inital state of these flags is well defined.

These flags are already actively used in the wild, it's lilely to only way to tune some Python options.

Not documenting them only makes Python harder to embed, it doesn't prevent users to use it.

Moreover, variables are exported as public symbols (prefixed with "Py", not "_Py").

==============================

Python has variables for the global configuration to control different features
and options. By default, these flags are controlled by :ref:`command line
options <using-on-interface-options>`.

When a flag is set by an option, the value of the flag is the number of times
that the option was set. For example, ``-b`` sets :c:data:`Py_BytesWarningFlag`
to 1 and ``-bb`` sets :c:data:`Py_BytesWarningFlag` to 2.

.. c:var:: Py_BytesWarningFlag

Issue a warning when comparing :class:`bytes` or :class:`bytearray` with
:class:`str` or :class:`bytes` with :class:`int`. Issue an error if greater
or equal to ``2``.

Set by the :option:`-b` option.

.. c:var:: Py_DebugFlag

Turn on parser debugging output (for wizards only, depending on compilation
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the word wizards and thought of magicians :) Do you mean something like setup wizard?
Or only a real wizard/magician can use this flag? 😅
Maybe this is a common terminology and I'm just too new.. 🤷‍♀️

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Victor meant real wizards, like his and me. 🎩

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok in that case I suggest something less ambiguous, like "for expert" or "for experienced core devs" only.
This sentence also missed a period (.) at the end.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wording already is used two times in the documentation. In particularly in the documentation for the -d option. This is very old documentation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In sum, this option is only for those who understand this jargon. 😉

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I copied the doc from the -d option documentation.

Ok, I replaced "for wizards only" with "for expert only".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would keep the original wording. It is used also in the man page since the initial edition in 1994.

options).

Set by the :option:`-d` option and the :envvar:`PYTHONDEBUG` environment
variable.

.. c:var:: Py_DontWriteBytecodeFlag

If set to non-zero, Python won't try to write ``.pyc`` files on the
import of source modules.

Set by the :option:`-B` option and the :envvar:`PYTHONDONTWRITEBYTECODE`
environment variable.

.. c:var:: Py_HashRandomizationFlag

Set to ``1`` if the :envvar:`PYTHONHASHSEED` environment variable is set to
a non-empty string.

If the flag is non-zero, read the :envvar:`PYTHONHASHSEED` environment
variable to initialize the secret hash seed.

.. c:var:: Py_IgnoreEnvironmentFlag

Ignore all :envvar:`PYTHON*` environment variables, e.g.
:envvar:`PYTHONPATH` and :envvar:`PYTHONHOME`, that might be set.

Set by the :option:`-E` and :option:`-I` options.

.. c:var:: Py_InspectFlag

When a script is passed as first argument or the :option:`-c` option is used,
enter interactive mode after executing the script or the command, even when
:data:`sys.stdin` does not appear to be a terminal.

Set by the :option:`-i` option and the :envvar:`PYTHONINSPECT` environment
variable.

.. c:var:: Py_InteractiveFlag

Set by the :option:`-i` option.

.. c:var:: Py_IsolatedFlag

Run Python in isolated mode. In isolated mode :data:`sys.path` contains
neither the script's directory nor the user's site-packages directory.

Set by the :option:`-I` option.

.. versionadded:: 3.4

.. c:var:: Py_LegacyWindowsFSEncodingFlag

If the flag is non-zero, use the ``mbcs`` encoding instead of the UTF-8
encoding for the filesystem encoding.

Set to ``1`` if the :envvar:`PYTHONLEGACYWINDOWSFSENCODING` environment
variable is set to a non-empty string.

See also the :pep:`529`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See :pep:`529` for more details.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See also :pep:`529`.

(without "the")

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I picked: See :pep:529 for more details.


Availability: Windows.

.. c:var:: Py_LegacyWindowsStdioFlag

If the flag is non-zero, use :class:`io.FileIO` instead of
:class:`WindowsConsoleIO` for :mod:`sys` standard streams.

Set to ``1`` if the :envvar:`PYTHONLEGACYWINDOWSSTDIO` environment
variable is set to a non-empty string.

See also the :pep:`528`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove "the".


Availability: Windows.

.. c:var:: Py_NoSiteFlag

Disable the import of the module :mod:`site` and the site-dependent
manipulations of :data:`sys.path` that it entails. Also disable these
manipulations if :mod:`site` is explicitly imported later (call
:func:`site.main` if you want them to be triggered).

Set by the :option:`-S` option.

.. c:var:: Py_NoUserSiteDirectory

Don't add the :data:`user site-packages directory <site.USER_SITE>` to
:data:`sys.path`.

Set by the :option:`-s` and :option:`-I` options, and the
:envvar:`PYTHONNOUSERSITE` environment variable.

.. c:var:: Py_OptimizeFlag

Set by the :option:`-O` option and the :envvar:`PYTHONOPTIMIZE` environment
variable.

.. c:var:: Py_QuietFlag

Don't display the copyright and version messages even in interactive mode.

Set by the :option:`-q` option.

.. versionadded:: 3.2

.. c:var:: Py_UnbufferedStdioFlag

Force the stdout and stderr streams to be unbuffered.

Set by the :option:`-u` option and the :envvar:`PYTHONUNBUFFERED`
environment variable.

.. c:var:: Py_VerboseFlag

Print a message each time a module is initialized, showing the place
(filename or built-in module) from which it is loaded. If greater or equal
to ``2``, print a message for each file that is checked for when
searching for a module. Also provides information on module cleanup at exit.

Set by the :option:`-v` option and the :envvar:`PYTHONVERBOSE` environment
variable.


Other variables:

.. c:var:: Py_FrozenFlag

Suppress error messages when calculating the module search path in
:c:func:`Py_GetPath`.

Private flag used by ``_freeze_importlib`` and ``frozenmain`` programs.

.. c:var:: Py_UseClassExceptionsFlag
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that it's a good idea to document this flag, since I proposed to remove the flag https://bugs.python.org/issue32125

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed


Deprecated and ignored flag, used to enable class exceptions prior Python
2.0.

.. deprecated:: 2.0


Initializing and finalizing the interpreter
===========================================
Expand All @@ -27,9 +248,11 @@ Initializing and finalizing the interpreter
single: PySys_SetArgvEx()
single: Py_FinalizeEx()

Initialize the Python interpreter. In an application embedding Python, this
should be called before using any other Python/C API functions; with the
exception of :c:func:`Py_SetProgramName`, :c:func:`Py_SetPythonHome` and :c:func:`Py_SetPath`. This initializes
Initialize the Python interpreter. In an application embedding Python,
this should be called before using any other Python/C API functions; see
:ref:`Before Python Initialization <pre-init-safe>` for the few exceptions.

This initializes
the table of loaded modules (``sys.modules``), and creates the fundamental
modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`. It also initializes
the module search path (``sys.path``). It does not set ``sys.argv``; use
Expand Down Expand Up @@ -1092,7 +1315,7 @@ Python-level trace functions in previous versions.
+------------------------------+--------------------------------------+


.. c:var:: int PyTrace_CALL
.. c:var::: int PyTrace_CALL
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't the third colon redundant?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, it's a mistake. Fixed.


The value of the *what* parameter to a :c:type:`Py_tracefunc` function when a new
call to a function or method is being reported, or a new entry into a generator.
Expand All @@ -1101,7 +1324,7 @@ Python-level trace functions in previous versions.
frame.


.. c:var:: int PyTrace_EXCEPTION
.. c:var::: int PyTrace_EXCEPTION

The value of the *what* parameter to a :c:type:`Py_tracefunc` function when an
exception has been raised. The callback function is called with this value for
Expand All @@ -1112,31 +1335,31 @@ Python-level trace functions in previous versions.
these events; they are not needed by the profiler.


.. c:var:: int PyTrace_LINE
.. c:var::: int PyTrace_LINE

The value passed as the *what* parameter to a trace function (but not a
profiling function) when a line-number event is being reported.


.. c:var:: int PyTrace_RETURN
.. c:var::: int PyTrace_RETURN

The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a
call is returning without propagating an exception.


.. c:var:: int PyTrace_C_CALL
.. c:var::: int PyTrace_C_CALL

The v 6651 alue for the *what* parameter to :c:type:`Py_tracefunc` functions when a C
function is about to be called.


.. c:var:: int PyTrace_C_EXCEPTION
.. c:var::: int PyTrace_C_EXCEPTION

The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a C
function has raised an exception.


.. c:var:: int PyTrace_C_RETURN
.. c:var::: int PyTrace_C_RETURN

The value for the *what* parameter to :c:type:`Py_tracefunc` functions when a C
function has returned.
Expand Down
0