8000 Document functions safe to be called before Py_Initialize() · Issue #76305 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

Document functions safe to be called before Py_Initialize() #76305

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
vstinner opened this issue Nov 24, 2017 · 10 comments
Closed

Document functions safe to be called before Py_Initialize() #76305

vstinner opened this issue Nov 24, 2017 · 10 comments
Labels
3.7 (EOL) end of life docs Documentation in the Doc dir

Comments

@vstinner
Copy link
Member
BPO 32124
Nosy @ncoghlan, @vstinner, @ericsnowcurrently, @serhiy-storchaka
PRs
  • bpo-32124: Document C functions safe before init #4540
  • bpo-32124: C API preinit doc, change note formatting #4621
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2017-12-15.15:09:10.756>
created_at = <Date 2017-11-24.13:19:15.972>
labels = ['3.7', 'docs']
title = 'Document functions safe to be called before Py_Initialize()'
updated_at = <Date 2017-12-15.15:09:10.756>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2017-12-15.15:09:10.756>
actor = 'vstinner'
assignee = 'docs@python'
closed = True
closed_date = <Date 2017-12-15.15:09:10.756>
closer = 'vstinner'
components = ['Documentation']
creation = <Date 2017-11-24.13:19:15.972>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 32124
keywords = ['patch']
message_count = 10.0
messages = ['306894', '306898', '306902', '306912', '306916', '306917', '306921', '306922', '306925', '306937']
nosy_count = 5.0
nosy_names = ['ncoghlan', 'vstinner', 'docs@python', 'eric.snow', 'serhiy.storchaka']
pr_nums = ['4540', '4621']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue32124'
versions = ['Python 3.7']
    @vstinner
    Copy link
    Member Author

    Follow-up of bpo-32086, bpo-32096 and "[Python-Dev] Python initialization and embedded Python" thread:
    https://mail.python.org/pipermail/python-dev/2017-November/150605.html

    I propose to explicitly list functions that can be safetely called before Py_Initialize(). This safety warranty must be part of the C API.

    Maybe we should even test all tests function in test_capi using Programs/_testembed, as we did for Py_DecodeLocale() and Py_SetProgramName() in commit 9e87e77 ("pre_initialization_api" test).

    Attached PR adds proposed documentation. It also documents "global configuration variables" like Py_DebugFlag.
    8000

    @vstinner vstinner added the 3.7 (EOL) end of life label Nov 24, 2017
    @vstinner vstinner added the docs Documentation in the Doc dir label Nov 24, 2017
    @serhiy-storchaka
    Copy link
    Member

    Are you sure about PyMem_Malloc() and PyObject_Malloc()? What functions require them? I thought only PyMem_RawMalloc() can be called before Py_Initialize().

    I think that for all functions that *can* or *should* be called before Py_Initialize() this should be explicitly documented in the documentation of this function, like for Py_SetProgramName() and PyImport_AppendInittab().

    @vstinner
    Copy link
    Member Author

    Are you sure about PyMem_Malloc() and PyObject_Malloc()?

    Technically, the pymalloc memory allocator is initialized statically by the compiler, from the first instruction of the process.

    But maybe we should not suggest users to call them, especially because the allocator can be modified by the PYTHONMALLOC environment variable.

    What functions require them?

    No function to initalize Python require PyMem or PyObject allocators. Only PyMem_Raw allocator is needed.

    Py_EncodeLocale() uses it, but this function also uses Python objects (str, bytes), and so Py_EncodeLocale() must no be called before Py_Initialize().

    @serhiy-storchaka
    Copy link
    Member

    Should PyMem_SetAllocator() and PyObject_SetArenaAllocator() be called before Py_Initialize(), or they can be called after it?

    If PyMem_Malloc() and PyObject_Malloc() are not needed for pre-initialization, should we support calling them before Py_Initialize()? There are other functions and macros that can be safely used before Py_Initialize().

    @vstinner
    Copy link
    Member Author

    Should PyMem_SetAllocator() and PyObject_SetArenaAllocator() be called before Py_Initialize(), or they can be called after it?

    I'm quite sure that calling PyMem_SetAllocator() or PyObject_SetArenaAllocator() after Py_Initialize() will quickly crash.

    If PyMem_Malloc() and PyObject_Malloc() are not needed for pre-initialization, should we support calling them before Py_Initialize()?

    We don't have to support them.

    Ok, I remove them from the pre-init documentation.

    @serhiy-storchaka
    Copy link
    Member

    I'm quite sure that calling PyMem_SetAllocator() or
    PyObject_SetArenaAllocator() after Py_Initialize() will quickly crash.

    Then document this explicitly like for other functions that *should* be called
    before Py_Initialize() if called at all.

    @vstinner
    Copy link
    Member Author

    New changeset 84c4b19 by Victor Stinner in branch 'master':
    bpo-32124: Document C functions safe before init (bpo-4540)
    84c4b19

    @vstinner
    Copy link
    Member Author

    Then document this explicitly like for other functions that *should* be called before Py_Initialize() if called at all.

    I agree that it would be even better to document if a function must not be called after Py_Initialize().

    *But* I'm not sure of what I wrote, I have to check the code, and maybe even test manually to "see what happens" (ensure that it works) :-)

    So I decided to push my first PR, and will work on a second PR later.

    @ericsnowcurrently
    Copy link
    Member

    I've left a review (writing it as you merged the PR).

    My main concern is that we not promise more than we must. Every pre-init function or variable we promise to embedders represents global state that is hard to get rid of. It also entrenches pre-init API and state that we're aiming to deprecate (via PEP-432).

    @ncoghlan
    Copy link
    Contributor

    Key point to note regarding PEP-432: at least personally, I'm not actually aiming to deprecate the legacy embedding API.

    Instead, I'm just aiming to eventually stop *adding* to it, with new config structs replacing the current ad hoc mix of pre-init function calls, C globals, environment variables, and filesystem state.

    That means I'm quite willing to accept maintaining compatibility for applications using the current single phase initialisation approach as a design constraint for the PEP.

    We have a similar constraint in place for extension modules:
    even though any *new* features we introduce are likely to be dependent on switching over to PEP-489's multi-phase initialisation APIs, we still ensure that single-phase initialisation continues working for existing modules.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.7 (EOL) end of life docs Documentation in the Doc dir
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    4 participants
    @vstinner @ncoghlan @ericsnowcurrently @serhiy-storchaka
    0