Initialize/Finalize Python multiple time and import datetime each time lead to a memory corruption #118608
Comments
|
Which parts of the code use interned strings? |
|
If you mean cpython/Modules/_datetimemodule.c Lines 6871 to 6918 in a8e5fed |
|
With a debug build, I confirm that an assertion fails at the first PyObject *d = PyDateTime_DeltaType.tp_dict;
DATETIME_ADD_MACRO(d, "resolution", new_delta(0, 0, 1, 0));For me, the root issue is that datetime doesn't use heap types but static types. The symptom is that PyDict_SetItemString() calls PyUnicode_InternInPlace() and so stores interned strings, but interned strings are cleared at Python exit: in Py_Finalize(). Interned strings are just one symptom, but I expect other side effects of reusing static types between two Python executions. IMO only isolated extensions are safe to be used with the "sub-interpreter" case:
|
That's one way to see the issue. Another way to see is that Python doesn't support datetime with "sub-interpreters" (see my previous message). This issue is about supporting sub-interpreters in datetime :-) |
Yes, but you prevented me from the heap types conversion at #103092 (comment).
The main and non-isolated subinterpreters share the same module, which avoids redundant initializations. And many objects in a static type are carried over except interned strings. (3.12 needs PR #118618 to carry over static types) What case are you expecting?
Isolated sub-interpreters are not allowed to load a single-phase init module. |
Right, I am afraid of regression. We should do the conversion in a specific order, step by step, to limit risk and be able to rollback if everything goes wrong.
Objects must not be shared between interpreters, including types. Each interpreter should have its own types to prevent any race condition / concurrent accesses, especially when each interpreter has its own GIL. I don't recall details. It doesn't matter much. We both agree to isolate the extension. My proposed plan: #117398 (comment) |
|
Can the isolation be backported to 3.13? Your concern for the sub-interpreters is another specific issue, which is not confirmed on release builds right now. |
I prefer to not backport such large change. For 3.12 and 3.13, we may just stop clearing interned strings at exit, and only do that at Py_Main() exit. |
See also #113993 |
Crash report
What happened?
Example on Windows (debug build):
Output (7bbce38: CI failures with a similar test):
It is an expected error on a debug build, which comes form
_PyUnicode_ClearInterned()inunicodeobject.cof the commit ea2c001:Currently, release builds avoid the
datetimecrash by leaking all interned strings. Both the crash and the leak should be fixed before shipping 3.13 final.This dedicated issue is requested in:
CPython versions tested on:
3.12, CPython main branch
Operating systems tested on:
Linux, macOS, Windows
Linked PRs
The text was updated successfully, but these errors were encountered: