10000 gh-117953: Other Cleanups in the Extensions Machinery (gh-118206) · python/cpython@f201628 · GitHub
[go: up one dir, main page]
Skip to content

Commit f201628

Browse files
ericsnowcurrentlyericsnowcurrently
authored
gh-117953: Other Cleanups in the Extensions Machinery (gh-118206)
This change will make some later changes simpler.
1 parent f8290df commit f201628

File tree

3 files changed

+410
-102
lines changed

3 files changed

+410
-102
lines changed

‎Lib/test/test_import/__init__.py

Lines changed: 109 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2285,6 +2285,107 @@ def test_disallowed_reimport(self):
22852285

22862286

22872287
class TestSinglePhaseSnapshot(ModuleSnapshot):
2288+
"""A representation of a single-phase init module for testing.
2289+
2290+
Fields from ModuleSnapshot:
2291+
2292+
* id - id(mod)
2293+
* module - mod or a SimpleNamespace with __file__ & __spec__
2294+
* ns - a shallow copy of mod.__dict__
2295+
* ns_id - id(mod.__dict__)
2296+
* cached - sys.modules[name] (or None if not there or not snapshotable)
2297+
* cached_id - id(sys.modules[name]) (or None if not there)
2298+
2299+
Extra fields:
2300+
2301+
* summed - the result of calling "mod.sum(1, 2)"
2302+
* lookedup - the result of calling "mod.look_up_self()"
2303+
* lookedup_id - the object ID of self.lookedup
2304+
* state_initialized - the result of calling "mod.state_initialized()"
2305+
* init_count - (optional) the result of calling "mod.initialized_count()"
2306+
2307+
Overridden methods from ModuleSnapshot:
2308+
2309+
* from_module()
2310+
* parse()
2311+
2312+
Other methods from ModuleSnapshot:
2313+
2314+
* build_script()
2315+
* from_subinterp()
2316+
2317+
----
2318+
2319+
There are 5 modules in Modules/_testsinglephase.c:
2320+
2321+
* _testsinglephase
2322+
* has global state
2323+
* extra loads skip the init function, copy def.m_base.m_copy
2324+
* counts calls to init function
2325+
* _testsinglephase_basic_wrapper
2326+
* _testsinglephase by another name (and separate init function symbol)
2327+
* _testsinglephase_basic_copy
2328+
* same as _testsinglephase but with own def (and init func)
2329+
* _testsinglephase_with_reinit
2330+
* has no global or module state
2331+
* mod.state_initialized returns None
2332+
* an extra load in the main interpreter calls the cached init func
2333+
* an extra load in legacy subinterpreters does a full load
2334+
* _testsinglephase_with_state
2335+
* has module state
2336+
* an extra load in the main interpreter calls the cached init func
2337+
* an extra load in legacy subinterpreters does a full load
2338+
2339+
(See Modules/_testsinglephase.c for more info.)
2340+
2341+
For all those modules, the snapshot after the initial load (not in
2342+
the global extensions cache) would look like the following:
2343+
2344+
* initial load
2345+
* id: ID of nww module object
2346+
* ns: exactly what the module init put there
2347+
* ns_id: ID of new module's __dict__
2348+
* cached_id: same as self.id
2349+
* summed: 3 (never changes)
2350+
* lookedup_id: same as self.id
2351+
* state_initialized: a timestamp between the time of the load
2352+
and the time of the snapshot
2353+
* init_count: 1 (None for _testsinglephase_with_reinit)
2354+
2355+
For the other scenarios it varies.
2356+
2357+
For the _testsinglephase, _testsinglephase_basic_wrapper, and
2358+
_testsinglephase_basic_copy modules, the snapshot should look
2359+
like the following:
2360+
2361+
* reloaded
2362+
* id: no change
2363+
* ns: matches what the module init function put there,
2364+
including the IDs of all contained objects,
2365+
plus any extra attributes added before the reload
2366+
* ns_id: no change
2367+
* cached_id: no change
2368+
* lookedup_id: no change
2369+
* state_initialized: no change
2370+
* init_count: no change
2371+
* already loaded
2372+
* (same as initial load except for ns and state_initialized)
2373+
* ns: matches the initial load, incl. IDs of contained objects
2374+
* state_initialized: no change from initial load
2375+
2376+
For _testsinglephase_with_reinit:
2377+
2378+
* reloaded: same as initial load (old module & ns is discarded)
2379+
* already loaded: same as initial load (old module & ns is discarded)
2380+
2381+
For _testsinglephase_with_state:
2382+
2383+
* reloaded
2384+
* (same as initial load (old module & ns is discarded),
2385+
except init_count)
2386+
* init_count: increase by 1
2387+
* already loaded: same as reloaded
2388+
"""
22882389

22892390
@classmethod
22902391
def from_module(cls, mod):
@@ -2901,17 +3002,18 @@ def test_basic_multiple_interpreters_deleted_no_reset(self):
29013002
# * module's global state was initialized but cleared
29023003

29033004
# Start with an interpreter that gets destroyed right away.
2904-
base = self.import_in_subinterp(postscript='''
2905-
# Attrs set after loading are not in m_copy.
2906-
mod.spam = 'spam, spam, mash, spam, eggs, and spam'
2907-
''')
3005+
base = self.import_in_subinterp(
3006+
postscript='''
3007+
# Attrs set after loading are not in m_copy.
3008+
mod.spam = 'spam, spam, mash, spam, eggs, and spam'
3009+
''')
29083010
self.check_com F438 mon(base)
29093011
self.check_fresh(base)
29103012

29113013
# At this point:
29123014
# * alive in 0 interpreters
29133015
# * module def in _PyRuntime.imports.extensions
2914-
# * mod init func ran again
3016+
# * mod init func ran for the first time (since reset)
29153017
# * m_copy is NULL (claered when the interpreter was destroyed)
29163018
# * module's global state was initialized, not reset
29173019

@@ -2923,7 +3025,7 @@ def test_basic_multiple_interpreters_deleted_no_reset(self):
29233025
# At this point:
29243026
# * alive in 1 interpreter (interp1)
29253027
# * module def still in _PyRuntime.imports.extensions
2926-
# * mod init func ran again
3028+
# * mod init func ran for the second time (since reset)
29273029
# * m_copy was copied from interp1 (was NULL)
29283030
# * module's global state was updated, not reset
29293031

@@ -2935,7 +3037,7 @@ def test_basic_multiple_interpreters_deleted_no_reset(self):
29353037
# At this point:
29363038
# * alive in 2 interpreters (interp1, interp2)
29373039
# * module def still in _PyRuntime.imports.extensions
2938-
# * mod init func ran again
3040+
# * mod init func did not run again
29393041
# * m_copy was copied from interp2 (was from interp1)
29403042
# * module's global state was updated, not reset
29413043

‎Modules/_testsinglephase.c

Lines changed: 167 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,172 @@
11

22
/* Testing module for single-phase initialization of extension modules
3-
*/
3+
4+
This file contains 5 distinct modules, meaning each as its own name
5+
and its own init function (PyInit_...). The default import system will
6+
only find the one matching the filename: _testsinglephase. To load the
7+
others you must do so manually. For example:
8+
9+
```python
10+
name = '_testsinglephase_base_wrapper'
11+
filename = _testsinglephase.__file__
12+
loader = importlib.machinery.ExtensionFileLoader(name, filename)
13+
spec = importlib.util.spec_from_file_location(name, filename, loader=loader)
14+
mod = importlib._bootstrap._load(spec)
15+
```
16+
17+
Here are the 5 modules:
18+
19+
* _testsinglephase
20+
* def: _testsinglephase_basic,
21+
* m_name: "_testsinglephase"
22+
* m_size: -1
23+
* state
24+
* process-global
25+
* <int> initialized_count (default to -1; will never be 0)
26+
* <module_state> module (see module state below)
27+
* module state: no
28+
* initial __dict__: see common initial __dict__ below
29+
* init function
30+
1. create module
31+
2. clear <global>.module
32+
3. initialize <global>.module: see module state below
33+
4. initialize module: set initial __dict__
34+
5. increment <global>.initialized_count
35+
* functions
36+
* (3 common, see below)
37+
* initialized_count() - return <global>.module.initialized_count
38+
* import system
39+
* caches
40+
* global extensions cache: yes
41+
* def.m_base.m_copy: yes
42+
* def.m_base.m_init: no
43+
* per-interpreter cache: yes (all single-phase init modules)
44+
* load in main interpreter
45+
* initial (not already in global cache)
46+
1. get init function from shared object file
47+
2. run init function
48+
3. copy __dict__ into def.m_base.m_copy
49+
4. set entry in global cache
50+
5. set entry in per-interpreter cache
51+
6. set entry in sys.modules
52+
* reload (already in sys.modules)
53+
1. get def from global cache
54+
2. get module from sys.modules
55+
3. update module with contents of def.m_base.m_copy
56+
* already loaded in other interpreter (already in global cache)
57+
* same as reload, but create new module and update *it*
58+
* not in any sys.modules, still in global cache
59+
* same as already loaded
60+
* load in legacy (non-isolated) interpreter
61+
* same as main interpreter
62+
* unload: never (all single-phase init modules)
63+
* _testsinglephase_basic_wrapper
64+
* identical to _testsinglephase except module name
65+
* _testsinglephase_basic_copy
66+
* def: static local variable in init function
67+
* m_name: "_testsinglephase_basic_copy"
68+
* m_size: -1
69+
* state: same as _testsinglephase
70+
* init function: same as _testsinglephase
71+
* functions: same as _testsinglephase
72+
* import system: same as _testsinglephase
73+
* _testsinglephase_with_reinit
74+
* def: _testsinglephase_with_reinit,
75+
* m_name: "_testsinglephase_with_reinit"
76+
* m_size: 0
77+
* state
78+
* process-global state: no
79+
* module state: no
80+
* initial __dict__: see common initial __dict__ below
81+
* init function
82+
1. create module
83+
2. initialize temporary module state (local var): see module state below
84+
3. initialize module: set initial __dict__
85+
* functions: see common functions below
86+
* import system
87+
* caches
88+
* global extensions cache: only if loaded in main interpreter
89+
* def.m_base.m_copy: no
90+
* def.m_base.m_init: only if loaded in the main interpreter
91+
* per-interpreter cache: yes (all single-phase init modules)
92+
* load in main interpreter
93+
* initial (not already in global cache)
94+
* (same as _testsinglephase except step 3)
95+
1. get init function from shared object file
96+
2. run init function
97+
3. set def.m_base.m_init to the init function
98+
4. set entry in global cache
99+
5. set entry in per-interpreter cache
100+
6. set entry in sys.modules
101+
* reload (already in sys.modules)
102+
1. get def from global cache
103+
2. call def->m_base.m_init to get a new module object
104+
3. replace the existing module in sys.modules
105+
* already loaded in other interpreter (already in global cache)
106+
* same as reload (since will only be in cache for main interp)
107+
* not in any sys.modules, still in global cache
108+
* same as already loaded
109+
* load in legacy (non-isolated) interpreter
110+
* initial (not already in global cache)
111+
* (same as main interpreter except skip steps 3 & 4 there)
112+
1. get init function from shared object file
113+
2. run init function
114+
...
115+
5. set entry in per-interpreter cache
116+
6. set entry in sys.modules
117+
* reload (already in sys.modules)
118+
* same as initial (load from scratch)
119+
* already loaded in other interpreter (already in global cache)
120+
* same as initial (load from scratch)
121+
* not in any sys.modules, still in global cache
122+
* same as initial (load from scratch)
123+
* unload: never (all single-phase init modules)
124+
* _testsinglephase_with_state
125+
* def: _testsinglephase_with_state,
126+
* m_name: "_testsinglephase_with_state"
127+
* m_size: sizeof(module_state)
128+
* state
129+
* process-global: no
130+
* module state: see module state below
131+
* initial __dict__: see common initial __dict__ below
132+
* init function
133+
1. create module
134+
3. initialize module state: see module state below
135+
4. initialize module: set initial __dict__
136+
5. increment <global>.initialized_count
137+
* functions: see common functions below
138+
* import system: same as _testsinglephase_basic_copy
139+
140+
Module state:
141+
142+
* fields
143+
* <PyTime_t> initialized - when the module was first initialized
144+
* <PyObject> *error
145+
* <PyObject> *int_const
146+
* <PyObject> *str_const
147+
* initialization
148+
1. set state.initialized to the current time
149+
2. set state.error to a new exception class
150+
3. set state->int_const to int(1969)
151+
4. set state->str_const to "something different"
152+
153+
Common initial __dict__:
154+
155+
* error: state.error
156+
* int_const: state.int_const
157+
* str_const: state.str_const
158+
* _module_initialized: state.initialized
159+
160+
Common functions:
161+
162+
* look_up_self() - return the module from the per-interpreter "by-index" cache
163+
* sum() - return a + b
164+
* state_initialized() - return state->initialized (or None if m_size == 0)
165+
166+
See Python/import.c, especially the long comments, for more about
167+
single-phase init modules.
168+
*/
169+
4170
#ifndef Py_BUILD_CORE_BUILTIN
5171
# define Py_BUILD_CORE_MODULE 1
6172
#endif

0 commit comments

Comments
 (0)
0