8000 [PEP 558 - WIP] bpo-30744: Trace hooks no longer reset closure state by ncoghlan · Pull Request #3640 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

[PEP 558 - WIP] bpo-30744: Trace hooks no longer reset closure state #3640

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
7626a0e
bpo-30744: Trace hooks no longer reset closure state
ncoghlan Sep 18, 2017
3cbb73c
Disable the current broken writeback logic
ncoghlan Nov 5, 2017
01f3f34
Failing test case for writeback functionality
ncoghlan Nov 5, 2017
4f6dd93
Initial skeleton for a write-through proxy
ncoghlan Nov 5, 2017
cd45da7
Merge remote-tracking branch 'origin/master' into bpo-30744-make-loca…
ncoghlan Apr 21, 2019
acbf587
Finish adding the write-through proxy
ncoghlan Apr 21, 2019
d1a9014
Merge remote-tracking branch 'origin/master' into bpo-30744-make-loca…
ncoghlan May 21, 2019
5ea8bcf
Add test case for the PEP 558 locals() behaviour
ncoghlan May 22, 2019
fe92891
Actually implement most of the PEP and fix the tests
ncoghlan May 22, 2019
ac9e0bf
Fix segfault on cleared frames 8000
ncoghlan May 27, 2019
6774e71
Use correct printf formatting code
ncoghlan May 27, 2019
74c51e4
Initial skeleton for other mutable mapping methods
ncoghlan May 28, 2019
0e5fbf3
Break ref cycle when frame finishes executing
ncoghlan May 30, 2019
9e3ce53
Remove implicit frame locals update
ncoghlan May 30, 2019
9479557
Merge remote-tracking branch 'origin/master' into bpo-30744-make-loca…
ncoghlan May 30, 2019
8e886ef
Avoid double DECREF on error
ncoghlan May 30, 2019
5b63e7c
Attempt to make dealloc more robust under gc
ncoghlan May 30, 2019
f0ecee3
Merge remote-tracking branch 'origin/master' into bpo-30744-make-loca…
ncoghlan Dec 29, 2019
1752b54
Fix post-merge compilation errors
ncoghlan Dec 29, 2019
348a56d
Implement flp.pop()
ncoghlan Dec 29, 2019
93f512c
Merge remote-tracking branch 'origin/master' into bpo-30744-make-loca…
ncoghlan Dec 29, 2019
39ec4d8
Refactor ref map creation
ncoghlan Dec 29, 2019
7078632
Correctly manage fast local refcounts
ncoghlan Dec 29, 2019
ed5f86e
Allow closure updates after frame termination
ncoghlan Dec 29, 2019
a216747
Refactor frame post-eval cleanup
ncoghlan Dec 29, 2019
a0dc787
Use full word in API name
ncoghlan Dec 29, 2019
7b02bed
Update to match latest PEP draft
ncoghlan Dec 29, 2019
e9876b5
Update test_scope for snapshot semantics
ncoghlan Dec 29, 2019
0033c60
Update test_sys_settrace for snapshot semantics
ncoghlan Dec 29, 2019
a5a8b19
Fix pop/delete locals proxy bug
ncoghlan Dec 29, 2019
6c98f48
Update argument clinic output
ncoghlan Dec 30, 2019
617f6ab
Merge remote-tracking branch 'origin/master' into bpo-30744-make-loca…
ncoghlan Dec 30, 2019
729f121
Merge remote-tracking branch 'origin/master' into bpo-30744-make-loca…
ncoghlan Jan 18, 2020
619fb75
Merge remote-tracking branch 'origin/master' into bpo-30744-make-loca…
ncoghlan Feb 2, 2020
1fe964e
Migrate to revised public API design
ncoghlan Feb 2, 2020
b047ae4
Rename PyFrame_LocalsIsSnapshot to PyFrame_GetLocalsReturnsSnapshot
ncoghlan Feb 2, 2020
d1a8420
Mark fast locals proxy as an internal type
ncoghlan Feb 2, 2020
82108ff
Merge remote-tracking branch 'origin/master' into bpo-30744-make-loca…
ncoghlan Feb 15, 2020
eccb1ea
Update draft C API to match latest PEP text
ncoghlan Feb 16, 2020
c1933e7
Migrate exec() and eval() to PyLocals_Get()
ncoghlan Feb 16, 2020
161ad47
Merge remote-tracking branch 'origin/master' into bpo-30744-make-loca…
ncoghlan Feb 22, 2020
68f10ce
Avoid circular reference between locals proxy and frame
ncoghlan Feb 22, 2020
29ce344
Add back implicit view refresh in Python trace hook
ncoghlan Feb 22, 2020
ed6e53b
Attempt to tidy up Mac OS X compile warnings/errors
ncoghlan Feb 22, 2020
96c77cb
Merge remote-tracking branch 'origin/main' into bpo-30744-make-locals…
ncoghlan Jul 3, 2021
a348d08
Fix Argument Clinic checksum
ncoghlan Jul 3, 2021
69c8f19
Fix stable ABI minimum version
ncoghlan Jul 3, 2021
caeaf66
Bring implementation into line with latest PEP version
ncoghlan Jul 10, 2021
7ec5d26
Register new stable ABI additions
ncoghlan Jul 10, 2021
7ddc3eb
Add FLP str(), fix various value lookup issues
ncoghlan Jul 10, 2021
7400a46
Uninitialised fields will get you every time
ncoghlan Jul 10, 2021
8000 5eae0d5
Fix refcounting, bdb segfault, pdb functionality
ncoghlan Jul 10, 2021
40db4e7
Delegate more operations to the full dynamic snapshot
ncoghlan Jul 10, 2021
74b97a3
Add TODO item for false positives in containment checks
ncoghlan Jul 10, 2021
9f16513
Only ensure frame snapshot is up to date in O(n) proxy operations
ncoghlan Jul 10, 2021
c477e24
Keep locals snapshot up to date when reading/writing individual keys
ncoghlan Jul 11, 2021
dd94608
Avoid false positives in FLP contains method
ncoghlan Jul 11, 2021
b7eb662
Merge remote-tracking branch 'origin/main' into bpo-30744-make-locals…
ncoghlan Jul 17, 2021
1484c10
Finish public C API, start dict API tests
ncoghlan Jul 17, 2021
3891c13
Merge remote-tracking branch 'origin/main' into bpo-30744-make-locals…
ncoghlan Jul 17, 2021
760ffa9
Remove debugging print statement
ncoghlan Jul 17, 2021
ae6b013
Regenerated stable ABI files
ncoghlan Jul 18, 2021
b03309b
Rename _PyLocals_Kind APIs to avoid potential confusion
ncoghlan Jul 21, 2021
cde5035
Merge remote-tracking branch 'origin/main' into bpo-30744-make-locals…
ncoghlan Jul 21, 2021
66d058c
PyLocals_GetReturnsCopy -> PyLocals_GetKind()
ncoghlan Jul 21, 2021
67c3958
Share fast_refs mapping between proxy objects
ncoghlan Aug 21, 2021
034345f
Remove debugging print
ncoghlan Aug 21, 2021
3c49ff8
Defer value cache refresh until needed, start fleshing out dict API t…
ncoghlan Aug 21, 2021
ea5f943
Add dict union operations to proxy
ncoghlan Aug 21, 2021
fcf99ca
Implement and test locals proxy clear() method
ncoghlan Aug 21, 2021
16e0581
Remove pointless print() call
ncoghlan Aug 21, 2021
c356949
Implement proxy pop() tests
ncoghlan Aug 21, 2021
8a4e788
Implement and test proxy popitem()
ncoghlan Aug 21, 2021
706eec4
Test popitem with cells and extra variables
ncoghlan Aug 21, 2021
35a017c
Implement and test setdefault()
ncoghlan Aug 21, 2021
06c406c
Implement and test proxy __sizeof__()
ncoghlan Aug 21, 2021
31493b9
Add C API test for the LocalsToFast exception
ncoghlan Aug 21, 2021
b587a41
Force enum size
ncoghlan Aug 23, 2021
e1b505d
Clarify code comment
ncoghlan Aug 23, 2021
2b27389
Keep track of defined names even on cleared frames
ncoghlan Aug 26, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Actually implement most of the PEP and fix the tests
  • Loading branch information
ncoghlan committed May 22, 2019
commit fe928915d4c8fad7426ebcab33a115cf06252886
26 changes: 16 additions & 10 deletions Include/frameobject.h
8000
Original file line number Diff line number Diff line change
Expand Up @@ -71,16 +71,6 @@ PyAPI_FUNC(PyTryBlock *) PyFrame_BlockPop(PyFrameObject *);

PyAPI_FUNC(PyObject **) PyFrame_ExtendStack(PyFrameObject *, int, int);

/* Conversions between "fast locals" and locals in dictionary */

PyAPI_FUNC(void) PyFrame_LocalsToFast(PyFrameObject *, int);

PyAPI_FUNC(int) PyFrame_FastToLocalsWithError(PyFrameObject *f);
PyAPI_FUNC(void) PyFrame_FastToLocals(PyFrameObject *);
#ifdef Py_BUILD_CORE
int _PyFrame_FastToLocalsInternal(PyFrameObject *f, int); /* For sys.settrace */
#endif

PyAPI_FUNC(int) PyFrame_ClearFreeList(void);

PyAPI_FUNC(void) _PyFrame_DebugMallocStats(FILE *out);
Expand All @@ -89,13 +79,29 @@ PyAPI_FUNC(void) _PyFrame_DebugMallocStats(FILE *out);
PyAPI_FUNC(int) PyFrame_GetLineNumber(PyFrameObject *);


/* Conversions between "fast locals" and locals in dictionary */
PyAPI_FUNC(int) PyFrame_FastToLocalsWithError(PyFrameObject *f);
PyAPI_FUNC(void) PyFrame_FastToLocals(PyFrameObject *);

#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03080000
/* Fast locals proxy for reliable write-through from trace functions */
PyTypeObject PyFastLocalsProxy_Type;
#define _PyFastLocalsProxy_CheckExact(self) \
(Py_TYPE(self) == &PyFastLocalsProxy_Type)

/* Access the frame locals mapping */
PyAPI_FUNC(PyObject *) PyFrame_GetPyLocals(PyFrameObject *); // = locals()
PyAPI_FUNC(PyObject *) PyFrame_GetLocalsAttr(PyFrameObject *); // = frame.f_locals
#endif

#ifdef Py_BUILD_CORE
PyObject *_PyFrame_BorrowPyLocals(PyFrameObject *f); /* For PyEval_GetLocals() */
#endif


/* This always raises RuntimeError now (use PyFrame_GetLocalsAttr() instead) */
PyAPI_FUNC(void) PyFrame_LocalsToFast(PyFrameObject *, int);

#ifdef __cplusplus
}
#endif
Expand Down
2 changes: 2 additions & 0 deletions Lib/test/test_scope.py
Original file line number Diff line number Diff line change
Expand Up @@ -804,7 +804,9 @@ def function_local_semantics():
return locals()

global_ns = globals()
self.assertIsInstance(global_ns, dict)
local_ns = function_local_semantics()
self.assertIsInstance(local_ns, dict)
self.assertIs(local_ns["global_ns"], global_ns)
self.assertIs(local_ns["local_ns"], local_ns)
self.assertEqual(local_ns["known_var"], "original")
Expand Down
61 changes: 50 additions & 11 deletions Lib/test/test_sys_settrace.py
Original file line number Diff line number Diff line change
Expand Up @@ -1447,11 +1447,45 @@ def tracefunc(frame, event, arg):
self.assertEqual(x_from_nested_ref, [2, 6])
self.assertEqual(x_from_nested_locals, [3, 7])

def test_locals_writeback_support(self):
def test_locals_writethrough_proxy(self):
# To support interactive debuggers, trace functions are expected to
# be able to reliably modify function locals. However, this should
# NOT enable writebacks via locals() at function scope.
#
# This is a simple scenario to test the core behaviour of the
# writethrough proxy
local_var = "original"
# Check regular locals() snapshot
locals_snapshot = locals()
self.assertEqual(locals_snapshot["local_var"], "original")
locals_snapshot["local_var"] = "modified"
self.assertEqual(local_var, "original")
# Check writethrough proxy on frame
locals_proxy = sys._getframe().f_locals
self.assertEqual(locals_proxy["local_var"], "original")
locals_proxy["local_var"] = "modified"
self.assertEqual(local_var, "modified")
# Check handling of closures
def nested_scope():
nonlocal local_var
return local_var, locals(), sys._getframe().f_locals
closure_var, inner_snapshot, inner_proxy = nested_scope()
self.assertEqual(closure_var, "modified")
self.assertEqual(inner_snapshot["local_var"], "modified")
self.assertEqual(inner_proxy["local_var"], "modified")
inner_snapshot["local_var"] = "modified_again"
self.assertEqual(local_var, "modified")
self.assertEqual(inner_snapshot["local_var"], "modified_again")
self.assertEqual(inner_proxy["local_var"], "modified_again") # Q: Is this desirable?
inner_proxy["local_var"] = "modified_yet_again"
self.assertEqual(local_var, "modified_yet_again")
self.assertEqual(inner_snapshot["local_var"], "modified_yet_again")
self.assertEqual(inner_proxy["local_var"], "modified_yet_again")

def test_locals_writeback_complex_scenario(self):
# Further locals writeback testing using a more complex scenario
# involving multiple scopes of different kinds
#
# Note: the sample values have numbers in them so mixing up variable
# names in the checks can't accidentally make the test pass -
# you'd have to get both the name *and* expected number wrong
Expand All @@ -1471,7 +1505,7 @@ def outer():
a_nonlocal = 'original5' # We expect this to be retained
another_nonlocal = 'original6' # Trace func will modify this
def inner():
nonlocal another_nonlocal
nonlocal a_nonlocal, another_nonlocal
a_local = 'original7' # We expect this to be retained
another_local = 'original8' # Trace func will modify this
ns = locals()
Expand All @@ -1492,29 +1526,36 @@ def inner():
a_local, another_local, inner_ns = inner_result
a_local_via_ns = inner_ns['a_local']
a_nonlocal_via_inner_ns = inner_ns['a_nonlocal']
print(a_nonlocal_via_inner_ns)
another_nonlocal_via_inner_ns = inner_ns['another_nonlocal']
another_local_via_ns = inner_ns['another_local']
a_new_local_via_ns = inner_ns['a_new_local']
del outer, outer_result, outer_ns, inner_result, inner_ns
"""
)
def tracefunc(frame, event, arg):
ns = frame.f_locals
co_name = frame.f_code.co_name
if event == "return":
# We leave any state manipulation to the very end
ns = frame.f_locals
co_name = frame.f_code.co_name
# We leave most state manipulation to the very end
if co_name == "C":
# Modify class attributes
ns["another_attr"] = "modified4"
elif co_name == "inner":
# Modify local and nonlocal variables
# Modify variables in outer scope
ns["another_nonlocal"] = "modified6"
ns["another_local"] = "modified8"
outer_ns = frame.f_back.f_locals
outer_ns["outer_local"] = "modified10"
elif co_name == "<module>":
# Modify globals
ns["another_global"] = "modified2"
elif event == "line" and co_name == "inner":
# This is the one item we can't leave to the end, as the
# return tuple is already built by the time the return event
# fires, and we want to manipulate one of the entries in that
# So instead, we just mutate it on every line trace event
ns["another_local"] = "modified8"
print(event)
return tracefunc
actual_ns = {}
sys.settrace(tracefunc)
Expand All @@ -1525,9 +1566,6 @@ def tracefunc(frame, event, arg):
for k in list(actual_ns.keys()):
if k.startswith("_"):
del actual_ns[k]
# CURRENT FAILURE STATUS:
# - trace functions mutating state as expected
# - proxy is incorrectly being returned by locals(), so that also mutates
expected_ns = {
"a_global": "created1",
"another_global": "modified2",
Expand All @@ -1547,7 +1585,8 @@ def tracefunc(frame, event, arg):
"outer_local": "modified10",
"outer_local_via_ns": "modified10",
}
self.assertEqual(actual_ns, expected_ns)
# Expected is first so any error diff is easier to read
self.assertEqual(expected_ns, actual_ns)


if __name__ == "__main__":
Expand Down
Loading
0