8000 gh-119180: PEP 649 compiler changes by JelleZijlstra · Pull Request #119361 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-119180: PEP 649 compiler changes #119361

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 77 commits into from
Jun 11, 2024
Merged

gh-119180: PEP 649 compiler changes #119361

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
985f8df
Add __annotate__ descriptors
JelleZijlstra May 20, 2024
c822ffa
fix refleaks
JelleZijlstra May 20, 2024
e80095e
blurb
JelleZijlstra May 20, 2024
90ff2c4
Fix some tests
JelleZijlstra May 20, 2024
026c0ff
regen globals
JelleZijlstra May 20, 2024
7968744
Some initial work
JelleZijlstra May 20, 2024
4e54197
Add bytecode for adding annotate
JelleZijlstra May 20, 2024
4469b32
compiler changes
JelleZijlstra May 20, 2024
47d672e
Functions work
JelleZijlstra May 21, 2024
ab9359c
Raise AssertionError on the wrong format
JelleZijlstra May 21, 2024
e65d55a
Merge remote-tracking branch 'upstream/main' into pep649-compile
JelleZijlstra May 21, 2024
e50cd62
Modules and classes
JelleZijlstra May 21, 2024
afae5c0
attempts
JelleZijlstra May 21, 2024
e5a7b1a
Fixes
JelleZijlstra May 21, 2024
3f26d44
Merge remote-tracking branch 'upstream/main' into pep649-compile
JelleZijlstra May 21, 2024
31a4471
Fix code object name
JelleZijlstra May 21, 2024
fbb1d88
Start fixing test_type_annotations
JelleZijlstra May 22, 2024
f452eb2
Fix class scoping
JelleZijlstra May 22, 2024
8c4b4e3
unyielding
JelleZijlstra May 22, 2024
cbf9a3d
Fix test_typing
JelleZijlstra May 22, 2024
5d182fc
Fix test_type_parmas
JelleZijlstra May 22, 2024
ce98c19
Add test, add to inspect
JelleZijlstra May 22, 2024
e0578fc
Use inspect constants
JelleZijlstra May 22, 2024
ed16167
test_grammar tweaks
JelleZijlstra May 22, 2024
f38de20
fix test_positional_only_arg
JelleZijlstra May 22, 2024
87baca2
Fix test_module
JelleZijlstra May 22, 2024
355d3df
Fix symtable tests
JelleZijlstra May 22, 2024
f9d81b6
fix test_pydoc
JelleZijlstra May 22, 2024
dd1f64a
fix test_traceback
JelleZijlstra May 22, 2024
62f5b3b
fix test_opcodes
JelleZijlstra May 22, 2024
b66ad8b
Merge branch 'main' into pep649-compile
JelleZijlstra May 22, 2024
1a63f5d
Raise NotImplementedError
JelleZijlstra May 22, 2024
82c0dbc
Merge remote-tracking branch 'upstream/main' into pep649-compile
JelleZijlstra May 22, 2024
a0c39b5
blurb
JelleZijlstra May 22, 2024
083bbc5
Fix test_dis
JelleZijlstra May 22, 2024
de1b235
Remove broken tests
JelleZijlstra May 22, 2024
1c98fe5
Merge remote-tracking branch 'upstream/main' into pep649-compile
JelleZijlstra May 24, 2024
5f5cf11
No deferred evaluation in interactive mode
JelleZijlstra May 24, 2024
77f3b1c
gh-119443: Turn off from __future__ import annotations in REPL
JelleZijlstra May 24, 2024
4217830
Fix refleak
JelleZijlstra May 24, 2024
13f5d76
Fix another refleak
JelleZijlstra May 24, 2024
a121e1a
Merge remote-tracking branch 'upstream/main' into pep649-compile
JelleZijlstra May 24, 2024
242301c
Exit scope correctly on error
JelleZijlstra May 24, 2024
239ba23
fix test
JelleZijlstra May 24, 2024
b62e04c
Initialize field
JelleZijlstra May 25, 2024
c8a9294
Merge remote-tracking branch 'upstream/main' into pep649-compile
JelleZijlstra May 25, 2024
5ae206d
self-review
JelleZijlstra May 25, 2024
24fd328
Fix crash found by CIFuzz
JelleZijlstra May 25, 2024
0f9d0c5
Merge remote-tracking branch 'upstream/main' into pep649-compile
JelleZijlstra May 29, 2024
c181864
fix merge
JelleZijlstra May 29, 2024
0befff5
Name the function as __annotate__
JelleZijlstra May 29, 2024
ada6573
Merge remote-tracking branch 'upstream/main' into pep649-compile
JelleZijlstra May 29, 2024
ae7714c
Replace find_ann()
JelleZijlstra May 29, 2024
431811a
fix test
JelleZijlstra May 29, 2024
7ca24d3
Remove second compiler pass
JelleZijlstra May 29, 2024
2ab5d07
Fix refleak
JelleZijlstra May 29, 2024
3b4a645
Fix a test
Jelle 8000 Zijlstra May 29, 2024
1dfd02b
Fix bug when there are only non-simple annotations
JelleZijlstra May 29, 2024
daba318
Fix more tests
JelleZijlstra May 29, 2024
0daf0b1
Merge remote-tracking branch 'upstream/main' into pep649-compile
JelleZijlstra May 29, 2024
b066b3d
Remove REPL special case
JelleZijlstra May 29, 2024
7669361
no docstrings in the repl
JelleZijlstra May 29, 2024
c6a1b80
Fix pyrepl test
JelleZijlstra May 29, 2024
5cdbdd7
CR feedback
JelleZijlstra May 30, 2024
e748feb
Merge remote-tracking branch 'upstream/main' into pep649-compile
JelleZijlstra May 31, 2024
a2b4f9e
Merge remote-tracking branch 'upstream/main' into pep649-compile
JelleZijlstra Jun 1, 2024
bd469ab
Still generate __annotate__ if "from __future__ import annotations" i…
JelleZijlstra Jun 1, 2024
278de22
Regen globals
JelleZijlstra Jun 1, 2024
487ea34
Merge branch 'main' into pep649-compile
JelleZijlstra Jun 7, 2024
8f486ba
Merge remote-tracking branch 'upstream/main' into pep649-compile
JelleZijlstra Jun 8, 2024
6b563db
Feedback from Bénédikt Tran
JelleZijlstra Jun 8, 2024
0058b82
add test
JelleZijlstra Jun 9, 2024
517fb56
Merge remote-tracking branch 'upstream/main' into pep649-compile
JelleZijlstra Jun 9, 2024
21f93b6
Merge remote-tracking branch 'upstream/main' into pep649-compile
JelleZijlstra Jun 11, 2024
f72dbca
Undo changes to inspect that we do not want
JelleZijlstra Jun 11, 2024
8674eab
fix
JelleZijlstra Jun 11, 2024
ee11fd9
Fix more tests
JelleZijlstra Jun 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix more tests
  • Loading branch information
@JelleZijlstra
JelleZijlstra committed May 29, 2024
commit daba318a722cd3a6c88ca4aa14d52fbc45b90863
2 changes: 1 addition & 1 deletion Lib/test/test_dis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -363,7 +363,7 @@ def wrap_func_w_kwargs():
CALL 1
STORE_SUBSCR

2 LOAD_CONST 2 (<code object <annotations of top> at 0x..., file "<dis>", line 2>)
2 LOAD_CONST 2 (<code object __annotate__ at 0x..., file "<dis>", line 2>)
MAKE_FUNCTION
STORE_NAME 3 (__annotate__)
RETURN_CONST 3 (None)
Expand Down
2 changes: 2 additions & 0 deletions Lib/test/test_pyclbr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,8 @@ def ismethod(oclass, obj, name):

actualMethods = []
for m in py_item.__dict__.keys():
if m == "__annotate__":
continue
if ismethod(py_item, getattr(py_item, m), m):
actualMethods.append(m)
foundMethods = []
Expand Down
4 changes: 2 additions & 2 deletions Lib/test/test_pydoc/test_pydoc.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ class A(builtins.object)
class B(builtins.object)
| Methods defined here:
|
| __annotate__ = <annotations of B>(...)
| __annotate__(...)
Copy link
Member
@AlexWaygood AlexWaygood Jun 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(It would be great if we could generate a nice __text_signature__ for these generated methods, but that definitely doesn't need to be done in this PR)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're actually Python function, so I wonder why pydoc doesn't pick them up. Possibly because the parameter is called .format.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed:

>>> inspect.signature(f.__annotate__)
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    inspect.signature(f.__annotate__)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/Users/jelle/py/cpython/Lib/inspect.py", line 3329, in signature
    return Signature.from_callable(obj, follow_wrapped=follow_wrapped,
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                   globals=globals, locals=locals, eval_str=eval_str)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jelle/py/cpython/Lib/inspect.py", line 3055, in from_callable
    return _signature_from_callable(obj, sigcls=cls,
                                    follow_wrapper_chains=follow_wrapped,
                                    globals=globals, locals=locals, eval_str=eval_str)
  File "/Users/jelle/py/cpython/Lib/inspect.py", line 2558, in _signature_from_callable
    return _signature_from_function(sigcls, obj,
                                    skip_bound_arg=skip_bound_arg,
                                    globals=globals, locals=locals, eval_str=eval_str)
  File "/Users/jelle/py/cpython/Lib/inspect.py", line 2403, in _signature_from_function
    parameters.append(Parameter(name, annotation=annotation,
                      ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                kind=kind))
                                ^^^^^^^^^^
  File "/Users/jelle/py/cpython/Lib/inspect.py", line 2750, in __init__
    raise ValueError('{!r} is not a valid parameter name'.format(name))
ValueError: '.format' is not a valid parameter name

Maybe in a followup PR we can figure out a way to make this work. The parameter has to have an illegal name so it doesn't shadow any symbol name that the user might have used.

Copy link
Member
@AlexWaygood AlexWaygood Jun 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter has to have an illegal name so it doesn't shadow any symbol name that the user might have used.

I'm not sure I 100% understand this point, and no tests seem to fail if I make this change locally to your PR branch and recompile (and I verified that it means that __annotate__ methods have valid signatures as per inspect.signature):

diff --git a/Include/internal/pycore_global_strings.h b/Include/internal/pycore_global_strings.h
index 009802c4416..899b19ae941 100644
--- a/Include/internal/pycore_global_strings.h
+++ b/Include/internal/pycore_global_strings.h
@@ -45,7 +45,7 @@ struct _Py_global_strings {
         STRUCT_FOR_STR(dot, ".")
         STRUCT_FOR_STR(dot_locals, ".<locals>")
         STRUCT_FOR_STR(empty, "")
-        STRUCT_FOR_STR(format, ".format")
+        STRUCT_FOR_STR(format, "format")
         STRUCT_FOR_STR(generic_base, ".generic_base")
         STRUCT_FOR_STR(json_decoder, "json.decoder")
         STRUCT_FOR_STR(kwdefaults, ".kwdefaults")
diff --git a/Include/internal/pycore_runtime_init_generated.h b/Include/internal/pycore_runtime_init_generated.h
index ff5b6ee8e0f..e8b5173a457 100644
--- a/Include/internal/pycore_runtime_init_generated.h
+++ b/Include/internal/pycore_runtime_init_generated.h
@@ -554,7 +554,7 @@ extern "C" {
     INIT_STR(dot, "."), \
     INIT_STR(dot_locals, ".<locals>"), \
     INIT_STR(empty, ""), \
-    INIT_STR(format, ".format"), \
+    INIT_STR(format, "format"), \
     INIT_STR(generic_base, ".generic_base"), \
     INIT_STR(json_decoder, "json.decoder"), \
     INIT_STR(kwdefaults, ".kwdefaults"), \
diff --git a/Python/compile.c b/Python/compile.c
index c3372766d0b..7535067ded0 100644
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -1447,7 +1447,7 @@ compiler_setup_annotations_scope(struct compiler *c, location loc,
     }
     c->u->u_metadata.u_posonlyargcount = 1;
     // if .format != 1: raise NotImplementedError
-    _Py_DECLARE_STR(format, ".format");
+    _Py_DECLARE_STR(format, "format");
     ADDOP_I(c, loc, LOAD_FAST, 0);
     ADDOP_LOAD_CONST(c, loc, _PyLong_GetOne());
     ADDOP_I(c, loc, COMPARE_OP, (Py_NE << 5) | compare_masks[Py_NE]);
diff --git a/Python/symtable.c b/Python/symtable.c
index 23fc4a0ec03..0154092a905 100644
--- a/Python/symtable.c
+++ b/Python/symtable.c
@@ -2511,7 +2511,7 @@ symtable_visit_annotation(struct symtable *st, expr_ty annotation, void *key)
             }
         }
 
-        _Py_DECLARE_STR(format, ".format");
+        _Py_DECLARE_STR(format, "format");
         // The generated __annotate__ function takes a single parameter with the
         // internal name ".format".
         if (!symtable_add_def(st, &_Py_STR(format), DEF_PARAM,
@@ -2570,7 +2570,7 @@ symtable_visit_annotations(struct symtable *st, stmt_ty o, arguments_ty a, expr_
             return 0;
         }
     }
-    _Py_DECLARE_STR(format, ".format");
+    _Py_DECLARE_STR(format, "format");
     // We need to insert code that reads this "parameter" to the function.
     if (!symtable_add_def(st, &_Py_STR(format), DEF_PARAM, LOCATION(o))) {
         return 0;

This also might be worth a brief mention in PEP-749, since PEP-649 seems to specify that the signature of __annotate__ should be __annotate__(format: int).

In any event, this can definitely be tackled in a followup; I don't want to block this PR!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try something like this:

class format: pass

def f(x: format): pass

print(f.__annotations__)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, makes sense! Worth adding a test like that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, the problem is that any such test can only tell us about one specific name.

Copy link
Member
@AlexWaygood AlexWaygood Jun 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it doesn't verify any fundamental invariants about __annotate__ — but I still think it's useful to have such a test, so that the CI goes obviously red if someone else comes along in the future and tries to do the "obvious fix" to get inspect.signature() working for these methods.

Thanks for adding it!

|
| ----------------------------------------------------------------------
| Data descriptors defined here:
Expand Down Expand Up @@ -179,7 +179,7 @@ class A(builtins.object)

class B(builtins.object)
Methods defined here:
__annotate__ = (...)
__annotate__(...)
----------------------------------------------------------------------
Data descriptors defined here:
__dict__
Expand Down
2 changes: 1 addition & 1 deletion Lib/test/test_traceback.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -632,7 +632,7 @@ def foo(a: THIS_DOES_NOT_EXIST ) -> int:
' ~~~~~~~~^^\n'
f' File "{__file__}", line {lineno_f+3}, in f_with_type\n'
' foo.__annotations__\n'
f' File "{__file__}", line {lineno_f+1}, in <annotations of foo>\n'
f' File "{__file__}", line {lineno_f+1}, in __annotate__\n'
' def foo(a: THIS_DOES_NOT_EXIST ) -> int:\n'
' ^^^^^^^^^^^^^^^^^^^\n'
)
Expand Down
0