8000 gh-116022: Improve `repr()` of AST nodes by tomasr8 · Pull Request #117046 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

8000 gh-116022: Improve repr() of AST nodes #117046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 39 commits into from
Sep 18, 2024
Merged

gh-116022: Improve repr() of AST nodes #117046

merged 39 commits into from
Sep 18, 2024

Conversation

tomasr8
Copy link
Member
@tomasr8 tomasr8 commented Mar 19, 2024 •
edited
Loading

closes #116022

Based on the original issue the new repr() has:

  • limited depth: nodes which are to deep are represented as Node(...)
  • compressed list/tuple fields (e.g. body): more than 2 items are shown as [repr(first), ..., repr(last)]

Examples (output is formatted to be more readable):

>>> import ast
>>> ast.parse("x = 3")
Module(
    body=[Assign(targets=[Name(...)], value=Constant(...), type_comment=None)],
    type_ignores=[]
)


>>> ast.parse('x=7; y=4; c,z=3,4')
Module(
    body=[
        Assign(targets=[Name(...)], value=Constant(...), type_comment=None),
        ...,
        Assign(targets=[Tuple(...)], value=Tuple(...), type_comment=None)
    ],
    type_ignores=[]
)


>>> from pathlib import Path
>>> import typing
>>> ast.parse(Path(typing.__file__).read_text())
Module(
    body=[
        Expr(value=Constant(...)),
        ...,
        FunctionDef(
            name="__getattr__",
            args=arguments(...),
            body=[Expr(...), ..., Return(...)],
            decorator_list=[],
            returns=None,
            type_comment=None,
            type_params=[]
        )
    ],
    type_ignores=[]
)

Happy to take feedback :)

@tomasr8 tomasr8 requested a review from isidentical as a code owner March 19, 2024 21:29
@bedevere-app bedevere-app bot mentioned this pull request Mar 19, 2024
Closed
tomasr8
tomasr8 commented Mar 19, 2024
View reviewed changes
aisk
aisk reviewed Mar 20, 2024
View reviewed changes
tomasr8 and others added 5 commits March 20, 2024 19:04
@tomasr8 @aisk
Improve news entry
74759bc 
Co-authored-by: AN Long <aisk@users.noreply.github.com>
@tomasr8 @aisk
Follow PEP7
4ffc1fb 
Co-authored-by: AN Long <aisk@users.noreply.github.com>
@tomasr8
Fix test
6094100
@tomasr8
Regenerate files
5cde4fc
@JelleZijlstra
Merge branch 'main' into ast-repr
00133b8
JelleZijlstra
JelleZijlstra reviewed Apr 24, 2024
View reviewed changes
Copy link
Member
@JelleZijlstra JelleZijlstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, and sorry I missed this PR!

I think your approach makes sense but there's a number of small things missing in the C code, and we need good unit test coverage for the new code.

JelleZijlstra
JelleZijlstra reviewed May 2, 2024
View reviewed changes
@tomasr8
Copy link
Member Author
tomasr8 commented May 2, 2024

Thanks for taking a look!

I think your approach makes sense

If you had a different approach in mind, I'm happy to change it. This is what makes sense to me, but maybe it's not the best way to go about it..

I've been busy with work lately, but I'll try to add more tests to the PR in the coming days :)

JelleZijlstra
JelleZijlstra reviewed May 2, 2024
View reviewed changes
Copy link
Member
@JelleZijlstra JelleZijlstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should also be documented in the ast module (a versionchanged note saying that the repr() of AST nodes now includes the values of fields) and in the What's New for 3.13. (Or 3.14 if we don't get this ready before Tuesday.)

@jeremyhylton jeremyhylton self-requested a review May 23, 2024 17:59
devdanzin
devdanzin reviewed Jun 3, 2024
View reviewed changes
@tomasr8
Copy link
Member Author
tomasr8 commented Jul 14, 2024

I moved the tests to a separate data folder and added a CLI toggle to automatically regenerate them with ./python Lib/test/test_ast.py --snapshot-update. This is basically snapshot testing similar to how it's done in pytest for example. unittest unfortunately doesn't have this capability so it must be implemented separately in each module :/

The advantage of snapshot testing is that we don't need to write the test cases manually - both the initial tests and whenever they need to be updated. You can just run ./python Lib/test/test_ast.py --snapshot-update and compare the diff.

@ambv In case you have some thoughts on this

@tomasr8 tomasr8 requested a review from erlend-aasland as a code owner July 14, 2024 19:19
@erlend-aasland erlend-aasland removed their request for review July 15, 2024 12:05
@JelleZijlstra JelleZijlstra self-requested a review July 17, 2024 13:54
@tomasr8
Copy link
Member Author
tomasr8 commented Aug 3, 2024

^ fixed the conflicts in tests

vstinner
vstinner reviewed Aug 5, 2024
View reviewed changes
JelleZijlstra
JelleZijlstra reviewed Aug 14, 2024
View reviewed changes
Copy link
Member
@JelleZijlstra JelleZijlstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking pretty good!

Parser/asdl_c.py
}

if (!value_repr) {
Py_DECREF(name);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can DECREF name and value earlier so you don't have to repeat this. Looks like value is no longer needed after line 1602, so you could DECREF it there. name could be DECREFed after line 1623.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, I move those DECREFs up

@JelleZijlstra
Copy link
Member

There's some CI failure that looks like something needs to be tweaked in relation to the new file.

@tomasr8
Copy link
Member Author
tomasr8 commented Aug 14, 2024

There's some CI failure that looks like something needs to be tweaked in relation to the new file.

Thanks for pointing that out, should be fixed now :)

@JelleZijlstra
Copy link
Member

There's some CI failures related to conversion warnings, I think there are some changes landing soon around that so I'll wait for now.

@JelleZijlstra JelleZijlstra self-assigned this Sep 17, 2024
@JelleZijlstra JelleZijlstra merged commit 21d2a9a into python:main Sep 18, 2024
37 checks passed
@zware zware mentioned this pull request Sep 18, 2024
Merged
@tomasr8 tomasr8 deleted the ast-repr branch September 19, 2024 07:49
@tomasr8
Copy link
Member Author
tomasr8 commented Sep 19, 2024

Thanks @JelleZijlstra and everyone else for reviews and help on getting this done!

savannahostrowski pushed a commit to savannahostrowski/cpython that referenced this pull request Sep 22, 2024
@tomasr8 @aisk
@JelleZijlstra @AlexWaygood @picnixz @savannahostrowski
pythongh-116022: Improve repr() of AST nodes (python#117046)
491004a 
Co-authored-by: AN Long <aisk@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
@alex alex mentioned this pull request Oct 5, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vstinner vstinner vstinner left review comments

@aisk aisk aisk left review comments

@picnixz picnixz picnixz left review comments

@devdanzin devdanzin devdanzin left review comments

@JelleZijlstra JelleZijlstra JelleZijlstra approved these changes

@isidentical isidentical Awaiting requested review from isidentical isidentical is a code owner

@jeremyhylton jeremyhylton Awaiting requested review from jeremyhylton

Assignees

@JelleZijlstra JelleZijlstra

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Improve repr() of AST nodes
7 participants
@tomasr8 @JelleZijlstra @vstinner @aisk @picnixz @devdanzin @AlexWaygood
0