8000 Unable to parse parenthesized context managers · Issue #4145 · RustPython/RustPython · GitHub
[go: up one dir, main page]
Skip to content

Unable to parse parenthesized context managers #4145

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
charliermarsh opened this issue Aug 31, 2022 · 5 comments · Fixed by #4329
Closed

Unable to parse parenthesized context managers #4145

charliermarsh opened this issue Aug 31, 2022 · 5 comments · Fixed by #4329

Comments

@charliermarsh
Copy link
Collaborator
charliermarsh commented Aug 31, 2022 •
edited
Loading

In RustPython, this fails to parse:

from contextlib import contextmanager

@contextmanager
def ctx():
    yield

with (ctx() as foo):
    ...

...with:

---- parser::tests::test_parse_parenthesized stdout ----
thread 'parser::tests::test_parse_parenthesized' panicked at 'called `Result::unwrap()` on an `Err` value: BaseError { error: UnrecognizedToken(As, None), location: Location { row: 7, column: 13 }, source_path: "<test>" }', compiler/parser/src/parser.rs:209:58
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
@charliermarsh
Copy link
Collaborator Author

I tried to change WithStatement to parse via:

WithStatement: ast::Stmt = {
    <location:@L> <is_async:"async"?> "with" <items:OneOrMore<WithItem>> ":" <body:Suite> => {
        let type_comment = None;
        let node = if is_async.is_some() {
            ast::StmtKind::AsyncWith { items, body, type_comment }
        } else {
            ast::StmtKind::With { items, body, type_comment }
        };
        ast::Stmt::new(location, node)
    },
    <location:@L> <is_async:"async"?> "with" "(" <items:OneOrMore<WithItem>> ")" ":" <body:Suite> => {
        let type_comment = None;
        let node = if is_async.is_some() {
            ast::StmtKind::AsyncWith { items, body, type_comment }
        } else {
            ast::StmtKind::With { items, body, type_comment }
        };
        ast::Stmt::new(location, node)
    },
};

But LALRPOP is not happy:

python.lalrpop:683:5: 700:0: Conflict detected

    when in this state:
  NamedExpressionTest = Test (*) ["\n", "!=", "%", "%=", "&", "&=", "(", ")", "*", "**", "**=", "*=", "+", "+=", ",", "-", "-=", "->", ".", "...", "/", "//", "//=", "/=", ":", ":=", ";", "<", "<<", "<<=", "<=", "=", "==", ">", ">=", ">>", ">>=", "@", "@=", "False", "None", "True", "[", "]", "^", "^=", "and", "as", "assert", "async", "await", "break", "class", "continue", "def", "del", "elif", "else", "except", "finally", "for", "from", "from", "global", "if", "import", "in", "is", "lambda", "nonlocal", "not", "or", "pass", "raise", "return", "try", "while", "with", "yield", "{", "|", "|=", "}", "~", Dedent, Indent, StartExpression, StartInteractive, StartModule, bytes, complex, float, int, name, string, EOF]
  WithItem = Test (*) ["\n", "!=", "%", "%=", "&", "&=", "(", ")", "*", "**", "**=", "*=", "+", "+=", ",", "-", "-=", "->", ".", "...", "/", "//", "//=", "/=", ":", ":=", ";", "<", "<<", "<<=", "<=", "=", "==", ">", ">=", ">>", ">>=", "@", "@=", "False", "None", "True", "[", "]", "^", "^=", "and", "as", "assert", "async", "await", "break", "class", "continue", "def", "del", "elif", "else", "except", "finally", "for", "from", "from", "global", "if", "import", "in", "is", "lambda", "nonlocal", "not", "or", "pass", "raise", "return", "try", "while", "with", "yield", "{", "|", "|=", "}", "~", Dedent, Indent, StartExpression, StartInteractive, StartModule, bytes, complex, float, int, name, string, EOF]
  WithItem = Test (*) "as" Expression ["\n", "!=", "%", "%=", "&", "&=", "(", ")", "*", "**", "**=", "*=", "+", "+=", ",", "-", "-=", "->", ".", "...", "/", "//", "//=", "/=", ":", ":=", ";", "<", "<<", "<<=", "<=", "=", "==", ">", ">=", ">>", ">>=", "@", "@=", "False", "None", "True", "[", "]", "^", "^=", "and", "as", "assert", "async", "await", "break", "class", "continue", "def", "del", "elif", "else", "except", "finally", "for", "from", "from", "global", "if", "import", "in", "is", "lambda", "nonlocal", "not", "or", "pass", "raise", "return", "try", "while", "with", "yield", "{", "|", "|=", "}", "~", Dedent, Indent, StartExpression, StartInteractive, StartModule, bytes, complex, float, int, name, string, EOF]

  and looking at a token `"as"` we can reduce to a `NamedExpressionTest` but we can also shift
...

@charliermarsh charliermarsh mentioned this issue Aug 31, 2022
Closed
@youknowone
Copy link
Member

How about adding optional parenthesis to WithItem?

@charliermarsh
Copy link
Collaborator Author

Seeing the same (or similar issue), unable to compile.

Also, I think we do need the parens around the entire with-items, not any individual item:

Screen Shot 2022-09-16 at 5 46 58 AM

@andersk
Copy link
Contributor
andersk commented Sep 28, 2022 •
edited
Loading

This was actually one of the issues that motivated CPython 3.9’s switch from an LL(1) parser to a PEG parser (python/cpython#56991 (comment), PEP 617).

See also #2759.

@charliermarsh
Copy link
Collaborator Author

Woah, good find.

andersk added a commit to andersk/RustPython that referenced this issue Dec 12, 2022
@andersk
Parse Python 3.9+ parenthesized context managers
9d99fd3 
Since the upstream grammar for this is not LALR(1), we abuse LALRPOP
macros and the Into/TryInto traits to build a cover grammar that
converts to either tuples or `with` items after additional validation.
It’s annoying and ugly, but something like this is basically our only
option short of switching to a more powerful parser algorithm.

Fixes RustPython#4145.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
andersk added a commit to andersk/RustPython that referenced this issue Dec 12, 2022
@andersk
Parse Python 3.9+ parenthesized context managers
6375425 
Since the upstream grammar for this is not LR(1), we abuse LALRPOP
macros and the Into/TryInto traits to build a cover grammar that
converts to either tuples or `with` items after additional validation.
It’s annoying and ugly, but something like this is basically our only
option short of switching to a more powerful parser algorithm.

Fixes RustPython#4145.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
@andersk andersk mentioned this issue Dec 12, 2022
Merged
andersk added a commit to andersk/RustPython that referenced this issue Dec 12, 2022
@andersk
Parse Python 3.9+ parenthesized context managers
b7f4ff7 
Since the upstream grammar for this is not LR(1), we abuse LALRPOP
macros and the Into/TryInto traits to build a cover grammar that
converts to either tuples or `with` items after additional validation.
It’s annoying and ugly, but something like this is basically our only
option short of switching to a more powerful parser algorithm.

Fixes RustPython#4145.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Parse Python 3.9+ parenthesized context managers andersk/RustPython
3 participants
@andersk @youknowone @charliermarsh
0