[3.9] bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993)
#29071
Conversation
…alid continuation characters (pythonGH-28993) There are two errors that this commit fixes: * The parser was not correctly computing the offset and the string source for E_LINECONT errors due to the incorrect usage of strtok(). * The parser was not correctly unwinding the call stack when a tokenizer exception happened in rules involving optionals ('?', [...]) as we always make them return valid results by using the comma operator. We need to check first if we don't have an error before continuing.. (cherry picked from commit a106343) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
|
On Python 3.9, the actual position reported is (3, 22) instead of the expected (2, 2). That's both before and after this PR. The difference is that the PR solves an assertion failure (see my comment on the issue) so it's worth landing it anyway. I'll be changing the test here to point at (3, 22) then. |
| def test_error_offset_continuation_characters(self): | ||
| check = self.check | ||
| check('"\\\n"(1 for c in I,\\\n\\', 2, 2) | ||
| check('"\\\n"(1 for c in I,\\\n\\', 3, 22) |
There was a problem hiding this comment.
This is unfortunately a regression:
>>> try:
... compile('"\\\n"(1 for c in I,\\\n\\', "", "exec")
... except SyntaxError as e:
... f = e
...
>>> f.lineno
2
>>> f.offset
3
|
When you're done making the requested changes, leave the comment: And if you don't make the requested changes, you will be put in the comfy chair! |
|
There was a problem hiding this comment.
On the other hand the old error is wrong:
Python 3.9.7 (default, Oct 10 2021, 15:13:22)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> compile('"\\\n"(1 for c in I,\\\n\\', "","exec")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "", line 2
"\
"(1 for c in I,\
\
^
SyntaxError: Generator expression must be parenthesized
while the one in this PR is correct but the offsets are badly computed:
Python 3.10.0 (default, Oct 4 2021, 22:36:16) [GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> compile('"\\\n"(1 for c in I,\\\n\\', "","exec")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "", line 2
"(1 for c in I,\
^
SyntaxError: '(' was never closed
>>>
Right, I read (2, 3) as (3, 2) in the output. So yes, definitely there's a regression here. |
|
I have been investigating and this is actually a bug in the tokenizer that was solved by the big tokenizer refactor I did for 3.10. Backporting that is going to be a pain and quite invasive unfortunately :( |
|
@ambv: Please replace |
There are two errors that this commit fixes:
source for E_LINECONT errors due to the incorrect usage of strtok().
exception happened in rules involving optionals ('?', [...]) as we
always make them return valid results by using the comma operator. We
need to check first if we don't have an error before continuing..
(cherry picked from commit a106343)
Co-authored-by: Pablo Galindo Salgado Pablogsal@gmail.com
https://bugs.python.org/issue45494