8000 gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin by pablogsal · Pull Request #94386 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin #94386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jul 5, 2022
Merged

gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin #94386

merged 2 commits into from
Jul 5, 2022

Conversation

pablogsal
Copy link
Member
@pablogsal pablogsal commented Jun 28, 2022 •
edited
Loading

@pablogsal pablogsal requested a review from lysnikolaou as a code owner June 28, 2022 13:21
@pablogsal pablogsal added needs backport to 3.10 only security fixes needs backport to 3.11 only security fixes labels Jun 28, 2022
@pablogsal
Copy link
Member Author
pablogsal commented Jun 28, 2022 •
edited
Loading

I'm not adding a test (for now) because this doesn't reproduce nicely on the test suite.

@pablogsal pablogsal marked this pull request as draft June 28, 2022 13:46
@pablogsal
pythongh-94360: Fix a tokenizer crash when reading encoded files with…
a7cf92c 
… syntax errors from stdin

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
@pablogsal pablogsal marked this pull request as ready for review June 28, 2022 15:23
@zware
Copy link
Member
zware commented Jun 28, 2022

Confirmed that this resolves the crash as produced by my reduced reproducer in main 8000 and 3.11, and without the change to Parser/pegen_errors.c (which doesn't exist in 3.10) it also resolves both reproducers for 3.10.

@pablogsal
Copy link
Member Author

Thanks for checking @zware ! ♥️

ambv
ambv reviewed Jul 5, 2022
View reviewed changes
Parser/pegen_errors.c
@@ -259,15 +259,15 @@ get_error_line_from_tokenizer_buffers(Parser *p, Py_ssize_t lineno)
const char* buf_end = p->tok->fp_interactive ? p->tok->interactive_src_end : p->tok->inp;

for (int i = 0; i < relative_lineno - 1; i++) {
char *new_line = strchr(cur_line, '\n') + 1;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NULL + 1 is beautifully strongly typed, huh? 😎

< 8000 details-menu class="dropdown-menu dropdown-menu-sw show-more-popover color-fg-default" style="width:185px" src="" preload > Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6lujn3.jpg

ambv
ambv reviewed Jul 5, 2022
View reviewed changes
@pablogsal pablogsal merged commit 36fcde6 into python:main Jul 5, 2022
@pablogsal pablogsal deleted the gh-94360 branch July 5, 2022 16:39
@miss-islington
Copy link
Contributor

Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖

@bedevere-bot
Copy link

GH-94573 is a backport of this pull request to the 3.11 branch.

@miss-islington
Copy link
Contributor

Sorry, @pablogsal, I could not cleanly backport this to 3.10 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 36fcde61ba48c4e918830691ecf4092e4e3b9b99 3.10

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 5, 2022
@pablogsal @miss-islington
pythongh-94360: Fix a tokenizer crash when reading encoded files with…
2012ce5 
… syntax errors from stdin (pythonGH-94386)

* pythongh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>

* nitty nit

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
(cherry picked from commit 36fcde6)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
miss-islington added a commit that referenced this pull request Jul 5, 2022
@miss-islington @pablogsal
gh-94360: Fix a tokenizer crash when reading encoded files with synta…
d49c99f 
…x errors from stdin (GH-94386)

* gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>

* nitty nit

Co-authored-by: Łukasz Langa <lukasz@langa.pl>
(cherry picked from commit 36fcde6)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
@bedevere-bot
Copy link

GH-94574 is a backport of this pull request to the 3.10 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.10 only security fixes label Jul 5, 2022
pablogsal added a commit to pablogsal/cpython that referenced this pull request Jul 5, 2022
@pablogsal
[3.10] pythongh-94360: Fix a tokenizer crash when reading encoded fil…
005fc4b 
…es with syntax errors from stdin (pythonGH-94386)

* pythongh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>

* nitty nit

Co-authored-by: Łukasz Langa <lukasz@langa.pl>.
(cherry picked from commit 36fcde6)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
ambv pushed a commit that referenced this pull request Jul 5, 2022
@pablogsal
[3.10] gh-94360: Fix a tokenizer crash when reading encoded files wit…
697e78c 
…h syntax errors from stdin (GH-94386) (GH-94574)

Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>

(cherry picked from commit 36fcde6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
6DBF
Reviewers

@ambv ambv ambv left review comments

@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner

Assignees

@pablogsal pablogsal

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Tokenizer crash when redirecting input to stdin
5 participants
@pablogsal @zware @miss-islington @bedevere-bot @ambv
0