gh-104976: Ensure trailing 8000 dedent tokens are emitted as the previous tokenizer #104980
Conversation
7c0113e to
6db032a
Compare
|
@pablogsal Can you provide a very quick explanation of how this works and changes the location of the DEDENT token? Nothing too detailed, just a quick overview of the though process, cause I'm having difficulties understanding this. |
Yep: the objective is that inplicit dedents like these: are reported in the same line as the Unfortunately, this forces us to handle |
|
Tell me if something else is not clear:) |
Ah wait, I forgot to push some commits 😓 |
Aha! Ok, makes sense now. Thanks for the explanation though, man! |
8a23a18 to
19a58c5
Compare
Sorry for the confusion! 😅 Check again when you have time! |
…vious tokenizer Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
|
Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12. |
|
GH-105000 is a backport of this pull request to the 3.12 branch. |
…vious tokenizer (pythonGH-104980) (cherry picked from commit 46b52e6) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
Uh oh!
There was an error while loading. Please reload this page.