8000 gh-102856: Python tokenizer implementation for PEP 701 by mgmacias95 · Pull Request #104323 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-102856: Python tokenizer implementation for PEP 701 #104323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
May 21, 2023
Merged

gh-102856: Python tokenizer implementation for PEP 701 #104323

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
008f8e5
First iteration
mgmacias95 Apr 19, 2023
67a6ad6
Handle escaping {
mgmacias95 Apr 27, 2023
f58104d
nested expressions
mgmacias95 Apr 29, 2023
26102cc
Recursive expression tokenization
mgmacias95 May 2, 2023
a5f4b40
Remove intermediate token created for dev purposes
mgmacias95 May 2, 2023
598bab4
More improvements
mgmacias95 May 3, 2023
a0ed816
fix handling of } tokens
mgmacias95 May 7, 2023
90b4ab1
other tokenizer
pablogsal May 16, 2023
63ef1c1
Some progress
pablogsal May 17, 2023
6833b1a
Fix more bugs
pablogsal May 18, 2023
90da796
Fix more problems
pablogsal May 18, 2023
b5ccd94
Use IA to clean code
pablogsal May 18, 2023
b1c3b2a
Remove lel
pablogsal May 18, 2023
e941f12
Remove whitespace
pablogsal May 18, 2023
67a0239
Fix docs
mgmacias95 May 18, 2023
dcd221f
Moar tests and fix location error
pablogsal May 19, 2023
fd8b60a
Some cleanups
pablogsal May 19, 2023
f1a5090
pass the vacuum cleaner
pablogsal May 19, 2023
7fb58b0
Fix refleaks
mgmacias95 May 20, 2023
e1b5d35
📜🤖 Added by blurb_it.
blurb-it[bot] May 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove intermediate token created for dev purposes
  • Loading branch information
@mgmacias95 @pablogsal
mgmacias95 authored and pablogsal committed May 18, 2023
commit a5f4b408aed4a4006b8eb77a6d741029db630783
1 change: 0 additions & 1 deletion Grammar/Tokens
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ SOFT_KEYWORD
FSTRING_START
FSTRING_MIDDLE
FSTRING_END
FSTRING_EXPR
ERRORTOKEN

# These aren't used by the C tokenizer but are needed for tokenize.py
Expand Down
1 change: 0 additions & 1 deletion Include/internal/pycore_token.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,6 @@ extern "C" {
#define FSTRING_MIDDLE 62
#define FSTRING_END 63
#define ERRORTOKEN 64
#define FSTRING_EXPR 69
#define N_TOKENS 68
#define NT_OFFSET 256

Expand Down
0