8000 gh-104169: Refactor tokenizer into lexer and wrappers by lysnikolaou · Pull Request #110684 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-104169: Refactor tokenizer into lexer and wrappers #110684

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Oct 11, 2023
Merged

gh-104169: Refactor tokenizer into lexer and wrappers #110684

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
99a2b40
gh-104169: Refactor tokenizer into lexer and wrappers
lysnikolaou Oct 11, 2023
39257a0
Fix build - remove extern's and fix tok_new decl
lysnikolaou Oct 11, 2023
a4190a9
Renames to remove smelly symbols
lysnikolaou Oct 11, 2023
a949d15
Fix build errors with subdirs
lysnikolaou Oct 11, 2023
95662b8
Fix windows build
lysnikolaou Oct 11, 2023
ec41b56
fixup! Fix windows build
pablogsal Oct 11, 2023
815cb34
Fix ifdef
lysnikolaou Oct 11, 2023
20be8a0
Add empty lines at the end
lysnikolaou Oct 11, 2023
128fd02
📜🤖 Added by blurb_it.
blurb-it[bot] Oct 11, 2023
df2c8cb
Fix lint errors
lysnikolaou Oct 11, 2023
7e10dbf
Fix more windows stuff
lysnikolaou Oct 11, 2023
aeeb562
Fix c-analyzer
lysnikolaou Oct 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
📜🤖 Added by blurb_it.
  • Loading branch information
@blurb-it
blurb-it[bot] authored Oct 11, 2023
commit 128fd0272967907a32299e47e4e5137e20a35cb5
3 changes: 3 additions & 0 deletions Misc/NEWS.d/next/Core and Builtins/2023-10-11-12-48-03.gh-issue-104169.bPoX8u.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Split the tokenizer into two separate directories:
- One part includes the actual lexeme producing logic and lives in `Parser/lexer`.
- The second part wraps the lexer according to the different tokenization modes we have (string, utf-8, file, interactive, readline) and lives in `Parser/tokenizer`.
0