8000 gh-104169: Refactor tokenizer into lexer and wrappers by lysnikolaou · Pull Request #110684 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-104169: Refactor tokenizer into lexer and wrappers #110684

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Oct 11, 2023
Merged

gh-104169: Refactor tokenizer into lexer and wrappers #110684

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
99a2b40
gh-104169: Refactor tokenizer into lexer and wrappers
lysnikolaou Oct 11, 2023
39257a0
Fix build - remove extern's and fix tok_new decl
lysnikolaou Oct 11, 2023
a4190a9
Renames to remove smelly symbols
lysnikolaou Oct 11, 2023
a949d15
Fix build errors with subdirs
lysnikolaou Oct 11, 2023
95662b8
Fix windows build
lysnikolaou Oct 11, 2023
ec41b56
fixup! Fix windows build
pablogsal Oct 11, 2023
815cb34
Fix ifdef
lysnikolaou Oct 11, 2023
20be8a0
Add empty lines at the end
lysnikolaou Oct 11, 2023
128fd02
📜🤖 Added by blurb_it.
blurb-it[bot] Oct 11, 2023
df2c8cb
Fix lint errors
lysnikolaou Oct 11, 2023
7e10dbf
Fix more windows stuff
lysnikolaou Oct 11, 2023
aeeb562
Fix c-analyzer
lysnikolaou Oct 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix ifdef
  • Loading branch information
@lysnikolaou
lysnikolaou committed Oct 11, 2023
commit 815cb3476817b00b4b87adaba00e1803553fda0b
2 changes: 1 addition & 1 deletion Parser/tokenizer/helpers.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -514,7 +514,7 @@ _PyTokenizer_ensure_utf8(char *line, struct tok_state *tok)

/* ############## DEBUGGING STUFF ############## */

#if defined(Py_DEBUG)
#ifdef Py_DEBUG
void
_PyTokenizer_print_escape(FILE *f, const char *s, Py_ssize_t size)
{
Expand Down
2 changes: 1 addition & 1 deletion Parser/tokenizer/helpers.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ int _PyTokenizer_check_coding_spec(const char* line, Py_ssize_t size, struct tok
int set_readline(struct tok_state *, const char *));
int _PyTokenizer_ensure_utf8(char *line, struct tok_state *tok);

#if Py_DEBUG
#ifdef Py_DEBUG
void _PyTokenizer_print_escape(FILE *f, const char *s, Py_ssize_t size);
void _PyTokenizer_tok_dump(int type, char *start, char *end);
#endif
Expand Down
0