8000 3.12+: tokenize of mixed tabs and spaces now produces an error · Issue #105713 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

3.12+: tokenize of mixed tabs and spaces now produces an error #105713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
asottile opened this issue Jun 12, 2023 · 1 comment
Closed

3.12+: tokenize of mixed tabs and spaces now produces an error #105713

asottile opened this issue Jun 12, 2023 · 1 comment
Assignees
@pablogsal @lysnikolaou
Labels
3.12 only security fixes 3.13 bugs and security fixes type-bug An unexpected behavior, bug, or error

Comments

@asottile
Copy link
Contributor
asottile commented Jun 12, 2023 •
edited by bedevere-bot
Loading

I read the parts in here and it does mention that some ERRORTOKENs will be converted to exceptions but it doesn't seem to cover this case.

this breaks the E101 check in pycodestyle (or I guess more specifically turns it into a syntax error)

this is probably fine? but should be called out in the documentation ?

Bug report

if True:
    print(1)  # indented with spaces
	print(2)  # indented with tab
$ python3.11 -m tokenize t.py 
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,2:            NAME           'if'           
1,3-1,7:            NAME           'True'         
1,7-1,8:            OP             ':'            
1,8-1,9:            NEWLINE        '\n'           
2,0-2,4:            INDENT         '    '         
2,4-2,9:            NAME           'print'        
2,9-2,10:           OP             '('            
2,10-2,11:          NUMBER         '1'            
2,11-2,12:          OP             ')'            
2,12-2,13:          NEWLINE        '\n'           
3,0-3,1:            INDENT         '\t'           
3,1-3,6:            NAME           'print'        
3,6-3,7:            OP             '('            
3,7-3,8:            NUMBER         '2'            
3,8-3,9:            OP             ')'            
3,9-3,10:           NEWLINE        '\n'           
4,0-4,0:            DEDENT         ''             
4,0-4,0:            DEDENT         ''             
4,0-4,0:            ENDMARKER      ''             
$ python3.12 -m tokenize t.py 
t.py:3:10: error: inconsistent use of tabs and spaces in indentation

Your environment

  • CPython versions tested on: d310fc7
  • Operating system and architecture: ubuntu 22.04 x86_64

Linked PRs
@asottile asottile added the type-bug An unexpected behavior, bug, or error label Jun 12, 2023
@AlexWaygood AlexWaygood added 3.12 only security fixes 3.13 bugs and security fixes labels Jun 12, 2023
@lysnikolaou
Copy link
Member

This is probably alright, yes. I'll open a PR to mention it in the docs.

lysnikolaou added a commit to lysnikolaou/cpython that referenced this issue Jun 13, 2023
@lysnikolaou
[3.12] pythongh-105713: Mention in docs what mixing spaces/tabs now r…
fac6825 
…aises
lysnikolaou added a commit to lysnikolaou/cpython that referenced this issue Jun 13, 2023
@lysnikolaou
pythongh-105713: Document that tokenize raises when mixing tabs/spaces
b77999a
@bedevere-bot bedevere-bot mentioned this issue Jun 13, 2023
Merged
@miss-islington miss-islington mentioned this issue Jun 13, 2023
Merged
lysnikolaou added a commit that referenced this issue Jun 13, 2023
@lysnikolaou @AlexWaygood
gh-105713: Document that tokenize raises when mixing tabs/spaces (#10…
ed8217b 
…5723)

* gh-105713: Document that tokenize raises when mixing tabs/spaces
* Update Doc/whatsnew/3.12.rst

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 13, 2023
@lysnikolaou @AlexWaygood @miss-islington
pythongh-105713: Document that tokenize raises when mixing tabs/spaces (
0cd0a54 
pythonGH-105723)

* pythongh-105713: Document that tokenize raises when mixing tabs/spaces
* Update Doc/whatsnew/3.12.rst

(cherry picked from commit ed8217b)

Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
lysnikolaou added a commit that referenced this issue Jun 13, 2023
@miss-islington @lysnikolaou @AlexWaygood
[3.12] gh-105713: Document that tokenize raises when mixing tabs/spac…
92929fd 
…es (GH-105723) (#105725)

* gh-105713: Document that tokenize raises when mixing tabs/spaces
* Update Doc/whatsnew/3.12.rst

(cherry picked from commit ed8217b)

Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pablogsal pablogsal

@lysnikolaou lysnikolaou

Labels
3.12 only security fixes 3.13 bugs and security fixes type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@asottile @pablogsal @lysnikolaou @AlexWaygood
0