8000 3.12+: tokenize of mixed tabs and spaces now produces an error · Issue #105713 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content

3.12+: tokenize of mixed tabs and spaces now produces an error #105713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
asottile opened this issue Jun 12, 2023 · 1 comment
Closed

3.12+: tokenize of mixed tabs and spaces now produces an error #105713

asottile opened this issue Jun 12, 2023 · 1 comment
Assignees
Labels
3.12 only security fixes 3.13 bugs and security fixes type-bug An unexpected behavior, bug, or error

Comments

@asottile
Copy link
Contributor
asottile commented Jun 12, 2023

I read the parts in here and it does mention that some ERRORTOKENs will be converted to exceptions but it doesn't seem to cover this case.

this breaks the E101 check in pycodestyle (or I guess more specifically turns it into a syntax error)

this is probably fine? but should be called out in the documentation ?

Bug report

if True:
    print(1)  # indented with spaces
	print(2)  # indented with tab
$ python3.11 -m tokenize t.py 
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,2:            NAME           'if'           
1,3-1,7:            NAME           'True'         
1,7-1,8:            OP             ':'            
1,8-1,9:            NEWLINE        '\n'           
2,0-2,4:            INDENT         '    '         
2,4-2,9:            NAME           'print'        
2,9-2,10:           OP             '('            
2,10-2,11:          NUMBER         '1'            
2,11-2,12:          OP             ')'            
2,12-2,13:          NEWLINE        '\n'           
3,0-3,1:            INDENT         '\t'           
3,1-3,6:            NAME           'print'        
3,6-3,7:            OP             '('            
3,7-3,8:            NUMBER         '2'            
3,8-3,9:            OP             ')'            
3,9-3,10:           NEWLINE        '\n'           
4,0-4,0:            DEDENT         ''             
4,0-4,0:            DEDENT         ''             
4,0-4,0:            ENDMARKER      ''             
$ python3.12 -m tokenize t.py 
t.py:3:10: error: inconsistent use of tabs and spaces in indentation

Your environment

  • CPython versions tested on: d310fc7
  • Operating system and architecture: ubuntu 22.04 x86_64

Linked PRs

@asottile asottile added the type-bug An unexpected behavior, bug, or error label Jun 12, 2023
@AlexWaygood AlexWaygood added 3.12 only security fixes 3.13 bugs and security fixes labels Jun 12, 2023
@lysnikolaou
Copy link
Member

This is probably alright, yes. I'll open a PR to mention it in the docs.

lysnikolaou added a commit to lysnikolaou/cpython that referenced this issue Jun 13, 2023
lysnikolaou added a commit that referenced this issue Jun 13, 2023
…5723)

* gh-105713: Document that tokenize raises when mixing tabs/spaces
* Update Doc/whatsnew/3.12.rst

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jun 13, 2023
pythonGH-105723)

* pythongh-105713: Document that tokenize raises when mixing tabs/spaces
* Update Doc/whatsnew/3.12.rst

(cherry picked from commit ed8217b)

Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
lysnikolaou added a commit that referenced this issue Jun 13, 2023
…es (GH-105723) (#105725)

* gh-105713: Document that tokenize raises when mixing tabs/spaces
* Update Doc/whatsnew/3.12.rst

(cherry picked from commit ed8217b)

Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.12 only security fixes 3.13 bugs and security fixes type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

4 participants
0