Closed
Description
Bug report
tokenize adds a concrete newline character when it is not present. this breaks any sort of roundtrip source generation and pycodestyle's end-of-file checker
here's an example of a file containing a single byte (generated via echo -n 1 > t.py)
# hd -c t.py
00000000 31 |1|
0000000 1
0000001
# python3.12 -m tokenize t.py
0,0-0,0: ENCODING 'utf-8'
1,0-1,1: NUMBER '1'
1,1-1,2: NEWLINE '\n'
2,0-2,0: ENDMARKER ''
# python3.11 -m tokenize t.py
0,0-0,0: ENCODING 'utf-8'
1,0-1,1: NUMBER '1'
1,1-1,2: NEWLINE ''
2,0-2,0: ENDMARKER '' Your environment
- CPython versions tested on: 3.12 dbd7d7c
- Operating system and architecture: ubuntu 22.04 LTS x86_64
cc @pablogsal
Linked PRs
Metadata
Metadata
Assignees
Projects
Status
Done