10000 [3.12] gh-116666: Add "token" glossary term (GH-130888) (GH-131368) · python/cpython@7a8097c · GitHub
[go: up one dir, main page]
Skip to content

Commit 7a8097c

Browse files
miss-islingtonencukouAA-Turner
authored
[3.12] gh-116666: Add "token" glossary term (GH-130888) (GH-131368)
gh-116666: Add "token" glossary term (GH-130888) Add glossary entry for `token`, and link to it. Avoid talking about tokens in the SyntaxError intro (errors.rst); at this point tokenization is too much of a technical detail. (Even to an advanced reader, the fact that a *single* token is highlighted isn't too relevant. Also, we don't need to guarantee that it's a single token.) (cherry picked from commit 30d5205) Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
1 parent 9d23a6d commit 7a8097c

File tree

4 files changed

+28
-11
lines changed

4 files changed

+28
-11
lines changed

‎Doc/glossary.rst

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -734,6 +734,10 @@ Glossary
734734
thread removes *key* from *mapping* after the test, but before the lookup.
735735
This issue can be solved with locks or by using the EAFP approach.
736736

737+
lexical analyzer
738+
739+
Formal name for the *tokenizer*; see :term:`token`.
740+
737741
list
738742
A built-in Python :term:`sequence`. Despite its name it is more akin
739743
to an array in other languages than to a linked list since access to
@@ -1212,6 +1216,17 @@ Glossary
12121216
See also :term:`binary file` for a file object able to read and write
12131217
:term:`bytes-like objects <bytes-like object>`.
12141218

1219+
token
1220+
1221+
A small unit of source code, generated by the
1222+
:ref:`lexical analyzer <lexical>` (also called the *tokenizer*).
1223+
Names, numbers, strings, operators,
1224+
newlines and similar are represented by tokens.
1225+
1226+
The :mod:`tokenize` module exposes Python's lexical analyzer.
1227+
The :mod:`token` module contains information on the various types
1228+
of tokens.
1229+
12151230
triple-quoted string
12161231
A string which is bound by three instances of either a quotation mark
12171232
(") or an apostrophe ('). While they don't provide any functionality

‎Doc/reference/lexical_analysis.rst

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,9 @@ Lexical analysis
88
.. index:: lexical analysis, parser, token
99

1010
A Python program is read by a *parser*. Input to the parser is a stream of
11-
*tokens*, generated by the *lexical analyzer*. This chapter describes how the
12-
lexical analyzer breaks a file into tokens.
11+
:term:`tokens <token>`, generated by the *lexical analyzer* (also known as
12+
the *tokenizer*).
13+
This chapter describes how the lexical analyzer breaks a file into tokens.
1314

1415
Python reads program text as Unicode code points; the encoding of a source file
1516
can be given by an encoding declaration and defaults to UTF-8, see :pep:`3120`

‎Doc/tutorial/errors.rst

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,12 @@ complaint you get while you are still learning Python::
2424
SyntaxError: invalid syntax
2525

2626
The parser repeats the offending line and displays little arrows pointing
27-
at the token in the line where the error was detected. The error may be
28-
caused by the absence of a token *before* the indicated token. In the
29-
example, the error is detected at the function :func:`print`, since a colon
30-
(``':'``) is missing before it. File name and line number are printed so you
31-
know where to look in case the input came from a script.
27+
at the place where the error was detected. Note that this is not always the
28+
place that needs to be fixed. In the example, the error is detected at the
29+
function :func:`print`, since a colon (``':'``) is missing just before it.
30+
31+
The file name (``<stdin>`` in our example) and line number are printed so you
32+
know where to look in case th 8000 e input came from a file.
3233

3334

3435
.. _tut-exceptions:

‎Doc/tutorial/interactive.rst

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -37,10 +37,10 @@ Alternatives to the Interactive Interpreter
3737

3838
This facility is an enormous step forward compared to earlier versions of the
3939
interpreter; however, some wishes are left: It would be nice if the proper
40-
indentation were suggested on continuation lines (the parser knows if an indent
41-
token is required next). The completion mechanism might use the interpreter's
42-
symbol table. A command to check (or even suggest) matching parentheses,
43-
quotes, etc., would also be useful.
40+
indentation were suggested on continuation lines (the parser knows if an
41+
:data:`~token.INDENT` token is required next). The completion mechanism might
42+
use the interpreter's symbol table. A command to check (or even suggest)
43+
matching parentheses, quotes, etc., would also be useful.
4444

4545
One alternative enhanced interactive interpreter that has been around for quite
4646
some time is IPython_, which features tab completion, object exploration and

0 commit comments

Comments
 (0)
0