Fix compilation issue and improve text

python · pablogsal · May 24, 2023 · May 23, 2023 · May 23, 2023 · May 23, 2023
commit 1182d56b2307d0b3e1f4c30bedd9b60c6c54172e
diff --git a/Doc/whatsnew/3.12.rst b/Doc/whatsnew/3.12.rst
@@ -136,22 +136,70 @@ Improved Error Messages
 New Features
 ============
 
-* Add :ref:`perf_profiling` through the new
-  environment variable :envvar:`PYTHONPERFSUPPORT`,
-  the new command-line option :option:`-X perf <-X>`,
-  as well as the new :func:`sys.activate_stack_trampoline`,
-  :func:`sys.deactivate_stack_trampoline`,
-  and :func:`sys.is_stack_trampoline_active` APIs.
-  (Design by Pablo Galindo. Contributed by Pablo Galindo and Christian Heimes
-  with contributions from Gregory P. Smith [Google] and Mark Shannon
-  in :gh:`96123`.)
-* The extraction methods in :mod:`tarfile`, and :func:`shutil.unpack_archive`,
-  have a new a *filter* argument that allows limiting tar features than may be
-  surprising or dangerous, such as creating files outside the destination
-  directory.
-  See :ref:`tarfile-extraction-filter` for details.
-  In Python 3.14, the default will switch to ``'data'``.
-  (Contributed by Petr Viktorin in :pep:`706`.)
+.. _whatsnew312-pep701:
+
+PEP 701: Syntactic formalization of f-strings
+---------------------------------------------
+
+:pep:`701` lifts some restrictions on the usage of f-strings. Expression components
+inside f-strings can now be any valid Python expression including backslashes,
+unicode escaped sequences, multi-line expressions, comments and strings reusing the
+same quote as the containing f-string. Let's cover these in detail:
+
+* Quote reuse: in Python 3.11, reusing the same quotes as the contaning f-string
+  raises a :exc:`SyntaxError`, forcing the user to either use other available
+  quotes (like using double quotes or triple quites if the f-string uses single
+  quites). In Python 3.12, you can now do things like this:
+
+    >>> songs = ['Take me back to Eden', 'Alkaline', 'Ascensionism']
+    >>> f"This is the playlist: {", ".join(things)}"
+    'This is the playlist: Take me back to Eden, Alkaline, Ascensionism
+
+  Note that before this change there was no explicit limit in how f-strings can
+  be nested, but the fact that string quotes cannot be reused inside the
+  expression component of f-strings made it impossible to nest f-strings
+  arbitrarily. In fact, this is the most nested-fstring that can be written:
+
+    >>> f"""{f'''{f'{f"{1+1}"}'}'''}"""
+    '2'
+
+  As now f-strings can contain any valid Python expression inside expression
+  components, it is now possible to nest f-strings arbitrarily:
+
+    >>> f"{f"{f"{f"{f"{f"{1+1}"}"}"}"}"}"
+    '2'
+
+* Multi-line expressions and comments: In Python 3.11, f-strings expressions
+  must be defined in a single line even if outside f-strings expressions could
+  span multiple lines (like literal lists being defined over multiple lines),
+  making them harder to read. In Python 3.12 you can now define expressions
+  spaning multiple lines and include comments on them:
+
+    >>> f"This is the playlist: {", ".join([
+    ...     'Take me back to Eden',  # My, my, those eyes like fire
+    ...     'Alkaline',              # Not acid nor alkaline
+    ...     'Ascensionism'           # Take to the broken skies at last
+    ... ])}"
+    'This is the playlist: Take me back to Eden, Alkaline, Ascensionism'
+
+* Backslashes and unicode characters: before Python 3.12 f-string expressions
+  couldn't contain any ``\`` character. This also affected unicode escaped
+  sequences (such as ``\N{snowman}``) as these contain the ``\N`` part that
+  previously could not be part of expression components of f-strings. Now, you
+  can define expressions like this:
+
+    >>> print(f"This is the playlist: {"\n".join(songs)}")
+    This is the playlist: Take me back to Eden
+    Alkaline
+    Ascensionism
+    >>> print(f"This is the playlist: {"\N{BLACK HEART SUIT}".join(songs)}")
+    This is the playlist: Take me back to Eden♥Alkaline♥Ascensionism
+
+See :pep:`701` for more details.
+
+(Contributed by Pablo Galindo, Batuhan Taskaya, Lysandros Nikolaou, Cristián
+Maureira-Fredes and Marta Gómez in :gh:`102856`. PEP written by Pablo Galindo,
+Batuhan Taskaya, Lysandros Nikolaou and Marta Gómez).
 
 .. _whatsnew312-pep709:
 
@@ -177,50 +225,6 @@ Inlining does result in a few visible behavior changes:
 
 Contributed by Carl Meyer and Vladimir Matveev in :pep:`709`.
 
-PEP 701: Syntactic formalization of f-strings
----------------------------------------------
-
-:pep:`701` lifts some restrictions in the usage of f-strings. The expressions
-inside f-strings can now be any Python expression: including backslashes,
-unicode characters, multi-line expressions, comments and strings reusing the
-same quote. Let's see some examples:
-
-* Quote reuse: in Python 3.11, reusing the same quotes as the f-string raised a
-  :exc:`SyntaxError`, forcing the user to either use single quotes or escape the quotes
-  inside the expression. In Python 3.12, you can now do things like this:
-
-    >>> things = ['a', 'b', 'c']
-    >>> f"These are the things: {", ".join(things)}"
-    'These are the things: a, b, c'
-
-* Multi-line expressions and comments: In Python 3.11, f-strings expressions
-  must be defined in a single line, making them harder to read. In Python 3.12
-  you can now define expressions in multiple lines and include comments on them:
-
-    >>> f"These are the things: {", ".join([
-    ...     'a',  # A
-    ...     'b',  # B
-    ...     'c'   # C
-    ... ])}"
-    'These are the things: a, b, c'
-
-* Backslashes and unicode characters: f-string expressions couldn't contain
-  any \ character (except for escaping quotes). This also affected unicode
-  characters (such as `\N{snowman}`). Now, you can define expressions like this:
-
-    >>> print(f"These are the things: {"\n".join(things)}")
-    These are the things: a
-    b
-    c
-    >>> print(f"These are the things: {"\N{snowman}".join(things)}")
-    These are the things: a☃b☃c
-
-See :pep:`701` for more details.
-
-(Contributed by Pablo Galindo, Batuhan Taskaya, Lysandros Nikolaou, Cristián
-Maureira-Fredes and Marta Gómez in :gh:`102856`. PEP written by Pablo Galindo,
-Batuhan Taskaya and Lysandros Nikolaou)
-
 PEP 688: Making the buffer protocol accessible in Python
 --------------------------------------------------------
 
@@ -264,6 +268,24 @@ See :pep:`692` for more details.
 
 (PEP written by Franek Magiera)
 
+
+* Add :ref:`perf_profiling` through the new
+  environment variable :envvar:`PYTHONPERFSUPPORT`,
+  the new command-line option :option:`-X perf <-X>`,
+  as well as the new :func:`sys.activate_stack_trampoline`,
+  :func:`sys.deactivate_stack_trampoline`,
+  and :func:`sys.is_stack_trampoline_active` APIs.
+  (Design by Pablo Galindo. Contributed by Pablo Galindo and Christian Heimes
+  with contributions from Gregory P. Smith [Google] and Mark Shannon
+  in :gh:`96123`.)
+* The extraction methods in :mod:`tarfile`, and :func:`shutil.unpack_archive`,
+  have a new a *filter* argument that allows limiting tar features than may be
+  surprising or dangerous, such as creating files outside the destination
+  directory.
+  See :ref:`tarfile-extraction-filter` for details.
+  In Python 3.14, the default will switch to ``'data'``.
+  (Contributed by Petr Viktorin in :pep:`706`.)
+
 Other Language Changes
 ======================
 
@@ -342,12 +364,6 @@ array
 * The :class:`array.array` class now supports subscripting, making it a
   :term:`generic type`. (Contributed by Jelle Zijlstra in :gh:`98658`.)
 
-tokenize
---------
-
-* The :mod:`tokenize` module includes the changes introduced in :pep:`701`. (
-  Contributed by Marta Gómez Macías and Pablo Galindo in :gh:`102856`.)
-
 asyncio
 -------
 
@@ -593,6 +609,12 @@ tkinter
   like ``create_*()`` methods.
   (Contributed by Serhiy Storchaka in :gh:`94473`.)
 
+tokenize
+--------
+
+* The :mod:`tokenize` module includes the changes introduced in :pep:`701`. (
+  Contributed by Marta Gómez Macías and Pablo Galindo in :gh:`102856`.)
+
 types
 -----
 
@@ -737,9 +759,10 @@ Optimizations
 * Speed up :class:`asyncio.Task` creation by deferring expensive string formatting.
   (Contributed by Itamar O in :gh:`103793`.)
 
-* The :func:
ED4F
`tokenize.tokenize` and :func:`tokenize.generate_tokens` calls are
-  now 64% faster due to the changes introduced in :pep:`701`. (Contributed by
-  Marta Gómez Macías and Pablo Galindo in :gh:`102856`.)
+* The :func:`tokenize.tokenize` and :func:`tokenize.generate_tokens` functions are
+  up to 64% faster as a side effect of the changes required to cover :pep:`701` in
+  the :mod:`tokenize` module. (Contributed by Marta Gómez Macías and Pablo Galindo
+  in :gh:`102856`.)
 
 
 CPython bytecode changes
@@ -1255,12 +1278,33 @@ Changes in the Python API
   that may be surprising or dangerous.
   See :ref:`tarfile-extraction-filter` for details.
 
-* The behaviour of :func:`tokenize.tokenize` and
-  :func:`tokenize.generate_tokens` is now changed due to the changes introduced
-  in :pep:`701`. In addition to that, final ``DEDENT`` tokens are now within
-  the file bounds. This means that for a file containing 3 lines, the old
-  tokenizer returned a ``DEDENT`` token in line 4 whilst the new tokenizer
-  returns it in line 3.
+* The output of the :func:`tokenize.tokenize` and :func:`tokenize.generate_tokens`
+  functions is now changed due to the changes introduced in :pep:`701`. This
+  means that ``STRING`` tokens are not emited anymore for f-strings and the
+  tokens described in :pep:`701` are now produced instead: ``FSTRING_START``,
+  ``FSRING_MIDDLE`` and ``FSTRING_END`` are now emited for f-string "string"
+  parts in addition to the the apropiate tokens for the tokenization in the
+  expression components. For example for the f-string ``f"start {1+1} end"``
+  the old version of the tokenizer emitted::
+
+    1,0-1,18:           STRING         'f"start {1+1} end"'
+
+  while the new version emits::
+
+      1,0-1,2:            FSTRING_START  'f"'
+      1,2-1,8:            FSTRING_MIDDLE 'start '
+      1,8-1,9:            OP             '{'
+      1,9-1,10:           NUMBER         '1'
+      1,10-1,11:          OP             '+'
+      1,11-1,12:          NUMBER         '1'
+      1,12-1,13:          OP             '}'
+      1,13-1,17:          FSTRING_MIDDLE ' end'
+      1,17-1,18:          FSTRING_END    '"'
+
+  Aditionally, final ``DEDENT`` tokens are now emited within the bounds of the
+  input. This means that for a file containing 3 lines, the old version of the
+  tokenizer returned a ``DEDENT`` token in line 4 whilst the new version returns
+  the token in line 3.
 
 Build Changes
 =============