WHAT’S NEW IN PYTHON 3.

11
YOU WILL LEARN ABOUT:
1. Changes to Python in the 3.11 release
2. How the 1.2x speed-up was achieved
3. New, more precise messaging in tracebacks
4. The addition of TOML to the standard library
5. New features with Exceptions and the new TaskGroup
mechanism

OVERVIEW
● Python 3.11 took a little longer than usual, but there is good stuff in it
● Big focus on performance improvement: 1.2x speed-up on average
● Further improvement to error messaging
● TOML has been added to the standard library (read-only)
● New features in Exceptions:
● ExceptionGroup
● Adding notes

OVERVIEW
● New mechanism in asyncio for creating tasks
● Context manager based
● Takes advantage of Exception groups
● Additions to the typing module:
● Self
● Variadic types
● Dataclass transform

NEXT UP...

Zoom zoom zoom

TABLE OF CONTENTS
1. Overview
2. Faster, Faster, Faster
3. Improved Traceback Messages
4. TOML
5. Exceptions & Task Groups
6. Typing Features
7. Odds and Ends
8. Summary and Further Investigation

FASTER CODE
● On-going Faster CPython Project
● Python 3.11 is 10-60% faster than Python 3.10
● Average 1.22x speed-up on standard benchmark suite

ADAPTIVE INTERPRETER
● PEP 659: Specializing Adaptive Interpreter
● Interpreter now adapts types or values for performance gains
● “Specializes code aggressively, but over a very small region”
● Byte-code gets adjusted after running several times
● LOAD_ATTR → LOAD_ATTR_ADAPTIVE
● LOAD_ATTR_INSTANCE_VALUE
● LOAD_ATTR_MODULE
● LOAD_ATTR_SLOT

ZERO COST EXCEPTIONS

● Reduce the cost of try blocks when there is no exception
● Inspired by similar feature in Java & C++
● Compiler generates a table of where to jump to in case of exceptions
● Almost zero cost when no exception
● Still a cost handling the exception
● May help speed of regular functions as the corresponding frame objects
can be smaller (previously had 240 bytes exception handling overhead)

FASTER STARTUP
● Python caches your bytecode in the __pycache__ directory
● At start-up it:
● Reads the cache
● Unmarshals the objects
● Allocates the code on the heap
● Evaluates the code
● Python 3.11 is freezing key core modules
● Code is statically allocated
● Can be directly loaded
● 10-15% improvement on interpreter loading times

BUT WAIT, THERE’S MORE

● Improvements to the function frame creation process
● Optimizations on:
● Recursive calls
● How ASCII is normalized to Unicode
● math module’s comb() and perm() functions
● Regular expressions

NEXT UP...

More error message improvements

TABLE OF CONTENTS
1. Overview
2. Faster, Faster, Faster
3. Improved Traceback Messages
4. TOML
5. Exceptions & Task Groups
6. Typing Features
7. Odds and Ends
8. Summary and Further Investigation

BETTER TRACEBACKS
● PEP 657: Include Fine-Grained Error Locations in Tracebacks
● Continuation of error message improvement from Python 3.10
● Traceback messages now have more detail as to where in the statement a
problem occurred

NEXT UP...

TOML in the standard library

TABLE OF CONTENTS
1. Overview
2. Faster, Faster, Faster
3. Improved Traceback Messages
4. TOML
5. Exceptions & Task Groups
6. Typing Features
7. Odds and Ends
8. Summary and Further Investigation

TOML
● TOML: Tom’s Obvious, Minimal Language
● Configuration file format
● Similar to Windows INI files
● Third-party libraries, now part of the standard library
● Being used by many packaging tools

DATA TYPES
● Comments
● Key-value pairs
● Arrays
● Tables
● Inline tables
● Integers, floats, booleans
● Dates, times

PYTHON’S TOML
● Commonly used by a lot of Python tools:
● black, mypy, pytest, tox, pylint, isort, ...
● Only supports binary files to ensure UTF-8 encoding
● Interface is consistent with json.load and pickle.load
● Python’s standard library implementation is read-only
● Third-party style-preserving TOML library: TOML Kit
https://github.com/sdispater/tomlkit

NEXT UP...

Exceptional exceptions
TABLE OF CONTENTS
1. Overview
2. Faster, Faster, Faster
3. Improved Traceback Messages
4. TOML
5. Exceptions & Task Groups
6. Typing Features
7. Odds and Ends
8. Summary and Further Investigation

EXCEPTION NOTES
● PEP 678: Enriching Exceptions with Notes
● Add information to an exception
● Especially helpful for re-raising

EXCEPTION GROUPS
● Exception handling code may itself cause an exception
● This can be hard to understand
● An ExceptionGroup is a way of grouping exceptions together
● The new except* syntax allows you to react to a subset of an
ExceptionGroup

TASK GROUPS
● The ascynio library has a new way of creating tasks
● TaskGroup is used as a context manager
● Uses an ExceptionGroup when problems happen in an asynchronous
task

COROUTINE PRIMER
● ascynio is an abstraction for parallel execution of code
● Uses the async and await keywords
● Tasks need to be created, then waited upon until completion

async def write_letters(letters):

abcABdeC

 COMPARISON
async def bootstrap():
tasks = [
asyncio.create_task(write_letters(string.ascii_lowercase)),
asyncio.create_task(write_letters(string.ascii_uppercase)),
]

await asyncio.gather(*tasks)

async def bootstrap():

async with asyncio.TaskGroup() as group:
New in
group.create_task(write_letters(string.ascii_lowercase))
3.11
group.create_task(write_letters(string.ascii_uppercase))

NEXT UP...

More types of typing

TABLE OF CONTENTS
1. Overview
2. Faster, Faster, Faster
3. Improved Traceback Messages
4. TOML
5. Exceptions & Task Groups
6. Typing Features
7. Odds and Ends
8. Summary and Further Investigation

SELF TYPE
● typing.Self indicates the use of a class’s own type
● Useful with factory methods
import json
from dataclasses import dataclass
from typing import Self

@dataclass
class Person:
name: str

pe
age: int

ty
y
M
@classmethod
def from_json(cls, data: str) -> Self:
p = json.loads(data)
return Person(p["name"], p["int"])

VARIADIC TYPE
● typing.TypeVar indicates parameterization with a single type
● typing.TypeVarTuple indicates parameterization with an arbitrary number of types

from typing import TypeVar, TypeVarTuple

gs
g

in
T = TypeVar('T')

in

Th
Th
Ts = TypeVarTuple('Ts')

y
e

an
On

M
def move_first_element_to_last(tup: tuple[T, *Ts]) -> tuple[*Ts, T]:
return (*tup[1:], tup[0])

DATACLASS TRANSFORM
● Many libraries use data class like semantics
● Django, DRF, Pydantic, Ninja...
● typing.datclass_transform indicates to type checkers that a class
should be treated as a data class
● Decorator for: functions, classes, metaclasses
● Example:
● A transformed ModelBase class indicates children are data-class-
like
● A CustomerModel(ModelBase) class with id and name
attributes
● Type checker expects the __init__ method to take id and name

EVEN MORE TYPING

● LiteralString matches if a string is only a literal in the code
literal = "a string"
not a literal ➡ sql_injection = "SELECT * FROM " + table

● Additions to TypedDict: Required[] and NotRequired[]

class Movie(TypedDict):
title: str
year: NotRequired[int]

NEXT UP...

Stuff, and things

TABLE OF CONTENTS
1. Overview
2. Faster, Faster, Faster
3. Improved Traceback Messages
4. TOML
5. Exceptions & Task Groups
6. Typing Features
7. Odds and Ends
8. Summary and Further Investigation

ISO DATE PARSING

● ISO Date format: Year-Month-Date Time
● datetime parsing function is now capable of handling more of the
specification

NEGATIVE ZERO
● Floating point numbers often aren’t what you expect
>>> 0.1 + 0.2
0.30000000000000004

● Representation of negative in floating point is independent of the number

● Negative zero
● New format in f-strings to handle the unexpected

 DEPRECATING DEAD BATTERIES

● Clean-up effort underway to remove old items from the standard library
● There are hundreds of modules in the standard library
● All of which need to be maintained
● PEP 594: Removing dead batteries from the standard library
● aifc ● crypt ● nis ● sunau
● audioop ● imghdr ● ossaudiodev ● telnetlib
● cgi ● mailcap ● pipes ● uu
● cgitb ● msilib ● sndhdr ● xdrlib
● chunk ● nntplib ● spwd

NEXT UP...

Should you?
TABLE OF CONTENTS
1. Overview
2. Faster, Faster, Faster
3. Improved Traceback Messages
4. TOML
5. Exceptions & Task Groups
6. Typing Features
7. Odds and Ends
8. Summary and Further Investigation

SUMMARY
● Python 3.11 has lots of new things
● Big focus on making Python faster
● Continued improvement on error messages, tracebacks
● Addition of TOML into the standard library
● Exception groups and notes
● New task creation mechanism for asyncio
● More additions to the typing module

SHOULD YOU UPGRADE?

● If you’re already on Python 3.10, the interpreter changes should be
seamless
● Performance gains are nice
● Improved error messaging can make a big difference
● Test all your things!

MORE ON: PYTHON 3.11 RELEASE

● Release notes:
https://docs.python.org/3.11/whatsnew/3.11.html
● PEG parser added in 3.9:
https://peps.python.org/pep-0617/

MORE ON: FASTER CODE EXECUTION

● Mark Shannon @ EuroPython 2022
https://www.youtube.com/watch?v=4ytEjwywUMM&t=9748s
● Brandt Bucher @ Talk Python
https://talkpython.fm/episodes/show/381/python-perf-specializing-adaptive-interpreter
● Guido and Mark @ Talk Python
https://talkpython.fm/episodes/show/339/making-python-faster-with-guido-and-mark

MORE ON: TRACEBACKS AND ERRORS

● Better Exceptions Library:
https://github.com/qix-/better-exceptions
● Friendly:
https://friendly-traceback.github.io/docs/

MORE ON: TOML

● TOML specification:
https://toml.io/en/
● tomli as a backport:
https://github.com/hukkin/tomli
● TOML Kit, for writing with style preservation:
https://github.com/sdispater/tomlkit

OTHER REALPYTHON CONTENT

● Python and TOML: New Best Friends:
https://realpython.com/python-toml/
● Async IO in Python: A Complete Walkthrough:
(tutorial) https://realpython.com/async-io-python/
(course) https://realpython.com/courses/python-3-concurrency-asyncio-module/
● Speed Up Your Python Program With Concurrency:
(tutorial) https://realpython.com/python-concurrency/
(course) https://realpython.com/courses/speed-python-concurrency/
● Python Type Checking (Guide):
(tutorial) https://realpython.com/python-type-checking/
(course) https://realpython.com/courses/python-type-checking/

Dankie ju faleminderit faleminderit ‫ شكرا‬Grazias Շնորհակալ