diff --git a/pep-0692.rst b/pep-0692.rst index 187ef1b553b..4fc44dff92b 100644 --- a/pep-0692.rst +++ b/pep-0692.rst @@ -2,13 +2,15 @@ PEP: 692 Title: Using TypedDict for more precise \*\*kwargs typing Author: Franek Magiera Sponsor: Jelle Zijlstra -Discussions-To: https://mail.python.org/archives/list/typing-sig@python.org/thread/U42MJE6QZYWPVIFHJIGIT7OE52ZGIQV3/ +Discussions-To: https://discuss.python.org/t/pep-692-using-typeddict-for-more-precise-kwargs-typing/17314 Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 29-May-2022 Python-Version: 3.12 Post-History: `29-May-2022 `__, + `12-Jul-2022 `__, + `12-Jul-2022 `__, Abstract @@ -20,7 +22,9 @@ be very limiting. Therefore, in this PEP we propose a new way to enable more precise ``**kwargs`` typing. The new approach revolves around using ``TypedDict`` to type ``**kwargs`` that comprise keyword arguments of different types. It also involves introducing a grammar change and a new dunder -``__unpack__``. +``__typing_unpack__``. + +.. _pep-692-motivation: Motivation ========== @@ -39,9 +43,31 @@ type annotating ``**kwargs`` is not possible. This is especially a problem for already existing codebases where the need of refactoring the code in order to introduce proper type annotations may be considered not worth the effort. This in turn prevents the project from getting all of the benefits that type hinting -can provide. As a consequence, there has been a `lot of discussion `__ -around supporting more precise ``**kwargs`` typing and it became a -feature that would be valuable for a large part of the Python community. +can provide. + +Moreover, ``**kwargs`` can be used to reduce the amount of code needed in +cases when there is a top-level function that is a part of a public API and it +calls a bunch of helper functions, all of which expect the same keyword +arguments. Unfortunately, if those helper functions were to use ``**kwargs``, +there is no way to properly type hint them if the keyword arguments they expect +are of different types. In addition, even if the keyword arguments are of the +same type, there is no way to check whether the function is being called with +keyword names that it actually expects. + +As described in the :ref:`Intended Usage ` section, +using ``**kwargs`` is not always the best tool for the job. Despite that, it is +still a widely used pattern. As a consequence, there has been a lot of +discussion around supporting more precise ``**kwargs`` typing and it became a +feature that would be valuable for a large part of the Python community. This +is best illustrated by the `mypy GitHub issue 4441 `__ which +contains a lot of real world cases that could benefit from this propsal. + +One more use case worth mentioning for which ``**kwargs`` are also convenient, +is when a function should accommodate optional keyword-only arguments that +don't have default values. A need for a pattern like that can arise when values +that are usually used as defaults to indicate no user input, such as ``None``, +can be passed in by a user and should result in a valid, non-default behavior. +For example, this issue `came up `__ in the popular ``httpx`` library. Rationale ========= @@ -123,8 +149,8 @@ Keyword collisions A ``TypedDict`` that is used to type ``**kwargs`` could potentially contain keys that are already defined in the function's signature. If the duplicate -name is a standard argument, an error should be reported by type checkers. -If the duplicate name is a positional only argument, no errors should be +name is a standard parameter, an error should be reported by type checkers. +If the duplicate name is a positional-only parameter, no errors should be generated. For example:: def foo(name, **kwargs: **Movie) -> None: ... # WRONG! "name" will @@ -132,7 +158,7 @@ generated. For example:: # first parameter. def foo(name, /, **kwargs: **Movie) -> None: ... # OK! "name" is a - # positional argument, + # positional parameter, # so **kwargs can contain # a "name" keyword. @@ -207,9 +233,9 @@ Continuing the previous example:: dest = src # OK! -It is worth pointing out that the destination function's arguments that are to +It is worth pointing out that the destination function's parameters that are to be compatible with the keys and values from the ``TypedDict`` must be keyword -only arguments:: +only:: def dest(animal: Dog, string: str, number: int = ...): ... dest(animal_instance, "some string") # OK! @@ -336,35 +362,26 @@ would not cause errors at runtime during function invocation. Otherwise, the type checker should generate an error. In cases similar to the ``bar`` function above the problem could be worked -around by explicitly dereferencing desired fields and using them as parameters +around by explicitly dereferencing desired fields and using them as arguments to perform the function call:: def bar(**kwargs: **Animal): name = kwargs["name"] takes_name(name) +.. _pep-692-intended-usage: + Intended Usage -------------- - -This proposal will bring a large benefit to the codebases that already use -``**kwargs`` because of the flexibility that they provided in the initial -phases of the development, but now are mature enough to use a stricter -contract via type hints. - -Adding type hints directly in the source code as opposed to the ``*.pyi`` -stubs benefits anyone who reads the code as it is easier to understand. Given -that currently precise ``**kwargs`` type hinting is impossible in that case the -choices are to either not type hint ``**kwargs`` at all, which isn't ideal, or -to refactor the function to use explicit keyword arguments, which often exceeds -the scope of time and effort allocated to adding type hinting and, as any code -change, introduces risk for both project maintainers and users. In that case -hinting ``**kwargs`` using a ``TypedDict`` as described in this PEP will not -require refactoring and function body and function invocations could be -appropriately type checked. - -Another useful pattern that justifies using and typing ``**kwargs`` as proposed -is when the function's API should allow for optional keyword arguments that -don't have default values. +The intended use cases for this proposal are described in the +:ref:`pep-692-motivation` section. In summary, more precise ``**kwargs`` typing +can bring benefits to already existing codebases that decided to use +``**kwargs`` initially, but now are mature enough to use a stricter contract +via type hints. Using ``**kwargs`` can also help in reducing code duplication +and the amount of copy-pasting needed when there is a bunch of functions that +require the same set of keyword arguments. Finally, ``**kwargs`` are useful for +cases when a function needs to facilitate optional keyword arguments that don't +have obvious default values. However, it has to be pointed out that in some cases there are better tools for the job than using ``TypedDict`` to type ``**kwargs`` as proposed in this @@ -377,9 +394,9 @@ than using ``**kwargs`` and a ``TypedDict``:: Similarly, when type hinting third party libraries via stubs it is again better to state the function signature explicitly - this is the only way to type such -a function if it has default parameters. Another issue that may arise in this +a function if it has default arguments. Another issue that may arise in this case when trying to type hint the function with a ``TypedDict`` is that some -standard function arguments may be treated as keyword only:: +standard function parameters may be treated as keyword only:: def foo(name, year): ... # Function in a third party library. @@ -397,6 +414,9 @@ explicitly as:: def foo(name: str, year: int): ... +Also, for the benefit of IDEs and documentation pages, functions that are part +of the public API should prefer explicit keyword parameters whenever possible. + Grammar Changes =============== @@ -420,7 +440,7 @@ After: | '**' param_no_default param_no_default_double_star_annotation: - | param_double_star_annotation & ')' + | param_double_star_annotation ','? &')' param_double_star_annotation: NAME double_star_annotation @@ -463,13 +483,15 @@ previous example:: >>> def foo(**kwargs: **Movie): ... ... >>> foo.__annotations__ - {'kwargs': **Movie} + {'kwargs': Unpack[Movie]} -The double asterisk syntax should call the ``__unpack__`` special method on -the object it was used on. This means that ``def foo(**kwargs: **T): ...`` is -equivalent to ``def foo(**kwargs: T.__unpack__()): ...``. In addition, -``**Movie`` in the example above is the ``repr`` of the object that -``__unpack__()`` returns. +To accomplish this, we propose a new dunder called ``__typing_unpack__``. +The double asterisk syntax should result in a call to the ``__typing_unpack__`` +special method on an object it was used on. This means that at runtime, +``def foo(**kwargs: **T): ...`` is equivalent to +``def foo(**kwargs: type(T).__typing_unpack__(T)): ...``. +``TypedDict`` is the only type in the standard library that is expected to +implement ``__typing_unpack__``, which should return ``Unpack[self]``. Backwards Compatibility ----------------------- @@ -558,6 +580,7 @@ overloaded:: References ========== +.. _httpxIssue1384: https://github.com/encode/httpx/issues/1384 .. _mypyIssue4441: https://github.com/python/mypy/issues/4441 .. _mypyPull10576: https://github.com/python/mypy/pull/10576 .. _mypyExtensionsPull22: https://github.com/python/mypy_extensions/pull/22/files