From 3fa3e48cebe009363f5635a83c4de1e74472f8a7 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Sun, 5 Mar 2017 02:01:45 +0100 Subject: [PATCH 01/37] Collect various ideas --- pep-05xx.txt | 554 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 554 insertions(+) create mode 100644 pep-05xx.txt diff --git a/pep-05xx.txt b/pep-05xx.txt new file mode 100644 index 00000000000..a6961802f58 --- /dev/null +++ b/pep-05xx.txt @@ -0,0 +1,554 @@ +PEP: 5xx +Title: Protocols +Version: $Revision$ +Last-Modified: $Date$ +Author: Ivan Levkivskyi , + Jukka Lehtosalo , + Łukasz Langa +Discussions-To: Python-Dev +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 05-Mar-2017 +Python-Version: 3.7 + + +Abstract +======== + +This PEP describes the static and runtime behavior of protocol classes. + + +Rationale and Goals +=================== + +Currently, typing defines ABCs for several common Python protocols such as +Iterable and Sized. The problem with them is that a class has to be +explicitly marked to support them, which is arguably unpythonic and unlike +what you'd normally do in your non statically typed code. +For example, this conforms to the current PEP:: + + from typing import Sized, Iterable, Iterator + + class Bucket(Sized, Iterable[int]): + ... + def __len__(self) -> int: ... + def __iter__(self) -> Iterator[int]: ... + +My intention is that the above code could be written instead equivalently +without explicit base classes in the class definition, and Bucket would +still be implicitly considered a subtype of both Sized and Iterable[int] +by using structural subtyping:: + + from typing import Iterator + + class Bucket: + ... + def __len__(self) -> int: ... + def __iter__(self) -> Iterator[int]: ... + + +Non-Goals +--------- + +Runtime protocol classes are simple ABCs. Moreover as PEP 484 and PEP 526 +stated they are **completely optional**. + + +Nominal vs structural subtyping +=============================== + + +Existing approaches to structural subtyping +=========================================== + +* zope.interace +* TypeScript +* Go - everything is an interface +* ABCs +* collections.abc (with __subclasshook__) + + +Specification +============= + +Terminology +----------- + +Let's call them protocols. The reason is that the term iterator protocol, +for example, is widely understood in the community, and coming up with +a new term for this concept in a statically typed context would just create confusion. + +This has the drawback that the term 'protocol' becomes overloaded with +two subtly different meanings: the first is the traditional, well-known but +slightly fuzzy concept of protocols such as iterable; the second is the more +explicitly defined concept of protocols in statically typed code +(or more generally in code that just uses the typing module). +I argue that the distinction isn't important most of the time, and in other +cases people can just add a qualifier such as "protocol classes" +(for the new-style protocols) or "traditional/non-class/implicit protocols". + +Some terminology could be useful here for clarity. If a class includes a protocol +in its MRO, the class is an (explicit) subclass of the procotol. +If a class ia a structural subtype of a protocol, it is said to implement the protocol +and to be compatible with a protocol. +If a class is compatible with a protocol but the protocol is not included in the MRO, +the class is an implicit subclass of the protocol. + + +Defining a Protocol +------------------- + +There would be a new class typing.Protocol. If this is explicitly included +in the the base class list, the class is a protocol. Here is a simple example: + +from typing import Protocol + +class SupportsClose(Protocol): + def close(self) -> None: ... # See 3) for more about the '...'. + +Now if we define a class Resource with a close method that has a suitable signature, +it would implicitly be a subtype of SupportsClose, since we'd use structural +subtyping for protocol types: + +class Resource: + ... + + def close(self) -> None: + self.file.close() + self.lock.release() + +Protocol types can be used in annotations, of course, and for type checking:: + + def close_all(things: Iterable[SupportsClose]) -> None: + for t in things: + t.close() + + f = open('foo.txt') + r = Resource(...) + close_all([f, r]) # OK! + close_all([1]) # Error: 'int' has no 'close' method + +Note that both the user-defined class Resource and the IO type +(the return type of open) would be considered subtypes of SupportsClose +because they provide a suitable close method. + +If using the current typing module, our only option to implement +the above example would be to use an ABC (or type Any, but that would +compromise type checking). If we'd use an ABC, we'd have to explicitly +register the fact that these types are related, and this generally difficult +to do with library types as the type objects may be hidden deep in +the implementation of the library. Besides, this would be uglier than how +you'd actually write the code in straight, idiomatic dynamically typed Python. +The code with a protocol class matches common Python conventions much better. +It's also automatically extensible and works with additional, unrelated classes +that happen to implement the required interface. + + +Protocol members +---------------- + +I propose that most of the regular rules for classes still apply to +protocol classes (modulo a few exceptions that only apply to protocol classes). +I'd like protocols to also be ABCs, so all of these would be valid within +a protocol class:: + + # Variant 1 + def __len__(self) -> int: ... + + # Variant 2 + def __len__(self) -> int: pass + + # Variant 3 + @abstractmethod + def __len__(self) -> int: pass + + # Variant 4 + def __len__(self): pass + + # Variant 5 + def __len__(self) -> int: + return 0 + + # Variant 6 + def __len__(self) -> int: + raise NotImplementedError + +For variants 1, 2 and 3, a type checker should probably always require an explicit +implementation to be defined in a subclass that explicitly subclasses the protocol +(see below for more about this), because the implementations return None which +is not a valid return type for the method. For variants 4, 5 and 6, we can use +the provided implementation as a default implementation. +The default implementations won't be used if the subtype relationship is implicit +and only via structural subtyping -- the semantics of inheritance won't be changed. + +I also propose that a ... as the method body in a protocol type makes the method +implicitly abstract. This would only be checked statically, and there won't be any +runtime checking. The idea here is that most methods in a protocol will be abstract, +and having to always use @abstractmethod is a little verbose and ugly, and has +the issue of implicit None return types confusing things. +This would be the recommended way of defining methods in a protocol that +don't have an implementation, but the other approaches can be used for +legacy code or if people just feel like it. The recommended syntax would mirror +how methods are defined in stub files. + +Similar to methods, there will be multiple valid ways of defining data attributes +(or properties). All of these will be valid:: + + class Foo(Protocol): + a = ... # type: int # Variant 1 + b = 0 # Variant 2 + + # Variant 3 + @property + def c(self) -> int: ... + + # Variant 4 + @property + def c(self) -> int: + return 0 + + # Variant 5 + @property + def d(self) -> int: + raise NotImplementedError + + # Variant 6 + @abstractproperty + def e(self) -> int: ... + + # Variant 7 + @abstractproperty + def f(self) -> int: pass + +Also, properties with setters can also be defined. The first three variants +would be the recommended ways of defining attributes or properties, +but similar to 3), the others are possible and may be useful for supporting legacy code. + +When using an ... initializer, @abstractproperty or pass/... as property body +(and when the type does not include None), the data attribute is considered +abstract and must always be explicitly implemented in a compatible class. + +Attributes should not be defined in the body of a method by assignment via self. +This restriction is a little arbitrary, but my idea is that the protocol class +implementation is often not shared by subtypes so the interface +should not depend on the default implementation. +This is more of a style than a technical issue, as a type checker could infer +attributes from assignment statements within methods as well. + +When using the ... initializer, the ... initializer might leak into subclasses +at runtime, which is unfortunate:: + + class A(Protocol): + x = ... # type: int + + class B(A): + def __init__(self) -> None: + self.x = 1 + + b = B() + print(b.x) # 1 + print(B.x) # Ellipsis + +If we'd use a None initializer things wouldn't be any better. Maybe we can modify +the metaclass to recognize ... initializers and translate them to something else. +This needs to be documented, however. Also, in this proposal there is no way to +distinguish between class and instance data attributes. + +I'm not sure sure what to do with __init__. I guess a protocol could provide +a default implementation that could be used in explicit subclasses. + +Overall, I'm probably the least happy about this part of the proposal. + + +Explicitly declaring implementation +----------------------------------- + +when a class explicitelly inherits from protocol, +typechecker verifies that everithing is OK. +in this case normal methods are also inherited. + +when a protocol is defined, every method is resolved using normal mro procedure. +typechecker verifies that everythong is OK. +if concrete overrides abstact, then thats it -- it is nore more abstract +the same if vice versa (?) +set of all remaining abstracts defines the protocol + +I propose that protocols can be used as regular base classes. +I can see at least three things that support this decision. +First, a protocol class could define default implementations for some methods +(typing.Sequence would be an example if we decide to turn it into a protocol). +Second, we want a way of statically enforcing that a class actually implements +a protocol correctly (but there are other ways to achieve this effect +-- see below for alternatives). Third, this makes it possible to turn +an existing ABC into a protocol and just have things (mostly) work. +This would be important for the existing ABCs in typing what we may want +to change into protocols (see point 8 for more about this). +The general philosophy would be that Protocols are mostly like regular ABCs, +but a static type checker will handle them somewhat specially. + +Note that subclassing a protocol class would not turn the subclass into +a protocol unless it also has Protocol as an explicit base class. +I assume that we can use metaclass trickery to get this to work correctly. + +We could also explicitly add an assignment for checking +that a class implements a protocol. +I've seen a similar pattern in some Go code that I've reviewed. Example:: + + class A: + def __len__(self) -> float: + return ... + + _ = A() # type: Sized # Error: A.__len__ doesn't conform to 'Sized' + # (Incompatible return type 'float') + +I don't much care above the above example, as it moves the check away from +the class definition and it almost requires a comment as otherwise +the code probably wouldn't make any sense to an average reader +-- it looks like dead code. Besides, in the simplest form it requires us +to construct an instance of A which could problematic if this requires accessing +or allocating some resources such as files or sockets. +We could work around the latter by using a cast, for example, +but then the code would be really ugly. + + +Merging and extending protocols +------------------------------- + +I think that we should support subprotocols. A subprotocol can be defined +by having both one or more protocols as the explicit base classes and also +having typing.Protocol as an immediate base class:: + + from typing import Sized, Protocol + + class SizedAndCloseable(Sized, Protocol): + def close(self) -> None: ... + +Now the protocol SizedAndCloseable is a protocol with two methods, +__len__ and close. Alternatively, we could have implemented it like this, +assuming the existence of SupportsClose from an earlier example:: + + from typing import Sized + + class SupportsClose(...): ... # Like above + + class SizedAndCloseable(Sized, SupportsClose, Protocol): + pass + +The two definitions of SizedAndClosable would be equivalent. +Subclass relationships between protocols aren't meaningful when +considering subtyping, as we only use structural compatibility +as the criterion, not the MRO. + +If we omit Protocol in the base class list, this would be regular +(non-protocol) class that must implement Sized. If Protocol +is included in the base class list, all the other base classes +must be protocols. A protocol can't extend a regular class. + + +Generic and recursive Protocols +------------------------------- + +Protocol[T] as a shorthand for Protocol, Generic[T] + +Generic protocol are important. For example, SupportsAbs, Iterable and +Iterator would be generic. We could define them like this, +similar to generic ABCs:: + + T = TypeVar('T', covariant=True) + + class Iterable(Protocol[T]): + def __iter__(self) -> Iterator[T]: ... + +Should we support recursive protocols? + +Sure, why not. Just use the forward references. They might useful for +representing self-referential +data structures like trees in an abstract fashion, but I don't see them +used commonly in production code. + + +Interactions of Protocols +========================= + +cannot be instantiated. + +Subtyping relationships with other types +---------------------------------------- + +Subtyping: issubtype(A, Proto) is _always_ structural; +issubtype(Proto, Nom) always False. + + +All(ala Intersection) and Union +------------------------------- + +Union merges protocols: +open() -> Union[int, str] conforms to Union[Proto.open() -> int, Proto.open() -> str], +although it is not a subclass of any of two. + + +isinstance() and narrowing types +-------------------------------- + +isinstance(x, Proto) narrows type if defined with @auto_runtime; isinstance(Proto[int]) always fails + +We shouldn't implement any magic isinstance machinery, as performing a runtime +compatibility check is generally difficult: we might want to verify argument +counts to methods, names of arguments and even argument types, +depending the kind of protocol we are talking about, +but sometimes we wouldn't care about these, or we'd only care about some of these things. + +My preferred semantics would be to make isinstance fail by default for protocol types. +This would be in the spirit of duck typing -- protocols basically would be used to model +duck typing statically, not explicitly at runtime. + +However, it should be possible for protocol types to implement custom isinstance behavior +when this would make sense, similar to how Iterable and other ABCs in collections.abc +and typing already do it, but this should be specific to these particular classes. +We need this fallback option anyway for backward compatibility. + + +Type[...] with protocols +------------------------ + +Type[...] accepts only non-abstract (non-protocol?) classes + + +Runtime behavior of protocol classes +==================================== + +At runtime normal ABCs, cannot be instantiated. + + +@auto_runtime decorator (or make it always??) +----------------------- + +(like in collections.abc), typechecker ensures no empty bodies + + +Changes in typing module +------------------------ + +I think that at least these classes in typing should be protocols: +(probably all abstract change, concrete stay) + +* Sized +* Container +* Iterable +* Iterator +* Reversible +* SupportsAbs (and other Supports* classes) + +These classes are small and conceptually simple. It's easy to see which of +these protocols a class implements from the presence of a small number +of magic methods, which immediately suggest a protocol. + +I'm not sure about other classes such as Sequence, Set and IO. I believe +that these are sufficiently complex that it makes sense to require code +to be explicit about them, as there would not be any sufficiently obvious +and small set of 'marker' methods that tell that a class implements this protocol. +Also, it's too easy to leave some part of the protocol/interface unimplemented +by accident, and explicitly marking the subclass relationship allows type checkers +to pinpoint the missing implementations -- or they can be inherited from the ABC, +in case that is has default implementations. +So I'd currently vote against making these classes protocols. + + +Introspection +------------- + +Current typing is already good. + +The existing introspection machinery (dir, etc.) could be used with protocols, +but typing would not include an implementation of additional introspection +or runtime type checking capabilities for protocols. + +As all attributes need to be defined in the class body based on this proposal, +protocol classes would have better support for introspection than regular classes +where attributes can be defined implicitly -- protocol attributes can't be initialized +in ways that are not visible to introspection (using setattr, assignment via self, etc.). +Still, some things likes types of attributes wouldn't be visible at runtime, +so this would necessarily be somewhat limited. + + +Implementation details +---------------------- + +We'd need to implement at least the following things: + +* Define class Protocol (this could be simple, and would be similar to Generic). +* Implement metaclass functionality to detect whether a class is a protocol or not. +Maybe add a class attribute such as __protocol__ = True if that's the case. +Verify that a protocol class only has protocol base classes in the MRO (except for object)???. +* Optionally, override isinstance. +* Optionally, translate ... class attribute values to something else (properties?). + + +Postponed ideas +=============== + +Should we support optional attributes? +-------------------------------------- + +We can come up with examples where it would be handy to be able to say +that a method or data attribute does not need to be present in a class +implementing a protocol, but if it's present, it must conform to a specific +signature or type. One could use a hasattr check to determine whether +they can use the attribute on a particular instance. + +In the interest of simplicity, let's not support optional methods +or attributes. We can always revisit this later if there is an actual need. +The current realistic potential use cases for protocols that I've seen don't +require these. However, other languages have similar features and apparently +they are pretty commonly used. If I remember correctly, at least TypeScript +and Objective-C support a similar concept. + + +Should these be interoperable with other similar implementations? +----------------------------------------------------------------- + +The protocols as described here are basically a small extension to the existing +concept of ABCs. I argue that this is the way they should be understood, +instead of as something that replaces Zope interfaces, for example. + + +Should every class be a protocol by default? +-------------------------------------------- + +Some languages such as Go make structural subtyping the only or the primary +form of subtyping. We could achieve a similar result by making all classes protocols +by default (or even always). I argue that this would be a bad idea and classes should +need to be explicitly marked as protocols, as shown in my proposal above. + +* Protocols don't have some properties of regular classes. In particular, +isinstance is not well-defined for protocols, whereas it's well-defined +(and pretty commonly used) for regular classes. + +* Protocol classes should generally not have (many) method implementations, +as they describe an interface, not an implementation. +Most classes have many implementations, making them bad protocol classes. + +* Experience suggests that most classes aren't practical as protocols anyway, +mainly because their interfaces are too large, complex or implementation-oriented +(for example, they may include de facto private attributes and methods without a __ prefix). +Most actually useful protocols in existing Python code seem to be implicit. +The ABCs in typing and collections.abc are a kind-of exception, but even they are pretty +recent additions to Python and most programmers do not use them yet. + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From 51b0ca0c4ecc52aa237e3669a7b45a60c7a7bbe9 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Sun, 5 Mar 2017 11:08:11 +0100 Subject: [PATCH 02/37] Some formatting and reordering --- pep-05xx.txt | 378 +++++++++++++++++++++++++++------------------------ 1 file changed, 203 insertions(+), 175 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index a6961802f58..b959d73d27c 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -23,10 +23,10 @@ Rationale and Goals =================== Currently, typing defines ABCs for several common Python protocols such as -Iterable and Sized. The problem with them is that a class has to be +``Iterable`` and ``Sized``. The problem with them is that a class has to be explicitly marked to support them, which is arguably unpythonic and unlike what you'd normally do in your non statically typed code. -For example, this conforms to the current PEP:: +For example, this conforms to the PEP 484:: from typing import Sized, Iterable, Iterator @@ -36,9 +36,9 @@ For example, this conforms to the current PEP:: def __iter__(self) -> Iterator[int]: ... My intention is that the above code could be written instead equivalently -without explicit base classes in the class definition, and Bucket would -still be implicitly considered a subtype of both Sized and Iterable[int] -by using structural subtyping:: +without explicit base classes in the class definition, and ``Bucket`` would +still be implicitly considered a subtype of both ``Sized`` and +``Iterable[int]`` by using structural subtyping:: from typing import Iterator @@ -62,11 +62,11 @@ Nominal vs structural subtyping Existing approaches to structural subtyping =========================================== -* zope.interace +* ``zope.interace`` * TypeScript * Go - everything is an interface * ABCs -* collections.abc (with __subclasshook__) +* ``collections.abc`` (with ``__subclasshook__``) Specification @@ -77,7 +77,8 @@ Terminology Let's call them protocols. The reason is that the term iterator protocol, for example, is widely understood in the community, and coming up with -a new term for this concept in a statically typed context would just create confusion. +a new term for this concept in a statically typed context would just create +confusion. This has the drawback that the term 'protocol' becomes overloaded with two subtly different meanings: the first is the traditional, well-known but @@ -88,35 +89,38 @@ I argue that the distinction isn't important most of the time, and in other cases people can just add a qualifier such as "protocol classes" (for the new-style protocols) or "traditional/non-class/implicit protocols". -Some terminology could be useful here for clarity. If a class includes a protocol -in its MRO, the class is an (explicit) subclass of the procotol. -If a class ia a structural subtype of a protocol, it is said to implement the protocol -and to be compatible with a protocol. -If a class is compatible with a protocol but the protocol is not included in the MRO, -the class is an implicit subclass of the protocol. +Some terminology could be useful here for clarity. If a class includes a +protocol in its MRO, the class is an (explicit) subclass of the protocol. +If a class ia a structural subtype of a protocol, it is said to implement +the protocol and to be compatible with a protocol. +If a class is compatible with a protocol but the protocol is not included +in the MRO, the class is an implicit subclass of the protocol. + +Protocol members. Defining a Protocol ------------------- -There would be a new class typing.Protocol. If this is explicitly included -in the the base class list, the class is a protocol. Here is a simple example: +There will be a new class ``typing.Protocol``. If this is explicitly included +in the the base class list, the class is a protocol. +Here is a simple example:: -from typing import Protocol + from typing import Protocol -class SupportsClose(Protocol): - def close(self) -> None: ... # See 3) for more about the '...'. + class SupportsClose(Protocol): + def close(self) -> None: ... -Now if we define a class Resource with a close method that has a suitable signature, -it would implicitly be a subtype of SupportsClose, since we'd use structural -subtyping for protocol types: +Now if we define a class ``Resource`` with a close method that has +a suitable signature, it would implicitly be a subtype of ``SupportsClose``, +since we'd use structural subtyping for protocol types:: -class Resource: - ... + class Resource: + ... - def close(self) -> None: - self.file.close() - self.lock.release() + def close(self) -> None: + self.file.close() + self.lock.release() Protocol types can be used in annotations, of course, and for type checking:: @@ -129,29 +133,29 @@ Protocol types can be used in annotations, of course, and for type checking:: close_all([f, r]) # OK! close_all([1]) # Error: 'int' has no 'close' method -Note that both the user-defined class Resource and the IO type -(the return type of open) would be considered subtypes of SupportsClose -because they provide a suitable close method. +Note that both the user-defined class ``Resource`` and the ``IO`` type +(the return type of ``open()``) would be considered subtypes of +``SupportsClose`` because they provide a suitable close method. -If using the current typing module, our only option to implement -the above example would be to use an ABC (or type Any, but that would +If using the current ``typing`` module, our only option to implement +the above example would be to use an ABC (or type ``Any``, but that would compromise type checking). If we'd use an ABC, we'd have to explicitly register the fact that these types are related, and this generally difficult to do with library types as the type objects may be hidden deep in the implementation of the library. Besides, this would be uglier than how you'd actually write the code in straight, idiomatic dynamically typed Python. The code with a protocol class matches common Python conventions much better. -It's also automatically extensible and works with additional, unrelated classes -that happen to implement the required interface. +It's also automatically extensible and works with additional, +unrelated classes that happen to implement the required interface. Protocol members ---------------- I propose that most of the regular rules for classes still apply to -protocol classes (modulo a few exceptions that only apply to protocol classes). -I'd like protocols to also be ABCs, so all of these would be valid within -a protocol class:: +protocol classes (modulo a few exceptions that only apply to +protocol classes). I'd like protocols to also be ABCs, so all of these +would be valid within a protocol class:: # Variant 1 def __len__(self) -> int: ... @@ -174,26 +178,29 @@ a protocol class:: def __len__(self) -> int: raise NotImplementedError -For variants 1, 2 and 3, a type checker should probably always require an explicit -implementation to be defined in a subclass that explicitly subclasses the protocol -(see below for more about this), because the implementations return None which -is not a valid return type for the method. For variants 4, 5 and 6, we can use +For variants 1, 2 and 3, a type checker should probably always require +an explicit implementation to be defined in a subclass that explicitly +subclasses the protocol (see below for more about this), because +the implementations return ``None`` which is not a valid return type +for the method. For variants 4, 5 and 6, we can use the provided implementation as a default implementation. -The default implementations won't be used if the subtype relationship is implicit -and only via structural subtyping -- the semantics of inheritance won't be changed. - -I also propose that a ... as the method body in a protocol type makes the method -implicitly abstract. This would only be checked statically, and there won't be any -runtime checking. The idea here is that most methods in a protocol will be abstract, -and having to always use @abstractmethod is a little verbose and ugly, and has -the issue of implicit None return types confusing things. +The default implementations won't be used if the subtype relationship +is implicit and only via structural subtyping -- the semantics of inheritance +won't be changed. + +I also propose that a ``...`` as the method body in a protocol type makes +the method implicitly abstract. This would only be checked statically, +and there won't be any runtime checking. The idea here is that most methods +in a protocol will be abstract, and having to always use ``@abstractmethod`` +is a little verbose and ugly, and has the issue of implicit ``None`` +return types confusing things. This would be the recommended way of defining methods in a protocol that don't have an implementation, but the other approaches can be used for -legacy code or if people just feel like it. The recommended syntax would mirror -how methods are defined in stub files. +legacy code or if people just feel like it. The recommended syntax would +mirror how methods are defined in stub files. -Similar to methods, there will be multiple valid ways of defining data attributes -(or properties). All of these will be valid:: +Similar to methods, there will be multiple valid ways of defining data +attributes (or properties). All of these will be valid:: class Foo(Protocol): a = ... # type: int # Variant 1 @@ -223,21 +230,23 @@ Similar to methods, there will be multiple valid ways of defining data attribute Also, properties with setters can also be defined. The first three variants would be the recommended ways of defining attributes or properties, -but similar to 3), the others are possible and may be useful for supporting legacy code. - -When using an ... initializer, @abstractproperty or pass/... as property body -(and when the type does not include None), the data attribute is considered -abstract and must always be explicitly implemented in a compatible class. - -Attributes should not be defined in the body of a method by assignment via self. -This restriction is a little arbitrary, but my idea is that the protocol class -implementation is often not shared by subtypes so the interface -should not depend on the default implementation. +but similar to 3), the others are possible and may be useful for supporting +legacy code. + +When using an ... initializer, ``@abstractproperty`` or ``pass``/``...`` +as property body (and when the type does not include ``None``), the data +attribute is considered abstract and must always be explicitly implemented +in a compatible class. + +Attributes should not be defined in the body of a method by assignment +via ``self``. This restriction is a little arbitrary, but my idea is that +the protocol class implementation is often not shared by subtypes so +the interface should not depend on the default implementation. This is more of a style than a technical issue, as a type checker could infer attributes from assignment statements within methods as well. -When using the ... initializer, the ... initializer might leak into subclasses -at runtime, which is unfortunate:: +When using the ``...`` initializer, the ``... `` initializer might +leak into subclasses at runtime, which is unfortunate:: class A(Protocol): x = ... # type: int @@ -250,13 +259,14 @@ at runtime, which is unfortunate:: print(b.x) # 1 print(B.x) # Ellipsis -If we'd use a None initializer things wouldn't be any better. Maybe we can modify -the metaclass to recognize ... initializers and translate them to something else. -This needs to be documented, however. Also, in this proposal there is no way to -distinguish between class and instance data attributes. +If we'd use a None initializer things wouldn't be any better. +Maybe we can modify the metaclass to recognize ``...`` initializers and +translate them to something else. This needs to be documented, however. +Also, in this proposal there is no way to distinguish between class +and instance data attributes (ClassVar PEP 526). -I'm not sure sure what to do with __init__. I guess a protocol could provide -a default implementation that could be used in explicit subclasses. +I'm not sure sure what to do with ``__init__``. I guess a protocol could +provide a default implementation that could be used in explicit subclasses. Overall, I'm probably the least happy about this part of the proposal. @@ -264,24 +274,25 @@ Overall, I'm probably the least happy about this part of the proposal. Explicitly declaring implementation ----------------------------------- -when a class explicitelly inherits from protocol, -typechecker verifies that everithing is OK. +When a class explicitly inherits from protocol, +typechecker verifies that everything is OK. in this case normal methods are also inherited. -when a protocol is defined, every method is resolved using normal mro procedure. -typechecker verifies that everythong is OK. -if concrete overrides abstact, then thats it -- it is nore more abstract -the same if vice versa (?) +when a protocol is defined, every method is resolved using +normal MRO procedure. typechecker verifies that everything is OK. +if concrete overrides abstract, then thats it -- it is no more abstract +but not vice versa set of all remaining abstracts defines the protocol I propose that protocols can be used as regular base classes. I can see at least three things that support this decision. First, a protocol class could define default implementations for some methods -(typing.Sequence would be an example if we decide to turn it into a protocol). -Second, we want a way of statically enforcing that a class actually implements -a protocol correctly (but there are other ways to achieve this effect --- see below for alternatives). Third, this makes it possible to turn -an existing ABC into a protocol and just have things (mostly) work. +(``typing.Sequence`` would be an example if we decide to turn it +into a protocol). Second, we want a way of statically enforcing that a class +actually implements a protocol correctly (but there are other ways +to achieve this effect -- see below for alternatives). +Third, this makes it possible to turn an existing ABC into a protocol and +just have things (mostly) work. This would be important for the existing ABCs in typing what we may want to change into protocols (see point 8 for more about this). The general philosophy would be that Protocols are mostly like regular ABCs, @@ -291,26 +302,6 @@ Note that subclassing a protocol class would not turn the subclass into a protocol unless it also has Protocol as an explicit base class. I assume that we can use metaclass trickery to get this to work correctly. -We could also explicitly add an assignment for checking -that a class implements a protocol. -I've seen a similar pattern in some Go code that I've reviewed. Example:: - - class A: - def __len__(self) -> float: - return ... - - _ = A() # type: Sized # Error: A.__len__ doesn't conform to 'Sized' - # (Incompatible return type 'float') - -I don't much care above the above example, as it moves the check away from -the class definition and it almost requires a comment as otherwise -the code probably wouldn't make any sense to an average reader --- it looks like dead code. Besides, in the simplest form it requires us -to construct an instance of A which could problematic if this requires accessing -or allocating some resources such as files or sockets. -We could work around the latter by using a cast, for example, -but then the code would be really ugly. - Merging and extending protocols ------------------------------- @@ -324,9 +315,10 @@ having typing.Protocol as an immediate base class:: class SizedAndCloseable(Sized, Protocol): def close(self) -> None: ... -Now the protocol SizedAndCloseable is a protocol with two methods, -__len__ and close. Alternatively, we could have implemented it like this, -assuming the existence of SupportsClose from an earlier example:: +Now the protocol ``SizedAndCloseable`` is a protocol with two methods, +``__len__`` and ``close``. Alternatively, we could have implemented it +like this, assuming the existence of ``SupportsClose`` +from an earlier example:: from typing import Sized @@ -335,13 +327,13 @@ assuming the existence of SupportsClose from an earlier example:: class SizedAndCloseable(Sized, SupportsClose, Protocol): pass -The two definitions of SizedAndClosable would be equivalent. +The two definitions of ``SizedAndClosable`` would be equivalent. Subclass relationships between protocols aren't meaningful when considering subtyping, as we only use structural compatibility as the criterion, not the MRO. -If we omit Protocol in the base class list, this would be regular -(non-protocol) class that must implement Sized. If Protocol +If we omit ``Protocol`` in the base class list, this would be regular +(non-protocol) class that must implement ``Sized``. If ``Protocol`` is included in the base class list, all the other base classes must be protocols. A protocol can't extend a regular class. @@ -349,10 +341,10 @@ must be protocols. A protocol can't extend a regular class. Generic and recursive Protocols ------------------------------- -Protocol[T] as a shorthand for Protocol, Generic[T] +``Protocol[T, S, ...]`` as a shorthand for ``Protocol, Generic[T, S, ...]`` -Generic protocol are important. For example, SupportsAbs, Iterable and -Iterator would be generic. We could define them like this, +Generic protocols are important. For example, ``SupportsAbs``, ``Iterable`` +and ``Iterator`` would be generic. We could define them like this, similar to generic ABCs:: T = TypeVar('T', covariant=True) @@ -361,7 +353,6 @@ similar to generic ABCs:: def __iter__(self) -> Iterator[T]: ... Should we support recursive protocols? - Sure, why not. Just use the forward references. They might useful for representing self-referential data structures like trees in an abstract fashion, but I don't see them @@ -384,35 +375,41 @@ All(ala Intersection) and Union ------------------------------- Union merges protocols: -open() -> Union[int, str] conforms to Union[Proto.open() -> int, Proto.open() -> str], +open() -> Union[int, str] conforms to +Union[Proto.open() -> int, Proto.open() -> str], although it is not a subclass of any of two. -isinstance() and narrowing types --------------------------------- +``isinstance()`` and narrowing types +------------------------------------ -isinstance(x, Proto) narrows type if defined with @auto_runtime; isinstance(Proto[int]) always fails +``isinstance(x, Proto)`` and ``issublclass(C, Proto)`` +narrow type if defined with ``@auto_runtime``; ``isinstance(x, Proto[int])`` +always fails. -We shouldn't implement any magic isinstance machinery, as performing a runtime -compatibility check is generally difficult: we might want to verify argument -counts to methods, names of arguments and even argument types, +We shouldn't implement any magic ``isinstance()`` machinery, as performing +a runtime compatibility check is generally difficult: we might want to verify +argument counts to methods, names of arguments and even argument types, depending the kind of protocol we are talking about, -but sometimes we wouldn't care about these, or we'd only care about some of these things. - -My preferred semantics would be to make isinstance fail by default for protocol types. -This would be in the spirit of duck typing -- protocols basically would be used to model -duck typing statically, not explicitly at runtime. - -However, it should be possible for protocol types to implement custom isinstance behavior -when this would make sense, similar to how Iterable and other ABCs in collections.abc -and typing already do it, but this should be specific to these particular classes. +but sometimes we wouldn't care about these, or we'd only care about some +of these things. + +My preferred semantics would be to make ``isinstance()`` fail by default +for protocol types. This would be in the spirit of duck typing -- protocols +basically would be used to model duck typing statically, not explicitly +at runtime. + +However, it should be possible for protocol types to implement custom +``isinstance`` behavior when this would make sense, similar to how +``Iterable`` and other ABCs in ``collections.abc`` and typing already do it, +but this should be specific to these particular classes. We need this fallback option anyway for backward compatibility. -Type[...] with protocols ------------------------- +``Type[...]`` with protocols +---------------------------- -Type[...] accepts only non-abstract (non-protocol?) classes +``Type[...]`` accepts only non-abstract (non-protocol?) classes Runtime behavior of protocol classes @@ -421,10 +418,10 @@ Runtime behavior of protocol classes At runtime normal ABCs, cannot be instantiated. -@auto_runtime decorator (or make it always??) ------------------------ +``@auto_runtime`` decorator +--------------------------- -(like in collections.abc), typechecker ensures no empty bodies +(like in ``collections.abc``), typechecker ensures no empty bodies Changes in typing module @@ -433,24 +430,25 @@ Changes in typing module I think that at least these classes in typing should be protocols: (probably all abstract change, concrete stay) -* Sized -* Container -* Iterable -* Iterator -* Reversible -* SupportsAbs (and other Supports* classes) +* ``Sized`` +* ``Container`` +* ``Iterable`` +* ``Iterator`` +* ``Reversible`` +* ``SupportsAbs`` (and other ``Supports*`` classes) These classes are small and conceptually simple. It's easy to see which of these protocols a class implements from the presence of a small number of magic methods, which immediately suggest a protocol. -I'm not sure about other classes such as Sequence, Set and IO. I believe -that these are sufficiently complex that it makes sense to require code -to be explicit about them, as there would not be any sufficiently obvious -and small set of 'marker' methods that tell that a class implements this protocol. -Also, it's too easy to leave some part of the protocol/interface unimplemented -by accident, and explicitly marking the subclass relationship allows type checkers -to pinpoint the missing implementations -- or they can be inherited from the ABC, +I'm not sure about other classes such as ``Sequence``, ``Set`` and ``IO``. +I believe that these are sufficiently complex that it makes sense to require +code to be explicit about them, as there would not be any sufficiently obvious +and small set of 'marker' methods that tell that a class implements +this protocol. Also, it's too easy to leave some part of +the protocol/interface unimplemented by accident, and explicitly marking +the subclass relationship allows type checkers to pinpoint the missing +implementations -- or they can be inherited from the ABC, in case that is has default implementations. So I'd currently vote against making these classes protocols. @@ -460,16 +458,17 @@ Introspection Current typing is already good. -The existing introspection machinery (dir, etc.) could be used with protocols, -but typing would not include an implementation of additional introspection -or runtime type checking capabilities for protocols. +The existing introspection machinery (``dir``, etc.) could be used with +protocols, but typing would not include an implementation of additional +introspection or runtime type checking capabilities for protocols. As all attributes need to be defined in the class body based on this proposal, -protocol classes would have better support for introspection than regular classes -where attributes can be defined implicitly -- protocol attributes can't be initialized -in ways that are not visible to introspection (using setattr, assignment via self, etc.). -Still, some things likes types of attributes wouldn't be visible at runtime, -so this would necessarily be somewhat limited. +protocol classes would have better support for introspection than +regular classes where attributes can be defined implicitly -- protocol +attributes can't be initialized in ways that are not visible to introspection +(using ``setattr``, assignment via self, etc.). Still, some things like types +of attributes wouldn't be visible at runtime, so this would necessarily be +somewhat limited. Implementation details @@ -477,12 +476,15 @@ Implementation details We'd need to implement at least the following things: -* Define class Protocol (this could be simple, and would be similar to Generic). -* Implement metaclass functionality to detect whether a class is a protocol or not. -Maybe add a class attribute such as __protocol__ = True if that's the case. -Verify that a protocol class only has protocol base classes in the MRO (except for object)???. -* Optionally, override isinstance. -* Optionally, translate ... class attribute values to something else (properties?). +* Define class ``Protocol`` (this could be simple, and would be similar +to ``Generic``). +* Implement metaclass functionality to detect whether a class is +a protocol or not. Maybe add a class attribute such as ``__protocol__ = True`` +if that's the case. Verify that a protocol class only has protocol +base classes in the MRO (except for object). +* Optionally, override ``isinstance``. +* Optionally, translate ``...`` class attribute values to +something else (properties?). Postponed ideas @@ -494,7 +496,7 @@ Should we support optional attributes? We can come up with examples where it would be handy to be able to say that a method or data attribute does not need to be present in a class implementing a protocol, but if it's present, it must conform to a specific -signature or type. One could use a hasattr check to determine whether +signature or type. One could use a ``hasattr`` check to determine whether they can use the attribute on a particular instance. In the interest of simplicity, let's not support optional methods @@ -508,21 +510,23 @@ and Objective-C support a similar concept. Should these be interoperable with other similar implementations? ----------------------------------------------------------------- -The protocols as described here are basically a small extension to the existing -concept of ABCs. I argue that this is the way they should be understood, -instead of as something that replaces Zope interfaces, for example. +The protocols as described here are basically a small extension to +the existing concept of ABCs. I argue that this is the way they should +be understood, instead of as something that replaces Zope interfaces, +for example. -Should every class be a protocol by default? --------------------------------------------- +Make every class a protocol by default +-------------------------------------- Some languages such as Go make structural subtyping the only or the primary -form of subtyping. We could achieve a similar result by making all classes protocols -by default (or even always). I argue that this would be a bad idea and classes should -need to be explicitly marked as protocols, as shown in my proposal above. +form of subtyping. We could achieve a similar result by making all classes +protocols by default (or even always). I argue that this would be a bad idea +and classes should need to be explicitly marked as protocols, +as shown in my proposal above. * Protocols don't have some properties of regular classes. In particular, -isinstance is not well-defined for protocols, whereas it's well-defined +``isinstance`` is not well-defined for protocols, whereas it's well-defined (and pretty commonly used) for regular classes. * Protocol classes should generally not have (many) method implementations, @@ -530,11 +534,35 @@ as they describe an interface, not an implementation. Most classes have many implementations, making them bad protocol classes. * Experience suggests that most classes aren't practical as protocols anyway, -mainly because their interfaces are too large, complex or implementation-oriented -(for example, they may include de facto private attributes and methods without a __ prefix). +mainly because their interfaces are too large, complex or +implementation-oriented (for example, they may include de facto +private attributes and methods without a ``__`` prefix). Most actually useful protocols in existing Python code seem to be implicit. -The ABCs in typing and collections.abc are a kind-of exception, but even they are pretty -recent additions to Python and most programmers do not use them yet. +The ABCs in typing and ``collections.abc`` are a kind-of exception, but +even they are pretty recent additions to Python and most programmers +do not use them yet. + + +Use assignments to check explicitly that a class implements a protocol +---------------------------------------------------------------------- + +I've seen a similar pattern in some Go code that I've reviewed. Example:: + + class A: + def __len__(self) -> float: + return ... + + _ = A() # type: Sized # Error: A.__len__ doesn't conform to 'Sized' + # (Incompatible return type 'float') + +I don't much care above the above example, as it moves the check away from +the class definition and it almost requires a comment as otherwise +the code probably wouldn't make any sense to an average reader +-- it looks like dead code. Besides, in the simplest form it requires us +to construct an instance of A which could problematic if this requires +accessing or allocating some resources such as files or sockets. +We could work around the latter by using a cast, for example, +but then the code would be really ugly. Copyright From 993f7b3d716ab569a778881813353efb8b371980 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Mon, 6 Mar 2017 00:40:06 +0100 Subject: [PATCH 03/37] Add some examples --- pep-05xx.txt | 85 +++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 78 insertions(+), 7 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index b959d73d27c..5de061b9f28 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -16,7 +16,11 @@ Python-Version: 3.7 Abstract ======== -This PEP describes the static and runtime behavior of protocol classes. +PEP 484 has introduced type hints that can be used to specify type metadata +for static type checkers and other third party tools. However, PEP 484 +only supports the *nominal* subtyping. In this PEP we specify the static +and runtime semantics of protocol classes that will provide a support for +*structural* subtyping (static duck typing). Rationale and Goals @@ -48,6 +52,18 @@ still be implicitly considered a subtype of both ``Sized`` and def __iter__(self) -> Iterator[int]: ... +Nominal vs structural subtyping +------------------------------- + +As discussed in PEP 483, both nominal and structural subtyping have their +strengths and weaknesses. Therefore, in this PEP do we not propose to propose +to replace the nominal subtyping described by PEP 484 and currently supported +by the ``typing`` module. Instead, it is proposed that the protocol classes +complement normal classes, and the choice is left for a user to decide where +to apply a particular solution. See section "Rejected ideas" for additional +motivation. + + Non-Goals --------- @@ -55,17 +71,68 @@ Runtime protocol classes are simple ABCs. Moreover as PEP 484 and PEP 526 stated they are **completely optional**. -Nominal vs structural subtyping -=============================== - - Existing approaches to structural subtyping =========================================== -* ``zope.interace`` -* TypeScript +* ``zope.interace``:: + + from zope.interface import Interface + from zope.interface import Attribute + from zope.interface import implements + + class IHost(Interface): + """A host object""" + + name = Attribute("Name of host") + + def goodmorning(guest): + """Say good morning to guest""" + + class Host(object): + + implements(IHost) + + name = u'' + + def goodmorning(self, guest): + return "Good morning, %s!" % guest + +plus much more detailed runtime constraints:: + + from zope.interface import invariant + + def contacts_invariant(obj): + if not (obj.email or obj.phone): + raise Exception("At least one contact info is required") + + class IPerson(Interface): + + name = Attribute("Name") + email = Attribute("Email Address") + phone = Attribute("Phone Number") + + invariant(contacts_invariant) + +However, this all requires runtime validation. + +* TypeScript:: + + interface LabelledValue { + label: string; + } + + function printLabel(labelledObj: LabelledValue) { + console.log(labelledObj.label); + } + + let myObj = {size: 10, label: "Size 10 Object"}; + printLabel(myObj); + * Go - everything is an interface * ABCs + +require explicit subclassing or registration. + * ``collections.abc`` (with ``__subclasshook__``) @@ -270,6 +337,8 @@ provide a default implementation that could be used in explicit subclasses. Overall, I'm probably the least happy about this part of the proposal. +"Private" variables on self? treat (double?) underscores as private? + Explicitly declaring implementation ----------------------------------- @@ -411,6 +480,8 @@ We need this fallback option anyway for backward compatibility. ``Type[...]`` accepts only non-abstract (non-protocol?) classes +copy all stuff from my PR + Runtime behavior of protocol classes ==================================== From e384336dfaca871bad9b21d2945c7e115d08f687 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Mon, 6 Mar 2017 07:45:33 +0100 Subject: [PATCH 04/37] Add planned link templates --- pep-05xx.txt | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/pep-05xx.txt b/pep-05xx.txt index 5de061b9f28..f67d8663738 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -636,6 +636,49 @@ We could work around the latter by using a cast, for example, but then the code would be really ugly. +References +========== + +.. [mypy] + http://mypy-lang.org + +.. [wiki-structural-subtypin] + http://en.wikipedia.org/wiki/... + +.. [typeshed] + https://github.com/python/typeshed/ + +.. [issues] + https://github.com/python/typing/issues + +.. [pep482] + https://hg.python.org/peps/file/tip/pep-0482.txt + +.. [pep483] + https://hg.python.org/peps/file/tip/pep-0483.txt + +.. [pep484] + https://hg.python.org/peps/file/tip/pep-0484.txt + +.. [pep526] + https://hg.python.org/peps/file/tip/pep-0526.txt + +.. [typescript] + https://... + +.. [golang] + https://... + +.. [zope-interface] + https://... + +.. [abstract-base-classes] + https://... + +.. [collections-abc] + https://... + + Copyright ========= From 7ea5d41b74fb89f7443fa7312ccd534313bf55b4 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Mon, 6 Mar 2017 17:13:15 +0100 Subject: [PATCH 05/37] Add links + minor changes --- pep-05xx.txt | 73 ++++++++++++++++++++++++++++------------------------ 1 file changed, 40 insertions(+), 33 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index f67d8663738..88fda434db3 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -55,6 +55,8 @@ still be implicitly considered a subtype of both ``Sized`` and Nominal vs structural subtyping ------------------------------- +Although structural subtyping is natural for Python programmers +(explain once more what it is)... As discussed in PEP 483, both nominal and structural subtyping have their strengths and weaknesses. Therefore, in this PEP do we not propose to propose to replace the nominal subtyping described by PEP 484 and currently supported @@ -74,7 +76,7 @@ stated they are **completely optional**. Existing approaches to structural subtyping =========================================== -* ``zope.interace``:: +* Zope interfaces:: from zope.interface import Interface from zope.interface import Attribute @@ -97,7 +99,7 @@ Existing approaches to structural subtyping def goodmorning(self, guest): return "Good morning, %s!" % guest -plus much more detailed runtime constraints:: +plus interface contracts/constraints:: from zope.interface import invariant @@ -113,7 +115,7 @@ plus much more detailed runtime constraints:: invariant(contacts_invariant) -However, this all requires runtime validation. +Even more is possible. However, this all requires runtime validation. * TypeScript:: @@ -128,12 +130,14 @@ However, this all requires runtime validation. let myObj = {size: 10, label: "Size 10 Object"}; printLabel(myObj); +In addition optional members, always errors on reduntant members. + * Go - everything is an interface -* ABCs +* Python abstract base classes require explicit subclassing or registration. -* ``collections.abc`` (with ``__subclasshook__``) +* Abstract classes in ``collections.abc`` (with ``__subclasshook__``) Specification @@ -142,28 +146,30 @@ Specification Terminology ----------- -Let's call them protocols. The reason is that the term iterator protocol, +We propose to use the term *protocols* types supoorting structural subtyping. +The reason is that the term iterator protocol, for example, is widely understood in the community, and coming up with a new term for this concept in a statically typed context would just create confusion. -This has the drawback that the term 'protocol' becomes overloaded with +This has the drawback that the term *protocol* becomes overloaded with two subtly different meanings: the first is the traditional, well-known but -slightly fuzzy concept of protocols such as iterable; the second is the more -explicitly defined concept of protocols in statically typed code -(or more generally in code that just uses the typing module). -I argue that the distinction isn't important most of the time, and in other -cases people can just add a qualifier such as "protocol classes" -(for the new-style protocols) or "traditional/non-class/implicit protocols". - -Some terminology could be useful here for clarity. If a class includes a -protocol in its MRO, the class is an (explicit) subclass of the protocol. -If a class ia a structural subtype of a protocol, it is said to implement -the protocol and to be compatible with a protocol. +slightly fuzzy concept of protocols such as iterator; the second is the more +explicitly defined concept of protocols in statically typed code. +The distinction is not important most of the time, and in other +cases we propose to just add a qualifier such as *protocol classes* +when refering to the static type concept. + +If a class includes a protocol in its MRO, the class is an called an *explicit* +subclass of the protocol. If a class ia a structural subtype of a protocol, +it is said to implement the protocol and to be compatible with a protocol. If a class is compatible with a protocol but the protocol is not included -in the MRO, the class is an implicit subclass of the protocol. +in the MRO, the class is an *implicit* subclass of the protocol. -Protocol members. +The attributes (variables and methods) of a protocol that are mandatory +for other class in order to be considered a subtype are called +protocol members. Other members defined in a protocol class are called +non-protocol members. Defining a Protocol @@ -617,7 +623,8 @@ do not use them yet. Use assignments to check explicitly that a class implements a protocol ---------------------------------------------------------------------- -I've seen a similar pattern in some Go code that I've reviewed. Example:: +I've seen a similar pattern in some Go code that I've reviewed [golang]. +Example:: class A: def __len__(self) -> float: @@ -642,8 +649,8 @@ References .. [mypy] http://mypy-lang.org -.. [wiki-structural-subtypin] - http://en.wikipedia.org/wiki/... +.. [wiki-structural-subtyping] + https://en.wikipedia.org/wiki/Structural_type_system .. [typeshed] https://github.com/python/typeshed/ @@ -652,31 +659,31 @@ References https://github.com/python/typing/issues .. [pep482] - https://hg.python.org/peps/file/tip/pep-0482.txt + https://www.python.org/dev/peps/pep-0482/ .. [pep483] - https://hg.python.org/peps/file/tip/pep-0483.txt + https://www.python.org/dev/peps/pep-0483/ .. [pep484] - https://hg.python.org/peps/file/tip/pep-0484.txt + https://www.python.org/dev/peps/pep-0484/ .. [pep526] - https://hg.python.org/peps/file/tip/pep-0526.txt + https://www.python.org/dev/peps/pep-0526/ .. [typescript] - https://... + https://www.typescriptlang.org/docs/handbook/interfaces.html .. [golang] - https://... + https://golang.org/doc/effective_go.html#interfaces_and_types -.. [zope-interface] - https://... +.. [zope-interfaces] + https://zopeinterface.readthedocs.io/en/latest/ .. [abstract-base-classes] - https://... + https://docs.python.org/3/library/abc.html .. [collections-abc] - https://... + https://docs.python.org/3/library/collections.abc.html Copyright From cdcf62f7e479ffa70f88a2730111b07b73a052c6 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Mon, 6 Mar 2017 17:58:51 +0100 Subject: [PATCH 06/37] Polishing rationale --- pep-05xx.txt | 62 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 40 insertions(+), 22 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index 88fda434db3..d3a99864857 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -16,21 +16,21 @@ Python-Version: 3.7 Abstract ======== -PEP 484 has introduced type hints that can be used to specify type metadata +Type hints introduced in PEP 484 that can be used to specify type metadata for static type checkers and other third party tools. However, PEP 484 -only supports the *nominal* subtyping. In this PEP we specify the static -and runtime semantics of protocol classes that will provide a support for -*structural* subtyping (static duck typing). +only specifies the semantics of *nominal* subtyping. In this PEP we specify +static and runtime semantics of protocol classes that will provide a support +for *structural* subtyping (static duck typing). Rationale and Goals =================== -Currently, typing defines ABCs for several common Python protocols such as -``Iterable`` and ``Sized``. The problem with them is that a class has to be -explicitly marked to support them, which is arguably unpythonic and unlike -what you'd normally do in your non statically typed code. -For example, this conforms to the PEP 484:: +Currently, PEP 484 and ``typing`` module define abstract base classes for +several common Python protocols such as ``Iterable`` and ``Sized``. The problem +with them is that a class has to be explicitly marked to support them, which is +arguably unpythonic and unlike what one would normally do in non statically +typed code. For example, this conforms to the PEP 484:: from typing import Sized, Iterable, Iterator @@ -39,10 +39,13 @@ For example, this conforms to the PEP 484:: def __len__(self) -> int: ... def __iter__(self) -> Iterator[int]: ... -My intention is that the above code could be written instead equivalently -without explicit base classes in the class definition, and ``Bucket`` would -still be implicitly considered a subtype of both ``Sized`` and -``Iterable[int]`` by using structural subtyping:: +The same problem appears with user defined ABCs, they must be explicitly +subclassed. Moreover, extensive use of ABCs might impose additional runtime +costs. The intention of this PEP is to solve these three problems +by allowing to write the above code without explicit base classes in +the class definition, and ``Bucket`` would still be implicitly considered +a subtype of both ``Sized`` and ``Iterable[int]`` by static type checkers +using structural subtyping:: from typing import Iterator @@ -51,17 +54,23 @@ still be implicitly considered a subtype of both ``Sized`` and def __len__(self) -> int: ... def __iter__(self) -> Iterator[int]: ... + def collect(items: Iterable[int]) -> int: ... + collect(Bucket()) # Passes type check + +The same functionality will be provided for user defined protocols. + Nominal vs structural subtyping ------------------------------- -Although structural subtyping is natural for Python programmers -(explain once more what it is)... -As discussed in PEP 483, both nominal and structural subtyping have their -strengths and weaknesses. Therefore, in this PEP do we not propose to propose -to replace the nominal subtyping described by PEP 484 and currently supported -by the ``typing`` module. Instead, it is proposed that the protocol classes -complement normal classes, and the choice is left for a user to decide where +The structural subtyping is natural for Python programmers since it matches +the runtime semantics of duck typing: an object that has certain properties +is treated independently of its actual runtime class. +However, ss discussed in PEP 483, both nominal and structural subtyping have +their strengths and weaknesses. Therefore, in this PEP do we not propose +to replace the nominal subtyping described by PEP 484 with structural subtyping +completely. Instead, it is proposed that the protocol classes complement +normal classes, and the choice is left for a user to decide where to apply a particular solution. See section "Rejected ideas" for additional motivation. @@ -69,13 +78,22 @@ motivation. Non-Goals --------- -Runtime protocol classes are simple ABCs. Moreover as PEP 484 and PEP 526 -stated they are **completely optional**. +Att runtime protocol classes we be simple ABCs. There is no intent to provide +sophisticated runtime instace and class checks against protocol classes. +This would be difficult and error-prone and will contradict the logic +of PEP 484. As well, following PEP 484 and PEP 526 we state that protocols are +**completely optional** and there is no intent to make them required. +No runtime semantics will be imposed for variables or parameters annotated with +a protocol class, the actual checks will be performed by third party type +checkers and other tools. Existing approaches to structural subtyping =========================================== +Before describing the actual specification, we review existing approaches to +structural subtyping in Python and other languges: + * Zope interfaces:: from zope.interface import Interface From 1ffed9b9a3318fc6942b5431b02007460bec5d3e Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Mon, 6 Mar 2017 18:18:45 +0100 Subject: [PATCH 07/37] Some more reshuffling and formatting --- pep-05xx.txt | 182 +++++++++++++++++++++++++-------------------------- 1 file changed, 91 insertions(+), 91 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index d3a99864857..29b3b6134ac 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -94,66 +94,67 @@ Existing approaches to structural subtyping Before describing the actual specification, we review existing approaches to structural subtyping in Python and other languges: -* Zope interfaces:: +* Zope interfaces was probably historically first widely used approach to + structural subtyping. For example:: - from zope.interface import Interface - from zope.interface import Attribute - from zope.interface import implements + from zope.interface import Interface + from zope.interface import Attribute + from zope.interface import implements - class IHost(Interface): - """A host object""" + class IHost(Interface): + """A host object""" - name = Attribute("Name of host") + name = Attribute("Name of host") - def goodmorning(guest): - """Say good morning to guest""" + def goodmorning(guest): + """Say good morning to guest""" - class Host(object): + class Host(object): - implements(IHost) + implements(IHost) - name = u'' + name = u'' - def goodmorning(self, guest): - return "Good morning, %s!" % guest + def goodmorning(self, guest): + return "Good morning, %s!" % guest -plus interface contracts/constraints:: + plus interface contracts/constraints:: - from zope.interface import invariant + from zope.interface import invariant - def contacts_invariant(obj): - if not (obj.email or obj.phone): - raise Exception("At least one contact info is required") + def contacts_invariant(obj): + if not (obj.email or obj.phone): + raise Exception("At least one contact info is required") - class IPerson(Interface): + class IPerson(Interface): - name = Attribute("Name") - email = Attribute("Email Address") - phone = Attribute("Phone Number") + name = Attribute("Name") + email = Attribute("Email Address") + phone = Attribute("Phone Number") - invariant(contacts_invariant) + invariant(contacts_invariant) -Even more is possible. However, this all requires runtime validation. + Even more is possible. However, this all requires runtime validation. -* TypeScript:: +* TypeScript also provides support for user defined interfaces. + For example:: - interface LabelledValue { - label: string; - } + interface LabelledValue { + label: string; + } - function printLabel(labelledObj: LabelledValue) { - console.log(labelledObj.label); - } + function printLabel(labelledObj: LabelledValue) { + console.log(labelledObj.label); + } - let myObj = {size: 10, label: "Size 10 Object"}; - printLabel(myObj); + let myObj = {size: 10, label: "Size 10 Object"}; + printLabel(myObj); -In addition optional members, always errors on reduntant members. + In addition optional members, always errors on reduntant members. * Go - everything is an interface * Python abstract base classes - -require explicit subclassing or registration. + require explicit subclassing or registration. * Abstract classes in ``collections.abc`` (with ``__subclasshook__``) @@ -164,8 +165,8 @@ Specification Terminology ----------- -We propose to use the term *protocols* types supoorting structural subtyping. -The reason is that the term iterator protocol, +We propose to use the term *protocols* for types supoorting structural +subtyping. The reason is that the term iterator protocol, for example, is widely understood in the community, and coming up with a new term for this concept in a statically typed context would just create confusion. @@ -237,7 +238,7 @@ the implementation of the library. Besides, this would be uglier than how you'd actually write the code in straight, idiomatic dynamically typed Python. The code with a protocol class matches common Python conventions much better. It's also automatically extensible and works with additional, -unrelated classes that happen to implement the required interface. +unrelated classes that happen to implement the required protocol. Protocol members @@ -473,11 +474,25 @@ Union[Proto.open() -> int, Proto.open() -> str], although it is not a subclass of any of two. +``Type[...]`` with protocols +---------------------------- + +``Type[...]`` accepts only non-abstract (non-protocol?) classes + +copy all stuff from my PR + + +``@runtime`` decorator +--------------------------- + +(like in ``collections.abc``), typechecker ensures no empty bodies + + ``isinstance()`` and narrowing types ------------------------------------ ``isinstance(x, Proto)`` and ``issublclass(C, Proto)`` -narrow type if defined with ``@auto_runtime``; ``isinstance(x, Proto[int])`` +narrow type if defined with ``@runtime``; ``isinstance(x, Proto[int])`` always fails. We shouldn't implement any magic ``isinstance()`` machinery, as performing @@ -499,26 +514,11 @@ but this should be specific to these particular classes. We need this fallback option anyway for backward compatibility. -``Type[...]`` with protocols ----------------------------- - -``Type[...]`` accepts only non-abstract (non-protocol?) classes - -copy all stuff from my PR - - Runtime behavior of protocol classes ==================================== At runtime normal ABCs, cannot be instantiated. - -``@auto_runtime`` decorator ---------------------------- - -(like in ``collections.abc``), typechecker ensures no empty bodies - - Changes in typing module ------------------------ @@ -572,22 +572,49 @@ Implementation details We'd need to implement at least the following things: * Define class ``Protocol`` (this could be simple, and would be similar -to ``Generic``). + to ``Generic``). * Implement metaclass functionality to detect whether a class is -a protocol or not. Maybe add a class attribute such as ``__protocol__ = True`` -if that's the case. Verify that a protocol class only has protocol -base classes in the MRO (except for object). + a protocol or not. Maybe add a class attribute such as ``__protocol__ = True`` + if that's the case. Verify that a protocol class only has protocol + base classes in the MRO (except for object). * Optionally, override ``isinstance``. * Optionally, translate ``...`` class attribute values to -something else (properties?). + something else (properties?). Postponed ideas =============== -Should we support optional attributes? +Make every class a protocol by default -------------------------------------- +Some languages such as Go make structural subtyping the only or the primary +form of subtyping. We could achieve a similar result by making all classes +protocols by default (or even always). However we believe that this would be +a bad idea and classes should need to be explicitly marked as protocols, +as shown in this proposal: + +* Protocols don't have some properties of regular classes. In particular, + ``isinstance`` is not well-defined for protocols, whereas it's well-defined + (and pretty commonly used) for regular classes. + +* Protocol classes should generally not have (many) method implementations, + as they describe an interface, not an implementation. + Most classes have many implementations, making them bad protocol classes. + +* Experience suggests that many classes are not practical as protocols anyway, + mainly because their interfaces are too large, complex or + implementation-oriented (for example, they may include de facto + private attributes and methods without a ``__`` prefix). + Most actually useful protocols in existing Python code seem to be implicit. + The ABCs in typing and ``collections.abc`` are a kind-of exception, but + even they are pretty recent additions to Python and most programmers + do not use them yet. + + +Support optional protocol members +--------------------------------- + We can come up with examples where it would be handy to be able to say that a method or data attribute does not need to be present in a class implementing a protocol, but if it's present, it must conform to a specific @@ -602,8 +629,8 @@ they are pretty commonly used. If I remember correctly, at least TypeScript and Objective-C support a similar concept. -Should these be interoperable with other similar implementations? ------------------------------------------------------------------ +Should protocols be interoperable with similar implementations? +--------------------------------------------------------------- The protocols as described here are basically a small extension to the existing concept of ABCs. I argue that this is the way they should @@ -611,33 +638,6 @@ be understood, instead of as something that replaces Zope interfaces, for example. -Make every class a protocol by default --------------------------------------- - -Some languages such as Go make structural subtyping the only or the primary -form of subtyping. We could achieve a similar result by making all classes -protocols by default (or even always). I argue that this would be a bad idea -and classes should need to be explicitly marked as protocols, -as shown in my proposal above. - -* Protocols don't have some properties of regular classes. In particular, -``isinstance`` is not well-defined for protocols, whereas it's well-defined -(and pretty commonly used) for regular classes. - -* Protocol classes should generally not have (many) method implementations, -as they describe an interface, not an implementation. -Most classes have many implementations, making them bad protocol classes. - -* Experience suggests that most classes aren't practical as protocols anyway, -mainly because their interfaces are too large, complex or -implementation-oriented (for example, they may include de facto -private attributes and methods without a ``__`` prefix). -Most actually useful protocols in existing Python code seem to be implicit. -The ABCs in typing and ``collections.abc`` are a kind-of exception, but -even they are pretty recent additions to Python and most programmers -do not use them yet. - - Use assignments to check explicitly that a class implements a protocol ---------------------------------------------------------------------- From 72ceae6b4f141ec80e1780fd8812a59d842c6d76 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Mon, 6 Mar 2017 20:40:22 +0100 Subject: [PATCH 08/37] Add more examples --- pep-05xx.txt | 127 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 78 insertions(+), 49 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index 29b3b6134ac..28648618c01 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -55,7 +55,7 @@ using structural subtyping:: def __iter__(self) -> Iterator[int]: ... def collect(items: Iterable[int]) -> int: ... - collect(Bucket()) # Passes type check + result: int = collect(Bucket()) # Passes type check The same functionality will be provided for user defined protocols. @@ -94,35 +94,35 @@ Existing approaches to structural subtyping Before describing the actual specification, we review existing approaches to structural subtyping in Python and other languges: -* Zope interfaces was probably historically first widely used approach to - structural subtyping. For example:: - - from zope.interface import Interface - from zope.interface import Attribute - from zope.interface import implements +* Zope interfaces was one of the first widely used approaches to + structural subtyping in Python. It is implemented by providing special + classes to distinguish interface classes from normal classes, to mark + interface attributes and to excplicitly declare implementation. + For example:: - class IHost(Interface): - """A host object""" + from zope.interface import Interface, Attribute, implements - name = Attribute("Name of host") + class IEmployee(Interface): - def goodmorning(guest): - """Say good morning to guest""" + name = Attribute("Name of employee") - class Host(object): + def do(work): + """Do some work""" - implements(IHost) + class Employee(object): + implements(IEmployee) - name = u'' + name = 'Edwin' - def goodmorning(self, guest): - return "Good morning, %s!" % guest + def do(self, work): + return work.start() - plus interface contracts/constraints:: + Zope interfaces support various contracts and constraints for interface + classes. For example:: from zope.interface import invariant - def contacts_invariant(obj): + def required_contact(obj): if not (obj.email or obj.phone): raise Exception("At least one contact info is required") @@ -134,28 +134,51 @@ structural subtyping in Python and other languges: invariant(contacts_invariant) - Even more is possible. However, this all requires runtime validation. - -* TypeScript also provides support for user defined interfaces. - For example:: - - interface LabelledValue { + Even more detailed invariants are supported. However, Zope interface rely + entirely on runtime validation. Such focus on runtime properties goes beyond + the scope of current proposal, and static support for invariants might be + difficult to implement. However, the idea of marking an interface with + a special base class seems reasonable and easy to implement both statically + and at runtime. +* TypeScript also provides support for user defined classes and interfaces. + Explicit implementation declaration is not required, structural subtyping + is verified statically. For example:: + + interface LabelledItem { label: string; + size?: int; } - function printLabel(labelledObj: LabelledValue) { - console.log(labelledObj.label); + function printLabel(obj: LabelledValue) { + console.log(obj.label); } let myObj = {size: 10, label: "Size 10 Object"}; printLabel(myObj); - In addition optional members, always errors on reduntant members. + Note that optional interface members are supported. Also, TypeScript + prohibits reduntant members in implementations. While the idea of + optional members looks interesting, it would complicate this proposal and it + is not clear how usefull it will be. Therefore it is proposed to postpone + this, see "Rejected ideas". In general, the idea of static protocol + checking without runtime implications looks very promising, and actually + forms the basis of this proposal. +* Go uses a more radical approach and makes interfaces the primary way + to provide type information. Also, assignments are used to explicitely ensure + implementation:: + + type SomeInterface interface { + SomeMethod() ([]byte, error) + } + + if _, ok := someval.(SomeInterface); ok { + fmt.Printf("value implements some interface") + } -* Go - everything is an interface + Both these ideas seem to be not fit for Python as discussed in section + "Rejected ideas". * Python abstract base classes require explicit subclassing or registration. - * Abstract classes in ``collections.abc`` (with ``__subclasshook__``) @@ -191,7 +214,7 @@ protocol members. Other members defined in a protocol class are called non-protocol members. -Defining a Protocol +Defining a protocol ------------------- There will be a new class ``typing.Protocol``. If this is explicitly included @@ -223,7 +246,7 @@ Protocol types can be used in annotations, of course, and for type checking:: f = open('foo.txt') r = Resource(...) close_all([f, r]) # OK! - close_all([1]) # Error: 'int' has no 'close' method + close_all([1]) # Error: 'int' has no 'close' method Note that both the user-defined class ``Resource`` and the ``IO`` type (the return type of ``open()``) would be considered subtypes of @@ -360,9 +383,7 @@ and instance data attributes (ClassVar PEP 526). I'm not sure sure what to do with ``__init__``. I guess a protocol could provide a default implementation that could be used in explicit subclasses. -Overall, I'm probably the least happy about this part of the proposal. - -"Private" variables on self? treat (double?) underscores as private? +"Private" variables on self, treat double underscores as private. Explicitly declaring implementation @@ -432,7 +453,7 @@ is included in the base class list, all the other base classes must be protocols. A protocol can't extend a regular class. -Generic and recursive Protocols +Generic and recursive protocols ------------------------------- ``Protocol[T, S, ...]`` as a shorthand for ``Protocol, Generic[T, S, ...]`` @@ -453,8 +474,8 @@ data structures like trees in an abstract fashion, but I don't see them used commonly in production code. -Interactions of Protocols -========================= +Using protocols +=============== cannot be instantiated. @@ -465,8 +486,8 @@ Subtyping: issubtype(A, Proto) is _always_ structural; issubtype(Proto, Nom) always False. -All(ala Intersection) and Union -------------------------------- +``Union[]`` and ``All[]`` +------------------------- Union merges protocols: open() -> Union[int, str] conforms to @@ -474,27 +495,37 @@ Union[Proto.open() -> int, Proto.open() -> str], although it is not a subclass of any of two. -``Type[...]`` with protocols +``Type[]`` with protocols ---------------------------- -``Type[...]`` accepts only non-abstract (non-protocol?) classes +``Type[]`` accepts only non-abstract (non-protocol?) classes copy all stuff from my PR -``@runtime`` decorator ---------------------------- +``NewType()`` and type aliases +------------------------------ + +Protocols are essentially anonimous. To emphasize this point, static type +checkers might refuse the protocl classes inside ``NewType(...)`` to avoid an +illusion that a distictt type is provided. -(like in ``collections.abc``), typechecker ensures no empty bodies +On the contrary, type aliases are fully supported, including generic type +aliases. -``isinstance()`` and narrowing types ------------------------------------- +``@runtime`` decorator and narrowing types by ``isinstance()`` +-------------------------------------------------------------- + +``@runtime`` decorator automatically adds a ``__subclasshook__`` +(like in ``collections.abc``), typechecker ensures no empty bodies. ``isinstance(x, Proto)`` and ``issublclass(C, Proto)`` narrow type if defined with ``@runtime``; ``isinstance(x, Proto[int])`` always fails. +``Callable[..., Any]`` for methods (note arbitrary number of args). + We shouldn't implement any magic ``isinstance()`` machinery, as performing a runtime compatibility check is generally difficult: we might want to verify argument counts to methods, names of arguments and even argument types, @@ -597,11 +628,9 @@ as shown in this proposal: * Protocols don't have some properties of regular classes. In particular, ``isinstance`` is not well-defined for protocols, whereas it's well-defined (and pretty commonly used) for regular classes. - * Protocol classes should generally not have (many) method implementations, as they describe an interface, not an implementation. Most classes have many implementations, making them bad protocol classes. - * Experience suggests that many classes are not practical as protocols anyway, mainly because their interfaces are too large, complex or implementation-oriented (for example, they may include de facto From 6bea2e8e042d244c99e3b33ba692e2c73abec36e Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Mon, 6 Mar 2017 21:15:22 +0100 Subject: [PATCH 09/37] Add more examples to existing approaches --- pep-05xx.txt | 60 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 47 insertions(+), 13 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index 28648618c01..e1ebf18ad3a 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -134,7 +134,7 @@ structural subtyping in Python and other languges: invariant(contacts_invariant) - Even more detailed invariants are supported. However, Zope interface rely + Even more detailed invariants are supported. However, Zope interfaces rely entirely on runtime validation. Such focus on runtime properties goes beyond the scope of current proposal, and static support for invariants might be difficult to implement. However, the idea of marking an interface with @@ -161,25 +161,59 @@ structural subtyping in Python and other languges: optional members looks interesting, it would complicate this proposal and it is not clear how usefull it will be. Therefore it is proposed to postpone this, see "Rejected ideas". In general, the idea of static protocol - checking without runtime implications looks very promising, and actually - forms the basis of this proposal. + checking without runtime implications looks reasonable, and forms the basis + of this proposal. * Go uses a more radical approach and makes interfaces the primary way to provide type information. Also, assignments are used to explicitely ensure implementation:: type SomeInterface interface { - SomeMethod() ([]byte, error) + SomeMethod() ([]byte, error) } if _, ok := someval.(SomeInterface); ok { - fmt.Printf("value implements some interface") + fmt.Printf("value implements some interface") } - Both these ideas seem to be not fit for Python as discussed in section - "Rejected ideas". -* Python abstract base classes - require explicit subclassing or registration. -* Abstract classes in ``collections.abc`` (with ``__subclasshook__``) + Both these ideas are questionable in the context of this proposal, see + section "Rejected ideas". +* Python abstract base classes are the standard library tool to provide some + functionality similar to structural subtyping. The drawback of this approach + is the necessity to either subclass the abstract class or register an + implementation explicitly:: + + from abc import ABCMeta + + class MyTuple(metaclass=ABCMeta): + pass + + MyTuple.register(tuple) + + assert issubclass(tuple, MyABC) + assert isinstance((), MyABC) + + As mentioned in rationale, we want to avoid such necessity in static context. + However, in runtime context, ABCs are good candidates for protocol classes + and are already used extensively in ``typing`` module. + +* Abstract classes defined in ``collections.abc`` module are slightly more + advanced since they implement a custom ``__subclasshook__()`` method that + allows runtime structural checks without explicit registration:: + + from collections.abc import Iterable + + class MyIterable: + def __iter__(self): + return [] + + assert isinstance(MyIterable(), Iterable) + + This seems to be a good fit for both runtime and static behaviour of + protocols. The main goal of this proposal is to support such behavior + statically. In addition, to allow users to achieve (if necessary) such + runtime behaviour for user defined protocols, a simple class decorator + ``@runtime`` will be provided that automatically adds a ``__subclasshook__`` + that mimics the one in ``collections.abc``, see detailed specification below. Specification @@ -496,7 +530,7 @@ although it is not a subclass of any of two. ``Type[]`` with protocols ----------------------------- +------------------------- ``Type[]`` accepts only non-abstract (non-protocol?) classes @@ -613,8 +647,8 @@ We'd need to implement at least the following things: something else (properties?). -Postponed ideas -=============== +Rejected and postponed ideas +============================ Make every class a protocol by default -------------------------------------- From d5972c3c136577761894a6228cbbaba995c6c32a Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Mon, 6 Mar 2017 23:39:33 +0100 Subject: [PATCH 10/37] Typos, reordering, and few more details (backport) --- pep-05xx.txt | 158 ++++++++++++++++++++++++++++----------------------- 1 file changed, 86 insertions(+), 72 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index e1ebf18ad3a..162be68e79d 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -66,7 +66,7 @@ Nominal vs structural subtyping The structural subtyping is natural for Python programmers since it matches the runtime semantics of duck typing: an object that has certain properties is treated independently of its actual runtime class. -However, ss discussed in PEP 483, both nominal and structural subtyping have +However, as discussed in PEP 483, both nominal and structural subtyping have their strengths and weaknesses. Therefore, in this PEP do we not propose to replace the nominal subtyping described by PEP 484 with structural subtyping completely. Instead, it is proposed that the protocol classes complement @@ -78,8 +78,8 @@ motivation. Non-Goals --------- -Att runtime protocol classes we be simple ABCs. There is no intent to provide -sophisticated runtime instace and class checks against protocol classes. +At runtime protocol classes we be simple ABCs. There is no intent to provide +sophisticated runtime instance and class checks against protocol classes. This would be difficult and error-prone and will contradict the logic of PEP 484. As well, following PEP 484 and PEP 526 we state that protocols are **completely optional** and there is no intent to make them required. @@ -91,13 +91,13 @@ checkers and other tools. Existing approaches to structural subtyping =========================================== -Before describing the actual specification, we review existing approaches to -structural subtyping in Python and other languges: +Before describing the actual specification, we review existing approaches +related to structural subtyping in Python and other languages: * Zope interfaces was one of the first widely used approaches to structural subtyping in Python. It is implemented by providing special classes to distinguish interface classes from normal classes, to mark - interface attributes and to excplicitly declare implementation. + interface attributes and to explicitly declare implementation. For example:: from zope.interface import Interface, Attribute, implements @@ -135,12 +135,49 @@ structural subtyping in Python and other languges: invariant(contacts_invariant) Even more detailed invariants are supported. However, Zope interfaces rely - entirely on runtime validation. Such focus on runtime properties goes beyond - the scope of current proposal, and static support for invariants might be - difficult to implement. However, the idea of marking an interface with - a special base class seems reasonable and easy to implement both statically - and at runtime. -* TypeScript also provides support for user defined classes and interfaces. + entirely on runtime validation. Such focus on runtime properties goes + beyond the scope of current proposal, and static support for invariants + might be difficult to implement. However, the idea of marking an interface + with a special base class is reasonable and easy to implement both + statically and at runtime. +* Python abstract base classes are the standard library tool to provide some + functionality similar to structural subtyping. The drawback of this + approach is the necessity to either subclass the abstract class or + register an implementation explicitly:: + + from abc import ABCMeta + + class MyTuple(metaclass=ABCMeta): + pass + + MyTuple.register(tuple) + + assert issubclass(tuple, MyTuple) + assert isinstance((), MyTuple) + + As mentioned in rationale, we want to avoid such necessity, especially in + static context. However, in runtime context, ABCs are good candidates for + protocol classes and are already used extensively in ``typing`` module. +* Abstract classes defined in ``collections.abc`` module are slightly more + advanced since they implement a custom ``__subclasshook__()`` method that + allows runtime structural checks without explicit registration:: + + from collections.abc import Iterable + + class MyIterable: + def __iter__(self): + return [] + + assert isinstance(MyIterable(), Iterable) + + This seems to be a perfect fit for both runtime and static behavior of + protocols. The main goal of this proposal is to support such behavior + statically. In addition, to allow users achieving (if necessary) such + runtime behavior for user defined protocols, a simple class decorator + ``@runtime`` can be provided that automatically adds a ``__subclasshook__`` + that mimics the one in ``collections.abc``, see detailed specification + below. +* TypeScript provides support for user defined classes and interfaces. Explicit implementation declaration is not required, structural subtyping is verified statically. For example:: @@ -157,15 +194,15 @@ structural subtyping in Python and other languges: printLabel(myObj); Note that optional interface members are supported. Also, TypeScript - prohibits reduntant members in implementations. While the idea of - optional members looks interesting, it would complicate this proposal and it - is not clear how usefull it will be. Therefore it is proposed to postpone + prohibits redundant members in implementations. While the idea of + optional members looks interesting, it would complicate this proposal and + it is not clear how useful it will be. Therefore it is proposed to postpone this, see "Rejected ideas". In general, the idea of static protocol - checking without runtime implications looks reasonable, and forms the basis - of this proposal. + checking without runtime implications looks reasonable, and basically + this proposal follows the same line. * Go uses a more radical approach and makes interfaces the primary way - to provide type information. Also, assignments are used to explicitely ensure - implementation:: + to provide type information. Also, assignments are used to explicitly + ensure implementation:: type SomeInterface interface { SomeMethod() ([]byte, error) @@ -177,43 +214,6 @@ structural subtyping in Python and other languges: Both these ideas are questionable in the context of this proposal, see section "Rejected ideas". -* Python abstract base classes are the standard library tool to provide some - functionality similar to structural subtyping. The drawback of this approach - is the necessity to either subclass the abstract class or register an - implementation explicitly:: - - from abc import ABCMeta - - class MyTuple(metaclass=ABCMeta): - pass - - MyTuple.register(tuple) - - assert issubclass(tuple, MyABC) - assert isinstance((), MyABC) - - As mentioned in rationale, we want to avoid such necessity in static context. - However, in runtime context, ABCs are good candidates for protocol classes - and are already used extensively in ``typing`` module. - -* Abstract classes defined in ``collections.abc`` module are slightly more - advanced since they implement a custom ``__subclasshook__()`` method that - allows runtime structural checks without explicit registration:: - - from collections.abc import Iterable - - class MyIterable: - def __iter__(self): - return [] - - assert isinstance(MyIterable(), Iterable) - - This seems to be a good fit for both runtime and static behaviour of - protocols. The main goal of this proposal is to support such behavior - statically. In addition, to allow users to achieve (if necessary) such - runtime behaviour for user defined protocols, a simple class decorator - ``@runtime`` will be provided that automatically adds a ``__subclasshook__`` - that mimics the one in ``collections.abc``, see detailed specification below. Specification @@ -222,7 +222,7 @@ Specification Terminology ----------- -We propose to use the term *protocols* for types supoorting structural +We propose to use the term *protocols* for types supporting structural subtyping. The reason is that the term iterator protocol, for example, is widely understood in the community, and coming up with a new term for this concept in a statically typed context would just create @@ -234,13 +234,14 @@ slightly fuzzy concept of protocols such as iterator; the second is the more explicitly defined concept of protocols in statically typed code. The distinction is not important most of the time, and in other cases we propose to just add a qualifier such as *protocol classes* -when refering to the static type concept. +when referring to the static type concept. -If a class includes a protocol in its MRO, the class is an called an *explicit* -subclass of the protocol. If a class ia a structural subtype of a protocol, -it is said to implement the protocol and to be compatible with a protocol. -If a class is compatible with a protocol but the protocol is not included -in the MRO, the class is an *implicit* subclass of the protocol. +If a class includes a protocol in its MRO, the class is an called +an *explicit* subclass of the protocol. If a class is a structural subtype +of a protocol, it is said to implement the protocol and to be compatible +with a protocol. If a class is compatible with a protocol but the protocol +is not included in the MRO, the class is an *implicit* subclass +of the protocol. The attributes (variables and methods) of a protocol that are mandatory for other class in order to be considered a subtype are called @@ -394,7 +395,7 @@ the interface should not depend on the default implementation. This is more of a style than a technical issue, as a type checker could infer attributes from assignment statements within methods as well. -When using the ``...`` initializer, the ``... `` initializer might +When using the ``...`` initializer, the ``...`` initializer might leak into subclasses at runtime, which is unfortunate:: class A(Protocol): @@ -540,9 +541,9 @@ copy all stuff from my PR ``NewType()`` and type aliases ------------------------------ -Protocols are essentially anonimous. To emphasize this point, static type -checkers might refuse the protocl classes inside ``NewType(...)`` to avoid an -illusion that a distictt type is provided. +Protocols are essentially anonymous. To emphasize this point, static type +checkers might refuse the protocol classes inside ``NewType(...)`` to avoid an +illusion that a distinct type is provided. On the contrary, type aliases are fully supported, including generic type aliases. @@ -579,6 +580,17 @@ but this should be specific to these particular classes. We need this fallback option anyway for backward compatibility. +Using protocols in Python 3.5 and earlier +========================================= + +Variable annotation syntax was added in Python 3.6. To define the +protocols that have variable protocol members in earlier versions +we propose ... + +``typing`` module changes will be backported to earlier versions +via the backport available on PyPI. + + Runtime behavior of protocol classes ==================================== @@ -642,9 +654,11 @@ We'd need to implement at least the following things: a protocol or not. Maybe add a class attribute such as ``__protocol__ = True`` if that's the case. Verify that a protocol class only has protocol base classes in the MRO (except for object). -* Optionally, override ``isinstance``. -* Optionally, translate ``...`` class attribute values to - something else (properties?). +* Implement ``@runtime`` class decorator that makes ``isinstance`` behave as + for ``collections.abc`` by adding all abstract methods and all variables + from ``__annotations__`` to ``__subclsshook__``. +* In the backported version, translate ``...`` class attribute values to + abstract properties. Rejected and postponed ideas @@ -692,8 +706,8 @@ they are pretty commonly used. If I remember correctly, at least TypeScript and Objective-C support a similar concept. -Should protocols be interoperable with similar implementations? ---------------------------------------------------------------- +Should protocols be interoperable with other approaches? +-------------------------------------------------------- The protocols as described here are basically a small extension to the existing concept of ABCs. I argue that this is the way they should From 57d375ff9a660d8bd58853b746bd39d1a0d77ee4 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Tue, 7 Mar 2017 00:50:50 +0100 Subject: [PATCH 11/37] Update list of protocols in typing --- pep-05xx.txt | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index 162be68e79d..c57db4b802c 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -602,16 +602,24 @@ Changes in typing module I think that at least these classes in typing should be protocols: (probably all abstract change, concrete stay) +* ``Hashable`` +* ``SupportsAbs`` (and other ``Supports*`` classes) +* ``Iterable``, ``Iterator`` * ``Sized`` * ``Container`` -* ``Iterable`` -* ``Iterator`` +* ``Collection`` * ``Reversible`` -* ``SupportsAbs`` (and other ``Supports*`` classes) - -These classes are small and conceptually simple. It's easy to see which of -these protocols a class implements from the presence of a small number -of magic methods, which immediately suggest a protocol. +* ``Sequence``, ``MutableSequence`` +* ``AbstractSet``, ``MutableSet`` +* ``Mapping``, ``MutableMapping`` +* ``ItemsView`` (and other ``*View`` classes) +* ``AsyncIterable``, ``AsyncIterator`` +* ``Awaitable`` +* ``ContextManager`` + +Most of these classes are small and conceptually simple. It's easy to see +which of these protocols a class implements from the presence of a small +number of magic methods, which immediately suggest a protocol. I'm not sure about other classes such as ``Sequence``, ``Set`` and ``IO``. I believe that these are sufficiently complex that it makes sense to require From 9d4d68520da4bd0aa34f114e61502f8c1acdefcf Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Tue, 7 Mar 2017 15:51:57 +0100 Subject: [PATCH 12/37] Defining protocols plus minor changes and formatting --- pep-05xx.txt | 448 +++++++++++++++++++++++++++++---------------------- 1 file changed, 257 insertions(+), 191 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index c57db4b802c..d26c11ce277 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -2,9 +2,7 @@ PEP: 5xx Title: Protocols Version: $Revision$ Last-Modified: $Date$ -Author: Ivan Levkivskyi , - Jukka Lehtosalo , - Łukasz Langa +Author: Ivan Levkivskyi , Jukka Lehtosalo , Łukasz Langa Discussions-To: Python-Dev Status: Draft Type: Standards Track @@ -27,10 +25,11 @@ Rationale and Goals =================== Currently, PEP 484 and ``typing`` module define abstract base classes for -several common Python protocols such as ``Iterable`` and ``Sized``. The problem -with them is that a class has to be explicitly marked to support them, which is -arguably unpythonic and unlike what one would normally do in non statically -typed code. For example, this conforms to the PEP 484:: +several common Python protocols such as ``Iterable`` and ``Sized``. +The problem with them is that a class has to be explicitly marked to support +them, which is arguably unpythonic and unlike what one would normally do in +idiomatic dynamically typed Python code. For example, +this conforms to the PEP 484:: from typing import Sized, Iterable, Iterator @@ -40,8 +39,10 @@ typed code. For example, this conforms to the PEP 484:: def __iter__(self) -> Iterator[int]: ... The same problem appears with user defined ABCs, they must be explicitly -subclassed. Moreover, extensive use of ABCs might impose additional runtime -costs. The intention of this PEP is to solve these three problems +subclassed or registered. This is particularly difficult to do with library +types as the type objects may be hidden deep in the implementation +of the library. Moreover, extensive use of ABCs might impose additional +runtime costs. The intention of this PEP is to solve all these problems by allowing to write the above code without explicit base classes in the class definition, and ``Bucket`` would still be implicitly considered a subtype of both ``Sized`` and ``Iterable[int]`` by static type checkers @@ -57,7 +58,11 @@ using structural subtyping:: def collect(items: Iterable[int]) -> int: ... result: int = collect(Bucket()) # Passes type check -The same functionality will be provided for user defined protocols. +The same functionality will be provided for user defined protocols, as +specified below. The code with a protocol class matches common Python +conventions much better. It is also automatically extensible and works +with additional, unrelated classes that happen to implement +the required protocol. Nominal vs structural subtyping @@ -68,9 +73,9 @@ the runtime semantics of duck typing: an object that has certain properties is treated independently of its actual runtime class. However, as discussed in PEP 483, both nominal and structural subtyping have their strengths and weaknesses. Therefore, in this PEP do we not propose -to replace the nominal subtyping described by PEP 484 with structural subtyping -completely. Instead, it is proposed that the protocol classes complement -normal classes, and the choice is left for a user to decide where +to replace the nominal subtyping described by PEP 484 with structural +subtyping completely. Instead, it is proposed that the protocol classes +complement normal classes, and the choice is left for a user to decide where to apply a particular solution. See section "Rejected ideas" for additional motivation. @@ -112,7 +117,7 @@ related to structural subtyping in Python and other languages: class Employee(object): implements(IEmployee) - name = 'Edwin' + name = 'Anonymous' def do(self, work): return work.start() @@ -140,6 +145,7 @@ related to structural subtyping in Python and other languages: might be difficult to implement. However, the idea of marking an interface with a special base class is reasonable and easy to implement both statically and at runtime. + * Python abstract base classes are the standard library tool to provide some functionality similar to structural subtyping. The drawback of this approach is the necessity to either subclass the abstract class or @@ -158,6 +164,7 @@ related to structural subtyping in Python and other languages: As mentioned in rationale, we want to avoid such necessity, especially in static context. However, in runtime context, ABCs are good candidates for protocol classes and are already used extensively in ``typing`` module. + * Abstract classes defined in ``collections.abc`` module are slightly more advanced since they implement a custom ``__subclasshook__()`` method that allows runtime structural checks without explicit registration:: @@ -177,6 +184,7 @@ related to structural subtyping in Python and other languages: ``@runtime`` can be provided that automatically adds a ``__subclasshook__`` that mimics the one in ``collections.abc``, see detailed specification below. + * TypeScript provides support for user defined classes and interfaces. Explicit implementation declaration is not required, structural subtyping is verified statically. For example:: @@ -200,6 +208,7 @@ related to structural subtyping in Python and other languages: this, see "Rejected ideas". In general, the idea of static protocol checking without runtime implications looks reasonable, and basically this proposal follows the same line. + * Go uses a more radical approach and makes interfaces the primary way to provide type information. Also, assignments are used to explicitly ensure implementation:: @@ -252,179 +261,116 @@ non-protocol members. Defining a protocol ------------------- -There will be a new class ``typing.Protocol``. If this is explicitly included -in the the base class list, the class is a protocol. -Here is a simple example:: +Protocols are defined by including a special new class ``typing.Protocol`` +(an instance of ``abc.ABCMeta``) in the base classes list, preferably +at the beginning of the list. Here is a simple example:: from typing import Protocol class SupportsClose(Protocol): - def close(self) -> None: ... + def close(self) -> None: + ... -Now if we define a class ``Resource`` with a close method that has -a suitable signature, it would implicitly be a subtype of ``SupportsClose``, -since we'd use structural subtyping for protocol types:: +Now if one defines a class ``Resource`` with a ``close()`` method that has +a compatible signature, it would implicitly be a subtype of +``SupportsClose``, since the structural subtyping is used for +protocol types:: class Resource: ... - def close(self) -> None: self.file.close() self.lock.release() -Protocol types can be used in annotations, of course, and for type checking:: +Protocol types can be used in function annotations, variable annotations, +and for type checking:: def close_all(things: Iterable[SupportsClose]) -> None: for t in things: t.close() f = open('foo.txt') - r = Resource(...) + r = Resource() close_all([f, r]) # OK! close_all([1]) # Error: 'int' has no 'close' method -Note that both the user-defined class ``Resource`` and the ``IO`` type -(the return type of ``open()``) would be considered subtypes of -``SupportsClose`` because they provide a suitable close method. - -If using the current ``typing`` module, our only option to implement -the above example would be to use an ABC (or type ``Any``, but that would -compromise type checking). If we'd use an ABC, we'd have to explicitly -register the fact that these types are related, and this generally difficult -to do with library types as the type objects may be hidden deep in -the implementation of the library. Besides, this would be uglier than how -you'd actually write the code in straight, idiomatic dynamically typed Python. -The code with a protocol class matches common Python conventions much better. -It's also automatically extensible and works with additional, -unrelated classes that happen to implement the required protocol. +Note that both the user-defined class ``Resource`` and the built-in +``IO`` type (the return type of ``open()``) are considered subtypes of +``SupportsClose``, because they provide a ``close()`` method with +a compatible type signature. Protocol members ---------------- -I propose that most of the regular rules for classes still apply to -protocol classes (modulo a few exceptions that only apply to -protocol classes). I'd like protocols to also be ABCs, so all of these -would be valid within a protocol class:: - - # Variant 1 - def __len__(self) -> int: ... - - # Variant 2 - def __len__(self) -> int: pass - - # Variant 3 - @abstractmethod - def __len__(self) -> int: pass - - # Variant 4 - def __len__(self): pass - - # Variant 5 - def __len__(self) -> int: - return 0 - - # Variant 6 - def __len__(self) -> int: - raise NotImplementedError - -For variants 1, 2 and 3, a type checker should probably always require -an explicit implementation to be defined in a subclass that explicitly -subclasses the protocol (see below for more about this), because -the implementations return ``None`` which is not a valid return type -for the method. For variants 4, 5 and 6, we can use -the provided implementation as a default implementation. -The default implementations won't be used if the subtype relationship -is implicit and only via structural subtyping -- the semantics of inheritance -won't be changed. - -I also propose that a ``...`` as the method body in a protocol type makes -the method implicitly abstract. This would only be checked statically, -and there won't be any runtime checking. The idea here is that most methods -in a protocol will be abstract, and having to always use ``@abstractmethod`` -is a little verbose and ugly, and has the issue of implicit ``None`` -return types confusing things. -This would be the recommended way of defining methods in a protocol that -don't have an implementation, but the other approaches can be used for -legacy code or if people just feel like it. The recommended syntax would -mirror how methods are defined in stub files. - -Similar to methods, there will be multiple valid ways of defining data -attributes (or properties). All of these will be valid:: - - class Foo(Protocol): - a = ... # type: int # Variant 1 - b = 0 # Variant 2 - - # Variant 3 - @property - def c(self) -> int: ... - - # Variant 4 - @property - def c(self) -> int: - return 0 - - # Variant 5 - @property - def d(self) -> int: - raise NotImplementedError - - # Variant 6 - @abstractproperty - def e(self) -> int: ... - - # Variant 7 - @abstractproperty - def f(self) -> int: pass - -Also, properties with setters can also be defined. The first three variants -would be the recommended ways of defining attributes or properties, -but similar to 3), the others are possible and may be useful for supporting -legacy code. - -When using an ... initializer, ``@abstractproperty`` or ``pass``/``...`` -as property body (and when the type does not include ``None``), the data -attribute is considered abstract and must always be explicitly implemented -in a compatible class. - -Attributes should not be defined in the body of a method by assignment -via ``self``. This restriction is a little arbitrary, but my idea is that -the protocol class implementation is often not shared by subtypes so -the interface should not depend on the default implementation. -This is more of a style than a technical issue, as a type checker could infer -attributes from assignment statements within methods as well. - -When using the ``...`` initializer, the ``...`` initializer might -leak into subclasses at runtime, which is unfortunate:: - - class A(Protocol): - x = ... # type: int - - class B(A): - def __init__(self) -> None: - self.x = 1 +There are two ways to define methods that are protocol members: - b = B() - print(b.x) # 1 - print(B.x) # Ellipsis +* Use the ``@abstractmethod`` decorator +* Leave the method body empty, i.e. it should be either a single ``pass`` + statement, a single ``...`` ellipsis literal, or a single + ``raise NotImplementedError`` statement. -If we'd use a None initializer things wouldn't be any better. -Maybe we can modify the metaclass to recognize ``...`` initializers and -translate them to something else. This needs to be documented, however. -Also, in this proposal there is no way to distinguish between class -and instance data attributes (ClassVar PEP 526). +A combination of both may be used if necessary for runtime purposes. +If the method body is not empty but decorated by the ``@abstractmethod`` +decorator, then the corresponding definition serves as a default +method implementation. If some or all parameters are not annotated +their types are assumed to be ``Any`` (see PEP 484). All other methods +defined in the protocol class are considered non-protocol members. +Example:: -I'm not sure sure what to do with ``__init__``. I guess a protocol could -provide a default implementation that could be used in explicit subclasses. + from typing import Protocol -"Private" variables on self, treat double underscores as private. + class Example(Protocol): + @abstractmethod + def first(self) -> int: # This is a protocol member + return 1 + def second(self) -> int: # This one also + ... + def third(self) -> int: # And this one too + raise NotImplementedError('Implement this!') + + def auxiliary(self) -> str: # This is NOT a protocol member + x = 1 + +Having an empty method body in a protocol type makes the method +implicitly abstract only statically, and there won't be any runtime checking. +The rationale for this is that most methods in a protocol will be abstract, +and having to always use ``@abstractmethod`` is a little verbose and ugly. +Leaving empty method body is the recommended way of defining protocol members +that don't have an implementation. Such syntax mirrors how methods are +defined in stub files. Abstract static methods, abstract class methods, +and abstract properties are equally supported. + +To defined a variable protocol member one should use the PEP 526 variable +annotations in the class body. Attributes defined in the body of a method +by assignment via ``self`` considered non-protocol members. The rationale for +this is that the protocol class implementation is often not shared by +subtypes, so the interface should not depend on the default implementation. +An exception from these rules is an attribute starting with double +underscores and not ending with double underscores always considered +non-protocol member. Examples:: + + from typing import Protocol, List + + class Template(Protocol): + name: str # This is a protocol member + value: int = 0 # This one too + __private: bool # This is NOT a protocol member + + def method(self) -> None: + self.temp: List[int] = [] # And this one neither + +To distinguish between protocol class variables and protocol instance +variables, the special ``ClassVar`` annotation should be used as specified +by PEP 526. Explicitly declaring implementation ----------------------------------- -When a class explicitly inherits from protocol, +I propose that most of the regular rules for classes still apply to +protocol classes (modulo a few exceptions that only apply to +protocol classes) i.e. when a class explicitly inherits from protocol, typechecker verifies that everything is OK. in this case normal methods are also inherited. @@ -452,6 +398,19 @@ Note that subclassing a protocol class would not turn the subclass into a protocol unless it also has Protocol as an explicit base class. I assume that we can use metaclass trickery to get this to work correctly. +For variants 1, 2 and 3, a type checker should probably always require +an explicit implementation to be defined in a subclass that explicitly +subclasses the protocol (see below for more about this), because +the implementations return ``None`` which is not a valid return type +for the method. For variants 4, 5 and 6, we can use +the provided implementation as a default implementation. +The default implementations won't be used if the subtype relationship +is implicit and only via structural subtyping -- the semantics of inheritance +won't be changed. + +In particular, a protocol could provide a default implementation of +``__init__`` that could be used in explicit subclasses. + Merging and extending protocols ------------------------------- @@ -462,8 +421,9 @@ having typing.Protocol as an immediate base class:: from typing import Sized, Protocol - class SizedAndCloseable(Sized, Protocol): - def close(self) -> None: ... + class SizedAndCloseable(Protocol, Sized): + def close(self) -> None: + ... Now the protocol ``SizedAndCloseable`` is a protocol with two methods, ``__len__`` and ``close``. Alternatively, we could have implemented it @@ -474,7 +434,7 @@ from an earlier example:: class SupportsClose(...): ... # Like above - class SizedAndCloseable(Sized, SupportsClose, Protocol): + class SizedAndCloseable(Protocol, Sized, SupportsClose): pass The two definitions of ``SizedAndClosable`` would be equivalent. @@ -491,34 +451,51 @@ must be protocols. A protocol can't extend a regular class. Generic and recursive protocols ------------------------------- -``Protocol[T, S, ...]`` as a shorthand for ``Protocol, Generic[T, S, ...]`` - Generic protocols are important. For example, ``SupportsAbs``, ``Iterable`` -and ``Iterator`` would be generic. We could define them like this, -similar to generic ABCs:: +and ``Iterator`` are generic protocols. They are defined similar to normal +non-protocol generic types:: T = TypeVar('T', covariant=True) class Iterable(Protocol[T]): - def __iter__(self) -> Iterator[T]: ... + def __iter__(self) -> Iterator[T]: + ... + +Note that ``Protocol[T, S, ...]`` is allowed as a shorthand for +``Protocol, Generic[T, S, ...]``. Recursive protocols are also supported. +Forward references to the protocol class names can be given as strings as +specified by PEP 484. Recursive protocols will be useful for representing +self-referential data structures like trees in an abstract fashion:: -Should we support recursive protocols? -Sure, why not. Just use the forward references. They might useful for -representing self-referential -data structures like trees in an abstract fashion, but I don't see them -used commonly in production code. + class Traversable(Protocol): + leafs: Iterable['Traversable'] Using protocols =============== -cannot be instantiated. - Subtyping relationships with other types ---------------------------------------- -Subtyping: issubtype(A, Proto) is _always_ structural; -issubtype(Proto, Nom) always False. +Protocols cannot be instantiated, so that there are no values with +protocol types. For variables and parameters with protocol types, subtyping +relationships are subject to the following rules: + +* issubtype(Proto, Nom) always False. +* issubtype(A, Proto) is _always_ structural; +* Edge case: for recursive protocols, structural subtyping is decided + positively for all situations that are type safe. For example:: + + class Tree(Generic[T]): + def __init__(self, value: T, + leafs: 'List[Tree[T]]') -> None: + self.value = value + self.leafs = leafs + + def walk(graph: Traversable) -> None: + ... + tree: Tree[float] = Tree(0, []) + walk(tree) # OK, 'Tree[float]' is a subtype of 'Traversable' ``Union[]`` and ``All[]`` @@ -533,9 +510,31 @@ although it is not a subclass of any of two. ``Type[]`` with protocols ------------------------- -``Type[]`` accepts only non-abstract (non-protocol?) classes +Variables and parameters annotated with ``Type[Proto]`` accept only concrete +(non-protocol) subtypes of ``Proto``. For example:: -copy all stuff from my PR + class Proto(Protocol): + def meth(self) -> int: + ... + class Concrete: + def meth(self) -> int: + return 42 + + def fun(cls: Type[Proto]) -> int: + return cls().meth() # OK + fun(Proto) # Error + fun(Concrete) # OK + +The same rule applies to variables:: + + var: Type[Proto] + var = Proto # Error + var = Concrete # OK + var().meth() # OK + +Assigning protocol class to a variable is allowed if it is not explicitly +typed, such assignment create type aliases. For non-protocol classes, +the behavior of ``Type[]`` is not changed. ``NewType()`` and type aliases @@ -587,14 +586,88 @@ Variable annotation syntax was added in Python 3.6. To define the protocols that have variable protocol members in earlier versions we propose ... +Similar to methods, there will be multiple valid ways of defining data +attributes (or properties). All of these will be valid:: + + class Foo(Protocol): + a = ... # type: int # Variant 1 + b = 0 # Variant 2 + + # Variant 3 + @property + def c(self) -> int: ... + + # Variant 4 + @property + def c(self) -> int: + return 0 + + # Variant 5 + @property + def d(self) -> int: + raise NotImplementedError + + # Variant 6 + @abstractproperty + def e(self) -> int: ... + + # Variant 7 + @abstractproperty + def f(self) -> int: pass + +Also, properties with setters can also be defined. The first three variants +would be the recommended ways of defining attributes or properties, +but similar to 3), the others are possible and may be useful for supporting +legacy code. + +When using an ... initializer, ``@abstractproperty`` or ``pass``/``...`` +as property body (and when the type does not include ``None``), the data +attribute is considered abstract and must always be explicitly implemented +in a compatible class. + +When using the ``...`` initializer, the ``...`` initializer might +leak into subclasses at runtime, which is unfortunate:: + + class A(Protocol): + x = ... # type: int + + class B(A): + def __init__(self) -> None: + self.x = 1 + + b = B() + print(b.x) # 1 + print(B.x) # Ellipsis + +If we'd use a None initializer things wouldn't be any better. +Maybe we can modify the metaclass to recognize ``...`` initializers and +translate them to something else. This needs to be documented, however. + ``typing`` module changes will be backported to earlier versions via the backport available on PyPI. -Runtime behavior of protocol classes -==================================== +Runtime implementation of protocol classes +========================================== + +Implementation details +---------------------- + +The runtime implementation could be done in pure Python without any +effects on core interpreter and standard library except from +``typing`` module: + +* Define class ``typing.Protocol`` similar to ``typing.Generic``. +* Implement metaclass functionality to detect whether a class is + a protocol or not. Add a class attribute ``__protocol__ = True`` + if that is the case. Verify that a protocol class only has protocol + base classes in the MRO (except for object). +* Implement ``@runtime`` class decorator that makes ``isinstance()`` behave + as for ``collections.abc`` by adding all abstract methods and all variables + from ``__annotations__`` to ``__subclsshook__()``. +* In the backported version of ``typing``, translate ``...`` class attribute + values to abstract properties. -At runtime normal ABCs, cannot be instantiated. Changes in typing module ------------------------ @@ -615,12 +688,19 @@ I think that at least these classes in typing should be protocols: * ``ItemsView`` (and other ``*View`` classes) * ``AsyncIterable``, ``AsyncIterator`` * ``Awaitable`` +* ``Callable`` * ``ContextManager`` Most of these classes are small and conceptually simple. It's easy to see which of these protocols a class implements from the presence of a small number of magic methods, which immediately suggest a protocol. +Practically, few changes will be needed in ``typing`` since some of these +classes already behave this way at runtime. Most of these will need to be +updated only in the corresponding ``typeshed`` stub. + +At runtime normal ABCs, cannot be instantiated. + I'm not sure about other classes such as ``Sequence``, ``Set`` and ``IO``. I believe that these are sufficiently complex that it makes sense to require code to be explicit about them, as there would not be any sufficiently obvious @@ -650,23 +730,9 @@ attributes can't be initialized in ways that are not visible to introspection of attributes wouldn't be visible at runtime, so this would necessarily be somewhat limited. - -Implementation details ----------------------- - -We'd need to implement at least the following things: - -* Define class ``Protocol`` (this could be simple, and would be similar - to ``Generic``). -* Implement metaclass functionality to detect whether a class is - a protocol or not. Maybe add a class attribute such as ``__protocol__ = True`` - if that's the case. Verify that a protocol class only has protocol - base classes in the MRO (except for object). -* Implement ``@runtime`` class decorator that makes ``isinstance`` behave as - for ``collections.abc`` by adding all abstract methods and all variables - from ``__annotations__`` to ``__subclsshook__``. -* In the backported version, translate ``...`` class attribute values to - abstract properties. +``isinstance()`` and ``issubclass`` will _always_ fail with ``TypeError`` for +subscripted generic protocols, since a reliable answer could not be given +at runtime in this case. Rejected and postponed ideas From 82258d5e570ffc503054a0eeb2edc03ca53eb706 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Tue, 7 Mar 2017 18:14:08 +0100 Subject: [PATCH 13/37] Explicitly declaring implementation and other changes --- pep-05xx.txt | 162 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 106 insertions(+), 56 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index d26c11ce277..38d8a602012 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -360,56 +360,76 @@ non-protocol member. Examples:: def method(self) -> None: self.temp: List[int] = [] # And this one neither -To distinguish between protocol class variables and protocol instance -variables, the special ``ClassVar`` annotation should be used as specified -by PEP 526. +Note that unannotated variables are always assumed to have type ``Any``, +so that definition ``x = 0`` in a protocol body will create a protocol member +``x`` with type ``Any``, *not* ``int``. To distinguish between protocol class +variables and protocol instance variables, the special ``ClassVar`` +annotation should be used as specified by PEP 526. Explicitly declaring implementation ----------------------------------- -I propose that most of the regular rules for classes still apply to -protocol classes (modulo a few exceptions that only apply to -protocol classes) i.e. when a class explicitly inherits from protocol, -typechecker verifies that everything is OK. -in this case normal methods are also inherited. - -when a protocol is defined, every method is resolved using -normal MRO procedure. typechecker verifies that everything is OK. -if concrete overrides abstract, then thats it -- it is no more abstract -but not vice versa -set of all remaining abstracts defines the protocol - -I propose that protocols can be used as regular base classes. -I can see at least three things that support this decision. -First, a protocol class could define default implementations for some methods -(``typing.Sequence`` would be an example if we decide to turn it -into a protocol). Second, we want a way of statically enforcing that a class -actually implements a protocol correctly (but there are other ways -to achieve this effect -- see below for alternatives). -Third, this makes it possible to turn an existing ABC into a protocol and -just have things (mostly) work. -This would be important for the existing ABCs in typing what we may want -to change into protocols (see point 8 for more about this). -The general philosophy would be that Protocols are mostly like regular ABCs, -but a static type checker will handle them somewhat specially. +To explicitly declare that a certain class implements a given protocols, they +can be used as regular base classes. In this case: + +* A protocol class could define default implementations for some methods + (``typing.Sequence`` is an example with useful default methods). + In particular, a protocol could provide a default implementation + of ``__init__`` that could be used in explicit subclasses. + The default implementations will not be used if the subtype relationship + is implicit and only via structural subtyping -- the semantics of + inheritance is not changed. + +* Second, type checker will statically enforce that a class + actually implements a protocol correctly. In particular, type checker will + always require members without default implementation, to be defined in + a subclass that explicitly subclasses the protocol. + +The general philosophy is that protocols are mostly like regular ABCs, +but a static type checker will handle them specially. +In particular, a class can explicitly inherit from multiple protocols. +In this case every method is resolved using normal MRO procedure and type +checker verifies that all subtyping is correct. Note that a concrete method +with appropriate type signature can override abstract but not vice versa. +Examples:: + + from typing import Protocol, Tuple + + class PColor(Protocol): + rgb: Tuple[int, int, int] + def description(self) -> str: + ... + def simple_description(self) -> str: # This is non-protocol member + rerun 'nice color' + @abstractmethod + def to_bytes(self) -> bytes: + return b'' -Note that subclassing a protocol class would not turn the subclass into -a protocol unless it also has Protocol as an explicit base class. -I assume that we can use metaclass trickery to get this to work correctly. + class SimpleColor(PColor): + def __init__(se;f, red: int, green: int, blue: str) -> None: + self.rgb = red, green, blue # Error, last element is 'str' not 'int' + def description(self) -> str: + return super().simple_description() + # Error, 'to_bytes' is not implemented. + # Default implementation is not implicitly used. -For variants 1, 2 and 3, a type checker should probably always require -an explicit implementation to be defined in a subclass that explicitly -subclasses the protocol (see below for more about this), because -the implementations return ``None`` which is not a valid return type -for the method. For variants 4, 5 and 6, we can use -the provided implementation as a default implementation. -The default implementations won't be used if the subtype relationship -is implicit and only via structural subtyping -- the semantics of inheritance -won't be changed. + class A(Protocol): + def m(self) -> None: ... + def foo(self) -> int: ... + def bar(self) -> str: ... + class B(Protocol): + def m(self) -> None: print('hi') -In particular, a protocol could provide a default implementation of -``__init__`` that could be used in explicit subclasses. + class C(A, B): # Error, abstract 'A.m' can't override non-abstract 'B.m' + ... + class C(B, A): # OK + def bar(self) -> str: + return 'bar' + # Error, 'foo' is still not implemented. + +Note that subclassing a protocol class would not turn the subclass into +a protocol unless it also has ``Protocol`` as an explicit base class. Merging and extending protocols @@ -481,8 +501,10 @@ Protocols cannot be instantiated, so that there are no values with protocol types. For variables and parameters with protocol types, subtyping relationships are subject to the following rules: -* issubtype(Proto, Nom) always False. -* issubtype(A, Proto) is _always_ structural; +* A protocol is never a subtype of a concrete type. +* A concrete type or a protocol ``X`` is a subtype of another protocol ``P`` + if and only if ``X`` implements all protocol members of ``P``. In other + words a subtyping with respect to a protocol is always structural. * Edge case: for recursive protocols, structural subtyping is decided positively for all situations that are type safe. For example:: @@ -497,14 +519,50 @@ relationships are subject to the following rules: tree: Tree[float] = Tree(0, []) walk(tree) # OK, 'Tree[float]' is a subtype of 'Traversable' +Generic protocol types follow the same rules of variance as non-protocol +types. Protocols can be used in the all special type constructors provided +by ``typing`` module and follow corresponding subtyping rules. For example, +protocol ``PInt`` is a subtype of ``Union[PInt, int]``. + ``Union[]`` and ``All[]`` ------------------------- -Union merges protocols: -open() -> Union[int, str] conforms to -Union[Proto.open() -> int, Proto.open() -> str], -although it is not a subclass of any of two. +``Union`` of protocol classes behaves the same way as for non-protocol +classes. For example:: + + from typing import Union, Optional, Protocol + + class Exitable(Protocol): + def exit(self) -> int: + ... + class Quitable(Protocol): + def quit(self) -> Optional[int]: + ... + + def finish(task: Union[Exitable, Quitable]) -> int: + ... + class GoodJob: + ... + def quit(self) -> int: + return 0 + finish(GoodJob()) # OK + +In addition, we propose to add another special type construct +``All`` that represents intersection types. Although for normal types +it is not very useful, there are many situation where a variable should +implement more than one protocol. Annotation by ``All[Proto1, Proto2, ...]`` +means that a given variable or parameter must implement all protocols +``Ptoro1``, ``Proto2``, etc either implicitly or explicitly. Example:: + + from typing import Sequence, Hashable, All + + def cached_func(args: All[Sequence[float], Hashable]) -> float: + ... + cached_func((1, 2, 3)) # OK, tuple is hashable and sequence + +The interaction between union and intersection types is specified by PEP 483, +and basically reflects the corresponding interactions for sets. ``Type[]`` with protocols @@ -590,28 +648,20 @@ Similar to methods, there will be multiple valid ways of defining data attributes (or properties). All of these will be valid:: class Foo(Protocol): - a = ... # type: int # Variant 1 - b = 0 # Variant 2 - - # Variant 3 @property def c(self) -> int: ... - # Variant 4 @property def c(self) -> int: return 0 - # Variant 5 @property def d(self) -> int: raise NotImplementedError - # Variant 6 @abstractproperty def e(self) -> int: ... - # Variant 7 @abstractproperty def f(self) -> int: pass From 5d9fb7cfe0a4b6a74d1d390616142775d484c669 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Tue, 7 Mar 2017 19:45:03 +0100 Subject: [PATCH 14/37] More polishing --- pep-05xx.txt | 138 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 88 insertions(+), 50 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index 38d8a602012..9a40059b2c7 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -354,7 +354,7 @@ non-protocol member. Examples:: class Template(Protocol): name: str # This is a protocol member - value: int = 0 # This one too + value: int = 0 # This one too (with default) __private: bool # This is NOT a protocol member def method(self) -> None: @@ -373,57 +373,82 @@ Explicitly declaring implementation To explicitly declare that a certain class implements a given protocols, they can be used as regular base classes. In this case: -* A protocol class could define default implementations for some methods +* A class could use implementations of some methods in protocols, both + protocol members with default implementations and non-protocol members (``typing.Sequence`` is an example with useful default methods). - In particular, a protocol could provide a default implementation - of ``__init__`` that could be used in explicit subclasses. + In particular, a protocol could provide a default implementation of + ``__init__`` that could be used in explicit subclasses. The default implementations will not be used if the subtype relationship is implicit and only via structural subtyping -- the semantics of - inheritance is not changed. + inheritance is not changed. Examples:: + + from typing import Protocol + + class PColor(Protocol): + def description(self) -> str: + ... + def simple_description(self) -> str: # this is a non-protocol member + rerun 'nice color' + @abstractmethod + def add_int(self, x: int) -> int: # protocol member with default + return 42 + + class SimpleColor(PColor): + def description(self) -> str: + return super().simple_description() + def add_int(self, x: int) -> int: + return super().add_int(x) + x + + class BadColor: + def description(self) -> str: + return 'not so nice' + + simple: SimpleColor + def represent(c: PColor) -> None: + ... + represent(simple) # OK + represent(BadColor()) # Error, 'to_int' is not implemented implicitly * Second, type checker will statically enforce that a class actually implements a protocol correctly. In particular, type checker will - always require members without default implementation, to be defined in - a subclass that explicitly subclasses the protocol. + always require members without default implementation to be defined in + a subclass that explicitly subclasses the protocol:: -The general philosophy is that protocols are mostly like regular ABCs, -but a static type checker will handle them specially. -In particular, a class can explicitly inherit from multiple protocols. -In this case every method is resolved using normal MRO procedure and type -checker verifies that all subtyping is correct. Note that a concrete method -with appropriate type signature can override abstract but not vice versa. -Examples:: + from typing import Protocol, Tuple - from typing import Protocol, Tuple + class RGB(Protocol): + rgb: Tuple[int, int, int] + def intensity(self) -> int: + ... + @abstractmethod + def to_bytes(self) -> bytes: + return b'' - class PColor(Protocol): - rgb: Tuple[int, int, int] - def description(self) -> str: - ... - def simple_description(self) -> str: # This is non-protocol member - rerun 'nice color' - @abstractmethod - def to_bytes(self) -> bytes: - return b'' + class Point(RGB): + def __init__(self, red: int, green: int, blue: str) -> None: + self.rgb = red, green, blue # Error, 'blue' must be 'int' + # Error, 'intensity' is not defined! - class SimpleColor(PColor): - def __init__(se;f, red: int, green: int, blue: str) -> None: - self.rgb = red, green, blue # Error, last element is 'str' not 'int' - def description(self) -> str: - return super().simple_description() - # Error, 'to_bytes' is not implemented. - # Default implementation is not implicitly used. + # Note that 'to_bytes' is not required, this is safe + # since the default implementation is actually in the MRO. + +The general philosophy is that protocols are mostly like regular ABCs, +but a static type checker will handle them specially. +In particular, a class can explicitly inherit from multiple protocols. +In this case methods are resolved using normal MRO and type checker verifies +that all subtyping are correct. Note that a concrete method with appropriate +type signature can override protocol method but not vice versa. Examples:: class A(Protocol): def m(self) -> None: ... def foo(self) -> int: ... - def bar(self) -> str: ... class B(Protocol): - def m(self) -> None: print('hi') + def m(self) -> None: print('Hi') + def bar(self) -> str: ... - class C(A, B): # Error, abstract 'A.m' can't override non-abstract 'B.m' + class C(A, B): # Fails typecheck, protocol member 'A.m' can't override 'B.m' ... - class C(B, A): # OK + class D(B, A): # OK def bar(self) -> str: return 'bar' # Error, 'foo' is still not implemented. @@ -435,9 +460,9 @@ a protocol unless it also has ``Protocol`` as an explicit base class. Merging and extending protocols ------------------------------- -I think that we should support subprotocols. A subprotocol can be defined +Subprotocols are also supported. A subprotocol can be defined by having both one or more protocols as the explicit base classes and also -having typing.Protocol as an immediate base class:: +having ``typing.Protocol`` as an immediate base class:: from typing import Sized, Protocol @@ -446,9 +471,9 @@ having typing.Protocol as an immediate base class:: ... Now the protocol ``SizedAndCloseable`` is a protocol with two methods, -``__len__`` and ``close``. Alternatively, we could have implemented it -like this, assuming the existence of ``SupportsClose`` -from an earlier example:: +``__len__`` and ``close``. Alternatively, one can implement it +like this, assuming the existence of ``SupportsClose`` from +the previous example:: from typing import Sized @@ -457,12 +482,12 @@ from an earlier example:: class SizedAndCloseable(Protocol, Sized, SupportsClose): pass -The two definitions of ``SizedAndClosable`` would be equivalent. -Subclass relationships between protocols aren't meaningful when -considering subtyping, as we only use structural compatibility -as the criterion, not the MRO. +The two definitions of ``SizedAndClosable`` are equivalent. +Subclass relationships between protocols are not meaningful when +considering subtyping, since the structural compatibility is +the criterion, not the MRO. -If we omit ``Protocol`` in the base class list, this would be regular +If one omits ``Protocol`` in the base class list, this would be regular (non-protocol) class that must implement ``Sized``. If ``Protocol`` is included in the base class list, all the other base classes must be protocols. A protocol can't extend a regular class. @@ -599,11 +624,24 @@ the behavior of ``Type[]`` is not changed. ------------------------------ Protocols are essentially anonymous. To emphasize this point, static type -checkers might refuse the protocol classes inside ``NewType(...)`` to avoid an -illusion that a distinct type is provided. +checkers might refuse the protocol classes inside ``NewType()`` to avoid an +illusion that a distinct type is provided:: + + form typing import NewType , Protocol, Iterator + + class Id(Protocol): + code: int + secrets: Iterator[bytes] + + UserId = NewType('UserId', Id) # Error, can't provide distinct type On the contrary, type aliases are fully supported, including generic type -aliases. +aliases:: + + from typing import TypeVar, Reversible, Iterable, Sized + + T = TypeVar('T') + CompatReversible = Union[Reversible[T], All[Iterable[T], Sized]] ``@runtime`` decorator and narrowing types by ``isinstance()`` @@ -798,8 +836,8 @@ a bad idea and classes should need to be explicitly marked as protocols, as shown in this proposal: * Protocols don't have some properties of regular classes. In particular, - ``isinstance`` is not well-defined for protocols, whereas it's well-defined - (and pretty commonly used) for regular classes. + ``isinstance`` is not always well-defined for protocols, whereas it's + well-defined (and pretty commonly used) for regular classes. * Protocol classes should generally not have (many) method implementations, as they describe an interface, not an implementation. Most classes have many implementations, making them bad protocol classes. From a6e6d9ed6ca7af38258a79e0463c53a22469ae11 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Tue, 7 Mar 2017 20:32:00 +0100 Subject: [PATCH 15/37] Edit rejected/postponed ideas --- pep-05xx.txt | 51 ++++++++++++++++++++++++++------------------------- 1 file changed, 26 insertions(+), 25 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index 9a40059b2c7..a1bfe1e666e 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -836,18 +836,18 @@ a bad idea and classes should need to be explicitly marked as protocols, as shown in this proposal: * Protocols don't have some properties of regular classes. In particular, - ``isinstance`` is not always well-defined for protocols, whereas it's - well-defined (and pretty commonly used) for regular classes. -* Protocol classes should generally not have (many) method implementations, + ``isinstance()`` is not always well-defined for protocols, whereas it is + well-defined (and quite commonly used) for regular classes. +* Protocol classes should generally not have many method implementations, as they describe an interface, not an implementation. Most classes have many implementations, making them bad protocol classes. * Experience suggests that many classes are not practical as protocols anyway, mainly because their interfaces are too large, complex or implementation-oriented (for example, they may include de facto private attributes and methods without a ``__`` prefix). - Most actually useful protocols in existing Python code seem to be implicit. - The ABCs in typing and ``collections.abc`` are a kind-of exception, but - even they are pretty recent additions to Python and most programmers +* Most actually useful protocols in existing Python code seem to be implicit. + The ABCs in ``typing`` and ``collections.abc`` are rather an exception, but + even they are recent additions to Python and most programmers do not use them yet. @@ -856,48 +856,49 @@ Support optional protocol members We can come up with examples where it would be handy to be able to say that a method or data attribute does not need to be present in a class -implementing a protocol, but if it's present, it must conform to a specific +implementing a protocol, but if it is present, it must conform to a specific signature or type. One could use a ``hasattr`` check to determine whether they can use the attribute on a particular instance. -In the interest of simplicity, let's not support optional methods -or attributes. We can always revisit this later if there is an actual need. -The current realistic potential use cases for protocols that I've seen don't -require these. However, other languages have similar features and apparently -they are pretty commonly used. If I remember correctly, at least TypeScript -and Objective-C support a similar concept. +Other languages, such as TypeScript have similar features and +apparently they are pretty commonly used. The current realistic potential +use cases for protocols in Python don't require these. In the interest +of simplicity, we propose to not support optional methods or attributes. +We can always revisit this later if there is an actual need. Should protocols be interoperable with other approaches? -------------------------------------------------------- -The protocols as described here are basically a small extension to -the existing concept of ABCs. I argue that this is the way they should +The protocols as described here are basically a minimal extension to +the existing concept of ABCs. We argue that this is the way they should be understood, instead of as something that replaces Zope interfaces, -for example. +for example. Attempting such interoperabilities will significantly +complicate both the concept and the implementation. Use assignments to check explicitly that a class implements a protocol ---------------------------------------------------------------------- -I've seen a similar pattern in some Go code that I've reviewed [golang]. +In Go language [golang] the explicit checks for implementation are performed +via dummy assignments. Such way is also possible with the current proposal. Example:: class A: def __len__(self) -> float: return ... - _ = A() # type: Sized # Error: A.__len__ doesn't conform to 'Sized' - # (Incompatible return type 'float') + _: Sized = A() # Error: A.__len__ doesn't conform to 'Sized' + # (Incompatible return type 'float') -I don't much care above the above example, as it moves the check away from +We see such approach as not very clear, as it moves the check away from the class definition and it almost requires a comment as otherwise -the code probably wouldn't make any sense to an average reader --- it looks like dead code. Besides, in the simplest form it requires us -to construct an instance of A which could problematic if this requires +the code probably would not make any sense to an average reader +-- it looks like dead code. Besides, in the simplest form it requires one +to construct an instance of ``A`` which could be problematic if this requires accessing or allocating some resources such as files or sockets. -We could work around the latter by using a cast, for example, -but then the code would be really ugly. +We could work around the latter by using a cast, for example, but then +the code would be ugly. Therefore we discourage the use of such pattern. References From 3175013410a9e13683e509da10b4bff349693991 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Tue, 7 Mar 2017 23:57:27 +0100 Subject: [PATCH 16/37] Runtime things, reorder links --- pep-05xx.txt | 104 +++++++++++++++++++++++---------------------------- 1 file changed, 46 insertions(+), 58 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index a1bfe1e666e..e86d7950abe 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -742,7 +742,7 @@ Implementation details ---------------------- The runtime implementation could be done in pure Python without any -effects on core interpreter and standard library except from +effects on core interpreter and standard library except in ``typing`` module: * Define class ``typing.Protocol`` similar to ``typing.Generic``. @@ -760,8 +760,7 @@ effects on core interpreter and standard library except from Changes in typing module ------------------------ -I think that at least these classes in typing should be protocols: -(probably all abstract change, concrete stay) +The following classes in ``typing`` module will be protocols: * ``Hashable`` * ``SupportsAbs`` (and other ``Supports*`` classes) @@ -779,48 +778,40 @@ I think that at least these classes in typing should be protocols: * ``Callable`` * ``ContextManager`` -Most of these classes are small and conceptually simple. It's easy to see -which of these protocols a class implements from the presence of a small -number of magic methods, which immediately suggest a protocol. - +Most of these classes are small and conceptually simple. It is easy to see +what are the methods these protocols implement, and immediately recognize +the corresponding runtime protocol counterpart. Practically, few changes will be needed in ``typing`` since some of these -classes already behave this way at runtime. Most of these will need to be -updated only in the corresponding ``typeshed`` stub. - -At runtime normal ABCs, cannot be instantiated. +classes already behave the necessary way at runtime. Most of these will need +to be updated only in the corresponding ``typeshed`` stub. -I'm not sure about other classes such as ``Sequence``, ``Set`` and ``IO``. -I believe that these are sufficiently complex that it makes sense to require -code to be explicit about them, as there would not be any sufficiently obvious -and small set of 'marker' methods that tell that a class implements -this protocol. Also, it's too easy to leave some part of -the protocol/interface unimplemented by accident, and explicitly marking -the subclass relationship allows type checkers to pinpoint the missing -implementations -- or they can be inherited from the ABC, -in case that is has default implementations. -So I'd currently vote against making these classes protocols. +All other concrete generic classes such as ``List``, ``Set``, ``IO``, +``Deque``, etc are sufficiently complex that it makes sense to keep +them non-protocols (i.e. require code to be explicit about them). Also, it is +too easy to leave some methods unimplemented by accident, and explicitly +marking the subclass relationship allows type checkers to pinpoint the missing +implementations. Introspection ------------- -Current typing is already good. - -The existing introspection machinery (``dir``, etc.) could be used with -protocols, but typing would not include an implementation of additional -introspection or runtime type checking capabilities for protocols. - -As all attributes need to be defined in the class body based on this proposal, -protocol classes would have better support for introspection than -regular classes where attributes can be defined implicitly -- protocol -attributes can't be initialized in ways that are not visible to introspection -(using ``setattr``, assignment via self, etc.). Still, some things like types -of attributes wouldn't be visible at runtime, so this would necessarily be -somewhat limited. - -``isinstance()`` and ``issubclass`` will _always_ fail with ``TypeError`` for +The existing class introspection machinery (``dir``, ``__annotations__`` etc) +can be used with protocols, in addition all introspection tools implemented +in ``typing`` module will support protocols. Since all attributes need to be +defined in the class body based on this proposal, protocol classes will have +even better perspective for introspection than regular classes where +attributes can be defined implicitly -- protocol attributes can't be +initialized in ways that are not visible to introspection +(using ``setattr()``, assignment via ``self``, etc). Still, some things like +types of attributes will not be visible at runtime in Python 3.5 and earlier, +but this looks like a reasonable limitation. + +There will be only a limited support of ``isinstance()`` and ``issubclass()`` +as discussed above (these will _always_ fail with ``TypeError`` for subscripted generic protocols, since a reliable answer could not be given -at runtime in this case. +at runtime in this case). But together with other introspection tools this +give a reasonable perspective for runtime type checking tools. Rejected and postponed ideas @@ -857,7 +848,7 @@ Support optional protocol members We can come up with examples where it would be handy to be able to say that a method or data attribute does not need to be present in a class implementing a protocol, but if it is present, it must conform to a specific -signature or type. One could use a ``hasattr`` check to determine whether +signature or type. One could use a ``hasattr()`` check to determine whether they can use the attribute on a particular instance. Other languages, such as TypeScript have similar features and @@ -904,36 +895,21 @@ the code would be ugly. Therefore we discourage the use of such pattern. References ========== -.. [mypy] - http://mypy-lang.org +.. [pep484] + https://www.python.org/dev/peps/pep-0484/ + +.. [typing] + https://docs.python.org/3/library/typing.html .. [wiki-structural-subtyping] https://en.wikipedia.org/wiki/Structural_type_system -.. [typeshed] - https://github.com/python/typeshed/ - -.. [issues] - https://github.com/python/typing/issues - -.. [pep482] - https://www.python.org/dev/peps/pep-0482/ - .. [pep483] https://www.python.org/dev/peps/pep-0483/ -.. [pep484] - https://www.python.org/dev/peps/pep-0484/ - .. [pep526] https://www.python.org/dev/peps/pep-0526/ -.. [typescript] - https://www.typescriptlang.org/docs/handbook/interfaces.html - -.. [golang] - https://golang.org/doc/effective_go.html#interfaces_and_types - .. [zope-interfaces] https://zopeinterface.readthedocs.io/en/latest/ @@ -943,6 +919,18 @@ References .. [collections-abc] https://docs.python.org/3/library/collections.abc.html +.. [typescript] + https://www.typescriptlang.org/docs/handbook/interfaces.html + +.. [golang] + https://golang.org/doc/effective_go.html#interfaces_and_types + +.. [typeshed] + https://github.com/python/typeshed/ + +.. [mypy] + http://github.com/python/mypy/ + Copyright ========= From cbff6695cd55eedec4ea6191ec125afe966a308a Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Wed, 8 Mar 2017 00:49:36 +0100 Subject: [PATCH 17/37] Runtime decorator --- pep-05xx.txt | 60 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 38 insertions(+), 22 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index e86d7950abe..49492109d02 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -647,32 +647,48 @@ aliases:: ``@runtime`` decorator and narrowing types by ``isinstance()`` -------------------------------------------------------------- -``@runtime`` decorator automatically adds a ``__subclasshook__`` -(like in ``collections.abc``), typechecker ensures no empty bodies. - -``isinstance(x, Proto)`` and ``issublclass(C, Proto)`` -narrow type if defined with ``@runtime``; ``isinstance(x, Proto[int])`` -always fails. - -``Callable[..., Any]`` for methods (note arbitrary number of args). - -We shouldn't implement any magic ``isinstance()`` machinery, as performing -a runtime compatibility check is generally difficult: we might want to verify -argument counts to methods, names of arguments and even argument types, -depending the kind of protocol we are talking about, -but sometimes we wouldn't care about these, or we'd only care about some -of these things. - -My preferred semantics would be to make ``isinstance()`` fail by default -for protocol types. This would be in the spirit of duck typing -- protocols +The default semantics is ``isinstance()`` and ``issubclass()`` fail by default +for protocol types. This is in the spirit of duck typing -- protocols basically would be used to model duck typing statically, not explicitly at runtime. However, it should be possible for protocol types to implement custom -``isinstance`` behavior when this would make sense, similar to how -``Iterable`` and other ABCs in ``collections.abc`` and typing already do it, -but this should be specific to these particular classes. -We need this fallback option anyway for backward compatibility. +instance and class checks when this makes sense, similar to how ``Iterable`` +and other ABCs in ``collections.abc`` and ``typing`` already do it, +but this is only limited to non-generic and to unsubscripted generic protocols +(statically ``Iterable`` is equivalent to ``Iterable[Any]`). +The ``typing`` module will define a special ``@runtime`` class decorator that +automatically adds a ``__subclasshook__`` that provides the same semantics +for class and instance checks as for ``collections.abc`` classes:: + + from typing import runtime, Protocol + + @runtime + class Closeable(Protocol): + @abstarctmethod + def close(self): + ... + + assert isinstance(open('some/file'), Closeable) + +A static type checker will ensure that this decorator is only applied to +classes where all protocol members are defined by ``@abstractmethod`` for +methods and by a variable annotation for class and instance variables. +Static type checkers will understand ``isinstance(x, Proto)`` and +``issublclass(C, Proto)`` for protocols decorated with such decorator +(as they already do for ``Iterable`` etc). Static type checkers will +narrow types after such checks by the type erased ``Proto`` (i.e. with all +variables having type ``Any`` and all methods having type +``Callable[..., Any]``). Note that ``isinstance(x, Proto[int])`` etc will +always fail in agreement with PEP 484. Examples:: + + from typing import Iterable, Iterator, Sequence + + def process(items: Iterable[int]) -> None: + if isinstance(items, Iterator): + # 'items' have type 'Iterator[int]' here + elif isinstance(items, Sequence[int]): + # Error! Can't use 'isinstance()' with subscripted protocols Using protocols in Python 3.5 and earlier From dfccd065e136d300a7aa3240e65c0504eb30a6b8 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Wed, 8 Mar 2017 23:12:24 +0100 Subject: [PATCH 18/37] Backward compatible part and last bits --- pep-05xx.txt | 59 ++++++++++++++++++++-------------------------------- 1 file changed, 22 insertions(+), 37 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index 49492109d02..e1e6077f667 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -407,7 +407,7 @@ can be used as regular base classes. In this case: def represent(c: PColor) -> None: ... represent(simple) # OK - represent(BadColor()) # Error, 'to_int' is not implemented implicitly + represent(BadColor()) # Error, 'add_int' is not implemented implicitly * Second, type checker will statically enforce that a class actually implements a protocol correctly. In particular, type checker will @@ -451,7 +451,7 @@ type signature can override protocol method but not vice versa. Examples:: class D(B, A): # OK def bar(self) -> str: return 'bar' - # Error, 'foo' is still not implemented. + # Error, 'foo' is still not implemented (although 'm' is not needed). Note that subclassing a protocol class would not turn the subclass into a protocol unless it also has ``Protocol`` as an explicit base class. @@ -679,7 +679,7 @@ Static type checkers will understand ``isinstance(x, Proto)`` and (as they already do for ``Iterable`` etc). Static type checkers will narrow types after such checks by the type erased ``Proto`` (i.e. with all variables having type ``Any`` and all methods having type -``Callable[..., Any]``). Note that ``isinstance(x, Proto[int])`` etc will +``Callable[..., Any]``). Note that ``isinstance(x, Proto[int])`` etc. will always fail in agreement with PEP 484. Examples:: from typing import Iterable, Iterator, Sequence @@ -694,43 +694,30 @@ always fail in agreement with PEP 484. Examples:: Using protocols in Python 3.5 and earlier ========================================= -Variable annotation syntax was added in Python 3.6. To define the -protocols that have variable protocol members in earlier versions -we propose ... - -Similar to methods, there will be multiple valid ways of defining data -attributes (or properties). All of these will be valid:: +Variable annotation syntax was added in Python 3.6, so that the syntax +for defining variable protocol members proposed in section ... can't be used. +To define these in earlier versions of Python we propose to either use type +comments (and ellipsis literals if default value is not provided), or use +properties if it is necessary to indicate abstract status at runtime. +All of these will be valid:: class Foo(Protocol): + + x = ... # type: int + s = 'abc' # type: str # Protocol instance variable with default. + @property def c(self) -> int: ... @property - def c(self) -> int: + def c(self) -> int: # Default value can be provided for property. return 0 - @property - def d(self) -> int: - raise NotImplementedError - @abstractproperty def e(self) -> int: ... - @abstractproperty - def f(self) -> int: pass - -Also, properties with setters can also be defined. The first three variants -would be the recommended ways of defining attributes or properties, -but similar to 3), the others are possible and may be useful for supporting -legacy code. - -When using an ... initializer, ``@abstractproperty`` or ``pass``/``...`` -as property body (and when the type does not include ``None``), the data -attribute is considered abstract and must always be explicitly implemented -in a compatible class. - -When using the ``...`` initializer, the ``...`` initializer might -leak into subclasses at runtime, which is unfortunate:: +When using the ellipsis ``...`` initializer, it can leak into subclasses at +runtime, which is unfortunate:: class A(Protocol): x = ... # type: int @@ -743,12 +730,10 @@ leak into subclasses at runtime, which is unfortunate:: print(b.x) # 1 print(B.x) # Ellipsis -If we'd use a None initializer things wouldn't be any better. -Maybe we can modify the metaclass to recognize ``...`` initializers and -translate them to something else. This needs to be documented, however. - -``typing`` module changes will be backported to earlier versions -via the backport available on PyPI. +For this reason we propose to make ``Protocol`` metaclass recognize ``...`` +initializers and translate them to a special property that can't be read. +The ``typing`` module changes proposed in this PEP will be also backported to +earlier versions via the backport currently available on PyPI. Runtime implementation of protocol classes @@ -874,8 +859,8 @@ of simplicity, we propose to not support optional methods or attributes. We can always revisit this later if there is an actual need. -Should protocols be interoperable with other approaches? --------------------------------------------------------- +Make protocols interoperable with other approaches +-------------------------------------------------- The protocols as described here are basically a minimal extension to the existing concept of ABCs. We argue that this is the way they should From 60f4d52452b72ab67608d16cb80d4edebedd80eb Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Thu, 9 Mar 2017 08:54:15 +0100 Subject: [PATCH 19/37] Some clarifications --- pep-05xx.txt | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index e1e6077f667..b543e474b11 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -253,7 +253,7 @@ is not included in the MRO, the class is an *implicit* subclass of the protocol. The attributes (variables and methods) of a protocol that are mandatory -for other class in order to be considered a subtype are called +for other class in order to be considered a structural subtype are called protocol members. Other members defined in a protocol class are called non-protocol members. @@ -341,7 +341,7 @@ that don't have an implementation. Such syntax mirrors how methods are defined in stub files. Abstract static methods, abstract class methods, and abstract properties are equally supported. -To defined a variable protocol member one should use the PEP 526 variable +To defined a protocol variable one should use the PEP 526 variable annotations in the class body. Attributes defined in the body of a method by assignment via ``self`` considered non-protocol members. The rationale for this is that the protocol class implementation is often not shared by @@ -390,7 +390,7 @@ can be used as regular base classes. In this case: def simple_description(self) -> str: # this is a non-protocol member rerun 'nice color' @abstractmethod - def add_int(self, x: int) -> int: # protocol member with default + def add_int(self, x: int) -> int: # protocol method with default return 42 class SimpleColor(PColor): @@ -695,7 +695,7 @@ Using protocols in Python 3.5 and earlier ========================================= Variable annotation syntax was added in Python 3.6, so that the syntax -for defining variable protocol members proposed in section ... can't be used. +for defining protocol variables proposed in section ... can't be used. To define these in earlier versions of Python we propose to either use type comments (and ellipsis literals if default value is not provided), or use properties if it is necessary to indicate abstract status at runtime. @@ -709,12 +709,13 @@ All of these will be valid:: @property def c(self) -> int: ... - @property - def c(self) -> int: # Default value can be provided for property. + @abstractproperty + def c(self) -> int: # Default value can be provided for property. return 0 - @abstractproperty - def e(self) -> int: ... + @property + def e(self) -> int: # This is not a protocol member. + return 0 When using the ellipsis ``...`` initializer, it can leak into subclasses at runtime, which is unfortunate:: From 60e7f7fcd9adac85825535ea0ac5289f221fa3de Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Thu, 9 Mar 2017 09:18:29 +0100 Subject: [PATCH 20/37] Add links in text --- pep-05xx.txt | 82 +++++++++++++++++++++++++++------------------------- 1 file changed, 43 insertions(+), 39 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index b543e474b11..69a4e27b75c 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -24,11 +24,11 @@ for *structural* subtyping (static duck typing). Rationale and Goals =================== -Currently, PEP 484 and ``typing`` module define abstract base classes for -several common Python protocols such as ``Iterable`` and ``Sized``. -The problem with them is that a class has to be explicitly marked to support -them, which is arguably unpythonic and unlike what one would normally do in -idiomatic dynamically typed Python code. For example, +Currently, PEP 484 [pep484]_ and ``typing`` module [typing]_ define abstract +base classes for several common Python protocols such as ``Iterable`` and +``Sized``. The problem with them is that a class has to be explicitly marked +to support them, which is arguably unpythonic and unlike what one would +normally do in idiomatic dynamically typed Python code. For example, this conforms to the PEP 484:: from typing import Sized, Iterable, Iterator @@ -46,7 +46,7 @@ runtime costs. The intention of this PEP is to solve all these problems by allowing to write the above code without explicit base classes in the class definition, and ``Bucket`` would still be implicitly considered a subtype of both ``Sized`` and ``Iterable[int]`` by static type checkers -using structural subtyping:: +using structural [wiki-structural-subtyping]_ subtyping:: from typing import Iterator @@ -71,13 +71,13 @@ Nominal vs structural subtyping The structural subtyping is natural for Python programmers since it matches the runtime semantics of duck typing: an object that has certain properties is treated independently of its actual runtime class. -However, as discussed in PEP 483, both nominal and structural subtyping have -their strengths and weaknesses. Therefore, in this PEP do we not propose -to replace the nominal subtyping described by PEP 484 with structural -subtyping completely. Instead, it is proposed that the protocol classes -complement normal classes, and the choice is left for a user to decide where -to apply a particular solution. See section "Rejected ideas" for additional -motivation. +However, as discussed in PEP 483 [pep483]_, both nominal and structural +subtyping have their strengths and weaknesses. Therefore, in this PEP we +*do not propose* to replace the nominal subtyping described by PEP 484 with +structural subtyping completely. Instead, it is proposed that the protocol +classes complement normal classes, and the choice is left for a user to decide +where to apply a particular solution. See section "Rejected ideas" for +additional motivation. Non-Goals @@ -88,8 +88,8 @@ sophisticated runtime instance and class checks against protocol classes. This would be difficult and error-prone and will contradict the logic of PEP 484. As well, following PEP 484 and PEP 526 we state that protocols are **completely optional** and there is no intent to make them required. -No runtime semantics will be imposed for variables or parameters annotated with -a protocol class, the actual checks will be performed by third party type +No runtime semantics will be imposed for variables or parameters annotated +with a protocol class, the actual checks will be performed by third party type checkers and other tools. @@ -99,10 +99,10 @@ Existing approaches to structural subtyping Before describing the actual specification, we review existing approaches related to structural subtyping in Python and other languages: -* Zope interfaces was one of the first widely used approaches to - structural subtyping in Python. It is implemented by providing special - classes to distinguish interface classes from normal classes, to mark - interface attributes and to explicitly declare implementation. +* Zope interfaces [zope-interfaces]_ was one of the first widely used + approaches to structural subtyping in Python. It is implemented by providing + special classes to distinguish interface classes from normal classes, + to mark interface attributes and to explicitly declare implementation. For example:: from zope.interface import Interface, Attribute, implements @@ -146,10 +146,10 @@ related to structural subtyping in Python and other languages: with a special base class is reasonable and easy to implement both statically and at runtime. -* Python abstract base classes are the standard library tool to provide some - functionality similar to structural subtyping. The drawback of this - approach is the necessity to either subclass the abstract class or - register an implementation explicitly:: +* Python abstract base classes [abstract-base-classes]_ are the standard + library tool to provide some functionality similar to structural subtyping. + The drawback of this approach is the necessity to either subclass + the abstract class or register an implementation explicitly:: from abc import ABCMeta @@ -165,9 +165,10 @@ related to structural subtyping in Python and other languages: static context. However, in runtime context, ABCs are good candidates for protocol classes and are already used extensively in ``typing`` module. -* Abstract classes defined in ``collections.abc`` module are slightly more - advanced since they implement a custom ``__subclasshook__()`` method that - allows runtime structural checks without explicit registration:: +* Abstract classes defined in ``collections.abc`` module [collections-abc]_ + are slightly more advanced since they implement a custom + ``__subclasshook__()`` method that allows runtime structural checks without + explicit registration:: from collections.abc import Iterable @@ -185,9 +186,9 @@ related to structural subtyping in Python and other languages: that mimics the one in ``collections.abc``, see detailed specification below. -* TypeScript provides support for user defined classes and interfaces. - Explicit implementation declaration is not required, structural subtyping - is verified statically. For example:: +* TypeScript [typescript]_ provides support for user defined classes and + interfaces. Explicit implementation declaration is not required, + structural subtyping is verified statically. For example:: interface LabelledItem { label: string; @@ -209,8 +210,8 @@ related to structural subtyping in Python and other languages: checking without runtime implications looks reasonable, and basically this proposal follows the same line. -* Go uses a more radical approach and makes interfaces the primary way - to provide type information. Also, assignments are used to explicitly +* Go [golang]_ uses a more radical approach and makes interfaces the primary + way to provide type information. Also, assignments are used to explicitly ensure implementation:: type SomeInterface interface { @@ -341,7 +342,7 @@ that don't have an implementation. Such syntax mirrors how methods are defined in stub files. Abstract static methods, abstract class methods, and abstract properties are equally supported. -To defined a protocol variable one should use the PEP 526 variable +To defined a protocol variable one should use the PEP 526 [pep526]_ variable annotations in the class body. Attributes defined in the body of a method by assignment via ``self`` considered non-protocol members. The rationale for this is that the protocol class implementation is often not shared by @@ -757,6 +758,9 @@ effects on core interpreter and standard library except in from ``__annotations__`` to ``__subclsshook__()``. * In the backported version of ``typing``, translate ``...`` class attribute values to abstract properties. +* All structural subtyping checks will be performed by static type checkers, + such as ``mypy`` [mypy]_, no special support for protocol validation will + be provided at runtime. Changes in typing module @@ -785,7 +789,7 @@ what are the methods these protocols implement, and immediately recognize the corresponding runtime protocol counterpart. Practically, few changes will be needed in ``typing`` since some of these classes already behave the necessary way at runtime. Most of these will need -to be updated only in the corresponding ``typeshed`` stub. +to be updated only in the corresponding ``typeshed`` stubs [typeshed]_ . All other concrete generic classes such as ``List``, ``Set``, ``IO``, ``Deque``, etc are sufficiently complex that it makes sense to keep @@ -873,9 +877,9 @@ complicate both the concept and the implementation. Use assignments to check explicitly that a class implements a protocol ---------------------------------------------------------------------- -In Go language [golang] the explicit checks for implementation are performed -via dummy assignments. Such way is also possible with the current proposal. -Example:: +In Go language the explicit checks for implementation are performed +via dummy assignments [golang]_. Such way is also possible with the current +proposal. Example:: class A: def __len__(self) -> float: @@ -909,9 +913,6 @@ References .. [pep483] https://www.python.org/dev/peps/pep-0483/ -.. [pep526] - https://www.python.org/dev/peps/pep-0526/ - .. [zope-interfaces] https://zopeinterface.readthedocs.io/en/latest/ @@ -927,6 +928,9 @@ References .. [golang] https://golang.org/doc/effective_go.html#interfaces_and_types +.. [pep526] + https://www.python.org/dev/peps/pep-0526/ + .. [typeshed] https://github.com/python/typeshed/ From c90aa1cb9ed37ab1e82c1a9b741e82324a1a1b19 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Thu, 9 Mar 2017 12:32:50 +0100 Subject: [PATCH 21/37] Caption style, add cross-refs --- pep-05xx.txt | 46 ++++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 20 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index 69a4e27b75c..f02d2f22d51 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -48,7 +48,7 @@ the class definition, and ``Bucket`` would still be implicitly considered a subtype of both ``Sized`` and ``Iterable[int]`` by static type checkers using structural [wiki-structural-subtyping]_ subtyping:: - from typing import Iterator + from typing import Iterator, Iterable class Bucket: ... @@ -76,11 +76,11 @@ subtyping have their strengths and weaknesses. Therefore, in this PEP we *do not propose* to replace the nominal subtyping described by PEP 484 with structural subtyping completely. Instead, it is proposed that the protocol classes complement normal classes, and the choice is left for a user to decide -where to apply a particular solution. See section "Rejected ideas" for -additional motivation. +where to apply a particular solution. See section on `rejected`_ ideas at the +end of this PEP for additional motivation. -Non-Goals +Non-goals --------- At runtime protocol classes we be simple ABCs. There is no intent to provide @@ -93,7 +93,7 @@ with a protocol class, the actual checks will be performed by third party type checkers and other tools. -Existing approaches to structural subtyping +Existing Approaches to Structural Subtyping =========================================== Before describing the actual specification, we review existing approaches @@ -183,7 +183,7 @@ related to structural subtyping in Python and other languages: statically. In addition, to allow users achieving (if necessary) such runtime behavior for user defined protocols, a simple class decorator ``@runtime`` can be provided that automatically adds a ``__subclasshook__`` - that mimics the one in ``collections.abc``, see detailed specification + that mimics the one in ``collections.abc``, see detailed `discussion`_ below. * TypeScript [typescript]_ provides support for user defined classes and @@ -206,7 +206,7 @@ related to structural subtyping in Python and other languages: prohibits redundant members in implementations. While the idea of optional members looks interesting, it would complicate this proposal and it is not clear how useful it will be. Therefore it is proposed to postpone - this, see "Rejected ideas". In general, the idea of static protocol + this, see `rejected`_ ideas. In general, the idea of static protocol checking without runtime implications looks reasonable, and basically this proposal follows the same line. @@ -223,9 +223,11 @@ related to structural subtyping in Python and other languages: } Both these ideas are questionable in the context of this proposal, see - section "Rejected ideas". + section on `rejected`_ ideas. +.. _specification: + Specification ============= @@ -315,8 +317,8 @@ A combination of both may be used if necessary for runtime purposes. If the method body is not empty but decorated by the ``@abstractmethod`` decorator, then the corresponding definition serves as a default method implementation. If some or all parameters are not annotated -their types are assumed to be ``Any`` (see PEP 484). All other methods -defined in the protocol class are considered non-protocol members. +their types are assumed to be ``Any``, see PEP 484 [pep484]_. All other +methods defined in the protocol class are considered non-protocol members. Example:: from typing import Protocol @@ -517,7 +519,7 @@ self-referential data structures like trees in an abstract fashion:: leafs: Iterable['Traversable'] -Using protocols +Using Protocols =============== Subtyping relationships with other types @@ -645,6 +647,8 @@ aliases:: CompatReversible = Union[Reversible[T], All[Iterable[T], Sized]] +.. _discussion: + ``@runtime`` decorator and narrowing types by ``isinstance()`` -------------------------------------------------------------- @@ -692,15 +696,15 @@ always fail in agreement with PEP 484. Examples:: # Error! Can't use 'isinstance()' with subscripted protocols -Using protocols in Python 3.5 and earlier +Using Protocols in Python 3.5 and Earlier ========================================= Variable annotation syntax was added in Python 3.6, so that the syntax -for defining protocol variables proposed in section ... can't be used. -To define these in earlier versions of Python we propose to either use type -comments (and ellipsis literals if default value is not provided), or use -properties if it is necessary to indicate abstract status at runtime. -All of these will be valid:: +for defining protocol variables proposed in `specification`_ section can't +be used. To define these in earlier versions of Python we propose to either +use type comments (and ellipsis literals if default value is not provided), +or use abstract properties if it is necessary to indicate abstract status at +runtime. All of these are valid:: class Foo(Protocol): @@ -738,7 +742,7 @@ The ``typing`` module changes proposed in this PEP will be also backported to earlier versions via the backport currently available on PyPI. -Runtime implementation of protocol classes +Runtime Implementation of Protocol Classes ========================================== Implementation details @@ -820,8 +824,10 @@ at runtime in this case). But together with other introspection tools this give a reasonable perspective for runtime type checking tools. -Rejected and postponed ideas -============================ +.. _rejected: + +Rejected/Postponed Ideas +======================== Make every class a protocol by default -------------------------------------- From b008de13ef02759fe65186ee08a026863aa653c7 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Fri, 10 Mar 2017 01:05:57 +0100 Subject: [PATCH 22/37] Remove redundant links; + minor changes --- pep-05xx.txt | 112 +++++++++++++++++++++++++-------------------------- 1 file changed, 55 insertions(+), 57 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index f02d2f22d51..e08ada966ca 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -14,17 +14,19 @@ Python-Version: 3.7 Abstract ======== -Type hints introduced in PEP 484 that can be used to specify type metadata +Type hints introduced in PEP 484 can be used to specify type metadata for static type checkers and other third party tools. However, PEP 484 only specifies the semantics of *nominal* subtyping. In this PEP we specify static and runtime semantics of protocol classes that will provide a support for *structural* subtyping (static duck typing). +.. _rationale: + Rationale and Goals =================== -Currently, PEP 484 [pep484]_ and ``typing`` module [typing]_ define abstract +Currently, PEP 484 and ``typing`` module [typing]_ define abstract base classes for several common Python protocols such as ``Iterable`` and ``Sized``. The problem with them is that a class has to be explicitly marked to support them, which is arguably unpythonic and unlike what one would @@ -42,11 +44,13 @@ The same problem appears with user defined ABCs, they must be explicitly subclassed or registered. This is particularly difficult to do with library types as the type objects may be hidden deep in the implementation of the library. Moreover, extensive use of ABCs might impose additional -runtime costs. The intention of this PEP is to solve all these problems +runtime costs. + +The intention of this PEP is to solve all these problems by allowing to write the above code without explicit base classes in the class definition, and ``Bucket`` would still be implicitly considered a subtype of both ``Sized`` and ``Iterable[int]`` by static type checkers -using structural [wiki-structural-subtyping]_ subtyping:: +using structural [wiki-structural]_ subtyping:: from typing import Iterator, Iterable @@ -71,7 +75,7 @@ Nominal vs structural subtyping The structural subtyping is natural for Python programmers since it matches the runtime semantics of duck typing: an object that has certain properties is treated independently of its actual runtime class. -However, as discussed in PEP 483 [pep483]_, both nominal and structural +However, as discussed in PEP 483, both nominal and structural subtyping have their strengths and weaknesses. Therefore, in this PEP we *do not propose* to replace the nominal subtyping described by PEP 484 with structural subtyping completely. Instead, it is proposed that the protocol @@ -83,9 +87,9 @@ end of this PEP for additional motivation. Non-goals --------- -At runtime protocol classes we be simple ABCs. There is no intent to provide -sophisticated runtime instance and class checks against protocol classes. -This would be difficult and error-prone and will contradict the logic +At runtime, protocol classes will be simple ABCs. There is no intent to +provide sophisticated runtime instance and class checks against protocol +classes. This would be difficult and error-prone and will contradict the logic of PEP 484. As well, following PEP 484 and PEP 526 we state that protocols are **completely optional** and there is no intent to make them required. No runtime semantics will be imposed for variables or parameters annotated @@ -96,13 +100,13 @@ checkers and other tools. Existing Approaches to Structural Subtyping =========================================== -Before describing the actual specification, we review existing approaches -related to structural subtyping in Python and other languages: +Before describing the actual specification, we review and comment on existing +approaches related to structural subtyping in Python and other languages: * Zope interfaces [zope-interfaces]_ was one of the first widely used approaches to structural subtyping in Python. It is implemented by providing special classes to distinguish interface classes from normal classes, - to mark interface attributes and to explicitly declare implementation. + to mark interface attributes, and to explicitly declare implementation. For example:: from zope.interface import Interface, Attribute, implements @@ -137,16 +141,16 @@ related to structural subtyping in Python and other languages: email = Attribute("Email Address") phone = Attribute("Phone Number") - invariant(contacts_invariant) + invariant(required_contact) Even more detailed invariants are supported. However, Zope interfaces rely entirely on runtime validation. Such focus on runtime properties goes beyond the scope of current proposal, and static support for invariants might be difficult to implement. However, the idea of marking an interface - with a special base class is reasonable and easy to implement both + class with a special base class is reasonable and easy to implement both statically and at runtime. -* Python abstract base classes [abstract-base-classes]_ are the standard +* Python abstract base classes [abstract-classes]_ are the standard library tool to provide some functionality similar to structural subtyping. The drawback of this approach is the necessity to either subclass the abstract class or register an implementation explicitly:: @@ -161,8 +165,8 @@ related to structural subtyping in Python and other languages: assert issubclass(tuple, MyTuple) assert isinstance((), MyTuple) - As mentioned in rationale, we want to avoid such necessity, especially in - static context. However, in runtime context, ABCs are good candidates for + As mentioned in `rationale`_, we want to avoid such necessity, especially + in static context. However, in runtime context, ABCs are good candidates for protocol classes and are already used extensively in ``typing`` module. * Abstract classes defined in ``collections.abc`` module [collections-abc]_ @@ -235,7 +239,7 @@ Terminology ----------- We propose to use the term *protocols* for types supporting structural -subtyping. The reason is that the term iterator protocol, +subtyping. The reason is that the term *iterator protocol*, for example, is widely understood in the community, and coming up with a new term for this concept in a statically typed context would just create confusion. @@ -248,11 +252,11 @@ The distinction is not important most of the time, and in other cases we propose to just add a qualifier such as *protocol classes* when referring to the static type concept. -If a class includes a protocol in its MRO, the class is an called +If a class includes a protocol in its MRO, the class is called an *explicit* subclass of the protocol. If a class is a structural subtype of a protocol, it is said to implement the protocol and to be compatible with a protocol. If a class is compatible with a protocol but the protocol -is not included in the MRO, the class is an *implicit* subclass +is not included in the MRO, the class is an *implicit* subtype of the protocol. The attributes (variables and methods) of a protocol that are mandatory @@ -261,6 +265,8 @@ protocol members. Other members defined in a protocol class are called non-protocol members. +.. _definition: + Defining a protocol ------------------- @@ -317,7 +323,7 @@ A combination of both may be used if necessary for runtime purposes. If the method body is not empty but decorated by the ``@abstractmethod`` decorator, then the corresponding definition serves as a default method implementation. If some or all parameters are not annotated -their types are assumed to be ``Any``, see PEP 484 [pep484]_. All other +their types are assumed to be ``Any``, see PEP 484. All other methods defined in the protocol class are considered non-protocol members. Example:: @@ -340,11 +346,11 @@ implicitly abstract only statically, and there won't be any runtime checking. The rationale for this is that most methods in a protocol will be abstract, and having to always use ``@abstractmethod`` is a little verbose and ugly. Leaving empty method body is the recommended way of defining protocol members -that don't have an implementation. Such syntax mirrors how methods are -defined in stub files. Abstract static methods, abstract class methods, -and abstract properties are equally supported. +that don't have a default implementation. Such syntax mirrors how methods are +defined in stub files. Abstract static methods, and abstract class methods +are equally supported. -To defined a protocol variable one should use the PEP 526 [pep526]_ variable +To defined a protocol variable one should use the PEP 526 variable annotations in the class body. Attributes defined in the body of a method by assignment via ``self`` considered non-protocol members. The rationale for this is that the protocol class implementation is often not shared by @@ -364,9 +370,9 @@ non-protocol member. Examples:: self.temp: List[int] = [] # And this one neither Note that unannotated variables are always assumed to have type ``Any``, -so that definition ``x = 0`` in a protocol body will create a protocol member -``x`` with type ``Any``, *not* ``int``. To distinguish between protocol class -variables and protocol instance variables, the special ``ClassVar`` +so that a definition ``x = 0`` in a protocol body will create a protocol +member ``x`` with type ``Any``, *not* ``int``. To distinguish between protocol +class variables and protocol instance variables, the special ``ClassVar`` annotation should be used as specified by PEP 526. @@ -454,10 +460,10 @@ type signature can override protocol method but not vice versa. Examples:: class D(B, A): # OK def bar(self) -> str: return 'bar' - # Error, 'foo' is still not implemented (although 'm' is not needed). + # Error, 'foo' is not implemented (although 'm' is not needed here). Note that subclassing a protocol class would not turn the subclass into -a protocol unless it also has ``Protocol`` as an explicit base class. +a protocol unless it also has ``typing.Protocol`` as an explicit base class. Merging and extending protocols @@ -476,7 +482,7 @@ having ``typing.Protocol`` as an immediate base class:: Now the protocol ``SizedAndCloseable`` is a protocol with two methods, ``__len__`` and ``close``. Alternatively, one can implement it like this, assuming the existence of ``SupportsClose`` from -the previous example:: +the example in `definition`_ section:: from typing import Sized @@ -532,9 +538,10 @@ relationships are subject to the following rules: * A protocol is never a subtype of a concrete type. * A concrete type or a protocol ``X`` is a subtype of another protocol ``P`` if and only if ``X`` implements all protocol members of ``P``. In other - words a subtyping with respect to a protocol is always structural. + words subtyping with respect to a protocol is always structural. * Edge case: for recursive protocols, structural subtyping is decided - positively for all situations that are type safe. For example:: + positively for all situations that are type safe. Continuing + previous example:: class Tree(Generic[T]): def __init__(self, value: T, @@ -581,7 +588,7 @@ In addition, we propose to add another special type construct it is not very useful, there are many situation where a variable should implement more than one protocol. Annotation by ``All[Proto1, Proto2, ...]`` means that a given variable or parameter must implement all protocols -``Ptoro1``, ``Proto2``, etc either implicitly or explicitly. Example:: +``Ptoro1``, ``Proto2``, etc. either implicitly or explicitly. Example:: from typing import Sequence, Hashable, All @@ -619,7 +626,7 @@ The same rule applies to variables:: var().meth() # OK Assigning protocol class to a variable is allowed if it is not explicitly -typed, such assignment create type aliases. For non-protocol classes, +typed, such assignment creates a type alias. For non-protocol classes, the behavior of ``Type[]`` is not changed. @@ -652,8 +659,8 @@ aliases:: ``@runtime`` decorator and narrowing types by ``isinstance()`` -------------------------------------------------------------- -The default semantics is ``isinstance()`` and ``issubclass()`` fail by default -for protocol types. This is in the spirit of duck typing -- protocols +The default semantics is that ``isinstance()`` and ``issubclass()`` fail by +default for protocol types. This is in the spirit of duck typing -- protocols basically would be used to model duck typing statically, not explicitly at runtime. @@ -663,7 +670,7 @@ and other ABCs in ``collections.abc`` and ``typing`` already do it, but this is only limited to non-generic and to unsubscripted generic protocols (statically ``Iterable`` is equivalent to ``Iterable[Any]`). The ``typing`` module will define a special ``@runtime`` class decorator that -automatically adds a ``__subclasshook__`` that provides the same semantics +automatically adds a ``__subclasshook__()`` that provides the same semantics for class and instance checks as for ``collections.abc`` classes:: from typing import runtime, Protocol @@ -701,10 +708,10 @@ Using Protocols in Python 3.5 and Earlier Variable annotation syntax was added in Python 3.6, so that the syntax for defining protocol variables proposed in `specification`_ section can't -be used. To define these in earlier versions of Python we propose to either -use type comments (and ellipsis literals if default value is not provided), -or use abstract properties if it is necessary to indicate abstract status at -runtime. All of these are valid:: +be used in earlier versions. To define these in earlier versions of Python we +propose to either use type comments (and ellipsis literals if default value is +not provided), or use abstract properties if it is necessary to indicate +abstract status at runtime. All of these are valid:: class Foo(Protocol): @@ -761,9 +768,9 @@ effects on core interpreter and standard library except in as for ``collections.abc`` by adding all abstract methods and all variables from ``__annotations__`` to ``__subclsshook__()``. * In the backported version of ``typing``, translate ``...`` class attribute - values to abstract properties. + values to properties. * All structural subtyping checks will be performed by static type checkers, - such as ``mypy`` [mypy]_, no special support for protocol validation will + such as ``mypy`` [mypy]_, no additional support for protocol validation will be provided at runtime. @@ -786,14 +793,14 @@ The following classes in ``typing`` module will be protocols: * ``AsyncIterable``, ``AsyncIterator`` * ``Awaitable`` * ``Callable`` -* ``ContextManager`` +* ``ContextManager``, ``AsyncContextManager`` Most of these classes are small and conceptually simple. It is easy to see what are the methods these protocols implement, and immediately recognize the corresponding runtime protocol counterpart. Practically, few changes will be needed in ``typing`` since some of these classes already behave the necessary way at runtime. Most of these will need -to be updated only in the corresponding ``typeshed`` stubs [typeshed]_ . +to be updated only in the corresponding ``typeshed`` stubs [typeshed]_. All other concrete generic classes such as ``List``, ``Set``, ``IO``, ``Deque``, etc are sufficiently complex that it makes sense to keep @@ -818,7 +825,7 @@ types of attributes will not be visible at runtime in Python 3.5 and earlier, but this looks like a reasonable limitation. There will be only a limited support of ``isinstance()`` and ``issubclass()`` -as discussed above (these will _always_ fail with ``TypeError`` for +as discussed above (these will *always* fail with ``TypeError`` for subscripted generic protocols, since a reliable answer could not be given at runtime in this case). But together with other introspection tools this give a reasonable perspective for runtime type checking tools. @@ -907,22 +914,16 @@ the code would be ugly. Therefore we discourage the use of such pattern. References ========== -.. [pep484] - https://www.python.org/dev/peps/pep-0484/ - .. [typing] https://docs.python.org/3/library/typing.html -.. [wiki-structural-subtyping] +.. [wiki-structural] https://en.wikipedia.org/wiki/Structural_type_system -.. [pep483] - https://www.python.org/dev/peps/pep-0483/ - .. [zope-interfaces] https://zopeinterface.readthedocs.io/en/latest/ -.. [abstract-base-classes] +.. [abstract-classes] https://docs.python.org/3/library/abc.html .. [collections-abc] @@ -934,9 +935,6 @@ References .. [golang] https://golang.org/doc/effective_go.html#interfaces_and_types -.. [pep526] - https://www.python.org/dev/peps/pep-0526/ - .. [typeshed] https://github.com/python/typeshed/ From 02cca5c9086a5edba911da32856e03339804df66 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Fri, 10 Mar 2017 01:09:23 +0100 Subject: [PATCH 23/37] One more tiny change --- pep-05xx.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index e08ada966ca..7af852ab333 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -186,9 +186,9 @@ approaches related to structural subtyping in Python and other languages: protocols. The main goal of this proposal is to support such behavior statically. In addition, to allow users achieving (if necessary) such runtime behavior for user defined protocols, a simple class decorator - ``@runtime`` can be provided that automatically adds a ``__subclasshook__`` - that mimics the one in ``collections.abc``, see detailed `discussion`_ - below. + ``@runtime`` can be provided that automatically adds a + ``__subclasshook__()`` that mimics the one in ``collections.abc``, + see detailed `discussion`_ below. * TypeScript [typescript]_ provides support for user defined classes and interfaces. Explicit implementation declaration is not required, From 0f3732a13add7cea6c9cdd980816d7846a04f484 Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Thu, 9 Mar 2017 20:04:30 -0800 Subject: [PATCH 24/37] Copyediting changes I'm excited by this proposal but couldn't help but make some minor changes while reading through it. I hope they'll help make it more readable. --- pep-05xx.txt | 120 +++++++++++++++++++++++++-------------------------- 1 file changed, 60 insertions(+), 60 deletions(-) diff --git a/pep-05xx.txt b/pep-05xx.txt index 7af852ab333..06c13c3a871 100644 --- a/pep-05xx.txt +++ b/pep-05xx.txt @@ -26,12 +26,12 @@ for *structural* subtyping (static duck typing). Rationale and Goals =================== -Currently, PEP 484 and ``typing`` module [typing]_ define abstract +Currently, PEP 484 and the ``typing`` module [typing]_ define abstract base classes for several common Python protocols such as ``Iterable`` and ``Sized``. The problem with them is that a class has to be explicitly marked -to support them, which is arguably unpythonic and unlike what one would +to support them, which is unpythonic and unlike what one would normally do in idiomatic dynamically typed Python code. For example, -this conforms to the PEP 484:: +this conforms to PEP 484:: from typing import Sized, Iterable, Iterator @@ -40,7 +40,7 @@ this conforms to the PEP 484:: def __len__(self) -> int: ... def __iter__(self) -> Iterator[int]: ... -The same problem appears with user defined ABCs, they must be explicitly +The same problem appears with user-defined ABCs: they must be explicitly subclassed or registered. This is particularly difficult to do with library types as the type objects may be hidden deep in the implementation of the library. Moreover, extensive use of ABCs might impose additional @@ -62,7 +62,7 @@ using structural [wiki-structural]_ subtyping:: def collect(items: Iterable[int]) -> int: ... result: int = collect(Bucket()) # Passes type check -The same functionality will be provided for user defined protocols, as +The same functionality will be provided for user-defined protocols, as specified below. The code with a protocol class matches common Python conventions much better. It is also automatically extensible and works with additional, unrelated classes that happen to implement @@ -72,14 +72,14 @@ the required protocol. Nominal vs structural subtyping ------------------------------- -The structural subtyping is natural for Python programmers since it matches +Structural subtyping is natural for Python programmers since it matches the runtime semantics of duck typing: an object that has certain properties is treated independently of its actual runtime class. However, as discussed in PEP 483, both nominal and structural subtyping have their strengths and weaknesses. Therefore, in this PEP we *do not propose* to replace the nominal subtyping described by PEP 484 with -structural subtyping completely. Instead, it is proposed that the protocol -classes complement normal classes, and the choice is left for a user to decide +structural subtyping completely. Instead, protocol classes as specified in +this PEP complement normal classes, and users are free to choose where to apply a particular solution. See section on `rejected`_ ideas at the end of this PEP for additional motivation. @@ -93,7 +93,7 @@ classes. This would be difficult and error-prone and will contradict the logic of PEP 484. As well, following PEP 484 and PEP 526 we state that protocols are **completely optional** and there is no intent to make them required. No runtime semantics will be imposed for variables or parameters annotated -with a protocol class, the actual checks will be performed by third party type +with a protocol class. The actual checks will be performed by third-party type checkers and other tools. @@ -145,7 +145,7 @@ approaches related to structural subtyping in Python and other languages: Even more detailed invariants are supported. However, Zope interfaces rely entirely on runtime validation. Such focus on runtime properties goes - beyond the scope of current proposal, and static support for invariants + beyond the scope of the current proposal, and static support for invariants might be difficult to implement. However, the idea of marking an interface class with a special base class is reasonable and easy to implement both statically and at runtime. @@ -167,7 +167,7 @@ approaches related to structural subtyping in Python and other languages: As mentioned in `rationale`_, we want to avoid such necessity, especially in static context. However, in runtime context, ABCs are good candidates for - protocol classes and are already used extensively in ``typing`` module. + protocol classes and they are already used extensively in the ``typing`` module. * Abstract classes defined in ``collections.abc`` module [collections-abc]_ are slightly more advanced since they implement a custom @@ -191,15 +191,15 @@ approaches related to structural subtyping in Python and other languages: see detailed `discussion`_ below. * TypeScript [typescript]_ provides support for user defined classes and - interfaces. Explicit implementation declaration is not required, + interfaces. Explicit implementation declaration is not required and structural subtyping is verified statically. For example:: - interface LabelledItem { + interface LabeledItem { label: string; size?: int; } - function printLabel(obj: LabelledValue) { + function printLabel(obj: LabeledValue) { console.log(obj.label); } @@ -210,7 +210,7 @@ approaches related to structural subtyping in Python and other languages: prohibits redundant members in implementations. While the idea of optional members looks interesting, it would complicate this proposal and it is not clear how useful it will be. Therefore it is proposed to postpone - this, see `rejected`_ ideas. In general, the idea of static protocol + this; see `rejected`_ ideas. In general, the idea of static protocol checking without runtime implications looks reasonable, and basically this proposal follows the same line. @@ -226,8 +226,8 @@ approaches related to structural subtyping in Python and other languages: fmt.Printf("value implements some interface") } - Both these ideas are questionable in the context of this proposal, see - section on `rejected`_ ideas. + Both these ideas are questionable in the context of this proposal. See + the section on `rejected`_ ideas. .. _specification: @@ -323,7 +323,7 @@ A combination of both may be used if necessary for runtime purposes. If the method body is not empty but decorated by the ``@abstractmethod`` decorator, then the corresponding definition serves as a default method implementation. If some or all parameters are not annotated -their types are assumed to be ``Any``, see PEP 484. All other +their types are assumed to be ``Any`` (see PEP 484). All other methods defined in the protocol class are considered non-protocol members. Example:: @@ -345,14 +345,14 @@ Having an empty method body in a protocol type makes the method implicitly abstract only statically, and there won't be any runtime checking. The rationale for this is that most methods in a protocol will be abstract, and having to always use ``@abstractmethod`` is a little verbose and ugly. -Leaving empty method body is the recommended way of defining protocol members +Leaving the method body empty is the recommended way of defining protocol members that don't have a default implementation. Such syntax mirrors how methods are -defined in stub files. Abstract static methods, and abstract class methods +defined in stub files. Abstract staticmethods and abstract classmethods are equally supported. -To defined a protocol variable one should use the PEP 526 variable +To define a protocol variable, one should use PEP 526 variable annotations in the class body. Attributes defined in the body of a method -by assignment via ``self`` considered non-protocol members. The rationale for +by assignment via ``self`` are considered non-protocol members. The rationale for this is that the protocol class implementation is often not shared by subtypes, so the interface should not depend on the default implementation. An exception from these rules is an attribute starting with double @@ -418,10 +418,10 @@ can be used as regular base classes. In this case: represent(simple) # OK represent(BadColor()) # Error, 'add_int' is not implemented implicitly -* Second, type checker will statically enforce that a class - actually implements a protocol correctly. In particular, type checker will +* Type checkers will statically enforce that the class + actually implements the protocol correctly. In particular, type checkers will always require members without default implementation to be defined in - a subclass that explicitly subclasses the protocol:: + subclasses that explicitly subclass the protocol:: from typing import Protocol, Tuple @@ -438,15 +438,15 @@ can be used as regular base classes. In this case: self.rgb = red, green, blue # Error, 'blue' must be 'int' # Error, 'intensity' is not defined! - # Note that 'to_bytes' is not required, this is safe + # Note that 'to_bytes' is not required, but this is safe # since the default implementation is actually in the MRO. The general philosophy is that protocols are mostly like regular ABCs, but a static type checker will handle them specially. In particular, a class can explicitly inherit from multiple protocols. -In this case methods are resolved using normal MRO and type checker verifies +In this case methods are resolved using normal MRO and the type checker verifies that all subtyping are correct. Note that a concrete method with appropriate -type signature can override protocol method but not vice versa. Examples:: +type signature can override a protocol method but not vice versa. Examples:: class A(Protocol): def m(self) -> None: ... @@ -462,7 +462,7 @@ type signature can override protocol method but not vice versa. Examples:: return 'bar' # Error, 'foo' is not implemented (although 'm' is not needed here). -Note that subclassing a protocol class would not turn the subclass into +Subclassing a protocol class would not turn the subclass into a protocol unless it also has ``typing.Protocol`` as an explicit base class. @@ -470,7 +470,7 @@ Merging and extending protocols ------------------------------- Subprotocols are also supported. A subprotocol can be defined -by having both one or more protocols as the explicit base classes and also +by having both one or more protocols as immediate base classes and also having ``typing.Protocol`` as an immediate base class:: from typing import Sized, Protocol @@ -493,10 +493,10 @@ the example in `definition`_ section:: The two definitions of ``SizedAndClosable`` are equivalent. Subclass relationships between protocols are not meaningful when -considering subtyping, since the structural compatibility is +considering subtyping, since structural compatibility is the criterion, not the MRO. -If one omits ``Protocol`` in the base class list, this would be regular +If one omits ``Protocol`` in the base class list, this would be a regular (non-protocol) class that must implement ``Sized``. If ``Protocol`` is included in the base class list, all the other base classes must be protocols. A protocol can't extend a regular class. @@ -531,14 +531,14 @@ Using Protocols Subtyping relationships with other types ---------------------------------------- -Protocols cannot be instantiated, so that there are no values with +Protocols cannot be instantiated, so there are no values with protocol types. For variables and parameters with protocol types, subtyping relationships are subject to the following rules: * A protocol is never a subtype of a concrete type. * A concrete type or a protocol ``X`` is a subtype of another protocol ``P`` if and only if ``X`` implements all protocol members of ``P``. In other - words subtyping with respect to a protocol is always structural. + words, subtyping with respect to a protocol is always structural. * Edge case: for recursive protocols, structural subtyping is decided positively for all situations that are type safe. Continuing previous example:: @@ -555,8 +555,8 @@ relationships are subject to the following rules: walk(tree) # OK, 'Tree[float]' is a subtype of 'Traversable' Generic protocol types follow the same rules of variance as non-protocol -types. Protocols can be used in the all special type constructors provided -by ``typing`` module and follow corresponding subtyping rules. For example, +types. Protocols can be used in all special type constructors provided +by the ``typing`` module and follow the corresponding subtyping rules. For example, protocol ``PInt`` is a subtype of ``Union[PInt, int]``. @@ -626,7 +626,7 @@ The same rule applies to variables:: var().meth() # OK Assigning protocol class to a variable is allowed if it is not explicitly -typed, such assignment creates a type alias. For non-protocol classes, +typed, and such assignment creates a type alias. For non-protocol classes, the behavior of ``Type[]`` is not changed. @@ -634,7 +634,7 @@ the behavior of ``Type[]`` is not changed. ------------------------------ Protocols are essentially anonymous. To emphasize this point, static type -checkers might refuse the protocol classes inside ``NewType()`` to avoid an +checkers might refuse protocol classes inside ``NewType()`` to avoid an illusion that a distinct type is provided:: form typing import NewType , Protocol, Iterator @@ -659,15 +659,15 @@ aliases:: ``@runtime`` decorator and narrowing types by ``isinstance()`` -------------------------------------------------------------- -The default semantics is that ``isinstance()`` and ``issubclass()`` fail by -default for protocol types. This is in the spirit of duck typing -- protocols +The default semantics is that ``isinstance()`` and ``issubclass()`` fail +for protocol types. This is in the spirit of duck typing -- protocols basically would be used to model duck typing statically, not explicitly at runtime. However, it should be possible for protocol types to implement custom instance and class checks when this makes sense, similar to how ``Iterable`` and other ABCs in ``collections.abc`` and ``typing`` already do it, -but this is only limited to non-generic and to unsubscripted generic protocols +but this is limited to non-generic and to unsubscripted generic protocols (statically ``Iterable`` is equivalent to ``Iterable[Any]`). The ``typing`` module will define a special ``@runtime`` class decorator that automatically adds a ``__subclasshook__()`` that provides the same semantics @@ -677,7 +677,7 @@ for class and instance checks as for ``collections.abc`` classes:: @runtime class Closeable(Protocol): - @abstarctmethod + @abstractmethod def close(self): ... @@ -688,7 +688,7 @@ classes where all protocol members are defined by ``@abstractmethod`` for methods and by a variable annotation for class and instance variables. Static type checkers will understand ``isinstance(x, Proto)`` and ``issublclass(C, Proto)`` for protocols decorated with such decorator -(as they already do for ``Iterable`` etc). Static type checkers will +(as they already do for ``Iterable`` etc.). Static type checkers will narrow types after such checks by the type erased ``Proto`` (i.e. with all variables having type ``Any`` and all methods having type ``Callable[..., Any]``). Note that ``isinstance(x, Proto[int])`` etc. will @@ -709,8 +709,8 @@ Using Protocols in Python 3.5 and Earlier Variable annotation syntax was added in Python 3.6, so that the syntax for defining protocol variables proposed in `specification`_ section can't be used in earlier versions. To define these in earlier versions of Python we -propose to either use type comments (and ellipsis literals if default value is -not provided), or use abstract properties if it is necessary to indicate +propose to either use type comments (and ellipsis literals if there is no +default value), or use abstract properties if it is necessary to indicate abstract status at runtime. All of these are valid:: class Foo(Protocol): @@ -729,7 +729,7 @@ abstract status at runtime. All of these are valid:: def e(self) -> int: # This is not a protocol member. return 0 -When using the ellipsis ``...`` initializer, it can leak into subclasses at +The ellipsis ``...`` initializer can leak into subclasses at runtime, which is unfortunate:: class A(Protocol): @@ -743,7 +743,7 @@ runtime, which is unfortunate:: print(b.x) # 1 print(B.x) # Ellipsis -For this reason we propose to make ``Protocol`` metaclass recognize ``...`` +For this reason we propose to make the ``Protocol`` metaclass recognize ``...`` initializers and translate them to a special property that can't be read. The ``typing`` module changes proposed in this PEP will be also backported to earlier versions via the backport currently available on PyPI. @@ -756,7 +756,7 @@ Implementation details ---------------------- The runtime implementation could be done in pure Python without any -effects on core interpreter and standard library except in +effects on the core interpreter and standard library except in ``typing`` module: * Define class ``typing.Protocol`` similar to ``typing.Generic``. @@ -770,7 +770,7 @@ effects on core interpreter and standard library except in * In the backported version of ``typing``, translate ``...`` class attribute values to properties. * All structural subtyping checks will be performed by static type checkers, - such as ``mypy`` [mypy]_, no additional support for protocol validation will + such as ``mypy`` [mypy]_. No additional support for protocol validation will be provided at runtime. @@ -814,17 +814,17 @@ Introspection ------------- The existing class introspection machinery (``dir``, ``__annotations__`` etc) -can be used with protocols, in addition all introspection tools implemented -in ``typing`` module will support protocols. Since all attributes need to be +can be used with protocols. In addition, all introspection tools implemented +in the ``typing`` module will support protocols. Since all attributes need to be defined in the class body based on this proposal, protocol classes will have even better perspective for introspection than regular classes where attributes can be defined implicitly -- protocol attributes can't be initialized in ways that are not visible to introspection -(using ``setattr()``, assignment via ``self``, etc). Still, some things like +(using ``setattr()``, assignment via ``self``, etc.). Still, some things like types of attributes will not be visible at runtime in Python 3.5 and earlier, but this looks like a reasonable limitation. -There will be only a limited support of ``isinstance()`` and ``issubclass()`` +There will be only limited support of ``isinstance()`` and ``issubclass()`` as discussed above (these will *always* fail with ``TypeError`` for subscripted generic protocols, since a reliable answer could not be given at runtime in this case). But together with other introspection tools this @@ -841,9 +841,9 @@ Make every class a protocol by default Some languages such as Go make structural subtyping the only or the primary form of subtyping. We could achieve a similar result by making all classes -protocols by default (or even always). However we believe that this would be -a bad idea and classes should need to be explicitly marked as protocols, -as shown in this proposal: +protocols by default (or even always). However we believe that it is better +to require classes to be explicitly marked as protocols, for the following +reasons: * Protocols don't have some properties of regular classes. In particular, ``isinstance()`` is not always well-defined for protocols, whereas it is @@ -870,7 +870,7 @@ implementing a protocol, but if it is present, it must conform to a specific signature or type. One could use a ``hasattr()`` check to determine whether they can use the attribute on a particular instance. -Other languages, such as TypeScript have similar features and +Languages such as TypeScript have similar features and apparently they are pretty commonly used. The current realistic potential use cases for protocols in Python don't require these. In the interest of simplicity, we propose to not support optional methods or attributes. @@ -901,14 +901,14 @@ proposal. Example:: _: Sized = A() # Error: A.__len__ doesn't conform to 'Sized' # (Incompatible return type 'float') -We see such approach as not very clear, as it moves the check away from +This approach moves the check away from the class definition and it almost requires a comment as otherwise the code probably would not make any sense to an average reader -- it looks like dead code. Besides, in the simplest form it requires one -to construct an instance of ``A`` which could be problematic if this requires +to construct an instance of ``A``, which could be problematic if this requires accessing or allocating some resources such as files or sockets. We could work around the latter by using a cast, for example, but then -the code would be ugly. Therefore we discourage the use of such pattern. +the code would be ugly. Therefore we discourage the use of this pattern. References From cb65bff9f4e891e33dc0240bd05949c8295e8b0a Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Fri, 10 Mar 2017 08:44:53 +0100 Subject: [PATCH 25/37] Rename PEP with a valid number to get the build running --- pep-05xx.txt => pep-0544.txt | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) rename pep-05xx.txt => pep-0544.txt (99%) diff --git a/pep-05xx.txt b/pep-0544.txt similarity index 99% rename from pep-05xx.txt rename to pep-0544.txt index 06c13c3a871..5e9f5a8d74e 100644 --- a/pep-05xx.txt +++ b/pep-0544.txt @@ -1,4 +1,4 @@ -PEP: 5xx +PEP: 544 Title: Protocols Version: $Revision$ Last-Modified: $Date$ @@ -11,6 +11,9 @@ Created: 05-Mar-2017 Python-Version: 3.7 +.. I have chosen the PEP number just to make the build running, + we will need to ask 'officially' for the PEP number. + Abstract ======== From 817bf2f95d6437df5bdf1a4c76ea9d49f7e42d2c Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Fri, 10 Mar 2017 08:58:59 +0100 Subject: [PATCH 26/37] Reflow to 79 characters --- pep-0544.txt | 44 +++++++++++++++++++++++--------------------- 1 file changed, 23 insertions(+), 21 deletions(-) diff --git a/pep-0544.txt b/pep-0544.txt index 5e9f5a8d74e..d47186fcdee 100644 --- a/pep-0544.txt +++ b/pep-0544.txt @@ -170,7 +170,8 @@ approaches related to structural subtyping in Python and other languages: As mentioned in `rationale`_, we want to avoid such necessity, especially in static context. However, in runtime context, ABCs are good candidates for - protocol classes and they are already used extensively in the ``typing`` module. + protocol classes and they are already used extensively in + the ``typing`` module. * Abstract classes defined in ``collections.abc`` module [collections-abc]_ are slightly more advanced since they implement a custom @@ -348,15 +349,15 @@ Having an empty method body in a protocol type makes the method implicitly abstract only statically, and there won't be any runtime checking. The rationale for this is that most methods in a protocol will be abstract, and having to always use ``@abstractmethod`` is a little verbose and ugly. -Leaving the method body empty is the recommended way of defining protocol members -that don't have a default implementation. Such syntax mirrors how methods are -defined in stub files. Abstract staticmethods and abstract classmethods -are equally supported. +Leaving the method body empty is the recommended way of defining protocol +members that don't have a default implementation. Such syntax mirrors how +methods are defined in stub files. Abstract staticmethods and abstract +classmethods are equally supported. To define a protocol variable, one should use PEP 526 variable annotations in the class body. Attributes defined in the body of a method -by assignment via ``self`` are considered non-protocol members. The rationale for -this is that the protocol class implementation is often not shared by +by assignment via ``self`` are considered non-protocol members. The rationale +for this is that the protocol class implementation is often not shared by subtypes, so the interface should not depend on the default implementation. An exception from these rules is an attribute starting with double underscores and not ending with double underscores always considered @@ -422,8 +423,8 @@ can be used as regular base classes. In this case: represent(BadColor()) # Error, 'add_int' is not implemented implicitly * Type checkers will statically enforce that the class - actually implements the protocol correctly. In particular, type checkers will - always require members without default implementation to be defined in + actually implements the protocol correctly. In particular, type checkers + will always require members without default implementation to be defined in subclasses that explicitly subclass the protocol:: from typing import Protocol, Tuple @@ -447,9 +448,10 @@ can be used as regular base classes. In this case: The general philosophy is that protocols are mostly like regular ABCs, but a static type checker will handle them specially. In particular, a class can explicitly inherit from multiple protocols. -In this case methods are resolved using normal MRO and the type checker verifies -that all subtyping are correct. Note that a concrete method with appropriate -type signature can override a protocol method but not vice versa. Examples:: +In this case methods are resolved using normal MRO and the type checker +verifies that all subtyping are correct. Note that a concrete method with +appropriate type signature can override a protocol method but not vice versa. +Examples:: class A(Protocol): def m(self) -> None: ... @@ -559,8 +561,8 @@ relationships are subject to the following rules: Generic protocol types follow the same rules of variance as non-protocol types. Protocols can be used in all special type constructors provided -by the ``typing`` module and follow the corresponding subtyping rules. For example, -protocol ``PInt`` is a subtype of ``Union[PInt, int]``. +by the ``typing`` module and follow the corresponding subtyping rules. +For example, protocol ``PInt`` is a subtype of ``Union[PInt, int]``. ``Union[]`` and ``All[]`` @@ -746,10 +748,10 @@ runtime, which is unfortunate:: print(b.x) # 1 print(B.x) # Ellipsis -For this reason we propose to make the ``Protocol`` metaclass recognize ``...`` -initializers and translate them to a special property that can't be read. -The ``typing`` module changes proposed in this PEP will be also backported to -earlier versions via the backport currently available on PyPI. +For this reason we propose to make the ``Protocol`` metaclass recognize +``...`` initializers and translate them to a special property that can't +be read. The ``typing`` module changes proposed in this PEP will be also +backported to earlier versions via the backport currently available on PyPI. Runtime Implementation of Protocol Classes @@ -818,9 +820,9 @@ Introspection The existing class introspection machinery (``dir``, ``__annotations__`` etc) can be used with protocols. In addition, all introspection tools implemented -in the ``typing`` module will support protocols. Since all attributes need to be -defined in the class body based on this proposal, protocol classes will have -even better perspective for introspection than regular classes where +in the ``typing`` module will support protocols. Since all attributes need +to be defined in the class body based on this proposal, protocol classes will +have even better perspective for introspection than regular classes where attributes can be defined implicitly -- protocol attributes can't be initialized in ways that are not visible to introspection (using ``setattr()``, assignment via ``self``, etc.). Still, some things like From 2d89ba950431889746810f7a39ac8117b40fff81 Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Fri, 10 Mar 2017 08:19:30 -0800 Subject: [PATCH 27/37] fix typo --- pep-0544.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0544.txt b/pep-0544.txt index d47186fcdee..3da9bd9c5fc 100644 --- a/pep-0544.txt +++ b/pep-0544.txt @@ -593,7 +593,7 @@ In addition, we propose to add another special type construct it is not very useful, there are many situation where a variable should implement more than one protocol. Annotation by ``All[Proto1, Proto2, ...]`` means that a given variable or parameter must implement all protocols -``Ptoro1``, ``Proto2``, etc. either implicitly or explicitly. Example:: +``Proto1``, ``Proto2``, etc. either implicitly or explicitly. Example:: from typing import Sequence, Hashable, All From 0efcbff45e26229082ec20b5d97a84f5819cba3a Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Fri, 10 Mar 2017 12:04:47 -0800 Subject: [PATCH 28/37] Some grammar tweaks --- pep-0544.txt | 42 +++++++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/pep-0544.txt b/pep-0544.txt index 3da9bd9c5fc..fdce24d94b3 100644 --- a/pep-0544.txt +++ b/pep-0544.txt @@ -50,8 +50,8 @@ of the library. Moreover, extensive use of ABCs might impose additional runtime costs. The intention of this PEP is to solve all these problems -by allowing to write the above code without explicit base classes in -the class definition, and ``Bucket`` would still be implicitly considered +by allowing users to write the above code without explicit base classes in +the class definition, allowing ``Bucket`` to be implicitly considered a subtype of both ``Sized`` and ``Iterable[int]`` by static type checkers using structural [wiki-structural]_ subtyping:: @@ -66,7 +66,7 @@ using structural [wiki-structural]_ subtyping:: result: int = collect(Bucket()) # Passes type check The same functionality will be provided for user-defined protocols, as -specified below. The code with a protocol class matches common Python +specified below. The above code with a protocol class matches common Python conventions much better. It is also automatically extensible and works with additional, unrelated classes that happen to implement the required protocol. @@ -158,9 +158,9 @@ approaches related to structural subtyping in Python and other languages: The drawback of this approach is the necessity to either subclass the abstract class or register an implementation explicitly:: - from abc import ABCMeta + from abc import ABC - class MyTuple(metaclass=ABCMeta): + class MyTuple(ABC): pass MyTuple.register(tuple) @@ -168,8 +168,8 @@ approaches related to structural subtyping in Python and other languages: assert issubclass(tuple, MyTuple) assert isinstance((), MyTuple) - As mentioned in `rationale`_, we want to avoid such necessity, especially - in static context. However, in runtime context, ABCs are good candidates for + As mentioned in the `rationale`_, we want to avoid such necessity, especially + in static context. However, in a runtime context, ABCs are good candidates for protocol classes and they are already used extensively in the ``typing`` module. @@ -319,9 +319,9 @@ Protocol members There are two ways to define methods that are protocol members: * Use the ``@abstractmethod`` decorator -* Leave the method body empty, i.e. it should be either a single ``pass`` - statement, a single ``...`` ellipsis literal, or a single - ``raise NotImplementedError`` statement. +* Leave the method body empty, i.e. it should be either docstring, + a single ``pass`` statement, a single ``...`` ellipsis literal, + or a single ``raise NotImplementedError`` statement. A combination of both may be used if necessary for runtime purposes. If the method body is not empty but decorated by the ``@abstractmethod`` @@ -383,7 +383,7 @@ annotation should be used as specified by PEP 526. Explicitly declaring implementation ----------------------------------- -To explicitly declare that a certain class implements a given protocols, they +To explicitly declare that a certain class implements the given protocols, they can be used as regular base classes. In this case: * A class could use implementations of some methods in protocols, both @@ -545,7 +545,7 @@ relationships are subject to the following rules: if and only if ``X`` implements all protocol members of ``P``. In other words, subtyping with respect to a protocol is always structural. * Edge case: for recursive protocols, structural subtyping is decided - positively for all situations that are type safe. Continuing + positively for all situations that are type safe. Continuing the previous example:: class Tree(Generic[T]): @@ -590,7 +590,7 @@ classes. For example:: In addition, we propose to add another special type construct ``All`` that represents intersection types. Although for normal types -it is not very useful, there are many situation where a variable should +it is not very useful, there are many situations where a variable should implement more than one protocol. Annotation by ``All[Proto1, Proto2, ...]`` means that a given variable or parameter must implement all protocols ``Proto1``, ``Proto2``, etc. either implicitly or explicitly. Example:: @@ -630,7 +630,7 @@ The same rule applies to variables:: var = Concrete # OK var().meth() # OK -Assigning protocol class to a variable is allowed if it is not explicitly +Assigning a protocol class to a variable is allowed if it is not explicitly typed, and such assignment creates a type alias. For non-protocol classes, the behavior of ``Type[]`` is not changed. @@ -672,8 +672,8 @@ at runtime. However, it should be possible for protocol types to implement custom instance and class checks when this makes sense, similar to how ``Iterable`` and other ABCs in ``collections.abc`` and ``typing`` already do it, -but this is limited to non-generic and to unsubscripted generic protocols -(statically ``Iterable`` is equivalent to ``Iterable[Any]`). +but this is limited to non-generic and unsubscripted generic protocols +(``Iterable`` is statically equivalent to ``Iterable[Any]`). The ``typing`` module will define a special ``@runtime`` class decorator that automatically adds a ``__subclasshook__()`` that provides the same semantics for class and instance checks as for ``collections.abc`` classes:: @@ -761,7 +761,7 @@ Implementation details ---------------------- The runtime implementation could be done in pure Python without any -effects on the core interpreter and standard library except in +effects on the core interpreter and standard library except in the ``typing`` module: * Define class ``typing.Protocol`` similar to ``typing.Generic``. @@ -779,8 +779,8 @@ effects on the core interpreter and standard library except in be provided at runtime. -Changes in typing module ------------------------- +Changes in the typing module +---------------------------- The following classes in ``typing`` module will be protocols: @@ -896,8 +896,8 @@ Use assignments to check explicitly that a class implements a protocol ---------------------------------------------------------------------- In Go language the explicit checks for implementation are performed -via dummy assignments [golang]_. Such way is also possible with the current -proposal. Example:: +via dummy assignments [golang]_. Such a way is also possible with the +current proposal. Example:: class A: def __len__(self) -> float: From 0de36bef2d8428efcf047252d40e903d52cda5da Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Sat, 11 Mar 2017 08:44:27 +0100 Subject: [PATCH 29/37] Implement Guido's idea of EIBTI plus minor comments --- pep-0544.txt | 205 +++++++++++++++++++++++---------------------------- 1 file changed, 91 insertions(+), 114 deletions(-) diff --git a/pep-0544.txt b/pep-0544.txt index fdce24d94b3..4d1603d4935 100644 --- a/pep-0544.txt +++ b/pep-0544.txt @@ -188,10 +188,9 @@ approaches related to structural subtyping in Python and other languages: This seems to be a perfect fit for both runtime and static behavior of protocols. The main goal of this proposal is to support such behavior - statically. In addition, to allow users achieving (if necessary) such - runtime behavior for user defined protocols, a simple class decorator - ``@runtime`` can be provided that automatically adds a - ``__subclasshook__()`` that mimics the one in ``collections.abc``, + statically. In addition, to allow users achieving such runtime behavior + for user defined protocols a ``__subclasshook__()`` that mimics the one in + ``collections.abc`` can be automatically added, see detailed `discussion`_ below. * TypeScript [typescript]_ provides support for user defined classes and @@ -295,8 +294,8 @@ protocol types:: self.file.close() self.lock.release() -Protocol types can be used in function annotations, variable annotations, -and for type checking:: +Apart from few restrictions explicitly mentioned below, protocol types can +be used in every context where a normal types can:: def close_all(things: Iterable[SupportsClose]) -> None: for t in things: @@ -316,17 +315,15 @@ a compatible type signature. Protocol members ---------------- -There are two ways to define methods that are protocol members: +The methods that are protocol members are defined using the +``@abstractmethod`` decorator. Such methods can have a default +implementation that will be type checked. Method body can be left empty, +i.e. be either only a docstring or a single ``...`` ellipsis literal. +In this case type checkers will recognize that there is no default +implementation, i.e. it can't be used via ``super()`` in explicit +subclasses. -* Use the ``@abstractmethod`` decorator -* Leave the method body empty, i.e. it should be either docstring, - a single ``pass`` statement, a single ``...`` ellipsis literal, - or a single ``raise NotImplementedError`` statement. - -A combination of both may be used if necessary for runtime purposes. -If the method body is not empty but decorated by the ``@abstractmethod`` -decorator, then the corresponding definition serves as a default -method implementation. If some or all parameters are not annotated +If some or all parameters of protocol method are not annotated their types are assumed to be ``Any`` (see PEP 484). All other methods defined in the protocol class are considered non-protocol members. Example:: @@ -337,24 +334,21 @@ Example:: @abstractmethod def first(self) -> int: # This is a protocol member return 1 + + @abstractmethod def second(self) -> int: # This one also ... - def third(self) -> int: # And this one too - raise NotImplementedError('Implement this!') def auxiliary(self) -> str: # This is NOT a protocol member x = 1 -Having an empty method body in a protocol type makes the method -implicitly abstract only statically, and there won't be any runtime checking. -The rationale for this is that most methods in a protocol will be abstract, -and having to always use ``@abstractmethod`` is a little verbose and ugly. -Leaving the method body empty is the recommended way of defining protocol -members that don't have a default implementation. Such syntax mirrors how -methods are defined in stub files. Abstract staticmethods and abstract -classmethods are equally supported. +Note that although formally the implicit return type of a method with +an empty body is ``None``, type checker will not warn about above example, +such convention mirrors how methods are defined in stub files. +Abstract staticmethods, abstract classmethods, and abstract properties are +equally supported. -To define a protocol variable, one should use PEP 526 variable +To define a protocol variable, one must use PEP 526 variable annotations in the class body. Attributes defined in the body of a method by assignment via ``self`` are considered non-protocol members. The rationale for this is that the protocol class implementation is often not shared by @@ -373,11 +367,9 @@ non-protocol member. Examples:: def method(self) -> None: self.temp: List[int] = [] # And this one neither -Note that unannotated variables are always assumed to have type ``Any``, -so that a definition ``x = 0`` in a protocol body will create a protocol -member ``x`` with type ``Any``, *not* ``int``. To distinguish between protocol -class variables and protocol instance variables, the special ``ClassVar`` -annotation should be used as specified by PEP 526. +To distinguish between protocol class variables and protocol instance +variables, the special ``ClassVar`` annotation should be used as specified +by PEP 526. Explicitly declaring implementation @@ -391,46 +383,49 @@ can be used as regular base classes. In this case: (``typing.Sequence`` is an example with useful default methods). In particular, a protocol could provide a default implementation of ``__init__`` that could be used in explicit subclasses. - The default implementations will not be used if the subtype relationship + The default implementations can not be used if the subtype relationship is implicit and only via structural subtyping -- the semantics of inheritance is not changed. Examples:: from typing import Protocol class PColor(Protocol): - def description(self) -> str: + @abstractmethod + def draw(self) -> str: ... - def simple_description(self) -> str: # this is a non-protocol member - rerun 'nice color' @abstractmethod - def add_int(self, x: int) -> int: # protocol method with default + def with_int(self, x: int) -> int: return 42 + def complex_method(self) -> int: + ... + # some complex code here + class SimpleColor(PColor): - def description(self) -> str: - return super().simple_description() - def add_int(self, x: int) -> int: - return super().add_int(x) + x + def draw(self) -> str: + return super().draw() # Error, no default implementation - class BadColor: - def description(self) -> str: - return 'not so nice' + def with_int(self, x: int) -> int: + return super().with_int(x) + x + + def other_method(self) -> None: + x = super().complex_method() + ... simple: SimpleColor def represent(c: PColor) -> None: ... represent(simple) # OK - represent(BadColor()) # Error, 'add_int' is not implemented implicitly * Type checkers will statically enforce that the class - actually implements the protocol correctly. In particular, type checkers - will always require members without default implementation to be defined in - subclasses that explicitly subclass the protocol:: + actually implements the protocol correctly:: from typing import Protocol, Tuple class RGB(Protocol): rgb: Tuple[int, int, int] + + @abstractmethod def intensity(self) -> int: ... @abstractmethod @@ -440,24 +435,32 @@ can be used as regular base classes. In this case: class Point(RGB): def __init__(self, red: int, green: int, blue: str) -> None: self.rgb = red, green, blue # Error, 'blue' must be 'int' - # Error, 'intensity' is not defined! - # Note that 'to_bytes' is not required, but this is safe - # since the default implementation is actually in the MRO. + def to_bytes(self) -> bytes: + return super().to_bytes() + + # Error, 'intensity' is not defined! The general philosophy is that protocols are mostly like regular ABCs, -but a static type checker will handle them specially. -In particular, a class can explicitly inherit from multiple protocols. +but a static type checker will handle them specially. Namely, type checkers +will enforce that the implementation does not have abstract members. + +A class can explicitly inherit from multiple protocols. In this case methods are resolved using normal MRO and the type checker verifies that all subtyping are correct. Note that a concrete method with appropriate type signature can override a protocol method but not vice versa. Examples:: class A(Protocol): - def m(self) -> None: ... + @abstractmethod + def m(self) -> str: ... + @abstractmethod def foo(self) -> int: ... + class B(Protocol): - def m(self) -> None: print('Hi') + def m(self) -> str: + return 'non abstract' + @abstractmethod def bar(self) -> str: ... class C(A, B): # Fails typecheck, protocol member 'A.m' can't override 'B.m' @@ -506,6 +509,10 @@ If one omits ``Protocol`` in the base class list, this would be a regular is included in the base class list, all the other base classes must be protocols. A protocol can't extend a regular class. +Note that rules around explicit subclassing are different from regular ABCs, +where abstractness is simply defined by having at least one abstract method +being unimplemented. Protocol classes must be marked *explicitly*. + Generic and recursive protocols ------------------------------- @@ -521,10 +528,12 @@ non-protocol generic types:: ... Note that ``Protocol[T, S, ...]`` is allowed as a shorthand for -``Protocol, Generic[T, S, ...]``. Recursive protocols are also supported. -Forward references to the protocol class names can be given as strings as -specified by PEP 484. Recursive protocols will be useful for representing -self-referential data structures like trees in an abstract fashion:: +``Protocol, Generic[T, S, ...]``. + +Recursive protocols are also supported. Forward references to the protocol +class names can be given as strings as specified by PEP 484. Recursive +protocols will be useful for representing self-referential data structures +like trees in an abstract fashion:: class Traversable(Protocol): leafs: Iterable['Traversable'] @@ -545,8 +554,8 @@ relationships are subject to the following rules: if and only if ``X`` implements all protocol members of ``P``. In other words, subtyping with respect to a protocol is always structural. * Edge case: for recursive protocols, structural subtyping is decided - positively for all situations that are type safe. Continuing the - previous example:: + positively for situations where such decision depends on itself. Continuing + the previous example:: class Tree(Generic[T]): def __init__(self, value: T, @@ -661,26 +670,19 @@ aliases:: .. _discussion: -``@runtime`` decorator and narrowing types by ``isinstance()`` --------------------------------------------------------------- - -The default semantics is that ``isinstance()`` and ``issubclass()`` fail -for protocol types. This is in the spirit of duck typing -- protocols -basically would be used to model duck typing statically, not explicitly -at runtime. +Narrowing types by ``isinstance()`` +----------------------------------- -However, it should be possible for protocol types to implement custom -instance and class checks when this makes sense, similar to how ``Iterable`` -and other ABCs in ``collections.abc`` and ``typing`` already do it, -but this is limited to non-generic and unsubscripted generic protocols +Protocol types will implement special instance and class checks, similar to +how ``Iterable`` and other ABCs in ``collections.abc`` and ``typing`` already +do it, but this is limited to non-generic and unsubscripted generic protocols (``Iterable`` is statically equivalent to ``Iterable[Any]`). -The ``typing`` module will define a special ``@runtime`` class decorator that -automatically adds a ``__subclasshook__()`` that provides the same semantics +The metaclass of ``typing.Protocol`` will automatically add +a ``__subclasshook__()`` that provides the same semantics for class and instance checks as for ``collections.abc`` classes:: from typing import runtime, Protocol - @runtime class Closeable(Protocol): @abstractmethod def close(self): @@ -688,16 +690,13 @@ for class and instance checks as for ``collections.abc`` classes:: assert isinstance(open('some/file'), Closeable) -A static type checker will ensure that this decorator is only applied to -classes where all protocol members are defined by ``@abstractmethod`` for -methods and by a variable annotation for class and instance variables. Static type checkers will understand ``isinstance(x, Proto)`` and -``issublclass(C, Proto)`` for protocols decorated with such decorator -(as they already do for ``Iterable`` etc.). Static type checkers will -narrow types after such checks by the type erased ``Proto`` (i.e. with all -variables having type ``Any`` and all methods having type -``Callable[..., Any]``). Note that ``isinstance(x, Proto[int])`` etc. will -always fail in agreement with PEP 484. Examples:: +``issublclass(C, Proto)`` for protocols (as they already do +for ``Iterable`` etc.). Static type checkers will narrow types after such +checks by the type erased ``Proto`` (i.e. with all variables having type +``Any`` and all methods having type ``Callable[..., Any]``). +Note that ``isinstance(x, Proto[int])`` etc. will always fail in agreement +with PEP 484. Examples:: from typing import Iterable, Iterator, Sequence @@ -713,17 +712,11 @@ Using Protocols in Python 3.5 and Earlier Variable annotation syntax was added in Python 3.6, so that the syntax for defining protocol variables proposed in `specification`_ section can't -be used in earlier versions. To define these in earlier versions of Python we -propose to either use type comments (and ellipsis literals if there is no -default value), or use abstract properties if it is necessary to indicate -abstract status at runtime. All of these are valid:: +be used in earlier versions. To define these in earlier versions of Python +one can use abstract properties:: class Foo(Protocol): - - x = ... # type: int - s = 'abc' # type: str # Protocol instance variable with default. - - @property + @abstractproperty def c(self) -> int: ... @abstractproperty @@ -731,26 +724,10 @@ abstract status at runtime. All of these are valid:: return 0 @property - def e(self) -> int: # This is not a protocol member. + def e(self) -> int: # Note, this is not a protocol member. return 0 -The ellipsis ``...`` initializer can leak into subclasses at -runtime, which is unfortunate:: - - class A(Protocol): - x = ... # type: int - - class B(A): - def __init__(self) -> None: - self.x = 1 - - b = B() - print(b.x) # 1 - print(B.x) # Ellipsis - -For this reason we propose to make the ``Protocol`` metaclass recognize -``...`` initializers and translate them to a special property that can't -be read. The ``typing`` module changes proposed in this PEP will be also +The ``typing`` module changes proposed in this PEP will be also backported to earlier versions via the backport currently available on PyPI. @@ -769,9 +746,9 @@ effects on the core interpreter and standard library except in the a protocol or not. Add a class attribute ``__protocol__ = True`` if that is the case. Verify that a protocol class only has protocol base classes in the MRO (except for object). -* Implement ``@runtime`` class decorator that makes ``isinstance()`` behave - as for ``collections.abc`` by adding all abstract methods and all variables - from ``__annotations__`` to ``__subclsshook__()``. +* Makes ``isinstance()`` behave as for ``collections.abc`` by adding all + abstract methods and all variables from ``__annotations__`` to + ``__subclsshook__()``. * In the backported version of ``typing``, translate ``...`` class attribute values to properties. * All structural subtyping checks will be performed by static type checkers, From 767c58bd7004ea74cc44c1c7437db945a8842735 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Sat, 11 Mar 2017 08:55:23 +0100 Subject: [PATCH 30/37] Fix typo --- pep-0544.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0544.txt b/pep-0544.txt index 4d1603d4935..e5b008efb35 100644 --- a/pep-0544.txt +++ b/pep-0544.txt @@ -691,7 +691,7 @@ for class and instance checks as for ``collections.abc`` classes:: assert isinstance(open('some/file'), Closeable) Static type checkers will understand ``isinstance(x, Proto)`` and -``issublclass(C, Proto)`` for protocols (as they already do +``issubclass(C, Proto)`` for protocols (as they already do for ``Iterable`` etc.). Static type checkers will narrow types after such checks by the type erased ``Proto`` (i.e. with all variables having type ``Any`` and all methods having type ``Callable[..., Any]``). From efc31547ab955a1180199f3aa7fb316724b6cebc Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Sun, 12 Mar 2017 16:49:28 +0100 Subject: [PATCH 31/37] Make implementation enforcement optional; fix order of Protocolbase --- pep-0544.txt | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/pep-0544.txt b/pep-0544.txt index e5b008efb35..a52f74839c9 100644 --- a/pep-0544.txt +++ b/pep-0544.txt @@ -417,8 +417,8 @@ can be used as regular base classes. In this case: ... represent(simple) # OK -* Type checkers will statically enforce that the class - actually implements the protocol correctly:: +* Type checkers might statically enforce that the class actually + implements the protocol correctly:: from typing import Protocol, Tuple @@ -439,11 +439,14 @@ can be used as regular base classes. In this case: def to_bytes(self) -> bytes: return super().to_bytes() - # Error, 'intensity' is not defined! + # Type checker might warn that 'intensity' is not defined The general philosophy is that protocols are mostly like regular ABCs, -but a static type checker will handle them specially. Namely, type checkers -will enforce that the implementation does not have abstract members. +but a static type checker will handle them specially. Subclassing a protocol +class would not turn the subclass into a protocol unless it also has +``typing.Protocol`` as an explicit base class. Without this base, subclassing +"downgrades" the class to a regular ABC that cannot be used with structural +subtyping. See section on `extending`_ for details of defining subprotocols. A class can explicitly inherit from multiple protocols. In this case methods are resolved using normal MRO and the type checker @@ -470,9 +473,8 @@ Examples:: return 'bar' # Error, 'foo' is not implemented (although 'm' is not needed here). -Subclassing a protocol class would not turn the subclass into -a protocol unless it also has ``typing.Protocol`` as an explicit base class. +.. _extending : Merging and extending protocols ------------------------------- @@ -483,7 +485,7 @@ having ``typing.Protocol`` as an immediate base class:: from typing import Sized, Protocol - class SizedAndCloseable(Protocol, Sized): + class SizedAndCloseable(Sized, Protocol): def close(self) -> None: ... @@ -496,7 +498,7 @@ the example in `definition`_ section:: class SupportsClose(...): ... # Like above - class SizedAndCloseable(Protocol, Sized, SupportsClose): + class SizedAndCloseable(Sized, SupportsClose, Protocol): pass The two definitions of ``SizedAndClosable`` are equivalent. From 7d714c3c5ec20677da91ef5ce8230c271f113a14 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Mon, 13 Mar 2017 08:59:59 +0100 Subject: [PATCH 32/37] Add missing @abstractmethod decorators --- pep-0544.txt | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/pep-0544.txt b/pep-0544.txt index a52f74839c9..8fb04419fb9 100644 --- a/pep-0544.txt +++ b/pep-0544.txt @@ -280,6 +280,7 @@ at the beginning of the list. Here is a simple example:: from typing import Protocol class SupportsClose(Protocol): + @abstractmethod def close(self) -> None: ... @@ -486,6 +487,7 @@ having ``typing.Protocol`` as an immediate base class:: from typing import Sized, Protocol class SizedAndCloseable(Sized, Protocol): + @abstractmethod def close(self) -> None: ... @@ -526,6 +528,7 @@ non-protocol generic types:: T = TypeVar('T', covariant=True) class Iterable(Protocol[T]): + @abstractmethod def __iter__(self) -> Iterator[T]: ... @@ -585,9 +588,11 @@ classes. For example:: from typing import Union, Optional, Protocol class Exitable(Protocol): + @abstracrmethod def exit(self) -> int: ... class Quitable(Protocol): + @abstractmethod def quit(self) -> Optional[int]: ... @@ -623,6 +628,7 @@ Variables and parameters annotated with ``Type[Proto]`` accept only concrete (non-protocol) subtypes of ``Proto``. For example:: class Proto(Protocol): + @abstractmethod def meth(self) -> int: ... class Concrete: From d4ab050c087c9906ba8854e37daa75b0324d7b27 Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Mon, 13 Mar 2017 09:03:35 +0100 Subject: [PATCH 33/37] Minor clarification --- pep-0544.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0544.txt b/pep-0544.txt index 8fb04419fb9..d4e8241e413 100644 --- a/pep-0544.txt +++ b/pep-0544.txt @@ -186,8 +186,8 @@ approaches related to structural subtyping in Python and other languages: assert isinstance(MyIterable(), Iterable) - This seems to be a perfect fit for both runtime and static behavior of - protocols. The main goal of this proposal is to support such behavior + Such behavior seems to be a perfect fit for both runtime and static behavior + of protocols. The main goal of this proposal is to support such behavior statically. In addition, to allow users achieving such runtime behavior for user defined protocols a ``__subclasshook__()`` that mimics the one in ``collections.abc`` can be automatically added, From d9d21c21f6ce3926b350cc7047d42018e6c94cac Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Thu, 16 Mar 2017 22:42:41 +0100 Subject: [PATCH 34/37] Implement Jukka's and David's comments; few more minor things --- pep-0544.txt | 301 +++++++++++++++++++++++++++------------------------ 1 file changed, 160 insertions(+), 141 deletions(-) diff --git a/pep-0544.txt b/pep-0544.txt index d4e8241e413..537159a304b 100644 --- a/pep-0544.txt +++ b/pep-0544.txt @@ -11,9 +11,6 @@ Created: 05-Mar-2017 Python-Version: 3.7 -.. I have chosen the PEP number just to make the build running, - we will need to ask 'officially' for the PEP number. - Abstract ======== @@ -94,10 +91,14 @@ At runtime, protocol classes will be simple ABCs. There is no intent to provide sophisticated runtime instance and class checks against protocol classes. This would be difficult and error-prone and will contradict the logic of PEP 484. As well, following PEP 484 and PEP 526 we state that protocols are -**completely optional** and there is no intent to make them required. -No runtime semantics will be imposed for variables or parameters annotated -with a protocol class. The actual checks will be performed by third-party type -checkers and other tools. +**completely optional**: + +* No runtime semantics will be imposed for variables or parameters annotated + with a protocol class. +* Any checks will be performed only by third-party type checkers and + other tools. +* Programmers are free to not use them even if they use type annotations. +* There is no intent to make protocols non-optional in the future. Existing Approaches to Structural Subtyping @@ -189,9 +190,8 @@ approaches related to structural subtyping in Python and other languages: Such behavior seems to be a perfect fit for both runtime and static behavior of protocols. The main goal of this proposal is to support such behavior statically. In addition, to allow users achieving such runtime behavior - for user defined protocols a ``__subclasshook__()`` that mimics the one in - ``collections.abc`` can be automatically added, - see detailed `discussion`_ below. + for user defined protocols a special ``@runtime`` decorator will be + provided, see detailed `discussion`_ below. * TypeScript [typescript]_ provides support for user defined classes and interfaces. Explicit implementation declaration is not required and @@ -264,8 +264,7 @@ of the protocol. The attributes (variables and methods) of a protocol that are mandatory for other class in order to be considered a structural subtype are called -protocol members. Other members defined in a protocol class are called -non-protocol members. +protocol members. .. _definition: @@ -275,12 +274,11 @@ Defining a protocol Protocols are defined by including a special new class ``typing.Protocol`` (an instance of ``abc.ABCMeta``) in the base classes list, preferably -at the beginning of the list. Here is a simple example:: +at the end of the list. Here is a simple example:: from typing import Protocol class SupportsClose(Protocol): - @abstractmethod def close(self) -> None: ... @@ -316,57 +314,45 @@ a compatible type signature. Protocol members ---------------- -The methods that are protocol members are defined using the -``@abstractmethod`` decorator. Such methods can have a default -implementation that will be type checked. Method body can be left empty, -i.e. be either only a docstring or a single ``...`` ellipsis literal. -In this case type checkers will recognize that there is no default -implementation, i.e. it can't be used via ``super()`` in explicit -subclasses. - -If some or all parameters of protocol method are not annotated -their types are assumed to be ``Any`` (see PEP 484). All other -methods defined in the protocol class are considered non-protocol members. -Example:: +All methods defined in the protocol class body are protocol members, both +normal and decorated with ``@abstractmethod``. If some or all parameters of +protocol method are not annotated, then their types are assumed to be ``Any`` +(see PEP 484). Bodies of protocol methods are type checked, except for methods +decorated with ``@abstractmethod`` with trivial bodies. A trivial body can +contain a docstring. Example:: from typing import Protocol + from abc import abstractmethod class Example(Protocol): - @abstractmethod def first(self) -> int: # This is a protocol member - return 1 + return 42 @abstractmethod - def second(self) -> int: # This one also - ... - - def auxiliary(self) -> str: # This is NOT a protocol member - x = 1 + def second(self) -> int: # Method without a default implementation + """Some method.""" Note that although formally the implicit return type of a method with -an empty body is ``None``, type checker will not warn about above example, -such convention mirrors how methods are defined in stub files. -Abstract staticmethods, abstract classmethods, and abstract properties are -equally supported. +a trivial body is ``None``, type checker will not warn about above example, +such convention is similar to how methods are defined in stub files. +Static methods, class methods, and properties are equally allowed +in protocols. To define a protocol variable, one must use PEP 526 variable annotations in the class body. Attributes defined in the body of a method -by assignment via ``self`` are considered non-protocol members. The rationale +by assignment via ``self`` are not allowed. The rationale for this is that the protocol class implementation is often not shared by subtypes, so the interface should not depend on the default implementation. -An exception from these rules is an attribute starting with double -underscores and not ending with double underscores always considered -non-protocol member. Examples:: +Examples:: from typing import Protocol, List class Template(Protocol): name: str # This is a protocol member value: int = 0 # This one too (with default) - __private: bool # This is NOT a protocol member def method(self) -> None: - self.temp: List[int] = [] # And this one neither + self.temp: List[int] = [] # Error in type checker To distinguish between protocol class variables and protocol instance variables, the special ``ClassVar`` annotation should be used as specified @@ -376,103 +362,77 @@ by PEP 526. Explicitly declaring implementation ----------------------------------- -To explicitly declare that a certain class implements the given protocols, they -can be used as regular base classes. In this case: +To explicitly declare that a certain class implements the given protocols, +they can be used as regular base classes. In this case a class could use +default implementations of protocol members. ``typing.Sequence`` is a good +example of a protocol with useful default methods. -* A class could use implementations of some methods in protocols, both - protocol members with default implementations and non-protocol members - (``typing.Sequence`` is an example with useful default methods). - In particular, a protocol could provide a default implementation of - ``__init__`` that could be used in explicit subclasses. - The default implementations can not be used if the subtype relationship - is implicit and only via structural subtyping -- the semantics of - inheritance is not changed. Examples:: - - from typing import Protocol +Abstract methods with trivial bodies are recognized by type checkers as +having no default implementation and can't be used via ``super()`` in +explicit subclasses. The default implementations can not be used if +the subtype relationship is implicit and only via structural +subtyping -- the semantics of inheritance is not changed. Examples:: class PColor(Protocol): @abstractmethod def draw(self) -> str: ... - @abstractmethod - def with_int(self, x: int) -> int: - return 42 - def complex_method(self) -> int: - ... # some complex code here - class SimpleColor(PColor): + class NiceColor(PColor): + def draw(self) -> str: + return "deep blue" + + class BadColor(PColor): def draw(self) -> str: return super().draw() # Error, no default implementation - def with_int(self, x: int) -> int: - return super().with_int(x) + x + class ImplicitColor: # Note no 'PColor' base here + def draw(self) -> str: + return "probably gray" + def comlex_method(self) -> int: + # class needs to implement this - def other_method(self) -> None: - x = super().complex_method() - ... + nice: NiceColor + another: ImplicitColor - simple: SimpleColor def represent(c: PColor) -> None: - ... - represent(simple) # OK + print(c.draw(), c.complex_method()) -* Type checkers might statically enforce that the class actually - implements the protocol correctly:: + represent(nice) # OK + represent(another) # Also OK - from typing import Protocol, Tuple +Note that there are no conceptual difference between explicit an implicit +subtypes, the main benefit of explicit subclassing is to get some protocol +methods "for free". In addition, type checkers can statically verify that +the class actually implements the protocol correctly:: class RGB(Protocol): rgb: Tuple[int, int, int] @abstractmethod def intensity(self) -> int: - ... - @abstractmethod - def to_bytes(self) -> bytes: - return b'' + return 0 class Point(RGB): def __init__(self, red: int, green: int, blue: str) -> None: self.rgb = red, green, blue # Error, 'blue' must be 'int' - def to_bytes(self) -> bytes: - return super().to_bytes() - # Type checker might warn that 'intensity' is not defined The general philosophy is that protocols are mostly like regular ABCs, but a static type checker will handle them specially. Subclassing a protocol class would not turn the subclass into a protocol unless it also has -``typing.Protocol`` as an explicit base class. Without this base, subclassing -"downgrades" the class to a regular ABC that cannot be used with structural +``typing.Protocol`` as an explicit base class. Without this base, the class +is "downgraded" to a regular ABC that cannot be used with structural subtyping. See section on `extending`_ for details of defining subprotocols. -A class can explicitly inherit from multiple protocols. -In this case methods are resolved using normal MRO and the type checker -verifies that all subtyping are correct. Note that a concrete method with -appropriate type signature can override a protocol method but not vice versa. -Examples:: - - class A(Protocol): - @abstractmethod - def m(self) -> str: ... - @abstractmethod - def foo(self) -> int: ... - - class B(Protocol): - def m(self) -> str: - return 'non abstract' - @abstractmethod - def bar(self) -> str: ... - - class C(A, B): # Fails typecheck, protocol member 'A.m' can't override 'B.m' - ... - class D(B, A): # OK - def bar(self) -> str: - return 'bar' - # Error, 'foo' is not implemented (although 'm' is not needed here). +A class can explicitly inherit from multiple protocols and also form normal +classes. In this case methods are resolved using normal MRO and a type checker +verifies that all subtyping are correct. The semantics of ``@abstractmethod`` +is not changed, all of them must be implemented by an explicit subclass +before it could be instantiated. .. _extending : @@ -487,14 +447,17 @@ having ``typing.Protocol`` as an immediate base class:: from typing import Sized, Protocol class SizedAndCloseable(Sized, Protocol): - @abstractmethod def close(self) -> None: ... Now the protocol ``SizedAndCloseable`` is a protocol with two methods, -``__len__`` and ``close``. Alternatively, one can implement it -like this, assuming the existence of ``SupportsClose`` from -the example in `definition`_ section:: +``__len__`` and ``close``. If one omits ``Protocol`` in the base class list, +this would be a regular (non-protocol) class that must implement ``Sized``. +If ``Protocol`` is included in the base class list, all the other base classes +must be protocols. A protocol can't extend a regular class. + +Alternatively, one can implement ``SizedAndCloseable`` like this, assuming +the existence of ``SupportsClose`` from the example in `definition`_ section:: from typing import Sized @@ -508,11 +471,6 @@ Subclass relationships between protocols are not meaningful when considering subtyping, since structural compatibility is the criterion, not the MRO. -If one omits ``Protocol`` in the base class list, this would be a regular -(non-protocol) class that must implement ``Sized``. If ``Protocol`` -is included in the base class list, all the other base classes -must be protocols. A protocol can't extend a regular class. - Note that rules around explicit subclassing are different from regular ABCs, where abstractness is simply defined by having at least one abstract method being unimplemented. Protocol classes must be marked *explicitly*. @@ -574,9 +532,11 @@ relationships are subject to the following rules: walk(tree) # OK, 'Tree[float]' is a subtype of 'Traversable' Generic protocol types follow the same rules of variance as non-protocol -types. Protocols can be used in all special type constructors provided -by the ``typing`` module and follow the corresponding subtyping rules. -For example, protocol ``PInt`` is a subtype of ``Union[PInt, int]``. +types. Protocol types can be used in all contexts where any other types +can be used, such as in ``Union``, ``ClassVar``, type variables bounds, etc. +Generic protocols follow the rules for generic abstract classes, except for +using structural compatibility instead of compatibility defined by +inheritance relationships. ``Union[]`` and ``All[]`` @@ -588,11 +548,9 @@ classes. For example:: from typing import Union, Optional, Protocol class Exitable(Protocol): - @abstracrmethod def exit(self) -> int: ... class Quitable(Protocol): - @abstractmethod def quit(self) -> Optional[int]: ... @@ -678,31 +636,37 @@ aliases:: .. _discussion: -Narrowing types by ``isinstance()`` ------------------------------------ +``@runtime`` decorator and narrowing types by ``isinstance()`` +-------------------------------------------------------------- + +The default semantics is that ``isinstance()`` and ``issubclass()`` fail +for protocol types. This is in the spirit of duck typing -- protocols +basically would be used to model duck typing statically, not explicitly +at runtime. -Protocol types will implement special instance and class checks, similar to -how ``Iterable`` and other ABCs in ``collections.abc`` and ``typing`` already -do it, but this is limited to non-generic and unsubscripted generic protocols +However, it should be possible for protocol types to implement custom +instance and class checks when this makes sense, similar to how ``Iterable`` +and other ABCs in ``collections.abc`` and ``typing`` already do it, +but this is limited to non-generic and unsubscripted generic protocols (``Iterable`` is statically equivalent to ``Iterable[Any]`). -The metaclass of ``typing.Protocol`` will automatically add -a ``__subclasshook__()`` that provides the same semantics -for class and instance checks as for ``collections.abc`` classes:: +The ``typing`` module will define a special ``@runtime`` class decorator +that provides the same semantics for class and instance checks as for +``collections.abc`` classes, essentially making them "runtime protocols":: from typing import runtime, Protocol + @runtime class Closeable(Protocol): - @abstractmethod def close(self): ... assert isinstance(open('some/file'), Closeable) Static type checkers will understand ``isinstance(x, Proto)`` and -``issubclass(C, Proto)`` for protocols (as they already do -for ``Iterable`` etc.). Static type checkers will narrow types after such -checks by the type erased ``Proto`` (i.e. with all variables having type -``Any`` and all methods having type ``Callable[..., Any]``). +``issubclass(C, Proto)`` for protocols defined with this decorator (as they +already do for ``Iterable`` etc.). Static type checkers will narrow types +after such checks by the type erased ``Proto`` (i.e. with all variables +having type ``Any`` and all methods having type ``Callable[..., Any]``). Note that ``isinstance(x, Proto[int])`` etc. will always fail in agreement with PEP 484. Examples:: @@ -714,9 +678,12 @@ with PEP 484. Examples:: elif isinstance(items, Sequence[int]): # Error! Can't use 'isinstance()' with subscripted protocols +Note that instance checks are not 100% reliable statically, this is why +this behavior is opt-in, see section on `rejected`_ ideas for examples. + -Using Protocols in Python 3.5 and Earlier -========================================= +Using Protocols in Python 2.7 - 3.5 +=================================== Variable annotation syntax was added in Python 3.6, so that the syntax for defining protocol variables proposed in `specification`_ section can't @@ -735,6 +702,7 @@ one can use abstract properties:: def e(self) -> int: # Note, this is not a protocol member. return 0 +In Python 2.7 the function type comments should be used as per PEP 484. The ``typing`` module changes proposed in this PEP will be also backported to earlier versions via the backport currently available on PyPI. @@ -754,11 +722,7 @@ effects on the core interpreter and standard library except in the a protocol or not. Add a class attribute ``__protocol__ = True`` if that is the case. Verify that a protocol class only has protocol base classes in the MRO (except for object). -* Makes ``isinstance()`` behave as for ``collections.abc`` by adding all - abstract methods and all variables from ``__annotations__`` to - ``__subclsshook__()``. -* In the backported version of ``typing``, translate ``...`` class attribute - values to properties. +* Implement ``@runtime`` that adds all attributes to ``__subclsshook__()``. * All structural subtyping checks will be performed by static type checkers, such as ``mypy`` [mypy]_. No additional support for protocol validation will be provided at runtime. @@ -826,6 +790,9 @@ give a reasonable perspective for runtime type checking tools. Rejected/Postponed Ideas ======================== +The ideas in this section were previously discussed in [several]_ +[discussions]_ [elsewhere]_. + Make every class a protocol by default -------------------------------------- @@ -836,8 +803,10 @@ to require classes to be explicitly marked as protocols, for the following reasons: * Protocols don't have some properties of regular classes. In particular, - ``isinstance()`` is not always well-defined for protocols, whereas it is - well-defined (and quite commonly used) for regular classes. + ``isinstance()``, as defined for normal classes, is based on the nominal + hierarchy. In order to make everything a protocol by default, and have + ``isinstance()`` work would require changing its semantics, + which won't happen. * Protocol classes should generally not have many method implementations, as they describe an interface, not an implementation. Most classes have many implementations, making them bad protocol classes. @@ -849,6 +818,10 @@ reasons: The ABCs in ``typing`` and ``collections.abc`` are rather an exception, but even they are recent additions to Python and most programmers do not use them yet. +* Many built-in functions only accept concrete instances of ``int`` + (and subclass instances), and similarly for other built-in classes. Making + ``int`` a structural type wouldn't be safe without major changes to the + Python runtime, which won't happen. Support optional protocol members @@ -901,6 +874,43 @@ We could work around the latter by using a cast, for example, but then the code would be ugly. Therefore we discourage the use of this pattern. +Support ``isinstance()`` checks by default +------------------------------------------ + +The problem with this is instance checks could be unreliable, except for +situations where there is a common signature convention such as ``Iterable``. +Another potentially problematic case is assignment of attributes +*after* instantiation:: + + class P(Protocol): + x: int + + class C: + def initialize(self) -> None: + self.x = 0 + + c = C() + isinstance(c1, P) # False + c.initialize() + isinstance(c, P) # True + + def f(x: Union[P, int]) -> None: + if isinstance(x, P): + # type of x is P here + ... + else: + # type of x is int here? + print(x + 1) + + f(C()) # oops + +We argue that requiring an explicit class decorator would be better, since +one can then attach warnings about problems like this in the documentation. +The user would be able to evaluate whether the benefits outweigh +the potential for confusion for each protocol and explicitly opt in -- but +the default behavior would be safer. + + References ========== @@ -931,6 +941,15 @@ References .. [mypy] http://github.com/python/mypy/ +.. [several] + https://mail.python.org/pipermail/python-ideas/2015-September/thread.html#35859 + +.. [discussions] + https://github.com/python/typing/issues/11 + +.. [elsewhere] + https://github.com/python/peps/pull/224 + Copyright ========= From 4dfbfb202cabd6058f00e55ecb7dfa01d7b73faf Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Fri, 17 Mar 2017 11:53:05 +0100 Subject: [PATCH 35/37] =?UTF-8?q?Implement=20most=20comments=20by=20=C5=81?= =?UTF-8?q?ukasz;=20few=20more=20to=20do?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- pep-0544.txt | 103 +++++++++++++++++++++++++-------------------------- 1 file changed, 51 insertions(+), 52 deletions(-) diff --git a/pep-0544.txt b/pep-0544.txt index 537159a304b..a846f123767 100644 --- a/pep-0544.txt +++ b/pep-0544.txt @@ -43,7 +43,7 @@ this conforms to PEP 484:: The same problem appears with user-defined ABCs: they must be explicitly subclassed or registered. This is particularly difficult to do with library types as the type objects may be hidden deep in the implementation -of the library. Moreover, extensive use of ABCs might impose additional +of the library. Also, extensive use of ABCs might impose additional runtime costs. The intention of this PEP is to solve all these problems @@ -62,6 +62,9 @@ using structural [wiki-structural]_ subtyping:: def collect(items: Iterable[int]) -> int: ... result: int = collect(Bucket()) # Passes type check +Note that ABCs in ``typing`` module already provide structural behavior +at runtime, ``isinstance(Bucket(), Iterable)`` returns ``True``. +The main goal of this proposal is to support such behavior statically. The same functionality will be provided for user-defined protocols, as specified below. The above code with a protocol class matches common Python conventions much better. It is also automatically extensible and works @@ -107,7 +110,7 @@ Existing Approaches to Structural Subtyping Before describing the actual specification, we review and comment on existing approaches related to structural subtyping in Python and other languages: -* Zope interfaces [zope-interfaces]_ was one of the first widely used +* ``zope.interface`` [zope-interfaces]_ was one of the first widely used approaches to structural subtyping in Python. It is implemented by providing special classes to distinguish interface classes from normal classes, to mark interface attributes, and to explicitly declare implementation. @@ -188,10 +191,10 @@ approaches related to structural subtyping in Python and other languages: assert isinstance(MyIterable(), Iterable) Such behavior seems to be a perfect fit for both runtime and static behavior - of protocols. The main goal of this proposal is to support such behavior - statically. In addition, to allow users achieving such runtime behavior - for user defined protocols a special ``@runtime`` decorator will be - provided, see detailed `discussion`_ below. + of protocols. As discussed in `rationale`_, we propose to add static support + for such behavior. In addition, to allow users to achieve such runtime + behavior for *user defined* protocols a special ``@runtime`` decorator will + be provided, see detailed `discussion`_ below. * TypeScript [typescript]_ provides support for user defined classes and interfaces. Explicit implementation declaration is not required and @@ -339,8 +342,8 @@ Static methods, class methods, and properties are equally allowed in protocols. To define a protocol variable, one must use PEP 526 variable -annotations in the class body. Attributes defined in the body of a method -by assignment via ``self`` are not allowed. The rationale +annotations in the class body. Additional attributes *only* defined in +the body of a method by assignment via ``self`` are not allowed. The rationale for this is that the protocol class implementation is often not shared by subtypes, so the interface should not depend on the default implementation. Examples:: @@ -403,7 +406,7 @@ subtyping -- the semantics of inheritance is not changed. Examples:: represent(nice) # OK represent(another) # Also OK -Note that there are no conceptual difference between explicit an implicit +Note that there is no conceptual difference between explicit and implicit subtypes, the main benefit of explicit subclassing is to get some protocol methods "for free". In addition, type checkers can statically verify that the class actually implements the protocol correctly:: @@ -421,13 +424,6 @@ the class actually implements the protocol correctly:: # Type checker might warn that 'intensity' is not defined -The general philosophy is that protocols are mostly like regular ABCs, -but a static type checker will handle them specially. Subclassing a protocol -class would not turn the subclass into a protocol unless it also has -``typing.Protocol`` as an explicit base class. Without this base, the class -is "downgraded" to a regular ABC that cannot be used with structural -subtyping. See section on `extending`_ for details of defining subprotocols. - A class can explicitly inherit from multiple protocols and also form normal classes. In this case methods are resolved using normal MRO and a type checker verifies that all subtyping are correct. The semantics of ``@abstractmethod`` @@ -435,14 +431,19 @@ is not changed, all of them must be implemented by an explicit subclass before it could be instantiated. -.. _extending : - Merging and extending protocols ------------------------------- -Subprotocols are also supported. A subprotocol can be defined -by having both one or more protocols as immediate base classes and also -having ``typing.Protocol`` as an immediate base class:: +The general philosophy is that protocols are mostly like regular ABCs, +but a static type checker will handle them specially. Subclassing a protocol +class would not turn the subclass into a protocol unless it also has +``typing.Protocol`` as an explicit base class. Without this base, the class +is "downgraded" to a regular ABC that cannot be used with structural +subtyping. + +A subprotocol can be defined by having *both* one or more protocols as +immediate base classes and also having ``typing.Protocol`` as an immediate +base class:: from typing import Sized, Protocol @@ -499,7 +500,7 @@ protocols will be useful for representing self-referential data structures like trees in an abstract fashion:: class Traversable(Protocol): - leafs: Iterable['Traversable'] + leaves: Iterable['Traversable'] Using Protocols @@ -516,13 +517,13 @@ relationships are subject to the following rules: * A concrete type or a protocol ``X`` is a subtype of another protocol ``P`` if and only if ``X`` implements all protocol members of ``P``. In other words, subtyping with respect to a protocol is always structural. -* Edge case: for recursive protocols, structural subtyping is decided - positively for situations where such decision depends on itself. Continuing - the previous example:: +* Edge case: for recursive protocols, a class is considered a subtype of + the protocol in situations where such decision depends on itself. + Continuing the previous example:: class Tree(Generic[T]): def __init__(self, value: T, - leafs: 'List[Tree[T]]') -> None: + leaves: 'List[Tree[T]]') -> None: self.value = value self.leafs = leafs @@ -539,8 +540,8 @@ using structural compatibility instead of compatibility defined by inheritance relationships. -``Union[]`` and ``All[]`` -------------------------- +Unions and intersections of protocols +------------------------------------- ``Union`` of protocol classes behaves the same way as for non-protocol classes. For example:: @@ -562,28 +563,28 @@ classes. For example:: return 0 finish(GoodJob()) # OK -In addition, we propose to add another special type construct -``All`` that represents intersection types. Although for normal types -it is not very useful, there are many situations where a variable should -implement more than one protocol. Annotation by ``All[Proto1, Proto2, ...]`` -means that a given variable or parameter must implement all protocols -``Proto1``, ``Proto2``, etc. either implicitly or explicitly. Example:: +One can use multiple inheritance to define an intersection of protocols. +Example:: - from typing import Sequence, Hashable, All + from typing import Sequence, Hashable + + class HashableFloats(Sequence[float], Hashable, Protocol): + pass - def cached_func(args: All[Sequence[float], Hashable]) -> float: + def cached_func(args: HashableFloats) -> float: ... - cached_func((1, 2, 3)) # OK, tuple is hashable and sequence + cached_func((1, 2, 3)) # OK, tuple is both hashable and sequence -The interaction between union and intersection types is specified by PEP 483, -and basically reflects the corresponding interactions for sets. +If the this will prove to be a widely used scenario. A special ``Intersection`` +type may be added in future as specified by PEP 483. ``Type[]`` with protocols ------------------------- Variables and parameters annotated with ``Type[Proto]`` accept only concrete -(non-protocol) subtypes of ``Proto``. For example:: +(non-protocol) subtypes of ``Proto``. The main reason for this is to allow +instantiation of parameters with such type. For example:: class Proto(Protocol): @abstractmethod @@ -605,9 +606,10 @@ The same rule applies to variables:: var = Concrete # OK var().meth() # OK -Assigning a protocol class to a variable is allowed if it is not explicitly -typed, and such assignment creates a type alias. For non-protocol classes, -the behavior of ``Type[]`` is not changed. +Assigning an ABC or a protocol class to a variable is allowed if it is +not explicitly typed, and such assignment creates a type alias. +For normal (non-abstract) classes, the behavior of ``Type[]`` is +not changed. ``NewType()`` and type aliases @@ -688,18 +690,15 @@ Using Protocols in Python 2.7 - 3.5 Variable annotation syntax was added in Python 3.6, so that the syntax for defining protocol variables proposed in `specification`_ section can't be used in earlier versions. To define these in earlier versions of Python -one can use abstract properties:: +one can use properties:: class Foo(Protocol): - @abstractproperty - def c(self) -> int: ... + @property + def c(self) -> int: + return 42 # Default value can be provided for property... @abstractproperty - def c(self) -> int: # Default value can be provided for property. - return 0 - - @property - def e(self) -> int: # Note, this is not a protocol member. + def d(self) -> int: # ... or it can be abstract return 0 In Python 2.7 the function type comments should be used as per PEP 484. @@ -722,7 +721,7 @@ effects on the core interpreter and standard library except in the a protocol or not. Add a class attribute ``__protocol__ = True`` if that is the case. Verify that a protocol class only has protocol base classes in the MRO (except for object). -* Implement ``@runtime`` that adds all attributes to ``__subclsshook__()``. +* Implement ``@runtime`` that adds all attributes to ``__subclasshook__()``. * All structural subtyping checks will be performed by static type checkers, such as ``mypy`` [mypy]_. No additional support for protocol validation will be provided at runtime. From d51420eb48a5a5907899a03632e978a30b115cfc Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Sat, 18 Mar 2017 02:32:24 +0100 Subject: [PATCH 36/37] More changes in response to comments --- pep-0544.txt | 49 +++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 43 insertions(+), 6 deletions(-) diff --git a/pep-0544.txt b/pep-0544.txt index a846f123767..10e1bfeba95 100644 --- a/pep-0544.txt +++ b/pep-0544.txt @@ -575,8 +575,9 @@ Example:: ... cached_func((1, 2, 3)) # OK, tuple is both hashable and sequence -If the this will prove to be a widely used scenario. A special ``Intersection`` -type may be added in future as specified by PEP 483. +If this will prove to be a widely used scenario, then a special +intersection type construct may be added in future as specified by PEP 483, +see `rejected`_ ideas for more details. ``Type[]`` with protocols @@ -844,10 +845,17 @@ Make protocols interoperable with other approaches The protocols as described here are basically a minimal extension to the existing concept of ABCs. We argue that this is the way they should -be understood, instead of as something that replaces Zope interfaces, +be understood, instead of as something that *replaces* Zope interfaces, for example. Attempting such interoperabilities will significantly complicate both the concept and the implementation. +On the other hand, Zope interfaces are conceptually a superset of protocols +defined here, but using an incompatible syntax to define them, +because before PEP 526 there was no straightforward way to annotate attributes. +In the 3.6+ world, ``zope.interface`` might potentially adopt the ``Protocol`` +syntax. In this case, type checkers could be taught to recognize interfaces +as protocols and make simple structural checks with respect to them. + Use assignments to check explicitly that a class implements a protocol ---------------------------------------------------------------------- @@ -878,6 +886,19 @@ Support ``isinstance()`` checks by default The problem with this is instance checks could be unreliable, except for situations where there is a common signature convention such as ``Iterable``. +For example:: + + class P(Protocol): + def common_method_name(self, x: int) -> int: ... + + class X: + + def common_method_name(self) -> None: ... # Note different signature + + def do_stuff(o: Union[P, X]) -> int: + if isinstance(o, P): + return o.common_method_name(1) # oops, what if it's an X instance? + Another potentially problematic case is assignment of attributes *after* instantiation:: @@ -895,10 +916,10 @@ Another potentially problematic case is assignment of attributes def f(x: Union[P, int]) -> None: if isinstance(x, P): - # type of x is P here + # static type of x is P here ... else: - # type of x is int here? + # type of x is "int" here? print(x + 1) f(C()) # oops @@ -907,7 +928,23 @@ We argue that requiring an explicit class decorator would be better, since one can then attach warnings about problems like this in the documentation. The user would be able to evaluate whether the benefits outweigh the potential for confusion for each protocol and explicitly opt in -- but -the default behavior would be safer. +the default behavior would be safer. Finally, it will be easy to make this +behaviour default if necessary, while it might be problematic to make it opt-in +after being default. + + +Provide a special intersection type construct +--------------------------------------------- + +There was an idea to allow ``Proto = All[Proto1, Proto2, ...]`` as a shorthand +for:: + + class Proto(Proto1, Proto2, ..., Protocol): + pass + +However, it is not yet clear how popular/useful it will be and implementing +this in type checkers for non-protocol classes could be difficult. Finally, it +will be very easy to add this later if needed. References From f6240c84cfde39d055df25d1443bdd706445beec Mon Sep 17 00:00:00 2001 From: Ivan Levkivskyi Date: Sat, 18 Mar 2017 02:42:56 +0100 Subject: [PATCH 37/37] Remove one reamining 'All' --- pep-0544.txt | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/pep-0544.txt b/pep-0544.txt index 10e1bfeba95..87b767f74e5 100644 --- a/pep-0544.txt +++ b/pep-0544.txt @@ -634,7 +634,9 @@ aliases:: from typing import TypeVar, Reversible, Iterable, Sized T = TypeVar('T') - CompatReversible = Union[Reversible[T], All[Iterable[T], Sized]] + class SizedIterable(Iterable[T], Sized, Protocol): + pass + CompatReversible = Union[Reversible[T], SizedIterable[T]] .. _discussion: @@ -929,7 +931,7 @@ one can then attach warnings about problems like this in the documentation. The user would be able to evaluate whether the benefits outweigh the potential for confusion for each protocol and explicitly opt in -- but the default behavior would be safer. Finally, it will be easy to make this -behaviour default if necessary, while it might be problematic to make it opt-in +behavior default if necessary, while it might be problematic to make it opt-in after being default.