Invariance, contravariance, and covariance for containers #10427
Comments
|
If the container and the variable That leaves class |
|
Thinking about this more, I think mypy's behavior is correct in its entirety. As I mentioned, I think the code is buggy in the cases where the type variable is defined as covariant and contravariant. In the case where it's invariant, you should be able to assign subtypes to the attribute I recommend closing because the current behavior in mypy looks right to me. |
|
I agree, that mypy is correct here and would also recommend closing. The basic assumption that the variances matter there are wrong. You should always be able to assign subtypes. The variances matter when it comes to inheritance for example. However having a covariant/contravariant
Does this mean that pyright flags those usages as incorrect? |
Similar to mypy, pyright validates variance only for protocol classes. I agree with you that enforcing this for non-protocols would cause a lot of churn for existing code bases. Pyright also implements PEP 695, which computes TypeVar variance automatically when the new syntax is used. |
I don't think what you say is true. EG, a function that accepts a mutable covariant argument can read from that argument and a function that accepts a mutable contravariant argument can write to that argument. Thus demonstrating that there are type safe usages for both co and contra type hinting of mutable variables. For example using the T, M, and B classes from before and 1st establishing a base case that everyone can agree is correct: TV = TypeVar("TV")
class BoxCovariantChecked(Generic[TV]):
def __init__(self, x: TV) -> None:
self._x = x
self._t = type(x)
def check(self, x: TV) -> None:
if not isinstance(x, self._t):
raise TypeError(f"x is of type {type(x)}, which is not a {self._t}.")
@property
def x(self) -> TV:
x = self._x
self.check(x)
return x
@x.setter
def x(self, x: TV) -> None:
self.check(x)
self._x = x
tb = BoxCovariantChecked(T())
tb.x = T()
print(tb.x) # <__main__.T object at 0x100c50290>
tb.x = M()
print(tb.x) # <__main__.M object at 0x100c50210>
tb.x = B()
print(tb.x) # <__main__.B object at 0x100c50290>
mb = BoxCovariantChecked(M())
# Runtime TypeError: x is of type <class '__main__.T'>, which is not a <class '__main__.M'>.
# mypy error: Incompatible types in assignment (expression has type "T", variable has type "M") [assignment]
# mb.x = T()
print(mb.x) # <__main__.M object at 0x102e9fc10>
mb.x = M()
print(mb.x) # <__main__.M object at 0x102e9fc50>
mb.x = B()
print(mb.x) # <__main__.B object at 0x102e9fc10>
bb = BoxCovariantChecked(B())
# Runtime TypeError: x is of type <class '__main__.T'>, which is not a <class '__main__.B'>.
# mypy error: Incompatible types in assignment (expression has type "T", variable has type "B") [assignment]
# bb.x = T()
print(bb.x) # <__main__.M object at 0x104e9fc10>
# Runtime TypeError: x is of type <class '__main__.M'>, which is not a <class '__main__.B'>.
# mypy error: Incompatible types in assignment (expression has type "M", variable has type "B") [assignment]
# bb.x = M()
print(bb.x) # <__main__.M object at 0x104e9fc50>
bb.x = B()
print(bb.x) # <__main__.B object at 0x104e9fc10>As can be seen in the above both the runtime type checks and mypy agree and both are correct. The lines that fail are commented out to allow the above code to run to completion, but see comments for the errors. Apply the above baseline classes and instances to a covariant function: Again runtime and mypy agree and they are correct. Again line that causes the error is commented out to allow code to run. But if we convert the read function into a write using contra-variance we get a mypy errors. Again problem lines commented out to allow code to run. Errors in comments. I still think my original example is correct, this second example shows the problem in another way that may be easier to follow. |
|
By "covariant function", I presume that you mean a function that makes use of a function-scoped type variable that is defined as covariant, as in your example Here's a simple thought experiment that should help make it clear why this is the case. As you probably know, the type parameter for The concept of variance confuses many Python developers, so you're not alone here. This is exacerbated by the way TypeVars have historically been defined in Python, where the definition is separated from the scope to which they are bound. PEP 695, which is implemented in Python 3.12, introduces a new dedicated syntax for type parameters that aims to reduce this confusion. It largely eliminate the need to understand variance. (Full disclosure: I'm the primary author of this PEP.) I don't think the error for def func(v: list[float]):
# Verify that all elements of the list are floats.
# (This is an invalid assumption for `list[float]`.)
assert all(type(x) == float for x in v)
func([1]) |
|
Thanks for pushing PEP 695, I think it will be a big improvement. I wanted to try out 695 on my examples; unfortunately, although I can get python 3.12.0rc1 installed it doesn't look like Mypy 1.5.1 supports 695? Is that correct? |
|
Pyright is currently the only Python type checker that supports PEP 695. (Full disclosure: I'm the primary author of pyright.) There is a tracking issue for mypy. I don't think anyone has started on it yet, so it could be quite a bit of time before you'll see support in mypy. |
|
OK will give that a go. |
|
Thanks for writing pyright; haven't used it before, but is seems very good :) Consider this code: """Think I pinched this example from a C# answer on Stackoverflow!"""
from dataclasses import dataclass, field
@dataclass
class Person:
name: str
@dataclass
class Teacher(Person):
...
@dataclass
class StudentTeacher(Teacher):
...
@dataclass
class School:
people: list[Person] = field(default_factory=list)
teachers: list[Teacher] = field(default_factory=list)
student_teachers: list[StudentTeacher] = field(default_factory=list)
@staticmethod
def add_to_list[T](items: list[T], item: T) -> None:
items.append(item)
def add_teacher(self, name: str) -> None:
teacher = Teacher(name)
School.add_to_list(self.people, teacher)
School.add_to_list(self.teachers, teacher)
# School.add_to_list(self.student_teachers, teacher) - "list[StudentTeacher]" is incompatible with "list[Teacher | StudentTeacher]"
def main():
school = School()
teacher = Teacher("Ellory")
school.people.append(teacher)
school.teachers.append(teacher)
# school.student_teachers.append(teacher) - "Teacher" is incompatible with "StudentTeacher"
school.add_teacher("Ash")
print(f"{school.people=}")
print(f"{school.teachers=}")
print(f"{school.student_teachers=}")
if __name__ == "__main__":
main()Which works and catches the errors, commented out, but with incorrect explanations e.g., If we compare the Java code for the key method, static <S, T extends S> void addToList(List<S> list, T item) { // Covariance only!
static void addToList(List<? super Teacher> list, Teacher item) { // The obvious way!
static <T> void addToList(List<? super T> list, T item) { // More general!All methods working is a nice feature because they all make sense and programmers might arrive at any of these options depending on code history and their experiences. PEP 695 code, as checked by pyright, isn't so straightforward. def add_to_list[S, T: S](items: list[S], item: T) -> None: # Would have thought this works, but no!
def add_to_list[T](items: list[T], item: Teacher) -> None: # This doesn't work, which is a pity because likely to be tried!
def add_to_list[T](items: list[T], item: T) -> None: # This is the only version that works and it's not intuitive!The last form, the only one that works!, doesn't read well since you would have thought the two Ts, I don't know if you have considered this for 695, but you could have default as covariant and annotate for contravariant ( def add_to_list[S, T: S](items: list[S], item: T) -> None: # S and T both covariant and T has an upper bound of S.
def add_to_list(items: list[_:>= Teacher], item: Teacher) -> None: # Anonymous super type, lower bounded by Teacher.
def add_to_list[T](items: list[_:>= T], item: T) -&
93A3
gt; None: # Anonymous super bounded T.An advantage of having types covariant by default is that it is clear that Boat may have sailed on changes to PEP though :(. |
|
The mypy issue tracker is perhaps not the best forum for this discussion, but I want to respond to your comments to clarify some misunderstandings. Hopefully this will assist others who come across this discussion. During development of PEP 695, there was discussion about explicit vs implicit (auto) variance. You can see that this suggestion appears in the rejected ideas section. The problem with explicit variance is that most developers get it wrong because they don't understand variance. It's a complex topic, and even people who think they understand it often do not ;). The decision was made to go with auto variance because this is something that type checkers can calculate. This is the approach taken by TypeScript, and it works well there.
This error makes sense if you understand the notion of invariance. Pyright even spells this out with the added note Note that variance doesn't apply to function-scoped type variables like If you want to report bugs or suggest improvements for pyright, please file issues in the pyright issue tracker. If you want to suggest extensions to the type system (including the PEP 695 syntax) feel free to post to the python/typing forum. I recommend closing this issue. Mypy is behaving correctly here. |
Mypy doesn't diagnose type problems correctly for containers with invariant, contravariant, nor covariant content types.
To Reproduce
Run the following code through Mypy (note Mypy errors in comments):
Expected Behavior
m_in.x = tis wrong since the variable is of typeInMTnotM.m_in.x = bshould be an error because aBis not anInMT(only anMis).m_contra.x = tshould not be an error because aTis aContraMT.m_contra.x = bshould be an error because aBis not anContraMT(only anMor aTare).CoMTnotM.Actual Behavior
See comments in code snippet.
Your Environment
The text was updated successfully, but these errors were encountered: