8000 Start using TypeAliasType in the semantic analyzer (#7923) · python/mypy@47bafd6 · GitHub
[go: up one dir, main page]
Skip to content

Commit 47bafd6

Browse files
ilevkivskyiilevkivskyi
authored
Start using TypeAliasType in the semantic analyzer (#7923)
This PR starts using the new `TypeAliasType` in the semantic analyzer. This PR doesn't yet pulls the trigger to enable the recursive types, but it is now essentially one line of code away. This PR: * Makes type analyzer return a `TypeAliasType` instead of eagerly expanding the alias target. * Refactors `TypeAliasExpr` to refer to `TypeAlias` (sorry for the noise). * Makes few minor fixes to make all existing tests pass. * Adds few logistic changes around `get_proper_type()` I found necessary while playing with actual recursive types over the weekend. Here are some strategical comments: * Taking into account how easy it was to make all existing tests pass, I don't think it is necessary to introduce a hidden option flag that would eagerly expand all type aliases after semantic analyzis. It would probably make sense to test this well locally before a public release. * There is a special case for no arguments generic aliases. Currently one is allowed to write `L = List; x: L[int]`, I preserve this by using eager expansion in this special case, otherwise it would complicate the whole logic significantly. This is also mostly a legacy thing because we have built-in aliases like `List = list` magically added by semantic analyzer. * I have found that just carelessly sprinkling `get_proper_type()` is not a best strategy. It saves all the existing special-casing but also introduces a risk for infinite recursion. In particular, "type ops tangle" should ideally always pass on the original alias type. Unfortunately, there is no way to fix/enforce this (without having some severe performance impact). Note it is mostly fine to " 8000 carelessly" use `get_proper_type()` in the "front end" (like `checker.py`, `checkexpr.py`, `checkmember.py` etc). Here is my plan for the next five PRs: 1. I am going to try merging `SubtypeVisitor` and `ProperSubtypeVisitor`, there is very large amount of code duplication (there is already an issue for this). 2. I am going to try to get rid of `sametypes.py` (I am going to open a new issue, see my arguments there). 3. I am going to enable the recursive aliases and add sufficiently many tests to be sure we are safe about infinite recursion in type ops. 4. I am going to change how named tuples and typed dicts are represented internally, currently they are stored as `TypeInfo`s, but will be stored as `TypeAlias`. Essentially there will be almost no difference between `A = Tuple[int, int]` and `A = NamedTuple('A', [('x', int), ('y', int)])`. This will allow typed dicts and named tuple participate in recursive types. 5. I am going to switch from using unbound type variables to bound type variables for generic type aliases, since now they are almost identical to `TypeInfo`s so it IMO it really makes sense to make them uniform (and avoid confusions and code duplication in future). 5a. Potentially as a follow-up I may add support for generic named tuples and typed dicts, since steps 4 plus 5 will make this almost trivial. There is another important thing to call out, previously unions never contained another unions as items (because constructor flattened them), and some code might implicitly rely on this. IMO we should probably update these places, since maintaining this guarantee may be painful. Yet another important thing is that this may break many plugins, so we need to announce this in #6617 when will merge this.
1 parent e97377c commit 47bafd6

15 files changed

+136
-86
lines changed

‎mypy/checkexpr.py

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -224,8 +224,7 @@ def analyze_ref_expr(self, e: RefExpr, lvalue: bool = False) -> Type:
224224
# Something that refers to a type alias appears in runtime context.
225225
# Note that we suppress bogus errors for alias redefinitions,
226226
# they are already reported in semanal.py.
227-
result = self.alias_type_in_runtime_context(node.target, node.alias_tvars,
228-
node.no_args, e,
227+
result = self.alias_type_in_runtime_context(node, node.no_args, e,
229228
alias_definition=e.is_alias_rvalue
230229
or lvalue)
231230
else:
@@ -2917,9 +2916,7 @@ def visit_type_application(self, tapp: TypeApplication) -> Type:
29172916
"""
29182917
if isinstance(tapp.expr, RefExpr) and isinstance(tapp.expr.node, TypeAlias):
29192918
# Subscription of a (generic) alias in runtime context, expand the alias.
2920-
target = tapp.expr.node.target
2921-
all_vars = tapp.expr.node.alias_tvars
2922-
item = expand_type_alias(target, all_vars, tapp.types, self.chk.fail,
2919+
item = expand_type_alias(tapp.expr.node, tapp.types, self.chk.fail,
29232920
tapp.expr.node.no_args, tapp)
29242921
item = get_proper_type(item)
29252922
if isinstance(item, Instance):
@@ -2951,10 +2948,10 @@ def visit_type_alias_expr(self, alias: TypeAliasExpr) -> Type:
29512948
both `reveal_type` instances will reveal the same type `def (...) -> builtins.list[Any]`.
29522949
Note that type variables are implicitly substituted with `Any`.
29532950
"""
2954-
return self.alias_type_in_runtime_context(alias.type, alias.tvars, alias.no_args,
2951+
return self.alias_type_in_runtime_context(alias.node, alias.no_args,
29552952
alias, alias_definition=True)
29562953

2957-
def alias_type_in_runtime_context(self, target: Type, alias_tvars: List[str],
2954+
def alias_type_in_runtime_context(self, alias: TypeAlias,
29582955
no_args: bool, ctx: Context,
29592956
*,
29602957
alias_definition: bool = False) -> Type:
@@ -2971,14 +2968,14 @@ class LongName(Generic[T]): ...
29712968
x = A()
29722969
y = cast(A, ...)
29732970
"""
2974-
if isinstance(target, Instance) and target.invalid: # type: ignore
2971+
if isinstance(alias.target, Instance) and alias.target.invalid: # type: ignore
29752972
# An invalid alias, error already has been reported
29762973
return AnyType(TypeOfAny.from_error)
29772974
# If this is a generic alias, we set all variables to `Any`.
29782975
# For example:
29792976
# A = List[Tuple[T, T]]
29802977
# x = A() <- same as List[Tuple[Any, Any]], see PEP 484.
2981-
item = get_proper_type(set_any_tvars(target, alias_tvars, ctx.line, ctx.column))
2978+
item = get_proper_type(set_any_tvars(alias, ctx.line, ctx.column))
29822979
if isinstance(item, Instance):
29832980
# Normally we get a callable type (or overloaded) with .is_type_obj() true
29842981
# representing the class's constructor

‎mypy/checkmember.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -502,8 +502,8 @@ def instance_alias_type(alias: TypeAlias,
502502
target = get_proper_type(alias.target) # type: Type
503503
assert isinstance(get_proper_type(target),
504504
Instance), "Must be called only with aliases to classes"
505-
target = set_any_tvars(target, alias.alias_tvars, alias.line, alias.column)
506-
assert isinstance(target, Instance) # type: ignore[misc]
505+
target = get_proper_type(set_any_tvars(alias, alias.line, alias.column))
506+
assert isinstance(target, Instance)
507507
tp = type_object_type(target.type, builtin_type)
508508
return expand_type_by_instance(tp, target)
509509

‎mypy/constraints.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -92,8 +92,7 @@ def infer_constraints(template: Type, actual: Type,
9292
"""
9393
if any(get_proper_type(template) == get_proper_type(t) for t in TypeState._inferring):
9494
return []
95-
if (isinstance(template, TypeAliasType) and isinstance(actual, TypeAliasType) and
96-
template.is_recursive and actual.is_recursive):
95+
if isinstance(template, TypeAliasType) and template.is_recursive:
9796
# This case requires special care because it may cause infinite recursion.
9897
TypeState._inferring.append(template)
9998
res = _infer_constraints(template, actual, direction)
@@ -105,6 +104,7 @@ def infer_constraints(template: Type, actual: Type,
105104
def _infer_constraints(template: Type, actual: Type,
106105
direction: int) -> List[Constraint]:
107106

107+
orig_template = template
108108
template = get_proper_type(template)
109109
actual = get_proper_type(actual)
110110

@@ -129,7 +129,7 @@ def _infer_constraints(template: Type, actual: Type,
129129
if direction == SUPERTYPE_OF and isinstance(actual, UnionType):
130130
res = []
131131
for a_item in actual.items:
132-
res.extend(infer_constraints(template, a_item, direction))
132+
res.extend(infer_constraints(orig_template, a_item, direction))
133133
return res
134134

135135
# Now the potential subtype is known not to be a Union or a type

‎mypy/nodes.py

Lines changed: 5 additions & 4 deletions
2121
Original file line numberDiff line numberDiff line change
@@ -2116,11 +2116,12 @@ class TypeAliasExpr(Expression):
21162116
# A = List[Any]
21172117
no_args = False # type: bool
21182118

2119-
def __init__(self, type: 'mypy.types.Type', tvars: List[str], no_args: bool) -> None:
2119+
def __init__(self, node: 'TypeAlias') -> None:
21202120
super().__init__()
2121-
self.type = type
2122-
self.tvars = tvars
2123-
self.no_args = no_args
+
self.type = node.target
2122+
self.tvars = node.alias_tvars
2123+
self.no_args = node.no_args
2124+
self.node = node
21242125

21252126
def accept(self, visitor: ExpressionVisitor[T]) -> T:
21262127
return visitor.visit_type_alias_expr(self)

‎mypy/semanal.py

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2495,18 +2495,20 @@ def check_and_set_up_type_alias(self, s: AssignmentStmt) -> bool:
24952495
context=s)
24962496
# When this type alias gets "inlined", the Any is not explicit anymore,
24972497
# so we need to replace it with non-explicit Anys.
2498-
res = make_any_non_explicit(res)
2498+
if not has_placeholder(res):
2499+
res = make_any_non_explicit(res)
24992500
no_args = isinstance(res, Instance) and not res.args # type: ignore
25002501
fix_instance_types(res, self.fail, self.note)
2502+
alias_node = TypeAlias(res, self.qualified_name(lvalue.name), s.line, s.column,
2503+
alias_tvars=alias_tvars, no_args=no_args)
25012504
if isinstance(s.rvalue, (IndexExpr, CallExpr)): # CallExpr is for `void = type(None)`
2502-
s.rvalue.analyzed = TypeAliasExpr(res, alias_tvars, no_args)
2505+
s.rvalue.analyzed = TypeAliasExpr(alias_node)
25032506
s.rvalue.analyzed.line = s.line
25042507
# we use the column from resulting target, to get better location for errors
25052508
s.rvalue.analyzed.column = res.column
25062509
elif isinstance(s.rvalue, RefExpr):
25072510
s.rvalue.is_alias_rvalue = True
2508-
alias_node = TypeAlias(res, self.qualified_name(lvalue.name), s.line, s.column,
2509-
alias_tvars=alias_tvars, no_args=no_args)
2511+
25102512
if existing:
25112513
# An alias gets updated.
25122514
updated = False

‎mypy/semanal_typeargs.py

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,12 @@
55
operations, including subtype checks.
66
"""
77

8-
from typing import List, Optional
8+
from typing import List, Optional, Set
99

1010
from mypy.nodes import TypeInfo, Context, MypyFile, FuncItem, ClassDef, Block
11-
from mypy.types import Type, Instance, TypeVarType, AnyType, get_proper_types
11+
from mypy.types import (
12+
Type, Instance, TypeVarType, AnyType, get_proper_types, TypeAliasType, get_proper_type
13+
)
1214
from mypy.mixedtraverser import MixedTraverserVisitor
1315
from mypy.subtypes import is_subtype
1416
from mypy.sametypes import is_same_type
@@ -27,6 +29,9 @@ def __init__(self, errors: Errors, options: Options, is_typeshed_file: bool) ->
2729
self.scope = Scope()
2830
# Should we also analyze function definitions, or only module top-levels?
2931
self.recurse_into_functions = True
32+
# Keep track of the type aliases already visited. This is needed to avoid
33+
# infinite recursion on types like A = Union[int, List[A]].
34+
self.seen_aliases = set() # type: Set[TypeAliasType]
3035

3136
def visit_mypy_file(self, o: MypyFile) -> None:
3237
self.errors.set_file(o.path, o.fullname, scope=self.scope)
@@ -48,6 +53,16 @@ def visit_block(self, o: Block) -> None:
4853
if not o.is_unreachable:
4954
super().visit_block(o)
5055

56+
def visit_type_alias_type(self, t: TypeAliasType) -> None:
57+
super().visit_type_alias_type(t)
58+
if t in self.seen_aliases:
59+
# Avoid infinite recursion on recursive type aliases.
60+
# Note: it is fine to skip the aliases we have already seen in non-recursive types,
61+
# since errors there have already already reported.
62+
return
63+
self.seen_aliases.add(t)
64+
get_proper_type(t).accept(self)
65+
5166
def visit_instance(self, t: Instance) -> None:
5267
# Type argument counts were checked in the main semantic analyzer pass. We assume
5368
# that the counts are correct here.

‎mypy/server/deps.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -884,6 +884,10 @@ def visit_type_alias_type(self, typ: TypeAliasType) -> List[str]:
884884
triggers = [trigger]
885885
for arg in typ.args:
886886
triggers.extend(self.get_type_triggers(arg))
887+
# TODO: Add guard for infinite recursion here. Moreover, now that type aliases
888+
# are its own kind of types we can simplify the logic to rely on intermediate
889+
# dependencies (like for instance types).
890+
triggers.extend(self.get_type_triggers(typ.alias.target))
887891
return triggers
888892

889893
def visit_any(self, typ: AnyType) -> List[str]:

‎mypy/subtypes.py

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
Type, AnyType, UnboundType, TypeVisitor, FormalArgument, NoneType,
88
Instance, TypeVarType, CallableType, TupleType, TypedDictType, UnionType, Overloaded,
99
ErasedType, PartialType, DeletedType, UninhabitedType, TypeType, is_named_instance,
10-
FunctionLike, TypeOfAny, LiteralType, ProperType, get_proper_type, TypeAliasType
10+
FunctionLike, TypeOfAny, LiteralType, get_proper_type, TypeAliasType
1111
)
1212
import mypy.applytype
1313
import mypy.constraints
@@ -103,6 +103,8 @@ def _is_subtype(left: Type, right: Type,
103103
ignore_pos_arg_names: bool = False,
104104
ignore_declared_variance: bool = False,
105105
ignore_promotions: bool = False) -> bool:
106+
orig_right = right
107+
orig_left = left
106108
left = get_proper_type(left)
107109
right = get_proper_type(right)
108110

@@ -113,7 +115,7 @@ def _is_subtype(left: Type, right: Type,
113115
# Normally, when 'left' is not itself a union, the only way
114116
# 'left' can be a subtype of the union 'right' is if it is a
115117
# subtype of one of the items making up the union.
116-
is_subtype_of_item = any(is_subtype(left, item,
118+
is_subtype_of_item = any(is_subtype(orig_left, item,
117119
ignore_type_params=ignore_type_params,
118120
ignore_pos_arg_names=ignore_pos_arg_names,
119121
ignore_declared_variance=ignore_declared_variance,
@@ -130,7 +132,7 @@ def _is_subtype(left: Type, right: Type,
130132
elif is_subtype_of_item:
131133
return True
132134
# otherwise, fall through
133-
return left.accept(SubtypeVisitor(right,
135+
return left.accept(SubtypeVisitor(orig_right,
134136
ignore_type_params=ignore_type_params,
135137
ignore_pos_arg_names=ignore_pos_arg_names,
136138
ignore_declared_variance=ignore_declared_variance,
@@ -155,13 +157,14 @@ def is_equivalent(a: Type, b: Type,
155157

156158
class SubtypeVisitor(TypeVisitor[bool]):
157159

158-
def __init__(self, right: ProperType,
160+
def __init__(self, right: Type,
159161
*,
160162
ignore_type_params: bool,
161163
ignore_pos_arg_names: bool = False,
162164
ignore_declared_variance: bool = False,
163165
ignore_promotions: bool = False) -> None:
164-
self.right = right
166+
self.right = get_proper_type(right)
167+
self.orig_right = right
165168
self.ignore_type_params = ignore_type_params
166169
self.ignore_pos_arg_names = ignore_pos_arg_names
167170
self.ignore_declared_variance = ignore_declared_variance
@@ -449,7 +452,7 @@ def visit_overloaded(self, left: Overloaded) -> bool:
449452
return False
450453

451454
def visit_union_type(self, left: UnionType) -> bool:
452-
return all(self._is_subtype(item, self.right) for item in left.items)
455+
return all(self._is_subtype(item, self.orig_right) for item in left.items)
453456

454457
def visit_partial_type(self, left: PartialType) -> bool:
455458
# This is indeterminate as we don't really know the complete type yet.
@@ -1083,7 +1086,8 @@ def restrict_subtype_away(t: Type, s: Type, *, ignore_promotions: bool = False)
10831086
s = get_proper_type(s)
10841087

10851088
if isinstance(t, UnionType):
1086-
new_items = [item for item in t.relevant_items()
1089+
new_items = [restrict_subtype_away(item, s, ignore_promotions=ignore_promotions)
1090+
for item in t.relevant_items()
10871091
if (isinstance(get_proper_type(item), AnyType) or
10881092
not covers_at_runtime(item, s, ignore_promotions))]
10891093
return UnionType.make_union(new_items)
@@ -1139,22 +1143,26 @@ def is_proper_subtype(left: Type, right: Type, *, ignore_promotions: bool = Fals
11391143

11401144
def _is_proper_subtype(left: Type, right: Type, *, ignore_promotions: bool = False,
11411145
erase_instances: bool = False) -> bool:
1146+
orig_left = left
1147+
orig_right = right
11421148
left = get_proper_type(left)
11431149
right = get_proper_type(right)
11441150

11451151
if isinstance(right, UnionType) and not isinstance(left, UnionType):
1146-
return any([is_proper_subtype(left, item, ignore_promotions=ignore_promotions,
1152+
return any([is_proper_subtype(orig_left, item, ignore_promotions=ignore_promotions,
11471153
erase_instances=erase_instances)
11481154
for item in right.items])
1149-
return left.accept(ProperSubtypeVisitor(right, ignore_promotions=ignore_promotions,
1155+
return left.accept(ProperSubtypeVisitor(orig_right,
1156+
ignore_promotions=ignore_promotions,
11501157
erase_instances=erase_instances))
11511158

11521159

11531160
class ProperSubtypeVisitor(TypeVisitor[bool]):
1154-
def __init__(self, right: ProperType, *,
1161+
def __init__(self, right: Type, *,
11551162
ignore_promotions: bool = False,
11561163
erase_instances: bool = False) -> None:
1157-
self.right = right
1164+
self.right = get_proper_type(right)
1165+
self.orig_right = right
11581166
self.ignore_promotions = ignore_promotions
11591167
self.erase_instances = erase_instances
11601168
self._subtype_kind = ProperSubtypeVisitor.build_subtype_kind(
@@ -1313,7 +1321,7 @@ def visit_overloaded(self, left: Overloaded) -> bool:
13131321
return False
13141322

13151323
def visit_union_type(self, left: UnionType) -> bool:
1316-
return all([self._is_proper_subtype(item, self.right) for item in left.items])
1324+
return all([self._is_proper_subtype(item, self.orig_right) for item in left.items])
13171325

13181326
def visit_partial_type(self, left: PartialType) -> bool:
13191327
# TODO: What's the right thing to do here?

‎mypy/treetransform.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -497,7 +497,7 @@ def visit_type_var_expr(self, node: TypeVarExpr) -> TypeVarExpr:
497497
self.type(node.upper_bound), variance=node.variance)
498498

499499
def visit_type_alias_expr(self, node: TypeAliasExpr) -> TypeAliasExpr:
500-
return TypeAliasExpr(node.type, node.tvars, node.no_args)
500+
return TypeAliasExpr(node.node)
501501

502502
def visit_newtype_expr(self, node: NewTypeExpr) -> NewTypeExpr:
503503
res = NewTypeExpr(node.name, node.old_type, line=node.line, column=node.column)

‎mypy/type_visitor.py

Lines changed: 19 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313

1414
from abc import abstractmethod
1515
from collections import OrderedDict
16-
from typing import Generic, TypeVar, cast, Any, List, Callable, Iterable, Optional
16+
from typing import Generic, TypeVar, cast, Any, List, Callable, Iterable, Optional, Set
1717
from mypy_extensions import trait
1818

1919
T = TypeVar('T')
@@ -246,14 +246,21 @@ def visit_type_alias_type(self, t: TypeAliasType) -> Type:
246246
class TypeQuery(SyntheticTypeVisitor[T]):
247247
"""Visitor for performing queries of types.
248248
249-
strategy is used to combine results for a series of types
249+
strategy is used to combine results for a series of types,
250+
common use cases involve a boolean query using `any` or `all`.
250251
251-
Common use cases involve a boolean query using `any` or `all`
252+
Note: this visitor keeps an internal state (tracks type aliases to avoid
253+
recursion), so it should *never* be re-used for querying different types,
254+
create a new visitor instance instead.
255+
256+
# TODO: check that we don't have existing violations of this rule.
252257
"""
253258

254259
def __init__(self, strategy: Callable[[Iterable[T]], T]) -> None:
255260
self.strategy = strategy
256-
self.seen = [] # type: List[Type]
261+
# Keep track of the type aliases already visited. This is needed to avoid
262+
# infinite recursion on types like A = Union[int, List[A]].
263+
self.seen_aliases = set() # type: Set[TypeAliasType]
257264

258265
def visit_unbound_type(self, t: UnboundType) -> T:
259266
return self.query_types(t.args)
@@ -329,14 +336,16 @@ def query_types(self, types: Iterable[Type]) -> T:
329336
"""Perform a query for a list of types.
330337
331338
Use the strategy to combine the results.
332-
Skip types already visited types to avoid infinite recursion.
333-
Note: types can be recursive until they are fully analyzed and "unentangled"
334-
in patches after the semantic analysis.
339+
Skip type aliases already visited types to avoid infinite recursion.
335340
"""
336341
res = [] # type: List[T]
337342
for t in types:
338-
if any(t is s for s in self.seen):
339-
continue
340-
self.seen.append(t)
343+
if isinstance(t, TypeAliasType):
344+
# Avoid infinite recursion for recursive type aliases.
345+
# TODO: Ideally we should fire subvisitors here (or use caching) if we care
346+
# about duplicates.
347+
if t in self.seen_aliases:
348+
continue
349+
self.seen_aliases.add(t)
341350
res.append(t.accept(self))
342351
return self.strategy(res)

0 commit comments

Comments
 (0)
0