8000 Fix 3903: untyped defs on derived classes by gantsevdenis · Pull Request #7548 · python/mypy · GitHub
[go: up one dir, main page]

Skip to content

Fix 3903: untyped defs on derived classes #7548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 10 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/source/command_line.rst
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,10 @@ definitions or calls.
It will assume all arguments have type ``Any`` and always infer ``Any``
as the return type.

``--inherit-signatures``
This allows to use base class function signature in derived classes, without explicitly

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Slight typo - "This allows the use of..." instead of "This allows to use..."

annotating the derived class.

``--disallow-untyped-decorators``
This flag reports an error whenever a function with type annotations
is decorated with a decorator without annotations.
Expand Down
4 changes: 3 additions & 1 deletion mypy/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -442,7 +442,9 @@ def add_invertible_flag(flag: str,
add_invertible_flag('--disallow-untyped-decorators', default=False, strict_flag=True,
help="Disallow decorating typed functions with untyped decorators",
group=untyped_group)

add_invertible_flag('--inherit-signatures', 8000 default=False, strict_flag=True,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this needs to be a strict flag, because --disallow-untyped-defs is already part of --strict IIRC.

help="Inherit signautes from base classes to child",
group=untyped_group)
none_group = parser.add_argument_group(
title='None and Optional handling',
description="Adjust how values of type 'None' are handled. For more context on "
Expand Down
4 changes: 4 additions & 0 deletions mypy/options.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ class BuildType:
"follow_imports_for_stubs",
"ignore_errors",
"ignore_missing_imports",
"inherit_signatures",
"local_partial_types",
"mypyc",
"no_implicit_optional",
Expand Down Expand Up @@ -104,6 +105,9 @@ def __init__(self) -> None:
# Type check unannotated functions
self.check_untyped_defs = False

# Inherit missing signatures from base classes
self.inherit_signatures = False

# Disallow decorating typed functions with untyped decorators
self.disallow_untyped_decorators = False

Expand Down
120 changes: 114 additions & 6 deletions mypy/semanal.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@
from contextlib import contextmanager

from typing import (
List, Dict, Set, Tuple, cast, TypeVar, Union, Optional, Callable, Iterator, Iterable,
)
List, Dict, Set, Tuple, cast, TypeVar, Union, Optional, Callable, Iterator, Iterable, Mapping)

from typing_extensions import Final

from mypy.nodes import (
Expand All @@ -75,7 +75,7 @@
nongen_builtins, get_member_expr_fullname, REVEAL_TYPE,
REVEAL_LOCALS, is_final_node, TypedDictExpr, type_aliases_target_versions,
EnumCallExpr, RUNTIME_PROTOCOL_DECOS, FakeExpression, Statement, AssignmentExpr,
)
ARG_OPT)
from mypy.tvar_scope import TypeVarScope
from mypy.typevars import fill_typevars
from mypy.visitor import NodeVisitor
Expand All @@ -87,8 +87,8 @@
FunctionLike, UnboundType, TypeVarDef, TupleType, UnionType, StarType,
CallableType, Overloaded, Instance, Type, AnyType, LiteralType, LiteralValue,
TypeTranslator, TypeOfAny, TypeType, NoneType, PlaceholderType, TPDICT_NAMES, ProperType,
get_proper_type, get_proper_types
)
get_proper_type, get_proper_types,
TypeList)
from mypy.typeops import function_type
from mypy.type_visitor import TypeQuery
from mypy.nodes import implicit_module_attrs
Expand Down Expand Up @@ -160,6 +160,45 @@
Tag = int


def substitute_unbounds(t: UnboundType, names_to_types: Mapping[str, Instance]) -> ProperType:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for misleading suggestion, doing the copy during semantic analyzis is still bad. I was thinking about superclass being in a different module that is type-checked before the module with subclass, but it obviously doesn't work always.

Also this function is just one big hack. I was thinking about something along the lines of mypy.checker.TypeChecker.bind_and_map_method().

For the latter to work well I see a possible strategy:

  1. Add two flags (or maybe better one three state flag) to functions indicating whether this function can be "improved", i.e. it has some untyped arguments, while a supertype has those typed, and whether the "copying" has already been performed.
  2. The first flag should be set during semantic analysis, this should be thought carefully to work well in import cycles.
  3. During type checking, whenever we encounter a reference to a method Sub.meth for which we know it can be "improved", but wasn't yet, we trigger a deferral.
  4. Finally, we perform the copying while analyzing the method itself (in a place where you did initially, but with a more sophisticated logic like currently)
  5. Be sure that aststrip.py resets the flags, and deps.py generates dependencies so that subclass method callsites are all rechecked when the signature in supertype changes (this may be already the case, at least partially).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gantsevdenis I can't find your comment that I have got by e-mail, but here is the response. Self-types is something like this:

T = TypeVar('T')
class Base:
    def copy(self: T) -> T: ...

class Sub(Base): ...
x = Sub().copy()  # inferred type of 'x' is 'Sub'

"""Replaces all type vars types with whatever ProperTypes given.

Used with --inherit-signatures option
Example, if we have:
class A(Generic[T,U]):
def foo(self) -> U:pass
def bar(self,*args)->Callable[[U],T]:pass
class B(A[int,str]):
def foo(self):pass
def bar(self,*args):pass

Then class B will be transformed to:
class B(A[int,str]):
def foo(self) -> str:pass
def bar(self,*args)->Callable[[str],int]:pass
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use PEP 8 everywhere including in docstring and in tests! (You can however skip empty lines in tests whenever it makes sense).

"""
workcopy = t.args.copy()
new_args = [] # type: List[Type]
while workcopy:
curr = workcopy.pop(0) # order is important, so keep it
if isinstance(curr, TypeList):
items = [] # type: List[Type]
for item in curr.items:
assert isinstance(item, UnboundType)
ret = substitute_unbounds(item, names_to_types)
items.append(ret)
curr.items = items
new_args.append(curr)
elif isinstance(curr, UnboundType):
ret = substitute_unbounds(curr, names_to_types)
new_args.append(ret)
else:
raise AssertionError(type(curr))
ret_type = names_to_types.get(t.name, t)
ret_type.args = new_args
return ret_type


class SemanticAnalyzer(NodeVisitor[None],
SemanticAnalyzerInterface,
SemanticAnalyzerPluginInterface):
Expand Down Expand Up @@ -477,6 +516,74 @@ def adjust_public_exports(self) -> None:
else:
g.module_public = False

def inherit_signature(self, cls_info: TypeInfo, func: FuncDef) -> None:
"""Copy type annotations from parent definition, if we can find it."""
parent, base_1 = None, None
base_0 = cls_info.bases[0]
for base_1 in cls_info.mro[1:]:
if func.name() in base_1.names:
parent = base_1.names[func.name()].node
if isinstance(parent, Decorator):
parent = parent.func
break
if not parent:
return None
assert base_1 is not None
assert isinstance(parent, FuncDef), "got %s" % type(parent)
pnb_args, cnb_args = len(parent.arg_kinds), len(func.arg_kinds)
if cnb_args < pnb_args:
self.fail("Signature of \"%s\" incompatible with supertype \"%s\"" %
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the message generating logic should live messages.py. Also I think we should emit a more specific error message, explaining that we can't copy the signature, and mentioning what went wrong (not enough arguments, etc).

Also in Python you can use 'single quotes to avoid "escaping" double quotes' (also below).

(func.name(), base_1.name()), func, code=None)
return None
fail = False

# check that child has a compatible signature
# for each parent's positional arg, we have (positional or optional)
for i, arg_kind in enumerate(parent.arg_kinds):
if func.arg_names[i] != parent.arg_names[i]:
fail = True
break
if arg_kind == ARG_POS:
if func.arg_kinds[i] not in (ARG_POS, ARG_OPT):
fail = True
break
elif arg_kind == ARG_OPT:
if func.arg_kinds[i] != ARG_OPT:
fail = True
break
else:
if func.arg_kinds[i] != parent.arg_kinds[i]:
fail = True
break
if fail:
self.fail("Signature of \"%s\" incompatible with supertype \"%s\"" %
(func.name(), base_1.name()), func, code=None)
return None

# copy parent's arg to child, mark as "Any" for all extra args
assert isinstance(parent.type, CallableType)
arg_kinds, arg_names, arg_types = [], [], []
for i in range(0, cnb_args):
arg_kinds.append(func.arg_kinds[i])
arg_names.append(func.arg_names[i])
if i < pnb_args:
arg_types.append(parent.type.arg_types[i])
else:
arg_types.append(AnyType(TypeOfAny.unannotated, None, None))
callable_type = CallableType(arg_types, arg_kinds, arg_names,
parent.type.ret_type,
parent.type.fallback)
func.unanalyzed_type = callable_type
func.type = callable_type
if isinstance(parent.type.ret_type, UnboundType):
assert isinstance(func.type.ret_type, UnboundType)
vals = cast(Iterable[Instance], base_0.args) # type: Iterable[Instance]
names_to_types = dict(zip(base_1.type_vars, vals)) # type: Mapping[str, Instance]
func.type.ret_type = substitute_unbounds(func.type.ret_type, names_to_types)
else:
func.type.ret_type = parent.type.ret_type
return None

@contextmanager
def file_context(self,
file_node: MypyFile,
Expand Down Expand Up @@ -530,7 +637,8 @@ def file_context(self,

def visit_func_def(self, defn: FuncDef) -> None:
self.statement = defn

if not defn.unanalyzed_type and self.options.inherit_signatures and self.type:
self.inherit_signature(self.type, defn)
# Visit default values because they may contain assignment expressions.
for arg in defn.arguments:
if arg.initializer:
Expand Down
94 changes: 94 additions & 0 deletions test-data/unit/check-classes.test
Original file line number Diff line number Diff line change
Expand Up @@ -6223,3 +6223,97 @@ class Foo:
reveal_type(Foo().x) # N: Revealed type is 'Union[Any, None]'
reveal_type(Foo().y) # N: Revealed type is 'builtins.list[Any]'
[builtins fixtures/list.pyi]

[case testSignatureInheritanceAcceptsNarrowParam]
# flags: --inherit-signatures
from typing import Any
class Base:
def foo(self, x:int, y:Any) -> int: return 5
class Child(Base):
def foo(self,x:int, y:str): return 0
[builtins fixtures/primitives.pyi]

[case testSignatureInheritanceAcceptsNarrowReturn]
# flags: --inherit-signatures
class ReturnBase: pass
class ReturnChild(ReturnBase): pass
class Base:
def foo(self) -> ReturnBase: return ReturnBase()
class Child(Base):
def foo(self): return ReturnChild()
[builtins fixtures/primitives.pyi]

[case testSignatureInheritanceDifferentNbArgs]
# flags: --inherit-signatures
class Base:
def foo(self, x:int) -> int: return 5
class Child(Base):
def foo(self, x, y): return 0 # E: Signature of "foo" incompatible with supertype "Base"
class Child2(Base):
def foo(self): return 0 # E: Signature of "foo" incompatible with supertype "Base"
[builtins fixtures/primitives.pyi]

[case testSigatureInheritanceDoesntCrashWithDecorator]
# flags: --inherit-signatures
def decor(f):
def inner(*args,**kwargs):
return f(*args,**kwargs)
return inner
class Base:
@decor
def foo(self, x:int) -> int: return 5
class Child(Base):
@decor
def foo(self, x): return "5" # E: Incompatible return value type (got "str", expected "int")
[builtins fixtures/dict.pyi]

[case testSignatureInheritanceAcceptsNarrowUnannotatedParam]
# flags: --inherit-signatures
class A:pass
class B(A):pass
class Base():
def foo(self, x:A) -> int: return 5
class Child(Base):
def foo(self, x): return 0
Child().foo(B())
[builtins fixtures/primitives.pyi]

[case testSignatureInheritanceRejectsTooWideParam]
# flags: --inherit-signatures
class A:pass
class B(A):pass
class Base():
def foo(self, x:B) -> int: return 5
class Child(Base):
def foo(self, x): return 0
Child().foo(A()) # E: Argument 1 to "foo" of "Child" has incompatible type "A"; expected "B"
[builtins fixtures/primitives.pyi]

[case testSignatureInheritanceAcceptsOptionalArgInChild]
# flags: --inherit-signatures
class A:pass
class Base():
def foo(self, x:A, y:int) -> int: return 5
class Child(Base):
def foo(self, x, y=42, z=""):
reveal_type(x) # N: Revealed type is '__main__.A'
reveal_type(y) # N: Revealed type is 'builtins.int'
reveal_type(z) # N: Revealed type is 'Any'
return 0

[case testSignatureInheritanceGenerics]
# flags: --inherit-signatures
from typing import TypeVar, Dict, Tuple
KT = TypeVar('KT')
VT = TypeVar('VT')
class MyDict(Dict[KT, VT]):
def return_KT(self) -> KT: pass
def return_VT(self) -> VT: pass
def return_VT_wrong(self) -> VT: pass
def return_Tuple(self)->Tuple[VT,KT]:pass
class StrDict(MyDict[str, int]):
def return_KT(self,*args,**kwargs): return "5"
def return_VT(self, *args,**kwargs): return 5
def return_VT_wrong(self,*args,**kwargs): return "" # E: Incompatible return value type (got "str", expected "int")
def return_Tuple(self,*args,**kwargs): return ("", 5) # E: Incompatible return value type (got "Tuple[str, int]", expected "Tuple[int, str]")
[builtins fixtures/dict.pyi]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Several important scenarios are still missing, for example:

  • Methods with self-types
  • Generic inheritance where subclass has new type variables with non-trivial mapping to old ones
  • Chained inheritance where C inherits B that inherits A
  • Multiple inheritance
  • Incremental mode tests, see check-incremental.test
  • Daemon mode tests, see fine-grained.test
  • Import cycles
  • Tests that inherited signature is used on external and internal access (including situations with deferrals)

0