diff --git a/.travis.yml b/.travis.yml index 4110653853bb..146bc7b49aa2 100644 --- a/.travis.yml +++ b/.travis.yml @@ -13,4 +13,4 @@ install: - python setup.py install script: - - python runtests.py -v + - python runtests.py diff --git a/README.md b/README.md index 6065e23d25fe..e8e32c749765 100644 --- a/README.md +++ b/README.md @@ -2,13 +2,16 @@ Mypy: Optional Static Typing for Python ======================================= [![Build Status](https://travis-ci.org/python/mypy.svg)](https://travis-ci.org/python/mypy) +[![Chat at https://gitter.im/python/mypy](https://badges.gitter.im/python/mypy.svg)](https://gitter.im/python/mypy?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) Got a question? File an issue! ------------------------------ -We don't have a mailing list; but we are always happy to answer questions -filed as issues in our trackers: +We don't have a mailing list; but we are always happy to answer +questions on [gitter chat](https://gitter.im/python/mypy) or filed as +issues in our trackers: + - [mypy tracker](https://github.com/python/mypy/issues) for mypy isues - [typeshed tracker](https://github.com/python/typeshed/issues) @@ -17,7 +20,6 @@ filed as issues in our trackers: for discussion of new type system features (PEP 484 changes) and runtime bugs in the typing module - What is mypy? ------------- diff --git a/docs/source/command_line.rst b/docs/source/command_line.rst index bfda57abe5d7..25b599a38c25 100644 --- a/docs/source/command_line.rst +++ b/docs/source/command_line.rst @@ -8,23 +8,24 @@ summary of command line flags can always be printed using the ``-h`` flag (or its long form ``--help``):: $ mypy -h - usage: mypy [-h] [-v] [-V] [--python-version x.y] [--platform PLATFORM] - [-2] [-s] [--almost-silent] [--disallow-untyped-calls] + usage: mypy [-h] [-v] [-V] [--python-version x.y] [--platform PLATFORM] [-2] + [-s] [--almost-silent] [--disallow-untyped-calls] [--disallow-untyped-defs] [--check-untyped-defs] [--disallow-subclassing-any] [--warn-incomplete-stub] [--warn-redundant-casts] [--warn-unused-ignores] - [--suppress-error-context] [--fast-parser] [-i] - [--cache-dir DIR] [--strict-optional] + [--hide-error-context] [--fast-parser] [-i] [--cache-dir DIR] + [--strict-optional] [--strict-optional-whitelist [GLOB [GLOB ...]]] [--pdb] [--show-traceback] [--stats] [--inferstats] [--custom-typing MODULE] [--scripts-are-modules] - [--config-file CONFIG_FILE] [--linecount-report DIR] - [--linecoverage-report DIR] [--old-html-report DIR] - [--memory-xml-report DIR] [--xml-report DIR] - [--xslt-html-report DIR] [--xslt-txt-report DIR] - [--html-report DIR] [--txt-report DIR] [-m MODULE] - [-c PROGRAM_TEXT] [-p PACKAGE] + [--config-file CONFIG_FILE] [--show-column-numbers] + [--linecount-report DIR] [--linecoverage-report DIR] + [--old-html-report DIR] [--memory-xml-report DIR] + [--xml-report DIR] [--xslt-html-report DIR] + [--xslt-txt-report DIR] [--html-report DIR] [--txt-report DIR] + [-m MODULE] [-c PROGRAM_TEXT] [-p PACKAGE] [files [files ...]] + (etc., too long to show everything here) Specifying files and directories to be checked @@ -306,6 +307,23 @@ Here are some more useful flags: default to using whatever operating system you are currently using. See :ref:`version_and_platform_checks` for more about this feature. +- ``--show-column-numbers`` will add column offsets to error messages, + for example, the following indicates an error in line 12, column 9 + (note that column offsets are 0-based): + + .. code-block:: python + + main.py:12:9: error: Unsupported operand types for / ("int" and "str") + +- ``--scripts-are-modules`` will give command line arguments that + appear to be scripts (i.e. files whose name does not end in ``.py``) + a module name derived from the script name rather than the fixed + name ``__main__``. This allows checking more than one script in a + single mypy invocation. (The default ``__main__`` is technically + more correct, but if you have many scripts that import a large + package, the behavior enabled by this flag is often more + convenient.) + .. _config-file-flag: - ``--config-file CONFIG_FILE`` causes configuration settings to be diff --git a/docs/source/common_issues.rst b/docs/source/common_issues.rst index 259300b450e3..c743e8653b7e 100644 --- a/docs/source/common_issues.rst +++ b/docs/source/common_issues.rst @@ -260,17 +260,16 @@ imports to a module and those imports cause cycles that didn't exist before. If those cycles become a problem when running your program, there's a trick: if the import is only needed for type annotations in forward references (string literals) or comments, you can write the -imports inside ``if False:`` so that they are not executed at runtime. -The reason this works is that mypy (currently) does not analyze -unreachable code like this. Example: +imports inside ``if TYPE_CHECKING:`` so that they are not executed at runtime. +Example: File ``foo.py``: .. code-block:: python - from typing import List + from typing import List, TYPE_CHECKING - if False: + if TYPE_CHECKING: import bar def listify(arg: 'bar.BarClass') -> 'List[bar.BarClass]': @@ -289,7 +288,5 @@ File ``bar.py``: .. note:: - It is possible that in the future, mypy will change its dead code - analysis and this trick will stop working. We will then offer an - alternative, e.g. a constant defined by the ``typing`` module that + The ``TYPE_CHECKING`` constant defined by the ``typing`` module is ``False`` at runtime but ``True`` while type checking. diff --git a/docs/source/config_file.rst b/docs/source/config_file.rst index cddde658171d..9688cbb622a2 100644 --- a/docs/source/config_file.rst +++ b/docs/source/config_file.rst @@ -87,7 +87,7 @@ The following global flags may only be set in the global section - ``cache_dir`` (string, default ``.mypy_cache``) stores module cache info in the given folder in incremental mode. -- ``suppress_error_context`` (Boolean, default False) suppresses +- ``hide_error_context`` (Boolean, default False) hides context notes before errors. diff --git a/docs/source/index.rst b/docs/source/index.rst index a223e2eac94c..c41e191a9633 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -28,6 +28,7 @@ Mypy is a static type checker for Python. additional_features command_line config_file + python36 faq cheat_sheet revision_history diff --git a/docs/source/python36.rst b/docs/source/python36.rst new file mode 100644 index 000000000000..3b67dd228fef --- /dev/null +++ b/docs/source/python36.rst @@ -0,0 +1,64 @@ +.. python36: + +New features in Python 3.6 +========================== + +Python 3.6 will be `released +`_ in December 2016. The +`first beta `_ +came out in September and adds some exciting features. Here's the +support matrix for these in mypy (to be updated with each new mypy +release). The intention is to support all of these by the time Python +3.6 is released. + +Syntax for variable annotations (`PEP 526 `_) +--------------------------------------------------------------------------------------- + +Python 3.6 feature: variables (in global, class or local scope) can +now have type annotations using either of the two forms: + +.. code-block:: python + + foo: Optional[int] + bar: List[str] = [] + +Mypy fully supports this syntax, interpreting them as equivalent to + +.. code-block:: python + + foo = None # type: Optional[int] + bar = [] # type: List[str] + +Literal string formatting (`PEP 498 `_) +--------------------------------------------------------------------------------- + +Python 3.6 feature: string literals of the form +``f"text {expression} text"`` evaluate ``expression`` using the +current evaluation context (locals and globals). + +Mypy does not yet support this. + +Underscores in numeric literals (`PEP 515 `_) +--------------------------------------------------------------------------------------- + +Python 3.6 feature: numeric literals can contain underscores, +e.g. ``1_000_000``. + +Mypy does not yet support this. + +Asynchronous generators (`PEP 525 `_) +------------------------------------------------------------------------------- + +Python 3.6 feature: coroutines defined with ``async def`` (PEP 492) +can now also be generators, i.e. contain ``yield`` expressions. + +Mypy does not yet support this. + +Asynchronous comprehensions (`PEP 530 `_) +----------------------------------------------------------------------------------- + +Python 3.6 feature: coroutines defined with ``async def`` (PEP 492) +can now also contain list, set and dict comprehensions that use +``async for`` syntax. + +Mypy does not yet support this. diff --git a/mypy.ini b/mypy.ini new file mode 100644 index 000000000000..8776f1f391a6 --- /dev/null +++ b/mypy.ini @@ -0,0 +1,5 @@ +[mypy] +show_none_errors = False + +[mypy-mypy/test/*] +show_none_errors = True diff --git a/mypy/binder.py b/mypy/binder.py index 2a987517e836..8056f85cab21 100644 --- a/mypy/binder.py +++ b/mypy/binder.py @@ -1,8 +1,9 @@ -from typing import (Any, Dict, List, Set, Iterator) +from typing import (Any, Dict, List, Set, Iterator, Optional, DefaultDict, Tuple, Union) from contextlib import contextmanager +from collections import defaultdict from mypy.types import Type, AnyType, PartialType -from mypy.nodes import (Node, Var) +from mypy.nodes import (Node, Expression, Var, RefExpr, SymbolTableNode) from mypy.subtypes import is_subtype from mypy.join import join_simple @@ -37,6 +38,7 @@ class A: reveal_type(lst[0].a) # str ``` """ + type_assignments = None # type: Optional[DefaultDict[Expression, List[Tuple[Type, Type]]]] def __init__(self) -> None: # The set of frames currently used. These map @@ -96,16 +98,16 @@ def _get(self, key: Key, index: int=-1) -> Type: return self.frames[i][key] return None - def push(self, expr: Node, typ: Type) -> None: - if not expr.literal: + def push(self, node: Node, typ: Type) -> None: + if not node.literal: return - key = expr.literal_hash + key = node.literal_hash if key not in self.declarations: - self.declarations[key] = self.get_declaration(expr) + self.declarations[key] = self.get_declaration(node) self._add_dependencies(key) self._push(key, typ) - def get(self, expr: Node) -> Type: + def get(self, expr: Union[Expression, Var]) -> Type: return self._get(expr.literal_hash) def cleanse(self, expr: Node) -> None: @@ -165,19 +167,29 @@ def pop_frame(self, fall_through: int = 0) -> Frame: return result - def get_declaration(self, expr: Any) -> Type: - if hasattr(expr, 'node') and isinstance(expr.node, Var): - type = expr.node.type + def get_declaration(self, node: Node) -> Type: + if isinstance(node, (RefExpr, SymbolTableNode)) and isinstance(node.node, Var): + type = node.node.type if isinstance(type, PartialType): return None return type else: return None - def assign_type(self, expr: Node, + @contextmanager + def accumulate_type_assignments(self) -> Iterator[DefaultDict[Expression, + List[Tuple[Type, Type]]]]: + self.type_assignments = defaultdict(list) + yield self.type_assignments + self.type_assignments = None + + def assign_type(self, expr: Expression, type: Type, declared_type: Type, restrict_any: bool = False) -> None: + if self.type_assignments is not None: + self.type_assignments[expr].append((type, declared_type)) + return if not expr.literal: return self.invalidate_dependencies(expr) @@ -197,7 +209,6 @@ def assign_type(self, expr: Node, # If x is Any and y is int, after x = y we do not infer that x is int. # This could be changed. - # Eric: I'm changing it in weak typing mode, since Any is so common. if (isinstance(self.most_recent_enclosing_type(expr, type), AnyType) and not restrict_any): @@ -212,7 +223,7 @@ def assign_type(self, expr: Node, # just copy this variable into a single stored frame. self.allow_jump(i) - def invalidate_dependencies(self, expr: Node) -> None: + def invalidate_dependencies(self, expr: Expression) -> None: """Invalidate knowledge of types that include expr, but not expr itself. For example, when expr is foo.bar, invalidate foo.bar.baz. @@ -223,7 +234,7 @@ def invalidate_dependencies(self, expr: Node) -> None: for dep in self.dependencies.get(expr.literal_hash, set()): self._cleanse_key(dep) - def most_recent_enclosing_type(self, expr: Node, type: Type) -> Type: + def most_recent_enclosing_type(self, expr: Expression, type: Type) -> Type: if isinstance(type, AnyType): return self.get_declaration(expr) key = expr.literal_hash diff --git a/mypy/build.py b/mypy/build.py index 285dbbad5c26..b7ee2d09ae5e 100644 --- a/mypy/build.py +++ b/mypy/build.py @@ -22,18 +22,14 @@ from os.path import dirname, basename from typing import (AbstractSet, Dict, Iterable, Iterator, List, - NamedTuple, Optional, Set, Tuple, Union, Mapping) + NamedTuple, Optional, Set, Tuple, Union) -from mypy.types import Type -from mypy.nodes import (MypyFile, Node, Import, ImportFrom, ImportAll, - SymbolTableNode, MODULE_REF) +from mypy.nodes import (MypyFile, Import, ImportFrom, ImportAll) from mypy.semanal import FirstPass, SemanticAnalyzer, ThirdPass from mypy.checker import TypeChecker from mypy.indirection import TypeIndirectionVisitor from mypy.errors import Errors, CompileError, DecodeError, report_internal_error -from mypy import fixup from mypy.report import Reports -from mypy import defaults from mypy import moduleinfo from mypy import util from mypy.fixup import fixup_module_pass_one, fixup_module_pass_two @@ -343,7 +339,7 @@ def __init__(self, data_dir: str, version_id: str) -> None: self.start_time = time.time() self.data_dir = data_dir - self.errors = Errors(options.suppress_error_context) + self.errors = Errors(options.hide_error_context, options.show_column_numbers) self.errors.set_ignore_prefix(ignore_prefix) self.lib_path = tuple(lib_path) self.source_set = source_set @@ -454,15 +450,15 @@ def module_not_found(self, path: str, line: int, id: str) -> None: if ((self.options.python_version[0] == 2 and moduleinfo.is_py2_std_lib_module(id)) or (self.options.python_version[0] >= 3 and moduleinfo.is_py3_std_lib_module(id))): self.errors.report( - line, "No library stub file for standard library module '{}'".format(id)) - self.errors.report(line, stub_msg, severity='note', only_once=True) + line, 0, "No library stub file for standard library module '{}'".format(id)) + self.errors.report(line, 0, stub_msg, severity='note', only_once=True) elif moduleinfo.is_third_party_module(id): - self.errors.report(line, "No library stub file for module '{}'".format(id)) - self.errors.report(line, stub_msg, severity='note', only_once=True) + self.errors.report(line, 0, "No library stub file for module '{}'".format(id)) + self.errors.report(line, 0, stub_msg, severity='note', only_once=True) else: - self.errors.report(line, "Cannot find module named '{}'".format(id)) - self.errors.report(line, '(Perhaps setting MYPYPATH ' - 'or using the "--silent-imports" flag would help)', + self.errors.report(line, 0, "Cannot find module named '{}'".format(id)) + self.errors.report(line, 0, '(Perhaps setting MYPYPATH ' + 'or using the "--silent-imports" flag would help)', severity='note', only_once=True) def report_file(self, file: MypyFile) -> None: @@ -770,7 +766,7 @@ def find_cache_meta(id: str, path: str, manager: BuildManager) -> Optional[Cache return m -def is_meta_fresh(meta: CacheMeta, id: str, path: str, manager: BuildManager) -> bool: +def is_meta_fresh(meta: Optional[CacheMeta], id: str, path: str, manager: BuildManager) -> bool: if meta is None: return False @@ -1140,10 +1136,8 @@ def __init__(self, # misspelled module name, missing stub, module not in # search path or the module has not been installed. if caller_state: - suppress_message = ((self.options.silent_imports and - not self.options.almost_silent) or - (caller_state.tree is not None and - 'import' in caller_state.tree.weak_opts)) + suppress_message = (self.options.silent_imports + and not self.options.almost_silent) if not suppress_message: save_import_context = manager.errors.import_context() manager.errors.set_import_context(caller_state.import_context) @@ -1190,11 +1184,11 @@ def skipping_ancestor(self, id: str, path: str, ancestor_for: 'State') -> None: manager = self.manager manager.errors.set_import_context([]) manager.errors.set_file(ancestor_for.xpath) - manager.errors.report(-1, "Ancestor package '%s' silently ignored" % (id,), + manager.errors.report(-1, -1, "Ancestor package '%s' silently ignored" % (id,), severity='note', only_once=True) - manager.errors.report(-1, "(Using --silent-imports, submodule passed on command line)", + manager.errors.report(-1, -1, "(Using --silent-imports, submodule passed on command line)", severity='note', only_once=True) - manager.errors.report(-1, "(This note brought to you by --almost-silent)", + manager.errors.report(-1, -1, "(This note brought to you by --almost-silent)", severity='note', only_once=True) def skipping_module(self, id: str, path: str) -> None: @@ -1204,11 +1198,13 @@ def skipping_module(self, id: str, path: str) -> None: manager.errors.set_import_context(self.caller_state.import_context) manager.errors.set_file(self.caller_state.xpath) line = self.caller_line - manager.errors.report(line, "Import of '%s' silently ignored" % (id,), + manager.errors.report(line, 0, + "Import of '%s' silently ignored" % (id,), severity='note') - manager.errors.report(line, "(Using --silent-imports, module not passed on command line)", + manager.errors.report(line, 0, + "(Using --silent-imports, module not passed on command line)", severity='note', only_once=True) - manager.errors.report(line, "(This note courtesy of --almost-silent)", + manager.errors.report(line, 0, "(This note courtesy of --almost-silent)", severity='note', only_once=True) manager.errors.set_import_context(save_import_context) @@ -1264,7 +1260,7 @@ def wrap_context(self) -> Iterator[None]: except CompileError: raise except Exception as err: - report_internal_error(err, self.path, 0, self.manager.errors) + report_internal_error(err, self.path, 0, self.manager.errors, self.options) self.manager.errors.set_import_context(save_import_context) self.check_blockers() @@ -1378,7 +1374,8 @@ def parse_file(self) -> None: if id == '': # Must be from a relative import. manager.errors.set_file(self.xpath) - manager.errors.report(line, "No parent module -- cannot perform relative import", + manager.errors.report(line, 0, + "No parent module -- cannot perform relative import", blocker=True) continue if id not in dep_line_map: @@ -1406,7 +1403,7 @@ def semantic_analysis(self) -> None: def semantic_analysis_pass_three(self) -> None: with self.wrap_context(): - self.manager.semantic_analyzer_pass3.visit_file(self.tree, self.xpath) + self.manager.semantic_analyzer_pass3.visit_file(self.tree, self.xpath, self.options) if self.options.dump_type_stats: dump_type_stats(self.tree, self.xpath) @@ -1492,7 +1489,7 @@ def load_graph(sources: List[BuildSource], manager: BuildManager) -> Graph: continue if st.id in graph: manager.errors.set_file(st.xpath) - manager.errors.report(-1, "Duplicate module named '%s'" % st.id) + manager.errors.report(-1, -1, "Duplicate module named '%s'" % st.id) manager.errors.raise_error() graph[st.id] = st new.append(st) @@ -1533,6 +1530,7 @@ def load_graph(sources: List[BuildSource], manager: BuildManager) -> Graph: for id, g in graph.items(): if g.has_new_submodules(): g.parse_file() + g.fix_suppressed_dependencies(graph) g.mark_interface_stale() return graph @@ -1617,7 +1615,9 @@ def process_graph(graph: Graph, manager: BuildManager) -> None: elif undeps: fresh_msg = "stale due to changed suppression (%s)" % " ".join(sorted(undeps)) elif stale_scc: - fresh_msg = "inherently stale (%s)" % " ".join(sorted(stale_scc)) + fresh_msg = "inherently stale" + if stale_scc != ascc: + fresh_msg += " (%s)" % " ".join(sorted(stale_scc)) if stale_deps: fresh_msg += " with stale deps (%s)" % " ".join(sorted(stale_deps)) else: diff --git a/mypy/checker.py b/mypy/checker.py index d9c9f5ddbecf..0b8681f43de9 100644 --- a/mypy/checker.py +++ b/mypy/checker.py @@ -1,18 +1,15 @@ """Mypy type checker.""" import itertools -import contextlib import fnmatch -import os -import os.path from typing import ( - Any, Dict, Set, List, cast, Tuple, TypeVar, Union, Optional, NamedTuple + Dict, Set, List, cast, Tuple, TypeVar, Union, Optional, NamedTuple ) from mypy.errors import Errors, report_internal_error from mypy.nodes import ( - SymbolTable, Node, MypyFile, Var, Expression, + SymbolTable, Node, MypyFile, Var, Expression, Lvalue, OverloadedFuncDef, FuncDef, FuncItem, FuncBase, TypeInfo, ClassDef, GDEF, Block, AssignmentStmt, NameExpr, MemberExpr, IndexExpr, TupleExpr, ListExpr, ExpressionStmt, ReturnStmt, IfStmt, @@ -41,7 +38,6 @@ from mypy.messages import MessageBuilder import mypy.checkexpr from mypy.checkmember import map_type_from_supertype -from mypy import defaults from mypy import messages from mypy.subtypes import ( is_subtype, is_equivalent, is_proper_subtype, @@ -50,9 +46,9 @@ from mypy.maptype import map_instance_to_supertype from mypy.semanal import self_type, set_callable_name, refers_to_fullname from mypy.erasetype import erase_typevars -from mypy.expandtype import expand_type_by_instance, expand_type +from mypy.expandtype import expand_type from mypy.visitor import NodeVisitor -from mypy.join import join_types +from mypy.join import join_types, join_type_list from mypy.treetransform import TransformVisitor from mypy.meet import meet_simple, nearest_builtin_ancestor, is_overlapping_types from mypy.binder import ConditionalTypeBinder @@ -103,8 +99,6 @@ class TypeChecker(NodeVisitor[Type]): dynamic_funcs = None # type: List[bool] # Stack of functions being type checked function_stack = None # type: List[FuncItem] - # Do weak type checking in this file - weak_opts = set() # type: Set[str] # Stack of collections of variables with partial types partial_types = None # type: List[Dict[Var, Context]] globals = None # type: SymbolTable @@ -143,7 +137,6 @@ def __init__(self, errors: Errors, modules: Dict[str, MypyFile]) -> None: self.type_context = [] self.dynamic_funcs = [] self.function_stack = [] - self.weak_opts = set() # type: Set[str] self.partial_types = [] self.deferred_nodes = [] self.pass_num = 0 @@ -157,7 +150,6 @@ def visit_file(self, file_node: MypyFile, path: str, options: Options) -> None: self.is_stub = file_node.is_stub self.errors.set_file(path) self.globals = file_node.names - self.weak_opts = file_node.weak_opts self.enter_partial_types() self.is_typeshed_stub = self.errors.is_typeshed_file(path) self.module_type_map = {} @@ -223,15 +215,15 @@ def accept(self, node: Node, type_context: Type = None) -> Type: try: typ = node.accept(self) except Exception as err: - report_internal_error(err, self.errors.file, node.line, self.errors) + report_internal_error(err, self.errors.file, node.line, self.errors, self.options) self.type_context.pop() self.store_type(node, typ) - if self.typing_mode_none(): + if not self.in_checked_function(): return AnyType() else: return typ - def accept_loop(self, body: Node, else_body: Node = None) -> Type: + def accept_loop(self, body: Union[IfStmt, Block], else_body: Block = None) -> Type: """Repeatedly type check a loop body until the frame doesn't change. Then check the else_body. @@ -1030,7 +1022,7 @@ def visit_assignment_stmt(self, s: AssignmentStmt) -> Type: Handle all kinds of assignment statements (simple, indexed, multiple). """ - self.check_assignment(s.lvalues[-1], s.rvalue, s.type is None) + self.check_assignment(s.lvalues[-1], s.rvalue, s.type is None, s.new_syntax) if len(s.lvalues) > 1: # Chained assignment (e.g. x = y = ...). @@ -1041,11 +1033,19 @@ def visit_assignment_stmt(self, s: AssignmentStmt) -> Type: for lv in s.lvalues[:-1]: self.check_assignment(lv, rvalue, s.type is None) - def check_assignment(self, lvalue: Node, rvalue: Node, infer_lvalue_type: bool = True) -> None: + def check_assignment(self, lvalue: Lvalue, rvalue: Expression, infer_lvalue_type: bool = True, + new_syntax: bool = False) -> None: """Type check a single assignment: lvalue = rvalue.""" if isinstance(lvalue, TupleExpr) or isinstance(lvalue, ListExpr): - self.check_assignment_to_multiple_lvalues(lvalue.items, rvalue, lvalue, - infer_lvalue_type) + if isinstance(rvalue, TupleExpr) or isinstance(rvalue, ListExpr): + self.check_multi_assign_literal(lvalue.items, rvalue, lvalue, infer_lvalue_type) + return + # Infer the type of an ordinary rvalue expression. + # TODO maybe elsewhere; redundant + rvalue_type = self.accept(rvalue) + self.check_multi_assign(lvalue.items, rvalue, rvalue_type, lvalue, + undefined_rvalue=False, + infer_lvalue_type=infer_lvalue_type) else: lvalue_type, index_lvalue, inferred = self.check_lvalue(lvalue) if lvalue_type: @@ -1078,7 +1078,8 @@ def check_assignment(self, lvalue: Node, rvalue: Node, infer_lvalue_type: bool = elif (is_literal_none(rvalue) and isinstance(lvalue, NameExpr) and isinstance(lvalue.node, Var) and - lvalue.node.is_initialized_in_class): + lvalue.node.is_initialized_in_class and + not new_syntax): # Allow None's to be assigned to class variables with non-Optional types. rvalue_type = lvalue_type else: @@ -1088,7 +1089,7 @@ def check_assignment(self, lvalue: Node, rvalue: Node, infer_lvalue_type: bool = self.binder.assign_type(lvalue, rvalue_type, lvalue_type, - self.typing_mode_weak()) + False) elif index_lvalue: self.check_indexed_assignment(index_lvalue, rvalue, rvalue) @@ -1096,42 +1097,33 @@ def check_assignment(self, lvalue: Node, rvalue: Node, infer_lvalue_type: bool = self.infer_variable_type(inferred, lvalue, self.accept(rvalue), rvalue) - def check_assignment_to_multiple_lvalues(self, lvalues: List[Node], rvalue: Node, - context: Context, - infer_lvalue_type: bool = True) -> None: - if isinstance(rvalue, TupleExpr) or isinstance(rvalue, ListExpr): - # Recursively go into Tuple or List expression rhs instead of - # using the type of rhs, because this allowed more fine grained - # control in cases like: a, b = [int, str] where rhs would get - # type List[object] - - rvalues = rvalue.items - - if self.check_rvalue_count_in_assignment(lvalues, len(rvalues), context): - star_index = next((i for i, lv in enumerate(lvalues) if - isinstance(lv, StarExpr)), len(lvalues)) - - left_lvs = lvalues[:star_index] - star_lv = cast(StarExpr, - lvalues[star_index]) if star_index != len(lvalues) else None - right_lvs = lvalues[star_index + 1:] - - left_rvs, star_rvs, right_rvs = self.split_around_star( - rvalues, star_index, len(lvalues)) - - lr_pairs = list(zip(left_lvs, left_rvs)) - if star_lv: - rv_list = ListExpr(star_rvs) - rv_list.set_line(rvalue.get_line()) - lr_pairs.append((star_lv.expr, rv_list)) - lr_pairs.extend(zip(right_lvs, right_rvs)) - - for lv, rv in lr_pairs: - self.check_assignment(lv, rv, infer_lvalue_type) - else: - self.check_multi_assignment(lvalues, rvalue, context, infer_lvalue_type) - - def check_rvalue_count_in_assignment(self, lvalues: List[Node], rvalue_count: int, + def check_multi_assign_literal(self, lvalues: List[Lvalue], + rvalue: Union[ListExpr, TupleExpr], + context: Context, infer_lvalue_type: bool = True) -> None: + # Recursively go into Tuple or List expression rhs instead of + # using the type of rhs, because this allowed more fine grained + # control in cases like: a, b = [int, str] where rhs would get + # type List[object] + # Tuple is also special cased to handle mutually nested lists and tuples + rvalues = rvalue.items + if self.check_rvalue_count_in_assignment(lvalues, len(rvalues), context): + star_index = next((i for (i, lv) in enumerate(lvalues) if isinstance(lv, StarExpr)), + len(lvalues)) + left_lvs = lvalues[:star_index] + star_lv = cast(StarExpr, lvalues[star_index]) if star_index != len(lvalues) else None + right_lvs = lvalues[star_index + 1:] + left_rvs, star_rvs, right_rvs = self.split_around_star( + rvalues, star_index, len(lvalues)) + lr_pairs = list(zip(left_lvs, left_rvs)) + if star_lv: + rv_list = ListExpr(star_rvs) + rv_list.set_line(rvalue.get_line()) + lr_pairs.append((star_lv.expr, rv_list)) + lr_pairs.extend(zip(right_lvs, right_rvs)) + for lv, rv in lr_pairs: + self.check_assignment(lv, rv, infer_lvalue_type) + + def check_rvalue_count_in_assignment(self, lvalues: List[Lvalue], rvalue_count: int, context: Context) -> bool: if any(isinstance(lvalue, StarExpr) for lvalue in lvalues): if len(lvalues) - 1 > rvalue_count: @@ -1144,33 +1136,64 @@ def check_rvalue_count_in_assignment(self, lvalues: List[Node], rvalue_count: in return False return True - def check_multi_assignment(self, lvalues: List[Node], - rvalue: Node, - context: Context, - infer_lvalue_type: bool = True, - msg: str = None) -> None: - """Check the assignment of one rvalue to a number of lvalues.""" - - # Infer the type of an ordinary rvalue expression. - rvalue_type = self.accept(rvalue) # TODO maybe elsewhere; redundant - undefined_rvalue = False - + def check_multi_assign_from_any(self, lvalues: List[Expression], rvalue: Expression, + rvalue_type: AnyType, context: Context, + undefined_rvalue: bool, + infer_lvalue_type: bool) -> None: + for lv in lvalues: + if isinstance(lv, StarExpr): + lv = lv.expr + self.check_assignment(lv, self.temp_node(AnyType(), context), infer_lvalue_type) + + def check_multi_assign(self, lvalues: List[Lvalue], rvalue: Expression, + rvalue_type: Type, context: Context, *, + undefined_rvalue: bool = False, + infer_lvalue_type: bool = True) -> None: if isinstance(rvalue_type, AnyType): - for lv in lvalues: - if isinstance(lv, StarExpr): - lv = lv.expr - self.check_assignment(lv, self.temp_node(AnyType(), context), infer_lvalue_type) + self.check_multi_assign_from_any(lvalues, rvalue, rvalue_type, + context, undefined_rvalue, infer_lvalue_type) elif isinstance(rvalue_type, TupleType): - self.check_multi_assignment_from_tuple(lvalues, rvalue, rvalue_type, + self.check_multi_assign_from_tuple(lvalues, rvalue, rvalue_type, + context, undefined_rvalue, infer_lvalue_type) + elif isinstance(rvalue_type, UnionType): + self.check_multi_assign_from_union(lvalues, rvalue, rvalue_type, + context, undefined_rvalue, infer_lvalue_type) + elif isinstance(rvalue_type, Instance) and self.instance_is_iterable(rvalue_type): + self.check_multi_assign_from_iterable(lvalues, rvalue, rvalue_type, context, undefined_rvalue, infer_lvalue_type) else: - self.check_multi_assignment_from_iterable(lvalues, rvalue_type, - context, infer_lvalue_type) + self.msg.type_not_iterable(rvalue_type, context) - def check_multi_assignment_from_tuple(self, lvalues: List[Node], rvalue: Node, - rvalue_type: TupleType, context: Context, - undefined_rvalue: bool, - infer_lvalue_type: bool = True) -> None: + def check_multi_assign_from_union(self, lvalues: List[Expression], rvalue: Expression, + rvalue_type: UnionType, context: Context, + undefined_rvalue: bool, + infer_lvalue_type: bool) -> None: + transposed = tuple([] for _ in lvalues) # type: Tuple[List[Type], ...] + with self.binder.accumulate_type_assignments() as assignments: + for item in rvalue_type.items: + self.check_multi_assign(lvalues, rvalue, item, context, + undefined_rvalue=True, + infer_lvalue_type=infer_lvalue_type) + for t, lv in zip(transposed, lvalues): + t.append(self.type_map.get(lv, AnyType())) + union_types = tuple(join_type_list(col) for col in transposed) + for expr, items in assignments.items(): + types, declared_types = zip(*items) + self.binder.assign_type(expr, + join_type_list(types), + join_type_list(declared_types), + False) + for union, lv in zip(union_types, lvalues): + _1, _2, inferred = self.check_lvalue(lv) + if inferred: + self.set_inferred_type(inferred, lv, union) + else: + self.store_type(lv, union) + + def check_multi_assign_from_tuple(self, lvalues: List[Lvalue], rvalue: Expression, + rvalue_type: TupleType, context: Context, + undefined_rvalue: bool, + infer_lvalue_type: bool) -> None: if self.check_rvalue_count_in_assignment(lvalues, len(rvalue_type.items), context): star_index = next((i for i, lv in enumerate(lvalues) if isinstance(lv, StarExpr)), len(lvalues)) @@ -1190,14 +1213,14 @@ def check_multi_assignment_from_tuple(self, lvalues: List[Node], rvalue: Node, for lv, rv_type in zip(left_lvs, left_rv_types): self.check_assignment(lv, self.temp_node(rv_type, context), infer_lvalue_type) if star_lv: - nodes = [self.temp_node(rv_type, context) for rv_type in star_rv_types] - list_expr = ListExpr(nodes) + list_expr = ListExpr([self.temp_node(rv_type, context) + for rv_type in star_rv_types]) list_expr.set_line(context.get_line()) self.check_assignment(star_lv.expr, list_expr, infer_lvalue_type) for lv, rv_type in zip(right_lvs, right_rv_types): self.check_assignment(lv, self.temp_node(rv_type, context), infer_lvalue_type) - def lvalue_type_for_inference(self, lvalues: List[Node], rvalue_type: TupleType) -> Type: + def lvalue_type_for_inference(self, lvalues: List[Lvalue], rvalue_type: TupleType) -> Type: star_index = next((i for i, lv in enumerate(lvalues) if isinstance(lv, StarExpr)), len(lvalues)) left_lvs = lvalues[:star_index] @@ -1208,7 +1231,7 @@ def lvalue_type_for_inference(self, lvalues: List[Node], rvalue_type: TupleType) type_parameters = [] # type: List[Type] - def append_types_for_inference(lvs: List[Node], rv_types: List[Type]) -> None: + def append_types_for_inference(lvs: List[Expression], rv_types: List[Type]) -> None: for lv, rv_type in zip(lvs, rv_types): sub_lvalue_type, index_expr, inferred = self.check_lvalue(lv) if sub_lvalue_type: @@ -1248,27 +1271,24 @@ def split_around_star(self, items: List[T], star_index: int, right = items[right_index:] return (left, star, right) - def type_is_iterable(self, type: Type) -> bool: - return (is_subtype(type, self.named_generic_type('typing.Iterable', - [AnyType()])) and - isinstance(type, Instance)) - - def check_multi_assignment_from_iterable(self, lvalues: List[Node], rvalue_type: Type, - context: Context, - infer_lvalue_type: bool = True) -> None: - if self.type_is_iterable(rvalue_type): - item_type = self.iterable_item_type(cast(Instance, rvalue_type)) - for lv in lvalues: - if isinstance(lv, StarExpr): - self.check_assignment(lv.expr, self.temp_node(rvalue_type, context), - infer_lvalue_type) - else: - self.check_assignment(lv, self.temp_node(item_type, context), - infer_lvalue_type) - else: - self.msg.type_not_iterable(rvalue_type, context) + def instance_is_iterable(self, instance: Instance) -> bool: + return is_subtype(instance, self.named_generic_type('typing.Iterable', + [AnyType()])) + + def check_multi_assign_from_iterable(self, lvalues: List[Expression], rvalue: Expression, + rvalue_type: Instance, context: Context, + undefined_rvalue: bool, + infer_lvalue_type: bool) -> None: + item_type = self.iterable_item_type(rvalue_type) + for lv in lvalues: + if isinstance(lv, StarExpr): + self.check_assignment(lv.expr, self.temp_node(rvalue_type, context), + infer_lvalue_type) + else: + self.check_assignment(lv, self.temp_node(item_type, context), + infer_lvalue_type) - def check_lvalue(self, lvalue: Node) -> Tuple[Type, IndexExpr, Var]: + def check_lvalue(self, lvalue: Lvalue) -> Tuple[Type, IndexExpr, Var]: lvalue_type = None # type: Type index_lvalue = None # type: IndexExpr inferred = None # type: Var @@ -1298,7 +1318,7 @@ def check_lvalue(self, lvalue: Node) -> Tuple[Type, IndexExpr, Var]: return lvalue_type, index_lvalue, inferred - def is_definition(self, s: Node) -> bool: + def is_definition(self, s: Lvalue) -> bool: if isinstance(s, NameExpr): if s.is_def: return True @@ -1314,13 +1334,10 @@ def is_definition(self, s: Node) -> bool: return s.is_def return False - def infer_variable_type(self, name: Var, lvalue: Node, + def infer_variable_type(self, name: Var, lvalue: Lvalue, init_type: Type, context: Context) -> None: """Infer the type of initialized variables from initializer type.""" - if self.typing_mode_weak(): - self.set_inferred_type(name, lvalue, AnyType()) - self.binder.assign_type(lvalue, init_type, self.binder.get_declaration(lvalue), True) - elif self.is_unusable_type(init_type): + if self.is_unusable_type(init_type): self.check_usable_type(init_type, context) self.set_inference_error_fallback_type(name, lvalue, init_type, context) elif isinstance(init_type, DeletedType): @@ -1341,7 +1358,7 @@ def infer_variable_type(self, name: Var, lvalue: Node, self.set_inferred_type(name, lvalue, init_type) - def infer_partial_type(self, name: Var, lvalue: Node, init_type: Type) -> bool: + def infer_partial_type(self, name: Var, lvalue: Lvalue, init_type: Type) -> bool: if isinstance(init_type, (NoneTyp, UninhabitedType)): partial_type = PartialType(None, name, [init_type]) elif isinstance(init_type, Instance): @@ -1360,7 +1377,7 @@ def infer_partial_type(self, name: Var, lvalue: Node, init_type: Type) -> bool: self.partial_types[-1][name] = lvalue return True - def set_inferred_type(self, var: Var, lvalue: Node, type: Type) -> None: + def set_inferred_type(self, var: Var, lvalue: Lvalue, type: Type) -> None: """Store inferred variable type. Store the type to both the variable node and the expression node that @@ -1370,7 +1387,7 @@ def set_inferred_type(self, var: Var, lvalue: Node, type: Type) -> None: var.type = type self.store_type(lvalue, type) - def set_inference_error_fallback_type(self, var: Var, lvalue: Node, type: Type, + def set_inference_error_fallback_type(self, var: Var, lvalue: Lvalue, type: Type, context: Context) -> None: """If errors on context line are ignored, store dummy type for variable. @@ -1385,7 +1402,7 @@ def set_inference_error_fallback_type(self, var: Var, lvalue: Node, type: Type, if context.get_line() in self.errors.ignored_lines[self.errors.file]: self.set_inferred_type(var, lvalue, AnyType()) - def narrow_type_from_binder(self, expr: Node, known_type: Type) -> Type: + def narrow_type_from_binder(self, expr: Expression, known_type: Type) -> Type: if expr.literal >= LITERAL_TYPE: restriction = self.binder.get(expr) if restriction: @@ -1393,8 +1410,8 @@ def narrow_type_from_binder(self, expr: Node, known_type: Type) -> Type: return ans return known_type - def check_simple_assignment(self, lvalue_type: Type, rvalue: Node, - context: Node, + def check_simple_assignment(self, lvalue_type: Type, rvalue: Expression, + context: Context, msg: str = messages.INCOMPATIBLE_TYPES_IN_ASSIGNMENT, lvalue_name: str = 'variable', rvalue_name: str = 'expression') -> Type: @@ -1405,8 +1422,6 @@ def check_simple_assignment(self, lvalue_type: Type, rvalue: Node, rvalue_type = self.accept(rvalue, lvalue_type) if isinstance(rvalue_type, DeletedType): self.msg.deleted_as_rvalue(rvalue_type, context) - if self.typing_mode_weak(): - return rvalue_type if isinstance(lvalue_type, DeletedType): self.msg.deleted_as_lvalue(lvalue_type, context) else: @@ -1416,7 +1431,7 @@ def check_simple_assignment(self, lvalue_type: Type, rvalue: Node, return rvalue_type def check_indexed_assignment(self, lvalue: IndexExpr, - rvalue: Node, context: Context) -> None: + rvalue: Expression, context: Context) -> None: """Type check indexed assignment base[index] = rvalue. The lvalue argument is the base[index] expression. @@ -1431,7 +1446,7 @@ def check_indexed_assignment(self, lvalue: IndexExpr, context) def try_infer_partial_type_from_indexed_assignment( - self, lvalue: IndexExpr, rvalue: Node) -> None: + self, lvalue: IndexExpr, rvalue: Expression) -> None: # TODO: Should we share some of this with try_infer_partial_type? if isinstance(lvalue.base, RefExpr) and isinstance(lvalue.base.node, Var): var = lvalue.base.node @@ -1501,7 +1516,7 @@ def visit_return_stmt(self, s: ReturnStmt) -> Type: if isinstance(return_type, (Void, NoneTyp, AnyType)): return None - if self.typing_mode_full(): + if self.in_checked_function(): self.fail(messages.RETURN_VALUE_EXPECTED, s) def wrap_generic_type(self, typ: Instance, rtyp: Instance, check_type: @@ -1536,10 +1551,7 @@ def visit_if_stmt(self, s: IfStmt) -> Type: for e, b in zip(s.expr, s.body): t = self.accept(e) self.check_usable_type(t, e) - if_map, else_map = find_isinstance_check( - e, self.type_map, - self.typing_mode_weak() - ) + if_map, else_map = find_isinstance_check(e, self.type_map) if if_map is None: # The condition is always false # XXX should issue a warning? @@ -1595,10 +1607,7 @@ def visit_assert_stmt(self, s: AssertStmt) -> Type: self.accept(s.expr) # If this is asserting some isinstance check, bind that type in the following code - true_map, _ = find_isinstance_check( - s.expr, self.type_map, - self.typing_mode_weak() - ) + true_map, _ = find_isinstance_check(s.expr, self.type_map) if true_map: for var, type in true_map.items(): @@ -1612,7 +1621,7 @@ def visit_raise_stmt(self, s: RaiseStmt) -> Type: if s.from_expr: self.type_check_raise(s.from_expr, s) - def type_check_raise(self, e: Node, s: RaiseStmt) -> None: + def type_check_raise(self, e: Expression, s: RaiseStmt) -> None: typ = self.accept(e) if isinstance(typ, FunctionLike): if typ.is_type_obj(): @@ -1703,7 +1712,7 @@ def visit_try_without_finally(self, s: TryStmt) -> bool: breaking_out = breaking_out and self.binder.last_pop_breaking_out return breaking_out - def visit_except_handler_test(self, n: Node) -> Type: + def visit_except_handler_test(self, n: Expression) -> Type: """Type check an exception handler test clause.""" type = self.accept(n) @@ -1739,7 +1748,7 @@ def visit_for_stmt(self, s: ForStmt) -> Type: self.analyze_index_variables(s.index, item_type, s) self.accept_loop(s.body, s.else_body) - def analyze_async_iterable_item_type(self, expr: Node) -> Type: + def analyze_async_iterable_item_type(self, expr: Expression) -> Type: """Analyse async iterable expression and return iterator item type.""" iterable = self.accept(expr) @@ -1758,7 +1767,7 @@ def analyze_async_iterable_item_type(self, expr: Node) -> Type: return self.check_awaitable_expr(awaitable, expr, messages.INCOMPATIBLE_TYPES_IN_ASYNC_FOR) - def analyze_iterable_item_type(self, expr: Node) -> Type: + def analyze_iterable_item_type(self, expr: Expression) -> Type: """Analyse iterable expression and return iterator item type.""" iterable = self.accept(expr) @@ -1793,7 +1802,7 @@ def analyze_iterable_item_type(self, expr: Node) -> Type: expr) return echk.check_call(method, [], [], expr)[0] - def analyze_index_variables(self, index: Node, item_type: Type, + def analyze_index_variables(self, index: Expression, item_type: Type, context: Context) -> None: """Type check or infer for loop or list comprehension index vars.""" self.check_assignment(index, self.temp_node(item_type, context)) @@ -1807,7 +1816,7 @@ def visit_del_stmt(self, s: DelStmt) -> Type: c.line = s.line return c.accept(self) else: - def flatten(t: Node) -> List[Node]: + def flatten(t: Expression) -> List[Expression]: """Flatten a nested sequence of tuples/lists into one list of nodes.""" if isinstance(t, TupleExpr) or isinstance(t, ListExpr): return [b for a in t.items for b in flatten(a)] @@ -1820,7 +1829,7 @@ def flatten(t: Node) -> List[Node]: self.binder.assign_type(elt, DeletedType(source=elt.name), self.binder.get_declaration(elt), - self.typing_mode_weak()) + False) return None def visit_decorator(self, e: Decorator) -> Type: @@ -2115,7 +2124,7 @@ def visit_yield_expr(self, e: YieldExpr) -> Type: expected_item_type = self.get_generator_yield_type(return_type, False) if e.expr is None: if (not isinstance(expected_item_type, (Void, NoneTyp, AnyType)) - and self.typing_mode_full()): + and self.in_checked_function()): self.fail(messages.YIELD_VALUE_EXPECTED, e) else: actual_item_type = self.accept(e.expr, expected_item_type) @@ -2226,32 +2235,17 @@ def store_type(self, node: Node, typ: Type) -> None: if typ is not None: self.module_type_map[node] = typ - def typing_mode_none(self) -> bool: - if self.is_dynamic_function() and not self.options.check_untyped_defs: - return not self.weak_opts - elif self.function_stack: - return False - else: - return False - - def typing_mode_weak(self) -> bool: - if self.is_dynamic_function() and not self.options.check_untyped_defs: - return bool(self.weak_opts) - elif self.function_stack: - return False - else: - return 'global' in self.weak_opts - - def typing_mode_full(self) -> bool: - if self.is_dynamic_function() and not self.options.check_untyped_defs: - return False - elif self.function_stack: - return True - else: - return 'global' not in self.weak_opts + def in_checked_function(self) -> bool: + """Should we type-check the current function? - def is_dynamic_function(self) -> bool: - return len(self.dynamic_funcs) > 0 and self.dynamic_funcs[-1] + - Yes if --check-untyped-defs is set. + - Yes outside functions. + - Yes in annotated functions. + - No otherwise. + """ + return (self.options.check_untyped_defs + or not self.dynamic_funcs + or not self.dynamic_funcs[-1]) def lookup(self, name: str, kind: int) -> SymbolTableNode: """Look up a definition from the symbol table with the given name. @@ -2332,7 +2326,7 @@ def check_usable_type(self, typ: Type, context: Context) -> None: if self.is_unusable_type(typ): self.msg.does_not_return_value(typ, context) - def temp_node(self, t: Type, context: Context = None) -> Node: + def temp_node(self, t: Type, context: Context = None) -> TempNode: """Create a temporary node with the given, fixed type.""" temp = TempNode(t) if context: @@ -2370,16 +2364,12 @@ def method_type(self, func: FuncBase) -> FunctionLike: # probably be better to have the dict keyed by the nodes' literal_hash # field instead. -# NB: This should be `TypeMap = Optional[Dict[Node, Type]]`! -# But see https://github.com/python/mypy/issues/1637 -TypeMap = Dict[Node, Type] +TypeMap = Optional[Dict[Node, Type]] -def conditional_type_map(expr: Node, +def conditional_type_map(expr: Expression, current_type: Optional[Type], proposed_type: Optional[Type], - *, - weak: bool = False ) -> Tuple[TypeMap, TypeMap]: """Takes in an expression, the current type of the expression, and a proposed type of that expression. @@ -2401,13 +2391,10 @@ def conditional_type_map(expr: Node, return {expr: proposed_type}, {} else: # An isinstance check, but we don't understand the type - if weak: - return {expr: AnyType()}, {expr: current_type} - else: - return {}, {} + return {}, {} -def is_literal_none(n: Node) -> bool: +def is_literal_none(n: Expression) -> bool: return isinstance(n, NameExpr) and n.fullname == 'builtins.None' @@ -2455,9 +2442,8 @@ def or_conditional_maps(m1: TypeMap, m2: TypeMap) -> TypeMap: return result -def find_isinstance_check(node: Node, +def find_isinstance_check(node: Expression, type_map: Dict[Node, Type], - weak: bool=False ) -> Tuple[TypeMap, TypeMap]: """Find any isinstance checks (within a chain of ands). Includes implicit and explicit checks for None. @@ -2476,7 +2462,7 @@ def find_isinstance_check(node: Node, if expr.literal == LITERAL_TYPE: vartype = type_map[expr] type = get_isinstance_type(node.args[1], type_map) - return conditional_type_map(expr, vartype, type, weak=weak) + return conditional_type_map(expr, vartype, type) elif (isinstance(node, ComparisonExpr) and any(is_literal_none(n) for n in node.operands) and experiments.STRICT_OPTIONAL): # Check for `x is None` and `x is not None`. @@ -2490,7 +2476,7 @@ def find_isinstance_check(node: Node, # two elements in node.operands, and at least one of them # should represent a None. vartype = type_map[expr] - if_vars, else_vars = conditional_type_map(expr, vartype, NoneTyp(), weak=weak) + if_vars, else_vars = conditional_type_map(expr, vartype, NoneTyp()) break if is_not: @@ -2507,49 +2493,31 @@ def find_isinstance_check(node: Node, else_map = {ref: else_type} if not isinstance(else_type, UninhabitedType) else None return if_map, else_map elif isinstance(node, OpExpr) and node.op == 'and': - left_if_vars, left_else_vars = find_isinstance_check( - node.left, - type_map, - weak, - ) - - right_if_vars, right_else_vars = find_isinstance_check( - node.right, - type_map, - weak, - ) + left_if_vars, left_else_vars = find_isinstance_check(node.left, type_map) + right_if_vars, right_else_vars = find_isinstance_check(node.right, type_map) # (e1 and e2) is true if both e1 and e2 are true, # and false if at least one of e1 and e2 is false. return (and_conditional_maps(left_if_vars, right_if_vars), or_conditional_maps(left_else_vars, right_else_vars)) elif isinstance(node, OpExpr) and node.op == 'or': - left_if_vars, left_else_vars = find_isinstance_check( - node.left, - type_map, - weak, - ) - - right_if_vars, right_else_vars = find_isinstance_check( - node.right, - type_map, - weak, - ) + left_if_vars, left_else_vars = find_isinstance_check(node.left, type_map) + right_if_vars, right_else_vars = find_isinstance_check(node.right, type_map) # (e1 or e2) is true if at least one of e1 or e2 is true, # and false if both e1 and e2 are false. return (or_conditional_maps(left_if_vars, right_if_vars), and_conditional_maps(left_else_vars, right_else_vars)) elif isinstance(node, UnaryExpr) and node.op == 'not': - left, right = find_isinstance_check(node.expr, type_map, weak) + left, right = find_isinstance_check(node.expr, type_map) return right, left # Not a supported isinstance check return {}, {} -def get_isinstance_type(node: Node, type_map: Dict[Node, Type]) -> Type: - type = type_map[node] +def get_isinstance_type(expr: Expression, type_map: Dict[Node, Type]) -> Type: + type = type_map[expr] if isinstance(type, TupleType): all_types = type.items @@ -2575,7 +2543,7 @@ def get_isinstance_type(node: Node, type_map: Dict[Node, Type]) -> Type: return UnionType(types) -def expand_node(defn: Node, map: Dict[TypeVarId, Type]) -> Node: +def expand_node(defn: FuncItem, map: Dict[TypeVarId, Type]) -> Node: visitor = TypeTransformVisitor(map) return defn.accept(visitor) diff --git a/mypy/checkexpr.py b/mypy/checkexpr.py index 21900ba7efd0..c65434420529 100644 --- a/mypy/checkexpr.py +++ b/mypy/checkexpr.py @@ -12,7 +12,7 @@ NameExpr, RefExpr, Var, FuncDef, OverloadedFuncDef, TypeInfo, CallExpr, Node, MemberExpr, IntExpr, StrExpr, BytesExpr, UnicodeExpr, FloatExpr, OpExpr, UnaryExpr, IndexExpr, CastExpr, RevealTypeExpr, TypeApplication, ListExpr, - TupleExpr, DictExpr, FuncExpr, SuperExpr, SliceExpr, Context, + TupleExpr, DictExpr, FuncExpr, SuperExpr, SliceExpr, Context, Expression, ListComprehension, GeneratorExpr, SetExpr, MypyFile, Decorator, ConditionalExpr, ComparisonExpr, TempNode, SetComprehension, DictionaryComprehension, ComplexExpr, EllipsisExpr, StarExpr, @@ -151,7 +151,7 @@ def analyze_ref_expr(self, e: RefExpr, lvalue: bool = False) -> Type: def analyze_var_ref(self, var: Var, context: Context) -> Type: if not var.type: - if not var.is_ready and self.chk.typing_mode_full(): + if not var.is_ready and self.chk.in_checked_function(): self.chk.handle_cannot_determine_type(var.name(), context) # Implicit 'Any' type. return AnyType() @@ -171,7 +171,7 @@ def visit_call_expr(self, e: CallExpr) -> Type: self.try_infer_partial_type(e) callee_type = self.accept(e.callee) if (self.chk.options.disallow_untyped_calls and - self.chk.typing_mode_full() and + self.chk.in_checked_function() and isinstance(callee_type, CallableType) and callee_type.implicit): return self.msg.untyped_function_call(callee_type, e) @@ -237,10 +237,10 @@ def check_call_expr_with_callee_type(self, callee_type: Type, return self.check_call(callee_type, e.args, e.arg_kinds, e, e.arg_names, callable_node=e.callee)[0] - def check_call(self, callee: Type, args: List[Node], + def check_call(self, callee: Type, args: List[Expression], arg_kinds: List[int], context: Context, arg_names: List[str] = None, - callable_node: Node = None, + callable_node: Expression = None, arg_messages: MessageBuilder = None) -> Tuple[Type, Type]: """Type check a call. @@ -309,7 +309,7 @@ def check_call(self, callee: Type, args: List[Node], messages=arg_messages) return self.check_call(target, args, arg_kinds, context, arg_names, arg_messages=arg_messages) - elif isinstance(callee, AnyType) or self.chk.typing_mode_none(): + elif isinstance(callee, AnyType) or not self.chk.in_checked_function(): self.infer_arg_types_in_context(None, args) return AnyType(), AnyType() elif isinstance(callee, UnionType): @@ -373,7 +373,7 @@ def analyze_type_type_callee(self, item: Type, context: Context) -> Type: return AnyType() def infer_arg_types_in_context(self, callee: Optional[CallableType], - args: List[Node]) -> List[Type]: + args: List[Expression]) -> List[Type]: """Infer argument expression types using a callable type as context. For example, if callee argument 2 has type List[int], infer the @@ -405,7 +405,7 @@ def infer_arg_types_in_context(self, callee: Optional[CallableType], return res def infer_arg_types_in_context2( - self, callee: CallableType, args: List[Node], arg_kinds: List[int], + self, callee: CallableType, args: List[Expression], arg_kinds: List[int], formal_to_actual: List[List[int]]) -> List[Type]: """Infer argument expression types using a callable type as context. @@ -471,7 +471,7 @@ def infer_function_type_arguments_using_context( error_context)) def infer_function_type_arguments(self, callee_type: CallableType, - args: List[Node], + args: List[Expression], arg_kinds: List[int], formal_to_actual: List[List[int]], context: Context) -> CallableType: @@ -481,7 +481,7 @@ def infer_function_type_arguments(self, callee_type: CallableType, Return a derived callable type that has the arguments applied. """ - if not self.chk.typing_mode_none(): + if self.chk.in_checked_function(): # Disable type errors during type inference. There may be errors # due to partial available context information at this time, but # these errors can be safely ignored as the arguments will be @@ -505,7 +505,7 @@ def infer_function_type_arguments(self, callee_type: CallableType, inferred_args = infer_function_type_arguments( callee_type, pass1_args, arg_kinds, formal_to_actual, - strict=self.chk.typing_mode_full()) # type: List[Type] + strict=self.chk.in_checked_function()) # type: List[Type] if 2 in arg_pass_nums: # Second pass of type inference. @@ -536,7 +536,7 @@ def infer_function_type_arguments(self, callee_type: CallableType, def infer_function_type_arguments_pass2( self, callee_type: CallableType, - args: List[Node], + args: List[Expression], arg_kinds: List[int], formal_to_actual: List[List[int]], inferred_args: List[Type], @@ -667,7 +667,7 @@ def check_argument_count(self, callee: CallableType, actual_types: List[Type], elif kind in [nodes.ARG_POS, nodes.ARG_OPT, nodes.ARG_NAMED] and is_duplicate_mapping( formal_to_actual[i], actual_kinds): - if (self.chk.typing_mode_full() or + if (self.chk.in_checked_function() or isinstance(actual_types[formal_to_actual[i][0]], TupleType)): if messages: messages.duplicate_argument_value(callee, i, context) @@ -965,8 +965,9 @@ def visit_op_expr(self, e: OpExpr) -> Type: if e.op == '*' and isinstance(e.left, ListExpr): # Expressions of form [...] * e get special type inference. return self.check_list_multiply(e) - if e.op == '%' and isinstance(e.left, (StrExpr, BytesExpr)): - return self.strfrm_checker.check_str_interpolation(cast(StrExpr, e.left), e.right) + if e.op == '%': + if isinstance(e.left, (StrExpr, BytesExpr, UnicodeExpr)): + return self.strfrm_checker.check_str_interpolation(e.left, e.right) left_type = self.accept(e.left) if e.op in nodes.op_methods: @@ -1051,7 +1052,7 @@ def get_operator_method(self, op: str) -> str: else: return nodes.op_methods[op] - def _check_op_for_errors(self, method: str, base_type: Type, arg: Node, + def _check_op_for_errors(self, method: str, base_type: Type, arg: Expression, context: Context ) -> Tuple[Tuple[Type, Type], MessageBuilder]: """Type check a binary operation which maps to a method call. @@ -1065,7 +1066,7 @@ def _check_op_for_errors(self, method: str, base_type: Type, arg: Node, local_errors) return result, local_errors - def check_op_local(self, method: str, base_type: Type, arg: Node, + def check_op_local(self, method: str, base_type: Type, arg: Expression, context: Context, local_errors: MessageBuilder) -> Tuple[Type, Type]: """Type check a binary operation which maps to a method call. @@ -1077,7 +1078,7 @@ def check_op_local(self, method: str, base_type: Type, arg: Node, return self.check_call(method_type, [arg], [nodes.ARG_POS], context, arg_messages=local_errors) - def check_op(self, method: str, base_type: Type, arg: Node, + def check_op(self, method: str, base_type: Type, arg: Expression, context: Context, allow_reverse: bool = False) -> Tuple[Type, Type]: """Type check a binary operation which maps to a method call. @@ -1098,10 +1099,7 @@ def check_op(self, method: str, base_type: Type, arg: Node, # If the right operand has type Any, we can't make any # conjectures about the type of the result, since the # operand could have a __r method that returns anything. - - # However, in weak mode, we do make conjectures. - if not self.chk.typing_mode_weak(): - result = AnyType(), result[1] + result = AnyType(), result[1] success = not local_errors.is_errors() else: result = AnyType(), AnyType() @@ -1184,14 +1182,12 @@ def check_boolean_op(self, e: OpExpr, context: Context) -> Type: if e.op == 'and': right_map, left_map = \ - mypy.checker.find_isinstance_check(e.left, self.chk.type_map, - self.chk.typing_mode_weak()) + mypy.checker.find_isinstance_check(e.left, self.chk.type_map) restricted_left_type = false_only(left_type) result_is_left = not left_type.can_be_true elif e.op == 'or': left_map, right_map = \ - mypy.checker.find_isinstance_check(e.left, self.chk.type_map, - self.chk.typing_mode_weak()) + mypy.checker.find_isinstance_check(e.left, self.chk.type_map) restricted_left_type = true_only(left_type) result_is_left = not left_type.can_be_false @@ -1277,9 +1273,9 @@ def visit_index_expr_helper(self, e: IndexExpr) -> Type: # It's actually a type application. return self.accept(e.analyzed) left_type = self.accept(e.base) - if isinstance(left_type, TupleType) and self.chk.typing_mode_full(): + if isinstance(left_type, TupleType) and self.chk.in_checked_function(): # Special case for tuples. They support indexing only by integer - # literals. (Except in weak type checking mode.) + # literals. index = e.index if isinstance(index, SliceExpr): return self.visit_tuple_slice_helper(left_type, index) @@ -1339,7 +1335,7 @@ def visit_tuple_slice_helper(self, left_type: TupleType, slic: SliceExpr) -> Typ return left_type.slice(begin, stride, end) - def _get_value(self, index: Node) -> Optional[int]: + def _get_value(self, index: Expression) -> Optional[int]: if isinstance(index, IntExpr): return index.value elif isinstance(index, UnaryExpr): @@ -1386,7 +1382,7 @@ def visit_list_expr(self, e: ListExpr) -> Type: def visit_set_expr(self, e: SetExpr) -> Type: return self.check_lst_expr(e.items, 'builtins.set', '', e) - def check_lst_expr(self, items: List[Node], fullname: str, + def check_lst_expr(self, items: List[Expression], fullname: str, tag: str, context: Context) -> Type: # Translate into type checking a generic function call. # Used for list and set expressions, as well as for tuples @@ -1475,8 +1471,8 @@ def visit_dict_expr(self, e: DictExpr) -> Type: Translate it into a call to dict(), with provisions for **expr. """ # Collect function arguments, watching out for **expr. - args = [] # type: List[Node] # Regular "key: value" - stargs = [] # type: List[Node] # For "**expr" + args = [] # type: List[Expression] # Regular "key: value" + stargs = [] # type: List[Expression] # For "**expr" for key, value in e.items: if key is None: stargs.append(value) @@ -1605,7 +1601,7 @@ def analyze_super(self, e: SuperExpr, is_lvalue: bool) -> Type: # There's an undefined base class, and we're # at the end of the chain. That's not an error. return AnyType() - if not self.chk.typing_mode_full(): + if not self.chk.in_checked_function(): return AnyType() return analyze_member_access(e.name, self_type(e.info), e, is_lvalue, True, False, @@ -1693,10 +1689,7 @@ def check_for_comp(self, e: Union[GeneratorExpr, DictionaryComprehension]) -> No self.accept(condition) # values are only part of the comprehension when all conditions are true - true_map, _ = mypy.checker.find_isinstance_check( - condition, self.chk.type_map, - self.chk.typing_mode_weak() - ) + true_map, _ = mypy.checker.find_isinstance_check(condition, self.chk.type_map) if true_map: for var, type in true_map.items(): @@ -1709,10 +1702,7 @@ def visit_conditional_expr(self, e: ConditionalExpr) -> Type: # Gain type information from isinstance if it is there # but only for the current expression - if_map, else_map = mypy.checker.find_isinstance_check( - e.cond, - self.chk.type_map, - self.chk.typing_mode_weak()) + if_map, else_map = mypy.checker.find_isinstance_check(e.cond, self.chk.type_map) if_type = self.analyze_cond_branch(if_map, e.if_expr, context=ctx) diff --git a/mypy/checkstrformat.py b/mypy/checkstrformat.py index 01a29d17b238..4f0c63c15025 100644 --- a/mypy/checkstrformat.py +++ b/mypy/checkstrformat.py @@ -2,13 +2,13 @@ import re -from typing import cast, List, Tuple, Dict, Callable +from typing import cast, List, Tuple, Dict, Callable, Union from mypy.types import ( Type, AnyType, TupleType, Instance, UnionType ) from mypy.nodes import ( - Node, StrExpr, BytesExpr, TupleExpr, DictExpr, Context + Node, StrExpr, BytesExpr, UnicodeExpr, TupleExpr, DictExpr, Context ) if False: # break import cycle only needed for mypy @@ -55,7 +55,12 @@ def __init__(self, self.exprchk = exprchk self.msg = msg - def check_str_interpolation(self, str: StrExpr, replacements: Node) -> Type: + # TODO: In Python 3, the bytes formatting has a more restricted set of options + # compared to string formatting. + # TODO: Bytes formatting in Python 3 is only supported in 3.5 and up. + def check_str_interpolation(self, + str: Union[StrExpr, BytesExpr, UnicodeExpr], + replacements: Node) -> Type: """Check the types of the 'replacements' in a string interpolation expression: str % replacements """ @@ -67,7 +72,15 @@ def check_str_interpolation(self, str: StrExpr, replacements: Node) -> Type: self.check_mapping_str_interpolation(specifiers, replacements) else: self.check_simple_str_interpolation(specifiers, replacements) - return self.named_type('builtins.str') + + if isinstance(str, BytesExpr): + return self.named_type('builtins.bytes') + elif isinstance(str, UnicodeExpr): + return self.named_type('builtins.unicode') + elif isinstance(str, StrExpr): + return self.named_type('builtins.str') + else: + assert False def parse_conversion_specifiers(self, format: str) -> List[ConversionSpecifier]: key_regex = r'(\(([^()]*)\))?' # (optional) parenthesised sequence of characters diff --git a/mypy/errors.py b/mypy/errors.py index f9da1db9e947..541e4ca61cd2 100644 --- a/mypy/errors.py +++ b/mypy/errors.py @@ -6,6 +6,8 @@ from typing import Tuple, List, TypeVar, Set, Dict, Optional +from mypy.options import Options + T = TypeVar('T') @@ -29,6 +31,9 @@ class ErrorInfo: # The line number related to this error within file. line = 0 # -1 if unknown + # The column number related to this error with file. + column = 0 # -1 if unknown + # Either 'error' or 'note'. severity = '' @@ -42,13 +47,14 @@ class ErrorInfo: only_once = False def __init__(self, import_ctx: List[Tuple[str, int]], file: str, typ: str, - function_or_member: str, line: int, severity: str, message: str, - blocker: bool, only_once: bool) -> None: + function_or_member: str, line: int, column: int, severity: str, + message: str, blocker: bool, only_once: bool) -> None: self.import_ctx = import_ctx self.file = file self.type = typ self.function_or_member = function_or_member self.line = line + self.column = column self.severity = severity self.message = message self.blocker = blocker @@ -90,9 +96,13 @@ class Errors: only_once_messages = None # type: Set[str] # Set to True to suppress "In function "foo":" messages. - suppress_error_context = False # type: bool + hide_error_context = False # type: bool + + # Set to True to show column numbers in error messages + show_column_numbers = False # type: bool - def __init__(self, suppress_error_context: bool = False) -> None: + def __init__(self, hide_error_context: bool = False, + show_column_numbers: bool = False) -> None: self.error_info = [] self.import_ctx = [] self.type_name = [None] @@ -100,10 +110,11 @@ def __init__(self, suppress_error_context: bool = False) -> None: self.ignored_lines = OrderedDict() self.used_ignored_lines = defaultdict(set) self.only_once_messages = set() - self.suppress_error_context = suppress_error_context + self.hide_error_context = hide_error_context + self.show_column_numbers = show_column_numbers def copy(self) -> 'Errors': - new = Errors(self.suppress_error_context) + new = Errors(self.hide_error_context, self.show_column_numbers) new.file = self.file new.import_ctx = self.import_ctx[:] new.type_name = self.type_name[:] @@ -169,7 +180,7 @@ def set_import_context(self, ctx: List[Tuple[str, int]]) -> None: """Replace the entire import context with a new value.""" self.import_ctx = ctx[:] - def report(self, line: int, message: str, blocker: bool = False, + def report(self, line: int, column: int, message: str, blocker: bool = False, severity: str = 'error', file: str = None, only_once: bool = False) -> None: """Report message at the given line using the current error context. @@ -187,7 +198,7 @@ def report(self, line: int, message: str, blocker: bool = False, if file is None: file = self.file info = ErrorInfo(self.import_context(), file, type, - self.function_or_member[-1], line, severity, message, + self.function_or_member[-1], line, column, severity, message, blocker, only_once) self.add_error_info(info) @@ -210,7 +221,7 @@ def generate_unused_ignore_notes(self) -> None: for line in ignored_lines - self.used_ignored_lines[file]: # Don't use report since add_error_info will ignore the error! info = ErrorInfo(self.import_context(), file, None, None, - line, 'note', "unused 'type: ignore' comment", + line, -1, 'note', "unused 'type: ignore' comment", False, False) self.error_info.append(info) @@ -245,10 +256,13 @@ def messages(self) -> List[str]: a = [] # type: List[str] errors = self.render_messages(self.sort_messages(self.error_info)) errors = self.remove_duplicates(errors) - for file, line, severity, message in errors: + for file, line, column, severity, message in errors: s = '' if file is not None: - if line is not None and line >= 0: + if self.show_column_numbers and line is not None and line >= 0 \ + and column is not None and column >= 0: + srcloc = '{}:{}:{}'.format(file, line, column) + elif line is not None and line >= 0: srcloc = '{}:{}'.format(file, line) else: srcloc = file @@ -258,16 +272,17 @@ def messages(self) -> List[str]: a.append(s) return a - def render_messages(self, errors: List[ErrorInfo]) -> List[Tuple[str, int, + def render_messages(self, errors: List[ErrorInfo]) -> List[Tuple[str, int, int, str, str]]: """Translate the messages into a sequence of tuples. - Each tuple is of form (path, line, message. The rendered + Each tuple is of form (path, line, col, message. The rendered sequence includes information about error contexts. The path item may be None. If the line item is negative, the line number is not defined for the tuple. """ - result = [] # type: List[Tuple[str, int, str, str]] # (path, line, severity, message) + result = [] # type: List[Tuple[str, int, int, str, str]] + # (path, line, column, severity, message) prev_import_context = [] # type: List[Tuple[str, int]] prev_function_or_member = None # type: str @@ -290,39 +305,39 @@ def render_messages(self, errors: List[ErrorInfo]) -> List[Tuple[str, int, # Remove prefix to ignore from path (if present) to # simplify path. path = remove_path_prefix(path, self.ignore_prefix) - result.append((None, -1, 'note', fmt.format(path, line))) + result.append((None, -1, -1, 'note', fmt.format(path, line))) i -= 1 file = self.simplify_path(e.file) # Report context within a source file. - if self.suppress_error_context: + if self.hide_error_context: pass elif (e.function_or_member != prev_function_or_member or e.type != prev_type): if e.function_or_member is None: if e.type is None: - result.append((file, -1, 'note', 'At top level:')) + result.append((file, -1, -1, 'note', 'At top level:')) else: - result.append((file, -1, 'note', 'In class "{}":'.format( + result.append((file, -1, -1, 'note', 'In class "{}":'.format( e.type))) else: if e.type is None: - result.append((file, -1, 'note', + result.append((file, -1, -1, 'note', 'In function "{}":'.format( e.function_or_member))) else: - result.append((file, -1, 'note', + result.append((file, -1, -1, 'note', 'In member "{}" of class "{}":'.format( e.function_or_member, e.type))) elif e.type != prev_type: if e.type is None: - result.append((file, -1, 'note', 'At top level:')) + result.append((file, -1, -1, 'note', 'At top level:')) else: - result.append((file, -1, 'note', + result.append((file, -1, -1, 'note', 'In class "{}":'.format(e.type))) - result.append((file, e.line, e.severity, e.message)) + result.append((file, e.line, e.column, e.severity, e.message)) prev_import_context = e.import_ctx prev_function_or_member = e.function_or_member @@ -348,22 +363,23 @@ def sort_messages(self, errors: List[ErrorInfo]) -> List[ErrorInfo]: i += 1 i += 1 - # Sort the errors specific to a file according to line number. - a = sorted(errors[i0:i], key=lambda x: x.line) + # Sort the errors specific to a file according to line number and column. + a = sorted(errors[i0:i], key=lambda x: (x.line, x.column)) result.extend(a) return result - def remove_duplicates(self, errors: List[Tuple[str, int, str, str]] - ) -> List[Tuple[str, int, str, str]]: + def remove_duplicates(self, errors: List[Tuple[str, int, int, str, str]] + ) -> List[Tuple[str, int, int, str, str]]: """Remove duplicates from a sorted error list.""" - res = [] # type: List[Tuple[str, int, str, str]] + res = [] # type: List[Tuple[str, int, int, str, str]] i = 0 while i < len(errors): dup = False j = i - 1 while (j >= 0 and errors[j][0] == errors[i][0] and errors[j][1] == errors[i][1]): - if errors[j] == errors[i]: + if (errors[j][3] == errors[i][3] and + errors[j][4] == errors[i][4]): # ignore column dup = True break j -= 1 @@ -406,24 +422,8 @@ def remove_path_prefix(path: str, prefix: str) -> str: return path -# Corresponds to command-line flag --pdb. -drop_into_pdb = False - -# Corresponds to command-line flag --show-traceback. -show_tb = False - - -def set_drop_into_pdb(flag: bool) -> None: - global drop_into_pdb - drop_into_pdb = flag - - -def set_show_tb(flag: bool) -> None: - global show_tb - show_tb = flag - - -def report_internal_error(err: Exception, file: str, line: int, errors: Errors) -> None: +def report_internal_error(err: Exception, file: str, line: int, + errors: Errors, options: Options) -> None: """Report internal error and exit. This optionally starts pdb or shows a traceback. @@ -448,14 +448,14 @@ def report_internal_error(err: Exception, file: str, line: int, errors: Errors) file=sys.stderr) # If requested, drop into pdb. This overrides show_tb. - if drop_into_pdb: + if options.pdb: print('Dropping into pdb', file=sys.stderr) import pdb pdb.post_mortem(sys.exc_info()[2]) # If requested, print traceback, else print note explaining how to get one. - if not show_tb: - if not drop_into_pdb: + if not options.show_traceback: + if not options.pdb: print('{}: note: please use --show-traceback to print a traceback ' 'when reporting a bug'.format(prefix), file=sys.stderr) diff --git a/mypy/expandtype.py b/mypy/expandtype.py index 3c608c4e1ad7..c299163b747b 100644 --- a/mypy/expandtype.py +++ b/mypy/expandtype.py @@ -66,14 +66,15 @@ def visit_erased_type(self, t: ErasedType) -> Type: def visit_instance(self, t: Instance) -> Type: args = self.expand_types(t.args) - return Instance(t.type, args, t.line) + return Instance(t.type, args, t.line, t.column) def visit_type_var(self, t: TypeVarType) -> Type: repl = self.variables.get(t.id, t) if isinstance(repl, Instance): inst = repl # Return copy of instance with type erasure flag on. - return Instance(inst.type, inst.args, inst.line, True) + return Instance(inst.type, inst.args, line=inst.line, + column=inst.column, erased=True) else: return repl @@ -93,7 +94,7 @@ def visit_tuple_type(self, t: TupleType) -> Type: def visit_union_type(self, t: UnionType) -> Type: # After substituting for type variables in t.items, # some of the resulting types might be subtypes of others. - return UnionType.make_simplified_union(self.expand_types(t.items), t.line) + return UnionType.make_simplified_union(self.expand_types(t.items), t.line, t.column) def visit_partial_type(self, t: PartialType) -> Type: return t diff --git a/mypy/exprtotype.py b/mypy/exprtotype.py index dc95f3d4ba02..764c716b1f96 100644 --- a/mypy/exprtotype.py +++ b/mypy/exprtotype.py @@ -1,7 +1,8 @@ -"""Translate an expression (Node) to a Type value.""" +"""Translate an Expression to a Type value.""" from mypy.nodes import ( - Node, NameExpr, MemberExpr, IndexExpr, TupleExpr, ListExpr, StrExpr, BytesExpr, EllipsisExpr + Expression, NameExpr, MemberExpr, IndexExpr, TupleExpr, + ListExpr, StrExpr, BytesExpr, EllipsisExpr ) from mypy.parsetype import parse_str_as_type, TypeParseError from mypy.types import Type, UnboundType, TypeList, EllipsisType @@ -11,7 +12,7 @@ class TypeTranslationError(Exception): """Exception raised when an expression is not valid as a type.""" -def expr_to_unanalyzed_type(expr: Node) -> Type: +def expr_to_unanalyzed_type(expr: Expression) -> Type: """Translate an expression to the corresponding type. The result is not semantically analyzed. It can be UnboundType or TypeList. diff --git a/mypy/fastparse.py b/mypy/fastparse.py index 7ad71be62462..85d7ac856672 100644 --- a/mypy/fastparse.py +++ b/mypy/fastparse.py @@ -14,11 +14,11 @@ UnaryExpr, FuncExpr, ComparisonExpr, StarExpr, YieldFromExpr, NonlocalDecl, DictionaryComprehension, SetComprehension, ComplexExpr, EllipsisExpr, YieldExpr, Argument, - AwaitExpr, + AwaitExpr, TempNode, Expression, Statement, ARG_POS, ARG_OPT, ARG_STAR, ARG_NAMED, ARG_STAR2 ) from mypy.types import ( - Type, CallableType, FunctionLike, AnyType, UnboundType, TupleType, TypeList, EllipsisType, + Type, CallableType, AnyType, UnboundType, TupleType, TypeList, EllipsisType, ) from mypy import defaults from mypy import experiments @@ -70,22 +70,18 @@ def parse(source: Union[str, bytes], fnam: str = None, errors: Errors = None, except (SyntaxError, TypeCommentParseError) as e: if errors: errors.set_file('' if fnam is None else fnam) - errors.report(e.lineno, e.msg) + errors.report(e.lineno, e.offset, e.msg) else: raise - return MypyFile([], - [], - False, - set(), - weak_opts=set()) + return MypyFile([], [], False, set()) def parse_type_comment(type_comment: str, line: int) -> Type: try: typ = ast35.parse(type_comment, '', 'eval') - except SyntaxError: - raise TypeCommentParseError(TYPE_COMMENT_SYNTAX_ERROR, line) + except SyntaxError as e: + raise TypeCommentParseError(TYPE_COMMENT_SYNTAX_ERROR, line, e.offset) else: assert isinstance(typ, ast35.Expression) return TypeConverter(line=line).visit(typ.body) @@ -95,7 +91,7 @@ def with_line(f: Callable[['ASTConverter', T], U]) -> Callable[['ASTConverter', @wraps(f) def wrapper(self: 'ASTConverter', ast: T) -> U: node = f(self, ast) - node.set_line(ast.lineno) + node.set_line(ast.lineno, ast.col_offset) return node return wrapper @@ -125,8 +121,21 @@ def generic_visit(self, node: ast35.AST) -> None: def visit_NoneType(self, n: Any) -> Optional[Node]: return None - def visit_list(self, l: Sequence[ast35.AST]) -> List[Node]: - return [self.visit(e) for e in l] + def translate_expr_list(self, l: Sequence[ast35.AST]) -> List[Expression]: + res = [] # type: List[Expression] + for e in l: + exp = self.visit(e) + assert exp is None or isinstance(exp, Expression) + res.append(exp) + return res + + def translate_stmt_list(self, l: Sequence[ast35.AST]) -> List[Statement]: + res = [] # type: List[Statement] + for e in l: + stmt = self.visit(e) + assert stmt is None or isinstance(stmt, Statement) + res.append(stmt) + return res op_map = { ast35.Add: '+', @@ -176,12 +185,12 @@ def from_comp_operator(self, op: ast35.cmpop) -> str: def as_block(self, stmts: List[ast35.stmt], lineno: int) -> Block: b = None if stmts: - b = Block(self.fix_function_overloads(self.visit_list(stmts))) + b = Block(self.fix_function_overloads(self.translate_stmt_list(stmts))) b.set_line(lineno) return b - def fix_function_overloads(self, stmts: List[Node]) -> List[Node]: - ret = [] # type: List[Node] + def fix_function_overloads(self, stmts: List[Statement]) -> List[Statement]: + ret = [] # type: List[Statement] current_overload = [] current_overload_name = None # mypy doesn't actually check that the decorator is literally @overload @@ -224,14 +233,14 @@ def translate_module_id(self, id: str) -> str: return 'builtins' return id - def visit_Module(self, mod: ast35.Module) -> Node: - body = self.fix_function_overloads(self.visit_list(mod.body)) + def visit_Module(self, mod: ast35.Module) -> MypyFile: + body = self.fix_function_overloads(self.translate_stmt_list(mod.body)) return MypyFile(body, self.imports, False, {ti.lineno for ti in mod.type_ignores}, - weak_opts=set()) + ) # --- stmt --- # FunctionDef(identifier name, arguments args, @@ -239,17 +248,17 @@ def visit_Module(self, mod: ast35.Module) -> Node: # arguments = (arg* args, arg? vararg, arg* kwonlyargs, expr* kw_defaults, # arg? kwarg, expr* defaults) @with_line - def visit_FunctionDef(self, n: ast35.FunctionDef) -> Node: + def visit_FunctionDef(self, n: ast35.FunctionDef) -> Union[FuncDef, Decorator]: return self.do_func_def(n) # AsyncFunctionDef(identifier name, arguments args, # stmt* body, expr* decorator_list, expr? returns, string? type_comment) @with_line - def visit_AsyncFunctionDef(self, n: ast35.AsyncFunctionDef) -> Node: + def visit_AsyncFunctionDef(self, n: ast35.AsyncFunctionDef) -> Union[FuncDef, Decorator]: return self.do_func_def(n, is_coroutine=True) def do_func_def(self, n: Union[ast35.FunctionDef, ast35.AsyncFunctionDef], - is_coroutine: bool = False) -> Node: + is_coroutine: bool = False) -> Union[FuncDef, Decorator]: """Helper shared between visit_FunctionDef and visit_AsyncFunctionDef.""" args = self.transform_args(n.args, n.lineno) @@ -260,7 +269,7 @@ def do_func_def(self, n: Union[ast35.FunctionDef, ast35.AsyncFunctionDef], try: func_type_ast = ast35.parse(n.type_comment, '', 'func_type') except SyntaxError: - raise TypeCommentParseError(TYPE_COMMENT_SYNTAX_ERROR, n.lineno) + raise TypeCommentParseError(TYPE_COMMENT_SYNTAX_ERROR, n.lineno, n.col_offset) assert isinstance(func_type_ast, ast35.FunctionType) # for ellipsis arg if (len(func_type_ast.argtypes) == 1 and @@ -268,8 +277,10 @@ def do_func_def(self, n: Union[ast35.FunctionDef, ast35.AsyncFunctionDef], arg_types = [a.type_annotation if a.type_annotation is not None else AnyType() for a in args] else: - arg_types = [a if a is not None else AnyType() for - a in TypeConverter(line=n.lineno).visit_list(func_type_ast.argtypes)] + translated_args = (TypeConverter(line=n.lineno) + .translate_expr_list(func_type_ast.argtypes)) + arg_types = [a if a is not None else AnyType() + for a in translated_args] return_type = TypeConverter(line=n.lineno).visit(func_type_ast.returns) # add implicit self type @@ -312,11 +323,11 @@ def do_func_def(self, n: Union[ast35.FunctionDef, ast35.AsyncFunctionDef], func_def.is_decorated = True func_def.set_line(n.lineno + len(n.decorator_list)) func_def.body.set_line(func_def.get_line()) - return Decorator(func_def, self.visit_list(n.decorator_list), var) + return Decorator(func_def, self.translate_expr_list(n.decorator_list), var) else: return func_def - def set_type_optional(self, type: Type, initializer: Node) -> None: + def set_type_optional(self, type: Type, initializer: Expression) -> None: if not experiments.STRICT_OPTIONAL: return # Indicate that type should be wrapped in an Optional if arg is initialized to None. @@ -372,7 +383,7 @@ def stringify_name(self, n: ast35.AST) -> str: # stmt* body, # expr* decorator_list) @with_line - def visit_ClassDef(self, n: ast35.ClassDef) -> Node: + def visit_ClassDef(self, n: ast35.ClassDef) -> ClassDef: self.class_nesting += 1 metaclass_arg = find(lambda x: x.arg == 'metaclass', n.keywords) metaclass = None @@ -382,48 +393,63 @@ def visit_ClassDef(self, n: ast35.ClassDef) -> Node: cdef = ClassDef(n.name, self.as_block(n.body, n.lineno), None, - self.visit_list(n.bases), + self.translate_expr_list(n.bases), metaclass=metaclass) - cdef.decorators = self.visit_list(n.decorator_list) + cdef.decorators = self.translate_expr_list(n.decorator_list) self.class_nesting -= 1 return cdef # Return(expr? value) @with_line - def visit_Return(self, n: ast35.Return) -> Node: + def visit_Return(self, n: ast35.Return) -> ReturnStmt: return ReturnStmt(self.visit(n.value)) # Delete(expr* targets) @with_line - def visit_Delete(self, n: ast35.Delete) -> Node: + def visit_Delete(self, n: ast35.Delete) -> DelStmt: if len(n.targets) > 1: - tup = TupleExpr(self.visit_list(n.targets)) + tup = TupleExpr(self.translate_expr_list(n.targets)) tup.set_line(n.lineno) return DelStmt(tup) else: return DelStmt(self.visit(n.targets[0])) - # Assign(expr* targets, expr value, string? type_comment) + # Assign(expr* targets, expr? value, string? type_comment, expr? annotation) @with_line - def visit_Assign(self, n: ast35.Assign) -> Node: + def visit_Assign(self, n: ast35.Assign) -> AssignmentStmt: typ = None - if n.type_comment: + if hasattr(n, 'annotation') and n.annotation is not None: # type: ignore + new_syntax = True + else: + new_syntax = False + if new_syntax and self.pyversion < (3, 6): + raise TypeCommentParseError('Variable annotation syntax is only ' + 'suppoted in Python 3.6, use type ' + 'comment instead', n.lineno, n.col_offset) + # typed_ast prevents having both type_comment and annotation. + if n.type_comment is not None: typ = parse_type_comment(n.type_comment, n.lineno) - - return AssignmentStmt(self.visit_list(n.targets), - self.visit(n.value), - type=typ) + elif new_syntax: + typ = TypeConverter(line=n.lineno).visit(n.annotation) # type: ignore + if n.value is None: # always allow 'x: int' + rvalue = TempNode(AnyType()) # type: Expression + else: + rvalue = self.visit(n.value) + lvalues = self.translate_expr_list(n.targets) + return AssignmentStmt(lvalues, + rvalue, + type=typ, new_syntax=new_syntax) # AugAssign(expr target, operator op, expr value) @with_line - def visit_AugAssign(self, n: ast35.AugAssign) -> Node: + def visit_AugAssign(self, n: ast35.AugAssign) -> OperatorAssignmentStmt: return OperatorAssignmentStmt(self.from_operator(n.op), self.visit(n.target), self.visit(n.value)) # For(expr target, expr iter, stmt* body, stmt* orelse, string? type_comment) @with_line - def visit_For(self, n: ast35.For) -> Node: + def visit_For(self, n: ast35.For) -> ForStmt: return ForStmt(self.visit(n.target), self.visit(n.iter), self.as_block(n.body, n.lineno), @@ -431,7 +457,7 @@ def visit_For(self, n: ast35.For) -> Node: # AsyncFor(expr target, expr iter, stmt* body, stmt* orelse) @with_line - def visit_AsyncFor(self, n: ast35.AsyncFor) -> Node: + def visit_AsyncFor(self, n: ast35.AsyncFor) -> ForStmt: r = ForStmt(self.visit(n.target), self.visit(n.iter), self.as_block(n.body, n.lineno), @@ -441,28 +467,28 @@ def visit_AsyncFor(self, n: ast35.AsyncFor) -> Node: # While(expr test, stmt* body, stmt* orelse) @with_line - def visit_While(self, n: ast35.While) -> Node: + def visit_While(self, n: ast35.While) -> WhileStmt: return WhileStmt(self.visit(n.test), self.as_block(n.body, n.lineno), self.as_block(n.orelse, n.lineno)) # If(expr test, stmt* body, stmt* orelse) @with_line - def visit_If(self, n: ast35.If) -> Node: + def visit_If(self, n: ast35.If) -> IfStmt: return IfStmt([self.visit(n.test)], [self.as_block(n.body, n.lineno)], self.as_block(n.orelse, n.lineno)) # With(withitem* items, stmt* body, string? type_comment) @with_line - def visit_With(self, n: ast35.With) -> Node: + def visit_With(self, n: ast35.With) -> WithStmt: return WithStmt([self.visit(i.context_expr) for i in n.items], [self.visit(i.optional_vars) for i in n.items], self.as_block(n.body, n.lineno)) # AsyncWith(withitem* items, stmt* body) @with_line - def visit_AsyncWith(self, n: ast35.AsyncWith) -> Node: + def visit_AsyncWith(self, n: ast35.AsyncWith) -> WithStmt: r = WithStmt([self.visit(i.context_expr) for i in n.items], [self.visit(i.optional_vars) for i in n.items], self.as_block(n.body, n.lineno)) @@ -471,12 +497,12 @@ def visit_AsyncWith(self, n: ast35.AsyncWith) -> Node: # Raise(expr? exc, expr? cause) @with_line - def visit_Raise(self, n: ast35.Raise) -> Node: + def visit_Raise(self, n: ast35.Raise) -> RaiseStmt: return RaiseStmt(self.visit(n.exc), self.visit(n.cause)) # Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody) @with_line - def visit_Try(self, n: ast35.Try) -> Node: + def visit_Try(self, n: ast35.Try) -> TryStmt: vs = [NameExpr(h.name) if h.name is not None else None for h in n.handlers] types = [self.visit(h.type) for h in n.handlers] handlers = [self.as_block(h.body, h.lineno) for h in n.handlers] @@ -490,19 +516,29 @@ def visit_Try(self, n: ast35.Try) -> Node: # Assert(expr test, expr? msg) @with_line - def visit_Assert(self, n: ast35.Assert) -> Node: + def visit_Assert(self, n: ast35.Assert) -> AssertStmt: return AssertStmt(self.visit(n.test)) # Import(alias* names) @with_line - def visit_Import(self, n: ast35.Import) -> Node: - i = Import([(self.translate_module_id(a.name), a.asname) for a in n.names]) + def visit_Import(self, n: ast35.Import) -> Import: + names = [] # type: List[Tuple[str, str]] + for alias in n.names: + name = self.translate_module_id(alias.name) + asname = alias.asname + if asname is None and name != alias.name: + # if the module name has been translated (and it's not already + # an explicit import-as), make it an implicit import-as the + # original name + asname = alias.name + names.append((name, asname)) + i = Import(names) self.imports.append(i) return i # ImportFrom(identifier? module, alias* names, int? level) @with_line - def visit_ImportFrom(self, n: ast35.ImportFrom) -> Node: + def visit_ImportFrom(self, n: ast35.ImportFrom) -> ImportBase: i = None # type: ImportBase if len(n.names) == 1 and n.names[0].name == '*': i = ImportAll(n.module, n.level) @@ -515,39 +551,39 @@ def visit_ImportFrom(self, n: ast35.ImportFrom) -> Node: # Global(identifier* names) @with_line - def visit_Global(self, n: ast35.Global) -> Node: + def visit_Global(self, n: ast35.Global) -> GlobalDecl: return GlobalDecl(n.names) # Nonlocal(identifier* names) @with_line - def visit_Nonlocal(self, n: ast35.Nonlocal) -> Node: + def visit_Nonlocal(self, n: ast35.Nonlocal) -> NonlocalDecl: return NonlocalDecl(n.names) # Expr(expr value) @with_line - def visit_Expr(self, n: ast35.Expr) -> Node: + def visit_Expr(self, n: ast35.Expr) -> ExpressionStmt: value = self.visit(n.value) return ExpressionStmt(value) # Pass @with_line - def visit_Pass(self, n: ast35.Pass) -> Node: + def visit_Pass(self, n: ast35.Pass) -> PassStmt: return PassStmt() # Break @with_line - def visit_Break(self, n: ast35.Break) -> Node: + def visit_Break(self, n: ast35.Break) -> BreakStmt: return BreakStmt() # Continue @with_line - def visit_Continue(self, n: ast35.Continue) -> Node: + def visit_Continue(self, n: ast35.Continue) -> ContinueStmt: return ContinueStmt() # --- expr --- # BoolOp(boolop op, expr* values) @with_line - def visit_BoolOp(self, n: ast35.BoolOp) -> Node: + def visit_BoolOp(self, n: ast35.BoolOp) -> OpExpr: # mypy translates (1 and 2 and 3) as (1 and (2 and 3)) assert len(n.values) >= 2 op = None @@ -559,17 +595,17 @@ def visit_BoolOp(self, n: ast35.BoolOp) -> Node: raise RuntimeError('unknown BoolOp ' + str(type(n))) # potentially inefficient! - def group(vals: List[Node]) -> Node: + def group(vals: List[Expression]) -> OpExpr: if len(vals) == 2: return OpExpr(op, vals[0], vals[1]) else: return OpExpr(op, vals[0], group(vals[1:])) - return group(self.visit_list(n.values)) + return group(self.translate_expr_list(n.values)) # BinOp(expr left, operator op, expr right) @with_line - def visit_BinOp(self, n: ast35.BinOp) -> Node: + def visit_BinOp(self, n: ast35.BinOp) -> OpExpr: op = self.from_operator(n.op) if op is None: @@ -579,7 +615,7 @@ def visit_BinOp(self, n: ast35.BinOp) -> Node: # UnaryOp(unaryop op, expr operand) @with_line - def visit_UnaryOp(self, n: ast35.UnaryOp) -> Node: + def visit_UnaryOp(self, n: ast35.UnaryOp) -> UnaryExpr: op = None if isinstance(n.op, ast35.Invert): op = '~' @@ -597,46 +633,48 @@ def visit_UnaryOp(self, n: ast35.UnaryOp) -> Node: # Lambda(arguments args, expr body) @with_line - def visit_Lambda(self, n: ast35.Lambda) -> Node: + def visit_Lambda(self, n: ast35.Lambda) -> FuncExpr: body = ast35.Return(n.body) body.lineno = n.lineno + body.col_offset = n.col_offset return FuncExpr(self.transform_args(n.args, n.lineno), self.as_block([body], n.lineno)) # IfExp(expr test, expr body, expr orelse) @with_line - def visit_IfExp(self, n: ast35.IfExp) -> Node: + def visit_IfExp(self, n: ast35.IfExp) -> ConditionalExpr: return ConditionalExpr(self.visit(n.test), self.visit(n.body), self.visit(n.orelse)) # Dict(expr* keys, expr* values) @with_line - def visit_Dict(self, n: ast35.Dict) -> Node: - return DictExpr(list(zip(self.visit_list(n.keys), self.visit_list(n.values)))) + def visit_Dict(self, n: ast35.Dict) -> DictExpr: + return DictExpr(list(zip(self.translate_expr_list(n.keys), + self.translate_expr_list(n.values)))) # Set(expr* elts) @with_line - def visit_Set(self, n: ast35.Set) -> Node: - return SetExpr(self.visit_list(n.elts)) + def visit_Set(self, n: ast35.Set) -> SetExpr: + return SetExpr(self.translate_expr_list(n.elts)) # ListComp(expr elt, comprehension* generators) @with_line - def visit_ListComp(self, n: ast35.ListComp) -> Node: + def visit_ListComp(self, n: ast35.ListComp) -> ListComprehension: return ListComprehension(self.visit_GeneratorExp(cast(ast35.GeneratorExp, n))) # SetComp(expr elt, comprehension* generators) @with_line - def visit_SetComp(self, n: ast35.SetComp) -> Node: + def visit_SetComp(self, n: ast35.SetComp) -> SetComprehension: return SetComprehension(self.visit_GeneratorExp(cast(ast35.GeneratorExp, n))) # DictComp(expr key, expr value, comprehension* generators) @with_line - def visit_DictComp(self, n: ast35.DictComp) -> Node: + def visit_DictComp(self, n: ast35.DictComp) -> DictionaryComprehension: targets = [self.visit(c.target) for c in n.generators] iters = [self.visit(c.iter) for c in n.generators] - ifs_list = [self.visit_list(c.ifs) for c in n.generators] + ifs_list = [self.translate_expr_list(c.ifs) for c in n.generators] return DictionaryComprehension(self.visit(n.key), self.visit(n.value), targets, @@ -648,7 +686,7 @@ def visit_DictComp(self, n: ast35.DictComp) -> Node: def visit_GeneratorExp(self, n: ast35.GeneratorExp) -> GeneratorExpr: targets = [self.visit(c.target) for c in n.generators] iters = [self.visit(c.iter) for c in n.generators] - ifs_list = [self.visit_list(c.ifs) for c in n.generators] + ifs_list = [self.translate_expr_list(c.ifs) for c in n.generators] return GeneratorExpr(self.visit(n.elt), targets, iters, @@ -656,35 +694,35 @@ def visit_GeneratorExp(self, n: ast35.GeneratorExp) -> GeneratorExpr: # Await(expr value) @with_line - def visit_Await(self, n: ast35.Await) -> Node: + def visit_Await(self, n: ast35.Await) -> AwaitExpr: v = self.visit(n.value) return AwaitExpr(v) # Yield(expr? value) @with_line - def visit_Yield(self, n: ast35.Yield) -> Node: + def visit_Yield(self, n: ast35.Yield) -> YieldExpr: return YieldExpr(self.visit(n.value)) # YieldFrom(expr value) @with_line - def visit_YieldFrom(self, n: ast35.YieldFrom) -> Node: + def visit_YieldFrom(self, n: ast35.YieldFrom) -> YieldFromExpr: return YieldFromExpr(self.visit(n.value)) # Compare(expr left, cmpop* ops, expr* comparators) @with_line - def visit_Compare(self, n: ast35.Compare) -> Node: + def visit_Compare(self, n: ast35.Compare) -> ComparisonExpr: operators = [self.from_comp_operator(o) for o in n.ops] - operands = self.visit_list([n.left] + n.comparators) + operands = self.translate_expr_list([n.left] + n.comparators) return ComparisonExpr(operators, operands) # Call(expr func, expr* args, keyword* keywords) # keyword = (identifier? arg, expr value) @with_line - def visit_Call(self, n: ast35.Call) -> Node: + def visit_Call(self, n: ast35.Call) -> CallExpr: def is_star2arg(k: ast35.keyword) -> bool: return k.arg is None - arg_types = self.visit_list( + arg_types = self.translate_expr_list( [a.value if isinstance(a, ast35.Starred) else a for a in n.args] + [k.value for k in n.keywords]) arg_kinds = ([ARG_STAR if isinstance(a, ast35.Starred) else ARG_POS for a in n.args] + @@ -696,7 +734,7 @@ def is_star2arg(k: ast35.keyword) -> bool: # Num(object n) -- a number as a PyObject. @with_line - def visit_Num(self, n: ast35.Num) -> Node: + def visit_Num(self, n: ast35.Num) -> Union[IntExpr, FloatExpr, ComplexExpr]: if isinstance(n.n, int): return IntExpr(n.n) elif isinstance(n.n, float): @@ -708,7 +746,7 @@ def visit_Num(self, n: ast35.Num) -> Node: # Str(string s) @with_line - def visit_Str(self, n: ast35.Str) -> Node: + def visit_Str(self, n: ast35.Str) -> Union[UnicodeExpr, StrExpr]: if self.pyversion[0] >= 3 or self.is_stub: # Hack: assume all string literals in Python 2 stubs are normal # strs (i.e. not unicode). All stubs are parsed with the Python 3 @@ -722,7 +760,7 @@ def visit_Str(self, n: ast35.Str) -> Node: # Bytes(bytes s) @with_line - def visit_Bytes(self, n: ast35.Bytes) -> Node: + def visit_Bytes(self, n: ast35.Bytes) -> Union[BytesExpr, StrExpr]: # The following line is a bit hacky, but is the best way to maintain # compatibility with how mypy currently parses the contents of bytes literals. contents = str(n.s)[2:-1] @@ -733,17 +771,17 @@ def visit_Bytes(self, n: ast35.Bytes) -> Node: return StrExpr(contents) # NameConstant(singleton value) - def visit_NameConstant(self, n: ast35.NameConstant) -> Node: + def visit_NameConstant(self, n: ast35.NameConstant) -> NameExpr: return NameExpr(str(n.value)) # Ellipsis @with_line - def visit_Ellipsis(self, n: ast35.Ellipsis) -> Node: + def visit_Ellipsis(self, n: ast35.Ellipsis) -> EllipsisExpr: return EllipsisExpr() # Attribute(expr value, identifier attr, expr_context ctx) @with_line - def visit_Attribute(self, n: ast35.Attribute) -> Node: + def visit_Attribute(self, n: ast35.Attribute) -> Union[MemberExpr, SuperExpr]: if (isinstance(n.value, ast35.Call) and isinstance(n.value.func, ast35.Name) and n.value.func.id == 'super'): @@ -753,40 +791,40 @@ def visit_Attribute(self, n: ast35.Attribute) -> Node: # Subscript(expr value, slice slice, expr_context ctx) @with_line - def visit_Subscript(self, n: ast35.Subscript) -> Node: + def visit_Subscript(self, n: ast35.Subscript) -> IndexExpr: return IndexExpr(self.visit(n.value), self.visit(n.slice)) # Starred(expr value, expr_context ctx) @with_line - def visit_Starred(self, n: ast35.Starred) -> Node: + def visit_Starred(self, n: ast35.Starred) -> StarExpr: return StarExpr(self.visit(n.value)) # Name(identifier id, expr_context ctx) @with_line - def visit_Name(self, n: ast35.Name) -> Node: + def visit_Name(self, n: ast35.Name) -> NameExpr: return NameExpr(n.id) # List(expr* elts, expr_context ctx) @with_line - def visit_List(self, n: ast35.List) -> Node: + def visit_List(self, n: ast35.List) -> ListExpr: return ListExpr([self.visit(e) for e in n.elts]) # Tuple(expr* elts, expr_context ctx) @with_line - def visit_Tuple(self, n: ast35.Tuple) -> Node: + def visit_Tuple(self, n: ast35.Tuple) -> TupleExpr: return TupleExpr([self.visit(e) for e in n.elts]) # --- slice --- # Slice(expr? lower, expr? upper, expr? step) - def visit_Slice(self, n: ast35.Slice) -> Node: + def visit_Slice(self, n: ast35.Slice) -> SliceExpr: return SliceExpr(self.visit(n.lower), self.visit(n.upper), self.visit(n.step)) # ExtSlice(slice* dims) - def visit_ExtSlice(self, n: ast35.ExtSlice) -> Node: - return TupleExpr(self.visit_list(n.dims)) + def visit_ExtSlice(self, n: ast35.ExtSlice) -> TupleExpr: + return TupleExpr(self.translate_expr_list(n.dims)) # Index(expr value) def visit_Index(self, n: ast35.Index) -> Node: @@ -804,12 +842,13 @@ def visit_raw_str(self, s: str) -> Type: return parse_type_comment(s.strip(), line=self.line) def generic_visit(self, node: ast35.AST) -> None: - raise TypeCommentParseError(TYPE_COMMENT_AST_ERROR, self.line) + raise TypeCommentParseError(TYPE_COMMENT_AST_ERROR, self.line, + getattr(node, 'col_offset', -1)) def visit_NoneType(self, n: Any) -> Type: return None - def visit_list(self, l: Sequence[ast35.AST]) -> List[Type]: + def translate_expr_list(self, l: Sequence[ast35.AST]) -> List[Type]: return [self.visit(e) for e in l] def visit_Name(self, n: ast35.Name) -> Type: @@ -831,15 +870,18 @@ def visit_Subscript(self, n: ast35.Subscript) -> Type: assert isinstance(value, UnboundType) assert not value.args + empty_tuple_index = False if isinstance(n.slice.value, ast35.Tuple): - params = self.visit_list(n.slice.value.elts) + params = self.translate_expr_list(n.slice.value.elts) + if len(n.slice.value.elts) == 0: + empty_tuple_index = True else: params = [self.visit(n.slice.value)] - return UnboundType(value.name, params, line=self.line) + return UnboundType(value.name, params, line=self.line, empty_tuple_index=empty_tuple_index) def visit_Tuple(self, n: ast35.Tuple) -> Type: - return TupleType(self.visit_list(n.elts), None, implicit=True, line=self.line) + return TupleType(self.translate_expr_list(n.elts), None, implicit=True, line=self.line) # Attribute(expr value, identifier attr, expr_context ctx) def visit_Attribute(self, n: ast35.Attribute) -> Type: @@ -856,10 +898,11 @@ def visit_Ellipsis(self, n: ast35.Ellipsis) -> Type: # List(expr* elts, expr_context ctx) def visit_List(self, n: ast35.List) -> Type: - return TypeList(self.visit_list(n.elts), line=self.line) + return TypeList(self.translate_expr_list(n.elts), line=self.line) class TypeCommentParseError(Exception): - def __init__(self, msg: str, lineno: int) -> None: + def __init__(self, msg: str, lineno: int, offset: int) -> None: self.msg = msg self.lineno = lineno + self.offset = offset diff --git a/mypy/fastparse2.py b/mypy/fastparse2.py index 6620e9894888..76e95bb4cd55 100644 --- a/mypy/fastparse2.py +++ b/mypy/fastparse2.py @@ -27,14 +27,13 @@ TupleExpr, GeneratorExpr, ListComprehension, ListExpr, ConditionalExpr, DictExpr, SetExpr, NameExpr, IntExpr, StrExpr, BytesExpr, UnicodeExpr, FloatExpr, CallExpr, SuperExpr, MemberExpr, IndexExpr, SliceExpr, OpExpr, - UnaryExpr, FuncExpr, ComparisonExpr, - StarExpr, YieldFromExpr, NonlocalDecl, DictionaryComprehension, + UnaryExpr, FuncExpr, ComparisonExpr, DictionaryComprehension, SetComprehension, ComplexExpr, EllipsisExpr, YieldExpr, Argument, - AwaitExpr, Expression, + Expression, Statement, ARG_POS, ARG_OPT, ARG_STAR, ARG_NAMED, ARG_STAR2 ) from mypy.types import ( - Type, CallableType, FunctionLike, AnyType, UnboundType, TupleType, TypeList, EllipsisType, + Type, CallableType, AnyType, UnboundType, ) from mypy import defaults from mypy import experiments @@ -44,7 +43,6 @@ try: from typed_ast import ast27 from typed_ast import ast35 - from typed_ast import conversions except ImportError: if sys.version_info.minor > 2: print('You must install the typed_ast package before you can run mypy' @@ -89,22 +87,18 @@ def parse(source: Union[str, bytes], fnam: str = None, errors: Errors = None, except (SyntaxError, TypeCommentParseError) as e: if errors: errors.set_file('' if fnam is None else fnam) - errors.report(e.lineno, e.msg) + errors.report(e.lineno, e.offset, e.msg) else: raise - return MypyFile([], - [], - False, - set(), - weak_opts=set()) + return MypyFile([], [], False, set()) def parse_type_comment(type_comment: str, line: int) -> Type: try: typ = ast35.parse(type_comment, '', 'eval') - except SyntaxError: - raise TypeCommentParseError(TYPE_COMMENT_SYNTAX_ERROR, line) + except SyntaxError as e: + raise TypeCommentParseError(TYPE_COMMENT_SYNTAX_ERROR, line, e.offset) else: assert isinstance(typ, ast35.Expression) return TypeConverter(line=line).visit(typ.body) @@ -114,7 +108,7 @@ def with_line(f: Callable[['ASTConverter', T], U]) -> Callable[['ASTConverter', @wraps(f) def wrapper(self: 'ASTConverter', ast: T) -> U: node = f(self, ast) - node.set_line(ast.lineno) + node.set_line(ast.lineno, ast.col_offset) return node return wrapper @@ -144,8 +138,21 @@ def generic_visit(self, node: ast27.AST) -> None: def visit_NoneType(self, n: Any) -> Optional[Node]: return None - def visit_list(self, l: Sequence[ast27.AST]) -> List[Node]: - return [self.visit(e) for e in l] + def translate_expr_list(self, l: Sequence[ast27.AST]) -> List[Expression]: + res = [] # type: List[Expression] + for e in l: + exp = self.visit(e) + assert isinstance(exp, Expression) + res.append(exp) + return res + + def translate_stmt_list(self, l: Sequence[ast27.AST]) -> List[Statement]: + res = [] # type: List[Statement] + for e in l: + stmt = self.visit(e) + assert isinstance(stmt, Statement) + res.append(stmt) + return res op_map = { ast27.Add: '+', @@ -194,12 +201,12 @@ def from_comp_operator(self, op: ast27.cmpop) -> str: def as_block(self, stmts: List[ast27.stmt], lineno: int) -> Block: b = None if stmts: - b = Block(self.fix_function_overloads(self.visit_list(stmts))) + b = Block(self.fix_function_overloads(self.translate_stmt_list(stmts))) b.set_line(lineno) return b - def fix_function_overloads(self, stmts: List[Node]) -> List[Node]: - ret = [] # type: List[Node] + def fix_function_overloads(self, stmts: List[Statement]) -> List[Statement]: + ret = [] # type: List[Statement] current_overload = [] current_overload_name = None # mypy doesn't actually check that the decorator is literally @overload @@ -242,14 +249,14 @@ def translate_module_id(self, id: str) -> str: return 'builtins' return id - def visit_Module(self, mod: ast27.Module) -> Node: - body = self.fix_function_overloads(self.visit_list(mod.body)) + def visit_Module(self, mod: ast27.Module) -> MypyFile: + body = self.fix_function_overloads(self.translate_stmt_list(mod.body)) return MypyFile(body, self.imports, False, {ti.lineno for ti in mod.type_ignores}, - weak_opts=set()) + ) # --- stmt --- # FunctionDef(identifier name, arguments args, @@ -257,7 +264,7 @@ def visit_Module(self, mod: ast27.Module) -> Node: # arguments = (arg* args, arg? vararg, arg* kwonlyargs, expr* kw_defaults, # arg? kwarg, expr* defaults) @with_line - def visit_FunctionDef(self, n: ast27.FunctionDef) -> Node: + def visit_FunctionDef(self, n: ast27.FunctionDef) -> Statement: converter = TypeConverter(line=n.lineno) args = self.transform_args(n.args, n.lineno) @@ -268,7 +275,7 @@ def visit_FunctionDef(self, n: ast27.FunctionDef) -> Node: try: func_type_ast = ast35.parse(n.type_comment, '', 'func_type') except SyntaxError: - raise TypeCommentParseError(TYPE_COMMENT_SYNTAX_ERROR, n.lineno) + raise TypeCommentParseError(TYPE_COMMENT_SYNTAX_ERROR, n.lineno, n.col_offset) assert isinstance(func_type_ast, ast35.FunctionType) # for ellipsis arg if (len(func_type_ast.argtypes) == 1 and @@ -277,7 +284,7 @@ def visit_FunctionDef(self, n: ast27.FunctionDef) -> Node: for a in args] else: arg_types = [a if a is not None else AnyType() for - a in converter.visit_list(func_type_ast.argtypes)] + a in converter.translate_expr_list(func_type_ast.argtypes)] return_type = converter.visit(func_type_ast.returns) # add implicit self type @@ -317,11 +324,11 @@ def visit_FunctionDef(self, n: ast27.FunctionDef) -> Node: func_def.is_decorated = True func_def.set_line(n.lineno + len(n.decorator_list)) func_def.body.set_line(func_def.get_line()) - return Decorator(func_def, self.visit_list(n.decorator_list), var) + return Decorator(func_def, self.translate_expr_list(n.decorator_list), var) else: return func_def - def set_type_optional(self, type: Type, initializer: Node) -> None: + def set_type_optional(self, type: Type, initializer: Expression) -> None: if not experiments.STRICT_OPTIONAL: return # Indicate that type should be wrapped in an Optional if arg is initialized to None. @@ -356,7 +363,7 @@ def get_type(i: int) -> Optional[Type]: return None args = [(convert_arg(arg), get_type(i)) for i, arg in enumerate(n.args)] - defaults = self.visit_list(n.defaults) + defaults = self.translate_expr_list(n.defaults) new_args = [] # type: List[Argument] num_no_defaults = len(args) - len(defaults) @@ -393,28 +400,28 @@ def stringify_name(self, n: ast27.AST) -> str: # stmt* body, # expr* decorator_list) @with_line - def visit_ClassDef(self, n: ast27.ClassDef) -> Node: + def visit_ClassDef(self, n: ast27.ClassDef) -> ClassDef: self.class_nesting += 1 cdef = ClassDef(n.name, self.as_block(n.body, n.lineno), None, - self.visit_list(n.bases), + self.translate_expr_list(n.bases), metaclass=None) - cdef.decorators = self.visit_list(n.decorator_list) + cdef.decorators = self.translate_expr_list(n.decorator_list) self.class_nesting -= 1 return cdef # Return(expr? value) @with_line - def visit_Return(self, n: ast27.Return) -> Node: + def visit_Return(self, n: ast27.Return) -> ReturnStmt: return ReturnStmt(self.visit(n.value)) # Delete(expr* targets) @with_line - def visit_Delete(self, n: ast27.Delete) -> Node: + def visit_Delete(self, n: ast27.Delete) -> DelStmt: if len(n.targets) > 1: - tup = TupleExpr(self.visit_list(n.targets)) + tup = TupleExpr(self.translate_expr_list(n.targets)) tup.set_line(n.lineno) return DelStmt(tup) else: @@ -422,25 +429,25 @@ def visit_Delete(self, n: ast27.Delete) -> Node: # Assign(expr* targets, expr value, string? type_comment) @with_line - def visit_Assign(self, n: ast27.Assign) -> Node: + def visit_Assign(self, n: ast27.Assign) -> AssignmentStmt: typ = None if n.type_comment: typ = parse_type_comment(n.type_comment, n.lineno) - return AssignmentStmt(self.visit_list(n.targets), + return AssignmentStmt(self.translate_expr_list(n.targets), self.visit(n.value), type=typ) # AugAssign(expr target, operator op, expr value) @with_line - def visit_AugAssign(self, n: ast27.AugAssign) -> Node: + def visit_AugAssign(self, n: ast27.AugAssign) -> OperatorAssignmentStmt: return OperatorAssignmentStmt(self.from_operator(n.op), self.visit(n.target), self.visit(n.value)) # For(expr target, expr iter, stmt* body, stmt* orelse, string? type_comment) @with_line - def visit_For(self, n: ast27.For) -> Node: + def visit_For(self, n: ast27.For) -> ForStmt: return ForStmt(self.visit(n.target), self.visit(n.iter), self.as_block(n.body, n.lineno), @@ -448,27 +455,27 @@ def visit_For(self, n: ast27.For) -> Node: # While(expr test, stmt* body, stmt* orelse) @with_line - def visit_While(self, n: ast27.While) -> Node: + def visit_While(self, n: ast27.While) -> WhileStmt: return WhileStmt(self.visit(n.test), self.as_block(n.body, n.lineno), self.as_block(n.orelse, n.lineno)) # If(expr test, stmt* body, stmt* orelse) @with_line - def visit_If(self, n: ast27.If) -> Node: + def visit_If(self, n: ast27.If) -> IfStmt: return IfStmt([self.visit(n.test)], [self.as_block(n.body, n.lineno)], self.as_block(n.orelse, n.lineno)) # With(withitem* items, stmt* body, string? type_comment) @with_line - def visit_With(self, n: ast27.With) -> Node: + def visit_With(self, n: ast27.With) -> WithStmt: return WithStmt([self.visit(n.context_expr)], [self.visit(n.optional_vars)], self.as_block(n.body, n.lineno)) @with_line - def visit_Raise(self, n: ast27.Raise) -> Node: + def visit_Raise(self, n: ast27.Raise) -> RaiseStmt: e = None if n.type is not None: e = n.type @@ -484,11 +491,11 @@ def visit_Raise(self, n: ast27.Raise) -> Node: # TryExcept(stmt* body, excepthandler* handlers, stmt* orelse) @with_line - def visit_TryExcept(self, n: ast27.TryExcept) -> Node: + def visit_TryExcept(self, n: ast27.TryExcept) -> TryStmt: return self.try_handler(n.body, n.handlers, n.orelse, [], n.lineno) @with_line - def visit_TryFinally(self, n: ast27.TryFinally) -> Node: + def visit_TryFinally(self, n: ast27.TryFinally) -> TryStmt: if len(n.body) == 1 and isinstance(n.body[0], ast27.TryExcept): return self.try_handler([n.body[0]], [], [], n.finalbody, n.lineno) else: @@ -499,7 +506,7 @@ def try_handler(self, handlers: List[ast27.ExceptHandler], orelse: List[ast27.stmt], finalbody: List[ast27.stmt], - lineno: int) -> Node: + lineno: int) -> TryStmt: def produce_name(item: ast27.ExceptHandler) -> Optional[NameExpr]: if item.name is None: return None @@ -520,7 +527,7 @@ def produce_name(item: ast27.ExceptHandler) -> Optional[NameExpr]: self.as_block(finalbody, lineno)) @with_line - def visit_Print(self, n: ast27.Print) -> Node: + def visit_Print(self, n: ast27.Print) -> ExpressionStmt: keywords = [] if n.dest is not None: keywords.append(ast27.keyword("file", n.dest)) @@ -534,10 +541,10 @@ def visit_Print(self, n: ast27.Print) -> Node: ast27.Name("print", ast27.Load(), lineno=n.lineno, col_offset=-1), n.values, keywords, None, None, lineno=n.lineno, col_offset=-1) - return self.visit(ast27.Expr(call, lineno=n.lineno, col_offset=-1)) + return self.visit_Expr(ast27.Expr(call, lineno=n.lineno, col_offset=-1)) @with_line - def visit_Exec(self, n: ast27.Exec) -> Node: + def visit_Exec(self, n: ast27.Exec) -> ExpressionStmt: new_globals = n.globals new_locals = n.locals @@ -547,7 +554,7 @@ def visit_Exec(self, n: ast27.Exec) -> Node: new_locals = ast27.Name("None", ast27.Load(), lineno=-1, col_offset=-1) # TODO: Comment in visit_Print also applies here - return self.visit(ast27.Expr( + return self.visit_Expr(ast27.Expr( ast27.Call( ast27.Name("exec", ast27.Load(), lineno=n.lineno, col_offset=-1), [n.body, new_globals, new_locals], @@ -556,9 +563,9 @@ def visit_Exec(self, n: ast27.Exec) -> Node: lineno=n.lineno, col_offset=-1)) @with_line - def visit_Repr(self, n: ast27.Repr) -> Node: + def visit_Repr(self, n: ast27.Repr) -> CallExpr: # TODO: Comment in visit_Print also applies here - return self.visit(ast27.Call( + return self.visit_Call(ast27.Call( ast27.Name("repr", ast27.Load(), lineno=n.lineno, col_offset=-1), n.value, [], None, None, @@ -566,19 +573,29 @@ def visit_Repr(self, n: ast27.Repr) -> Node: # Assert(expr test, expr? msg) @with_line - def visit_Assert(self, n: ast27.Assert) -> Node: + def visit_Assert(self, n: ast27.Assert) -> AssertStmt: return AssertStmt(self.visit(n.test)) # Import(alias* names) @with_line - def visit_Import(self, n: ast27.Import) -> Node: - i = Import([(self.translate_module_id(a.name), a.asname) for a in n.names]) + def visit_Import(self, n: ast27.Import) -> Import: + names = [] # type: List[Tuple[str, str]] + for alias in n.names: + name = self.translate_module_id(alias.name) + asname = alias.asname + if asname is None and name != alias.name: + # if the module name has been translated (and it's not already + # an explicit import-as), make it an implicit import-as the + # original name + asname = alias.name + names.append((name, asname)) + i = Import(names) self.imports.append(i) return i # ImportFrom(identifier? module, alias* names, int? level) @with_line - def visit_ImportFrom(self, n: ast27.ImportFrom) -> Node: + def visit_ImportFrom(self, n: ast27.ImportFrom) -> ImportBase: i = None # type: ImportBase if len(n.names) == 1 and n.names[0].name == '*': i = ImportAll(n.module, n.level) @@ -591,34 +608,34 @@ def visit_ImportFrom(self, n: ast27.ImportFrom) -> Node: # Global(identifier* names) @with_line - def visit_Global(self, n: ast27.Global) -> Node: + def visit_Global(self, n: ast27.Global) -> GlobalDecl: return GlobalDecl(n.names) # Expr(expr value) @with_line - def visit_Expr(self, n: ast27.Expr) -> Node: + def visit_Expr(self, n: ast27.Expr) -> ExpressionStmt: value = self.visit(n.value) return ExpressionStmt(value) # Pass @with_line - def visit_Pass(self, n: ast27.Pass) -> Node: + def visit_Pass(self, n: ast27.Pass) -> PassStmt: return PassStmt() # Break @with_line - def visit_Break(self, n: ast27.Break) -> Node: + def visit_Break(self, n: ast27.Break) -> BreakStmt: return BreakStmt() # Continue @with_line - def visit_Continue(self, n: ast27.Continue) -> Node: + def visit_Continue(self, n: ast27.Continue) -> ContinueStmt: return ContinueStmt() # --- expr --- # BoolOp(boolop op, expr* values) @with_line - def visit_BoolOp(self, n: ast27.BoolOp) -> Node: + def visit_BoolOp(self, n: ast27.BoolOp) -> OpExpr: # mypy translates (1 and 2 and 3) as (1 and (2 and 3)) assert len(n.values) >= 2 op = None @@ -630,17 +647,17 @@ def visit_BoolOp(self, n: ast27.BoolOp) -> Node: raise RuntimeError('unknown BoolOp ' + str(type(n))) # potentially inefficient! - def group(vals: List[Node]) -> Node: + def group(vals: List[Expression]) -> OpExpr: if len(vals) == 2: return OpExpr(op, vals[0], vals[1]) else: return OpExpr(op, vals[0], group(vals[1:])) - return group(self.visit_list(n.values)) + return group(self.translate_expr_list(n.values)) # BinOp(expr left, operator op, expr right) @with_line - def visit_BinOp(self, n: ast27.BinOp) -> Node: + def visit_BinOp(self, n: ast27.BinOp) -> OpExpr: op = self.from_operator(n.op) if op is None: @@ -650,7 +667,7 @@ def visit_BinOp(self, n: ast27.BinOp) -> Node: # UnaryOp(unaryop op, expr operand) @with_line - def visit_UnaryOp(self, n: ast27.UnaryOp) -> Node: + def visit_UnaryOp(self, n: ast27.UnaryOp) -> UnaryExpr: op = None if isinstance(n.op, ast27.Invert): op = '~' @@ -668,46 +685,48 @@ def visit_UnaryOp(self, n: ast27.UnaryOp) -> Node: # Lambda(arguments args, expr body) @with_line - def visit_Lambda(self, n: ast27.Lambda) -> Node: + def visit_Lambda(self, n: ast27.Lambda) -> FuncExpr: body = ast27.Return(n.body) body.lineno = n.lineno + body.col_offset = n.col_offset return FuncExpr(self.transform_args(n.args, n.lineno), self.as_block([body], n.lineno)) # IfExp(expr test, expr body, expr orelse) @with_line - def visit_IfExp(self, n: ast27.IfExp) -> Node: + def visit_IfExp(self, n: ast27.IfExp) -> ConditionalExpr: return ConditionalExpr(self.visit(n.test), self.visit(n.body), self.visit(n.orelse)) # Dict(expr* keys, expr* values) @with_line - def visit_Dict(self, n: ast27.Dict) -> Node: - return DictExpr(list(zip(self.visit_list(n.keys), self.visit_list(n.values)))) + def visit_Dict(self, n: ast27.Dict) -> DictExpr: + return DictExpr(list(zip(self.translate_expr_list(n.keys), + self.translate_expr_list(n.values)))) # Set(expr* elts) @with_line - def visit_Set(self, n: ast27.Set) -> Node: - return SetExpr(self.visit_list(n.elts)) + def visit_Set(self, n: ast27.Set) -> SetExpr: + return SetExpr(self.translate_expr_list(n.elts)) # ListComp(expr elt, comprehension* generators) @with_line - def visit_ListComp(self, n: ast27.ListComp) -> Node: + def visit_ListComp(self, n: ast27.ListComp) -> ListComprehension: return ListComprehension(self.visit_GeneratorExp(cast(ast27.GeneratorExp, n))) # SetComp(expr elt, comprehension* generators) @with_line - def visit_SetComp(self, n: ast27.SetComp) -> Node: + def visit_SetComp(self, n: ast27.SetComp) -> SetComprehension: return SetComprehension(self.visit_GeneratorExp(cast(ast27.GeneratorExp, n))) # DictComp(expr key, expr value, comprehension* generators) @with_line - def visit_DictComp(self, n: ast27.DictComp) -> Node: + def visit_DictComp(self, n: ast27.DictComp) -> DictionaryComprehension: targets = [self.visit(c.target) for c in n.generators] iters = [self.visit(c.iter) for c in n.generators] - ifs_list = [self.visit_list(c.ifs) for c in n.generators] + ifs_list = [self.translate_expr_list(c.ifs) for c in n.generators] return DictionaryComprehension(self.visit(n.key), self.visit(n.value), targets, @@ -719,7 +738,7 @@ def visit_DictComp(self, n: ast27.DictComp) -> Node: def visit_GeneratorExp(self, n: ast27.GeneratorExp) -> GeneratorExpr: targets = [self.visit(c.target) for c in n.generators] iters = [self.visit(c.iter) for c in n.generators] - ifs_list = [self.visit_list(c.ifs) for c in n.generators] + ifs_list = [self.translate_expr_list(c.ifs) for c in n.generators] return GeneratorExpr(self.visit(n.elt), targets, iters, @@ -727,20 +746,20 @@ def visit_GeneratorExp(self, n: ast27.GeneratorExp) -> GeneratorExpr: # Yield(expr? value) @with_line - def visit_Yield(self, n: ast27.Yield) -> Node: + def visit_Yield(self, n: ast27.Yield) -> YieldExpr: return YieldExpr(self.visit(n.value)) # Compare(expr left, cmpop* ops, expr* comparators) @with_line - def visit_Compare(self, n: ast27.Compare) -> Node: + def visit_Compare(self, n: ast27.Compare) -> ComparisonExpr: operators = [self.from_comp_operator(o) for o in n.ops] - operands = self.visit_list([n.left] + n.comparators) + operands = self.translate_expr_list([n.left] + n.comparators) return ComparisonExpr(operators, operands) # Call(expr func, expr* args, keyword* keywords) # keyword = (identifier? arg, expr value) @with_line - def visit_Call(self, n: ast27.Call) -> Node: + def visit_Call(self, n: ast27.Call) -> CallExpr: arg_types = [] # type: List[ast27.expr] arg_kinds = [] # type: List[int] signature = [] # type: List[Optional[str]] @@ -764,16 +783,16 @@ def visit_Call(self, n: ast27.Call) -> Node: signature.append(None) return CallExpr(self.visit(n.func), - self.visit_list(arg_types), + self.translate_expr_list(arg_types), arg_kinds, cast("List[str]", signature)) # Num(object n) -- a number as a PyObject. @with_line - def visit_Num(self, new: ast27.Num) -> Node: + def visit_Num(self, new: ast27.Num) -> Expression: value = new.n is_inverse = False - if new.n < 0: + if str(new.n).startswith('-'): # Hackish because of complex. value = -new.n is_inverse = True @@ -794,7 +813,7 @@ def visit_Num(self, new: ast27.Num) -> Node: # Str(string s) @with_line - def visit_Str(self, s: ast27.Str) -> Node: + def visit_Str(self, s: ast27.Str) -> Expression: # Hack: assume all string literals in Python 2 stubs are normal # strs (i.e. not unicode). All stubs are parsed with the Python 3 # parser, which causes unprefixed string literals to be interpreted @@ -818,12 +837,12 @@ def visit_Str(self, s: ast27.Str) -> Node: return UnicodeExpr(s.s) # Ellipsis - def visit_Ellipsis(self, n: ast27.Ellipsis) -> Node: + def visit_Ellipsis(self, n: ast27.Ellipsis) -> EllipsisExpr: return EllipsisExpr() # Attribute(expr value, identifier attr, expr_context ctx) @with_line - def visit_Attribute(self, n: ast27.Attribute) -> Node: + def visit_Attribute(self, n: ast27.Attribute) -> Expression: if (isinstance(n.value, ast27.Call) and isinstance(n.value.func, ast27.Name) and n.value.func.id == 'super'): @@ -833,36 +852,36 @@ def visit_Attribute(self, n: ast27.Attribute) -> Node: # Subscript(expr value, slice slice, expr_context ctx) @with_line - def visit_Subscript(self, n: ast27.Subscript) -> Node: + def visit_Subscript(self, n: ast27.Subscript) -> IndexExpr: return IndexExpr(self.visit(n.value), self.visit(n.slice)) # Name(identifier id, expr_context ctx) @with_line - def visit_Name(self, n: ast27.Name) -> Node: + def visit_Name(self, n: ast27.Name) -> NameExpr: return NameExpr(n.id) # List(expr* elts, expr_context ctx) @with_line - def visit_List(self, n: ast27.List) -> Node: + def visit_List(self, n: ast27.List) -> ListExpr: return ListExpr([self.visit(e) for e in n.elts]) # Tuple(expr* elts, expr_context ctx) @with_line - def visit_Tuple(self, n: ast27.Tuple) -> Node: + def visit_Tuple(self, n: ast27.Tuple) -> TupleExpr: return TupleExpr([self.visit(e) for e in n.elts]) # --- slice --- # Slice(expr? lower, expr? upper, expr? step) - def visit_Slice(self, n: ast27.Slice) -> Node: + def visit_Slice(self, n: ast27.Slice) -> SliceExpr: return SliceExpr(self.visit(n.lower), self.visit(n.upper), self.visit(n.step)) # ExtSlice(slice* dims) - def visit_ExtSlice(self, n: ast27.ExtSlice) -> Node: - return TupleExpr(self.visit_list(n.dims)) + def visit_ExtSlice(self, n: ast27.ExtSlice) -> TupleExpr: + return TupleExpr(self.translate_expr_list(n.dims)) # Index(expr value) - def visit_Index(self, n: ast27.Index) -> Node: + def visit_Index(self, n: ast27.Index) -> Expression: return self.visit(n.value) diff --git a/mypy/fixup.py b/mypy/fixup.py index 5c1fa9a1277f..eec31ec3390f 100644 --- a/mypy/fixup.py +++ b/mypy/fixup.py @@ -233,9 +233,9 @@ def lookup_qualified_stnode(modules: Dict[str, MypyFile], name: str) -> SymbolTa while True: assert '.' in head, "Cannot find %s" % (name,) head, tail = head.rsplit('.', 1) + rest.append(tail) mod = modules.get(head) if mod is not None: - rest.append(tail) break names = mod.names while True: diff --git a/mypy/lex.py b/mypy/lex.py index 41f426200f1f..f074de9c09b1 100644 --- a/mypy/lex.py +++ b/mypy/lex.py @@ -27,6 +27,7 @@ def __init__(self, string: str, pre: str = '') -> None: self.string = string self.pre = pre self.line = 0 + self.column = 0 def __repr__(self) -> str: """The representation is of form 'Keyword( if)'.""" @@ -273,9 +274,10 @@ def escape_repl(m: Match[str], prefix: str) -> str: class Lexer: """Lexical analyzer.""" - i = 0 # Current string index (into s) - s = '' # The string being analyzed - line = 0 # Current line number + i = 0 # Current string index (into s) + s = '' # The string being analyzed + line = 0 # Current line number + column = 0 # Current column number pre_whitespace = '' # Whitespace and comments before the next token enc = '' # Encoding @@ -339,6 +341,7 @@ def lex(self, text: Union[str, bytes], first_line: int) -> None: """Lexically analyze a string, storing the tokens at the tok list.""" self.i = 0 self.line = first_line + self.column = 0 if isinstance(text, bytes): if text.startswith(b'\xef\xbb\xbf'): @@ -612,6 +615,7 @@ def lex_triple_quoted_str(self, re3end: Pattern[str], prefix: str) -> None: line = self.line ss = self.s[self.i:self.i + len(prefix) + 3] self.i += len(prefix) + 3 + self.column += len(prefix) + 3 while True: m = re3end.match(self.s, self.i) if m is not None: @@ -625,6 +629,7 @@ def lex_triple_quoted_str(self, re3end: Pattern[str], prefix: str) -> None: ss += s self.line += 1 self.i += len(s) + self.column += len(s) lit = None # type: Token if 'b' in prefix or 'B' in prefix: lit = BytesLit(ss + m.group(0)) @@ -642,6 +647,7 @@ def lex_multiline_string_literal(self, re_end: Pattern[str], """ line = self.line self.i += len(prefix) + self.column += len(prefix) ss = prefix while True: m = self.match(re_end) @@ -652,6 +658,7 @@ def lex_multiline_string_literal(self, re_end: Pattern[str], ss += m self.line += 1 self.i += len(m) + self.column += len(m) if not m.endswith('\n') and not m.endswith('\r'): break self.add_special_token(StrLit(ss), line, 0) # TODO bytes @@ -740,15 +747,18 @@ def lex_break(self) -> None: last_tok.string += self.pre_whitespace + s self.i += len(s) self.line += 1 + self.column = 0 self.pre_whitespace = '' if was_semicolon: self.lex_indent() elif self.ignore_break(): self.add_pre_whitespace(s) self.line += 1 + self.column = 0 else: self.add_token(Break(s)) self.line += 1 + self.column = 0 self.lex_indent() def lex_semicolon(self) -> None: @@ -828,6 +838,7 @@ def add_pre_whitespace(self, s: str) -> None: """ self.pre_whitespace += s self.i += len(s) + self.column += len(s) type_ignore_exp = re.compile(r'[ \t]*#[ \t]*type:[ \t]*ignore\b') @@ -849,8 +860,10 @@ def add_token(self, tok: Token) -> None: delta += 1 self.ignored_lines.add(self.line - delta) tok.line = self.line + tok.column = self.column self.tok.append(tok) self.i += len(tok.string) + self.column += len(tok.string) self.pre_whitespace = '' def add_special_token(self, tok: Token, line: int, skip: int) -> None: @@ -864,6 +877,7 @@ def add_special_token(self, tok: Token, line: int, skip: int) -> None: tok.line = line self.tok.append(tok) self.i += skip + self.column += skip self.pre_whitespace = '' def ignore_break(self) -> bool: diff --git a/mypy/main.py b/mypy/main.py index f84cc46096d9..62d068c586c8 100644 --- a/mypy/main.py +++ b/mypy/main.py @@ -13,7 +13,7 @@ from mypy import git from mypy import experiments from mypy.build import BuildSource, BuildResult, PYTHON_EXTENSIONS -from mypy.errors import CompileError, set_drop_into_pdb, set_show_tb +from mypy.errors import CompileError from mypy.options import Options, BuildType from mypy.report import reporter_classes @@ -33,10 +33,6 @@ def main(script_path: str) -> None: else: bin_dir = None sources, options = process_options(sys.argv[1:]) - if options.pdb: - set_drop_into_pdb(True) - if options.show_traceback: - set_show_tb(True) f = sys.stdout try: res = type_check_only(sources, bin_dir, options) @@ -166,9 +162,9 @@ def process_options(args: List[str], help="warn about casting an expression to its inferred type") parser.add_argument('--warn-unused-ignores', action='store_true', help="warn about unneeded '# type: ignore' comments") - parser.add_argument('--suppress-error-context', action='store_true', - dest='suppress_error_context', - help="Suppress context notes before errors") + parser.add_argument('--hide-error-context', action='store_true', + dest='hide_error_context', + help="Hide context notes before errors") parser.add_argument('--fast-parser', action='store_true', help="enable experimental fast parser") parser.add_argument('-i', '--incremental', action='store_true', @@ -197,6 +193,9 @@ def process_options(args: List[str], parser.add_argument('--config-file', help="Configuration file, must have a [mypy] section " "(defaults to {})".format(defaults.CONFIG_FILE)) + parser.add_argument('--show-column-numbers', action='store_true', + dest='show_column_numbers', + help="Show column numbers in error messages") # hidden options # --shadow-file a.py tmp.py will typecheck tmp.py in place of a.py. # Useful for tools to make transformations to a file to get more diff --git a/mypy/messages.py b/mypy/messages.py index 48317a7616b9..b4828f1bfe08 100644 --- a/mypy/messages.py +++ b/mypy/messages.py @@ -148,6 +148,7 @@ def report(self, msg: str, context: Context, severity: str, file: str = None) -> """Report an error or note (unless disabled).""" if self.disable_count <= 0: self.errors.report(context.get_line() if context else -1, + context.get_column() if context else -1, msg.strip(), severity=severity, file=file) def fail(self, msg: str, context: Context, file: str = None) -> None: diff --git a/mypy/nodes.py b/mypy/nodes.py index dff502e47199..469b3087806f 100644 --- a/mypy/nodes.py +++ b/mypy/nodes.py @@ -19,6 +19,9 @@ class Context: @abstractmethod def get_line(self) -> int: pass + @abstractmethod + def get_column(self) -> int: pass + if False: # break import cycle only needed for mypy @@ -92,6 +95,7 @@ class Node(Context): """Common base class for all non-type parse tree nodes.""" line = -1 + column = -1 literal = LITERAL_NO literal_hash = None # type: Any @@ -102,26 +106,42 @@ def __str__(self) -> str: return repr(self) return ans - def set_line(self, target: Union[Token, 'Node', int]) -> 'Node': + def set_line(self, target: Union[Token, 'Node', int], column: int = None) -> None: + """If target is a node or token, pull line (and column) information + into this node. If column is specified, this will override any column + information coming from a node/token. + """ if isinstance(target, int): self.line = target else: self.line = target.line - return self + self.column = target.column + + if column is not None: + self.column = column def get_line(self) -> int: # TODO this should be just 'line' return self.line + def get_column(self) -> int: + # TODO this should be just 'column' + return self.column + def accept(self, visitor: NodeVisitor[T]) -> T: raise RuntimeError('Not implemented') -# These are placeholders for a future refactoring; see #1783. -# For now they serve as (unchecked) documentation of what various -# fields of Node subtypes are expected to contain. -Statement = Node -Expression = Node +class Statement(Node): + """A statement node.""" + + +class Expression(Node): + """An expression node.""" + + +# TODO: Union['NameExpr', 'TupleExpr', 'ListExpr', 'MemberExpr', 'IndexExpr']; see #1783. +Lvalue = Expression class SymbolNode(Node): @@ -171,20 +191,16 @@ class MypyFile(SymbolNode, Statement): ignored_lines = None # type: Set[int] # Is this file represented by a stub file (.pyi)? is_stub = False - # Do weak typing globally in the file? - weak_opts = None # type: Set[str] def __init__(self, defs: List[Statement], imports: List['ImportBase'], is_bom: bool = False, - ignored_lines: Set[int] = None, - weak_opts: Set[str] = None) -> None: + ignored_lines: Set[int] = None) -> None: self.defs = defs self.line = 1 # Dummy line number self.imports = imports self.is_bom = is_bom - self.weak_opts = weak_opts if ignored_lines: self.ignored_lines = ignored_lines else: @@ -378,17 +394,17 @@ def _initialization_statement(self) -> Optional['AssignmentStmt']: assign = AssignmentStmt([lvalue], rvalue) return assign - def set_line(self, target: Union[Token, Node, int]) -> Node: - super().set_line(target) + def set_line(self, target: Union[Token, Node, int], column: int = None) -> None: + super().set_line(target, column) if self.initializer: - self.initializer.set_line(self.line) + self.initializer.set_line(self.line, self.column) - self.variable.set_line(self.line) + self.variable.set_line(self.line, self.column) if self.initialization_statement: - self.initialization_statement.set_line(self.line) - self.initialization_statement.lvalues[0].set_line(self.line) + self.initialization_statement.set_line(self.line, self.column) + self.initialization_statement.lvalues[0].set_line(self.line, self.column) def serialize(self) -> JsonDict: # Note: we are deliberately not saving the type annotation since @@ -451,11 +467,10 @@ def __init__(self, arguments: List[Argument], body: 'Block', def max_fixed_argc(self) -> int: return self.max_pos - def set_line(self, target: Union[Token, Node, int]) -> Node: - super().set_line(target) + def set_line(self, target: Union[Token, Node, int], column: int = None) -> None: + super().set_line(target, column) for arg in self.arguments: - arg.set_line(self.line) - return self + arg.set_line(self.line, self.column) def is_dynamic(self) -> bool: return self.type is None @@ -761,16 +776,19 @@ class AssignmentStmt(Statement): An lvalue can be NameExpr, TupleExpr, ListExpr, MemberExpr, IndexExpr. """ - lvalues = None # type: List[Expression] + lvalues = None # type: List[Lvalue] rvalue = None # type: Expression # Declared type in a comment, may be None. type = None # type: mypy.types.Type + # This indicates usage of PEP 526 type annotation syntax in assignment. + new_syntax = False # type: bool - def __init__(self, lvalues: List[Expression], rvalue: Expression, - type: 'mypy.types.Type' = None) -> None: + def __init__(self, lvalues: List[Lvalue], rvalue: Expression, + type: 'mypy.types.Type' = None, new_syntax: bool = False) -> None: self.lvalues = lvalues self.rvalue = rvalue self.type = type + self.new_syntax = new_syntax def accept(self, visitor: NodeVisitor[T]) -> T: return visitor.visit_assignment_stmt(self) @@ -1744,12 +1762,12 @@ def accept(self, visitor: NodeVisitor[T]) -> T: return visitor.visit_newtype_expr(self) -class AwaitExpr(Node): +class AwaitExpr(Expression): """Await expression (await ...).""" - expr = None # type: Node + expr = None # type: Expression - def __init__(self, expr: Node) -> None: + def __init__(self, expr: Expression) -> None: self.expr = expr def accept(self, visitor: NodeVisitor[T]) -> T: @@ -1772,6 +1790,9 @@ class TempNode(Expression): def __init__(self, typ: 'mypy.types.Type') -> None: self.type = typ + def __repr__(self): + return 'TempNode(%s)' % str(self.type) + def accept(self, visitor: NodeVisitor[T]) -> T: return visitor.visit_temp_node(self) diff --git a/mypy/options.py b/mypy/options.py index 0eef6d7b8d56..1d0945645a83 100644 --- a/mypy/options.py +++ b/mypy/options.py @@ -98,8 +98,9 @@ def __init__(self) -> None: self.incremental = False self.cache_dir = defaults.CACHE_DIR self.debug_cache = False - self.suppress_error_context = False # Suppress "note: In function "foo":" messages. + self.hide_error_context = False # Hide "note: In function "foo":" messages. self.shadow_file = None # type: Optional[Tuple[str, str]] + self.show_column_numbers = False # type: bool def __eq__(self, other: object) -> bool: return self.__class__ == other.__class__ and self.__dict__ == other.__dict__ diff --git a/mypy/parse.py b/mypy/parse.py index 39979d7126b8..5739056e4b21 100644 --- a/mypy/parse.py +++ b/mypy/parse.py @@ -6,7 +6,7 @@ import re -from typing import List, Tuple, Any, Set, cast, Union, Optional +from typing import List, Tuple, Set, cast, Union, Optional from mypy import lex from mypy.lex import ( @@ -14,13 +14,12 @@ UnicodeLit, FloatLit, Op, Indent, Keyword, Punct, LexError, ComplexLit, EllipsisToken ) -import mypy.types from mypy.nodes import ( - MypyFile, Import, Node, ImportAll, ImportFrom, FuncDef, OverloadedFuncDef, - ClassDef, Decorator, Block, Var, OperatorAssignmentStmt, + MypyFile, Import, ImportAll, ImportFrom, FuncDef, OverloadedFuncDef, + ClassDef, Decorator, Block, Var, OperatorAssignmentStmt, Statement, ExpressionStmt, AssignmentStmt, ReturnStmt, RaiseStmt, AssertStmt, DelStmt, BreakStmt, ContinueStmt, PassStmt, GlobalDecl, - WhileStmt, ForStmt, IfStmt, TryStmt, WithStmt, + WhileStmt, ForStmt, IfStmt, TryStmt, WithStmt, Expression, TupleExpr, GeneratorExpr, ListComprehension, ListExpr, ConditionalExpr, DictExpr, SetExpr, NameExpr, IntExpr, StrExpr, BytesExpr, UnicodeExpr, FloatExpr, CallExpr, SuperExpr, MemberExpr, IndexExpr, SliceExpr, OpExpr, @@ -34,7 +33,7 @@ from mypy.errors import Errors, CompileError from mypy.types import Type, CallableType, AnyType, UnboundType from mypy.parsetype import ( - parse_type, parse_types, parse_signature, TypeParseError, parse_str_as_signature + parse_type, parse_types, parse_signature, TypeParseError ) from mypy.options import Options @@ -160,33 +159,12 @@ def parse(self, s: Union[str, bytes]) -> MypyFile: self.errors.raise_error() return file - def weak_opts(self) -> Set[str]: - """Do weak typing if any of the first ten tokens is a comment saying so. - - The comment can be one of: - # mypy: weak=global - # mypy: weak=local - # mypy: weak <- defaults to local - """ - regexp = re.compile(r'^[\s]*# *mypy: *weak(=?)([^\s]*)', re.M) - for t in self.tok[:10]: - for s in [t.string, t.pre]: - m = regexp.search(s) - if m: - opts = set(x for x in m.group(2).split(',') if x) - if not opts: - opts.add('local') - return opts - return set() - def parse_file(self) -> MypyFile: """Parse a mypy source file.""" is_bom = self.parse_bom() defs = self.parse_defs() - weak_opts = self.weak_opts() self.expect_type(Eof) - node = MypyFile(defs, self.imports, is_bom, self.ignored_lines, - weak_opts=weak_opts) + node = MypyFile(defs, self.imports, is_bom, self.ignored_lines) return node # Parse the initial part @@ -235,7 +213,7 @@ def translate_module_id(self, id: str) -> str: return 'builtins' return id - def parse_import_from(self) -> Node: + def parse_import_from(self) -> ImportBase: self.expect('from') # Build the list of beginning relative tokens. @@ -273,7 +251,7 @@ def parse_import_from(self) -> Node: if targets or self.current_str() == ',': self.fail('You cannot import any other modules when you ' 'import a custom typing module', - self.current().line) + self.current().line, self.current().column) node = Import([('typing', as_id)]) self.skip_until_break() break @@ -318,8 +296,8 @@ def parse_qualified_name(self) -> str: # Parsing global definitions - def parse_defs(self) -> List[Node]: - defs = [] # type: List[Node] + def parse_defs(self) -> List[Statement]: + defs = [] # type: List[Statement] while not self.eof(): try: defn, is_simple = self.parse_statement() @@ -340,7 +318,7 @@ def parse_class_def(self) -> ClassDef: metaclass = None try: - base_types = [] # type: List[Node] + base_types = [] # type: List[Expression] try: name_tok = self.expect_type(Name) name = name_tok.string @@ -391,10 +369,10 @@ def parse_class_keywords(self) -> Optional[str]: break return metaclass - def parse_super_type(self) -> Node: + def parse_super_type(self) -> Expression: return self.parse_expression(precedence[',']) - def parse_decorated_function_or_class(self) -> Node: + def parse_decorated_function_or_class(self) -> Union[Decorator, ClassDef]: decorators = [] no_type_checks = False while self.current_str() == '@': @@ -418,7 +396,7 @@ def parse_decorated_function_or_class(self) -> Node: cls.decorators = decorators return cls - def is_no_type_check_decorator(self, expr: Node) -> bool: + def is_no_type_check_decorator(self, expr: Expression) -> bool: if isinstance(expr, NameExpr): return expr.name == 'no_type_check' elif isinstance(expr, MemberExpr): @@ -427,7 +405,7 @@ def is_no_type_check_decorator(self, expr: Node) -> bool: else: return False - def parse_function(self, no_type_checks: bool=False) -> FuncDef: + def parse_function(self, no_type_checks: bool = False) -> FuncDef: def_tok = self.expect('def') is_method = self.is_class_body self.is_class_body = False @@ -445,7 +423,7 @@ def parse_function(self, no_type_checks: bool=False) -> FuncDef: # The function has a # type: ... signature. if typ: self.errors.report( - def_tok.line, 'Function has duplicate type signatures') + def_tok.line, def_tok.column, 'Function has duplicate type signatures') sig = cast(CallableType, comment_type) if sig.is_ellipsis_args: # When we encounter an ellipsis, fill in the arg_types with @@ -457,11 +435,12 @@ def parse_function(self, no_type_checks: bool=False) -> FuncDef: arg_names, sig.ret_type, None, - line=def_tok.line) + line=def_tok.line, + column=def_tok.column) elif is_method and len(sig.arg_kinds) < len(arg_kinds): self.check_argument_kinds(arg_kinds, [nodes.ARG_POS] + sig.arg_kinds, - def_tok.line) + def_tok.line, def_tok.column) # Add implicit 'self' argument to signature. first_arg = [AnyType()] # type: List[Type] typ = CallableType( @@ -470,17 +449,19 @@ def parse_function(self, no_type_checks: bool=False) -> FuncDef: arg_names, sig.ret_type, None, - line=def_tok.line) + line=def_tok.line, + column=def_tok.column) else: self.check_argument_kinds(arg_kinds, sig.arg_kinds, - def_tok.line) + def_tok.line, def_tok.column) typ = CallableType( sig.arg_types, arg_kinds, arg_names, sig.ret_type, None, - line=def_tok.line) + line=def_tok.line, + column=def_tok.column) # If there was a serious error, we really cannot build a parse tree # node. @@ -504,7 +485,7 @@ def parse_function(self, no_type_checks: bool=False) -> FuncDef: self.is_class_body = is_method def check_argument_kinds(self, funckinds: List[int], sigkinds: List[int], - line: int) -> None: + line: int, column: int) -> None: """Check that arguments are consistent. This verifies that they have the same number and the kinds correspond. @@ -515,9 +496,9 @@ def check_argument_kinds(self, funckinds: List[int], sigkinds: List[int], """ if len(funckinds) != len(sigkinds): if len(funckinds) > len(sigkinds): - self.fail("Type signature has too few arguments", line) + self.fail("Type signature has too few arguments", line, column) else: - self.fail("Type signature has too many arguments", line) + self.fail("Type signature has too many arguments", line, column) return for kind, token in [(nodes.ARG_STAR, '*'), (nodes.ARG_STAR2, '**')]: @@ -525,7 +506,7 @@ def check_argument_kinds(self, funckinds: List[int], sigkinds: List[int], (kind in funckinds and sigkinds.index(kind) != funckinds.index(kind))): self.fail( "Inconsistent use of '{}' in function " - "signature".format(token), line) + "signature".format(token), line, column) def parse_function_header( self, no_type_checks: bool=False) -> Tuple[str, @@ -588,21 +569,21 @@ def parse_args(self, no_type_checks: bool=False) -> Tuple[List[Argument], ret_type = None arg_kinds = [arg.kind for arg in args] - self.verify_argument_kinds(arg_kinds, lparen.line) + self.verify_argument_kinds(arg_kinds, lparen.line, lparen.column) annotation = self.build_func_annotation( - ret_type, args, lparen.line) + ret_type, args, lparen.line, lparen.column) return args, annotation, extra_stmts def build_func_annotation(self, ret_type: Type, args: List[Argument], - line: int, is_default_ret: bool = False) -> CallableType: + line: int, column: int, is_default_ret: bool = False) -> CallableType: arg_types = [arg.type_annotation for arg in args] # Are there any type annotations? if ((ret_type and not is_default_ret) or arg_types != [None] * len(arg_types)): # Yes. Construct a type for the function signature. - return self.construct_function_type(args, ret_type, line) + return self.construct_function_type(args, ret_type, line, column) else: return None @@ -688,7 +669,7 @@ def check_duplicate_argument_names(self, names: List[str]) -> None: for name in names: if name in found: self.fail('Duplicate argument name "{}"'.format(name), - self.current().line) + self.current().line, self.current().column) found.add(name) def parse_asterisk_arg(self, @@ -725,10 +706,11 @@ def parse_tuple_arg(self, index: int) -> Tuple[Argument, AssignmentStmt, List[st However, if the argument is (x,) then it *is* a (singleton) tuple. """ line = self.current().line + column = self.current().column # Generate a new argument name that is very unlikely to clash with anything. arg_name = '__tuple_arg_{}'.format(index + 1) if self.pyversion[0] >= 3: - self.fail('Tuples in argument lists only supported in Python 2 mode', line) + self.fail('Tuples in argument lists only supported in Python 2 mode', line, column) paren_arg = self.parse_parentheses() self.verify_tuple_arg(paren_arg) if isinstance(paren_arg, NameExpr): @@ -739,7 +721,7 @@ def parse_tuple_arg(self, index: int) -> Tuple[Argument, AssignmentStmt, List[st rvalue = NameExpr(arg_name) rvalue.set_line(line) decompose = AssignmentStmt([paren_arg], rvalue) - decompose.set_line(line) + decompose.set_line(line, column) kind = nodes.ARG_POS initializer = None if self.current_str() == '=': @@ -750,16 +732,16 @@ def parse_tuple_arg(self, index: int) -> Tuple[Argument, AssignmentStmt, List[st arg_names = self.find_tuple_arg_argument_names(paren_arg) return Argument(var, None, initializer, kind), decompose, arg_names - def verify_tuple_arg(self, paren_arg: Node) -> None: + def verify_tuple_arg(self, paren_arg: Expression) -> None: if isinstance(paren_arg, TupleExpr): if not paren_arg.items: - self.fail('Empty tuple not valid as an argument', paren_arg.line) + self.fail('Empty tuple not valid as an argument', paren_arg.line, paren_arg.column) for item in paren_arg.items: self.verify_tuple_arg(item) elif not isinstance(paren_arg, NameExpr): - self.fail('Invalid item in tuple argument', paren_arg.line) + self.fail('Invalid item in tuple argument', paren_arg.line, paren_arg.column) - def find_tuple_arg_argument_names(self, node: Node) -> List[str]: + def find_tuple_arg_argument_names(self, node: Expression) -> List[str]: result = [] # type: List[str] if isinstance(node, TupleExpr): for item in node.items: @@ -780,7 +762,7 @@ def parse_normal_arg(self, require_named: bool, else: type = self.parse_arg_type(allow_signature) - initializer = None # type: Node + initializer = None # type: Expression if self.current_str() == '=': self.expect('=') initializer = self.parse_expression(precedence[',']) @@ -796,7 +778,7 @@ def parse_normal_arg(self, require_named: bool, return Argument(variable, type, initializer, kind), require_named - def set_type_optional(self, type: Type, initializer: Node) -> None: + def set_type_optional(self, type: Type, initializer: Expression) -> None: if not experiments.STRICT_OPTIONAL: return # Indicate that type should be wrapped in an Optional if arg is initialized to None. @@ -804,7 +786,7 @@ def set_type_optional(self, type: Type, initializer: Node) -> None: if isinstance(type, UnboundType): type.optional = optional - def parse_parameter_annotation(self) -> Node: + def parse_parameter_annotation(self) -> Expression: if self.current_str() == ':': self.skip() return self.parse_expression(precedence[',']) @@ -816,21 +798,21 @@ def parse_arg_type(self, allow_signature: bool) -> Type: else: return None - def verify_argument_kinds(self, kinds: List[int], line: int) -> None: + def verify_argument_kinds(self, kinds: List[int], line: int, column: int) -> None: found = set() # type: Set[int] for i, kind in enumerate(kinds): if kind == nodes.ARG_POS and found & set([nodes.ARG_OPT, nodes.ARG_STAR, nodes.ARG_STAR2]): - self.fail('Invalid argument list', line) + self.fail('Invalid argument list', line, column) elif kind == nodes.ARG_STAR and nodes.ARG_STAR in found: - self.fail('Invalid argument list', line) + self.fail('Invalid argument list', line, column) elif kind == nodes.ARG_STAR2 and i != len(kinds) - 1: - self.fail('Invalid argument list', line) + self.fail('Invalid argument list', line, column) found.add(kind) def construct_function_type(self, args: List[Argument], ret_type: Type, - line: int) -> CallableType: + line: int, column: int) -> CallableType: # Complete the type annotation by replacing omitted types with 'Any'. arg_types = [arg.type_annotation for arg in args] for i in range(len(arg_types)): @@ -841,7 +823,7 @@ def construct_function_type(self, args: List[Argument], ret_type: Type, arg_kinds = [arg.kind for arg in args] arg_names = [arg.variable.name() for arg in args] return CallableType(arg_types, arg_kinds, arg_names, ret_type, None, name=None, - variables=None, line=line) + variables=None, line=line, column=column) # Parsing statements @@ -868,7 +850,7 @@ def parse_block(self, allow_type: bool = False) -> Tuple[Block, Type]: brk = self.expect_break() type = self.parse_type_comment(brk, signature=True) self.expect_indent() - stmt_list = [] # type: List[Node] + stmt_list = [] # type: List[Statement] while (not isinstance(self.current(), Dedent) and not isinstance(self.current(), Eof)): try: @@ -886,7 +868,7 @@ def parse_block(self, allow_type: bool = False) -> Tuple[Block, Type]: node.set_line(colon) return node, type - def try_combine_overloads(self, s: Node, stmt: List[Node]) -> bool: + def try_combine_overloads(self, s: Statement, stmt: List[Statement]) -> bool: if isinstance(s, Decorator) and stmt: fdef = s n = fdef.func.name() @@ -898,8 +880,8 @@ def try_combine_overloads(self, s: Node, stmt: List[Node]) -> bool: return True return False - def parse_statement(self) -> Tuple[Node, bool]: - stmt = None # type: Node + def parse_statement(self) -> Tuple[Statement, bool]: + stmt = None # type: Statement t = self.current() ts = self.current_str() is_simple = True # Is this a non-block statement? @@ -964,7 +946,9 @@ def parse_statement(self) -> Tuple[Node, bool]: stmt.set_line(t) return stmt, is_simple - def parse_expression_or_assignment(self) -> Node: + def parse_expression_or_assignment(self) -> Union[AssignmentStmt, + OperatorAssignmentStmt, + ExpressionStmt]: expr = self.parse_expression(star_expr_allowed=True) if self.current_str() == '=': return self.parse_assignment(expr) @@ -978,7 +962,7 @@ def parse_expression_or_assignment(self) -> Node: # Expression statement. return ExpressionStmt(expr) - def parse_assignment(self, lvalue: Any) -> Node: + def parse_assignment(self, lvalue: Expression) -> AssignmentStmt: """Parse an assignment statement. Assume that lvalue has been parsed already, and the current token is '='. @@ -1128,7 +1112,7 @@ def parse_for_stmt(self) -> ForStmt: node = ForStmt(index, expr, body, else_body) return node - def parse_for_index_variables(self) -> Node: + def parse_for_index_variables(self) -> Expression: # Parse index variables of a 'for' statement. index_items = [] force_tuple = False @@ -1148,7 +1132,7 @@ def parse_for_index_variables(self) -> Node: index = index_items[0] else: index = TupleExpr(index_items) - index.set_line(index_items[0].get_line()) + index.set_line(index_items[0]) return index @@ -1184,19 +1168,21 @@ def parse_if_stmt(self) -> IfStmt: else: return None - def parse_try_stmt(self) -> Node: + def parse_try_stmt(self) -> TryStmt: self.expect('try') body, _ = self.parse_block() is_error = False vars = [] # type: List[NameExpr] - types = [] # type: List[Node] + types = [] # type: List[Optional[Expression]] handlers = [] # type: List[Block] while self.current_str() == 'except': self.expect('except') if not isinstance(self.current(), Colon): try: t = self.current() - types.append(self.parse_expression(precedence[',']).set_line(t)) + expr = self.parse_expression(precedence[',']) + expr.set_line(t) + types.append(expr) if self.current_str() == 'as': self.expect('as') vars.append(self.parse_name_expr()) @@ -1287,9 +1273,9 @@ def parse_exec_stmt(self) -> ExecStmt: # Parsing expressions - def parse_expression(self, prec: int = 0, star_expr_allowed: bool = False) -> Node: + def parse_expression(self, prec: int = 0, star_expr_allowed: bool = False) -> Expression: """Parse a subexpression within a specific precedence context.""" - expr = None # type: Node + expr = None # type: Expression current = self.current() # Remember token for setting the line number. # Parse a "value" expression or unary operator expression and store @@ -1409,18 +1395,18 @@ def parse_expression(self, prec: int = 0, star_expr_allowed: bool = False) -> No return expr - def parse_parentheses(self) -> Node: + def parse_parentheses(self) -> Expression: self.skip() if self.current_str() == ')': # Empty tuple (). - expr = self.parse_empty_tuple_expr() # type: Node + expr = self.parse_empty_tuple_expr() # type: Expression else: # Parenthesised expression. expr = self.parse_expression(0, star_expr_allowed=True) self.expect(')') return expr - def parse_star_expr(self) -> Node: + def parse_star_expr(self) -> StarExpr: star = self.expect('*') expr = self.parse_expression(precedence['*u']) expr = StarExpr(expr) @@ -1433,7 +1419,7 @@ def parse_empty_tuple_expr(self) -> TupleExpr: node = TupleExpr([]) return node - def parse_list_expr(self) -> Node: + def parse_list_expr(self) -> Union[ListExpr, ListComprehension]: """Parse list literal or list comprehension.""" items = [] self.expect('[') @@ -1451,7 +1437,7 @@ def parse_list_expr(self) -> Node: expr = ListExpr(items) return expr - def parse_generator_expr(self, left_expr: Node) -> GeneratorExpr: + def parse_generator_expr(self, left_expr: Expression) -> GeneratorExpr: tok = self.current() indices, sequences, condlists = self.parse_comp_for() @@ -1459,10 +1445,10 @@ def parse_generator_expr(self, left_expr: Node) -> GeneratorExpr: gen.set_line(tok) return gen - def parse_comp_for(self) -> Tuple[List[Node], List[Node], List[List[Node]]]: + def parse_comp_for(self) -> Tuple[List[Expression], List[Expression], List[List[Expression]]]: indices = [] sequences = [] - condlists = [] # type: List[List[Node]] + condlists = [] # type: List[List[Expression]] while self.current_str() == 'for': conds = [] self.expect('for') @@ -1481,24 +1467,27 @@ def parse_comp_for(self) -> Tuple[List[Node], List[Node], List[List[Node]]]: return indices, sequences, condlists - def parse_expression_list(self) -> Node: + def parse_expression_list(self) -> Expression: prec = precedence[''] expr = self.parse_expression(prec) if self.current_str() != ',': return expr else: t = self.current() - return self.parse_tuple_expr(expr, prec).set_line(t) + tuple_expr = self.parse_tuple_expr(expr, prec) + tuple_expr.set_line(t) + return tuple_expr - def parse_conditional_expr(self, left_expr: Node) -> ConditionalExpr: + def parse_conditional_expr(self, left_expr: Expression) -> ConditionalExpr: self.expect('if') cond = self.parse_expression(precedence['']) self.expect('else') else_expr = self.parse_expression(precedence['']) return ConditionalExpr(cond, left_expr, else_expr) - def parse_dict_or_set_expr(self) -> Node: - items = [] # type: List[Tuple[Node, Node]] + def parse_dict_or_set_expr(self) -> Union[SetComprehension, SetExpr, + DictionaryComprehension, DictExpr]: + items = [] # type: List[Tuple[Expression, Expression]] self.expect('{') while self.current_str() != '}' and not self.eol(): key = self.parse_expression(precedence['']) @@ -1520,7 +1509,7 @@ def parse_dict_or_set_expr(self) -> Node: node = DictExpr(items) return node - def parse_set_expr(self, first: Node) -> SetExpr: + def parse_set_expr(self, first: Expression) -> SetExpr: items = [first] while self.current_str() != '}' and not self.eol(): self.expect(',') @@ -1531,13 +1520,13 @@ def parse_set_expr(self, first: Node) -> SetExpr: expr = SetExpr(items) return expr - def parse_set_comprehension(self, expr: Node) -> SetComprehension: + def parse_set_comprehension(self, expr: Expression) -> SetComprehension: gen = self.parse_generator_expr(expr) self.expect('}') set_comp = SetComprehension(gen) return set_comp - def parse_dict_comprehension(self, key: Node, value: Node, + def parse_dict_comprehension(self, key: Expression, value: Expression, colon: Token) -> DictionaryComprehension: indices, sequences, condlists = self.parse_comp_for() dic = DictionaryComprehension(key, value, indices, sequences, condlists) @@ -1545,7 +1534,7 @@ def parse_dict_comprehension(self, key: Node, value: Node, self.expect('}') return dic - def parse_tuple_expr(self, expr: Node, + def parse_tuple_expr(self, expr: Expression, prec: int = precedence[',']) -> TupleExpr: items = [expr] while True: @@ -1582,7 +1571,7 @@ def parse_int_expr(self) -> IntExpr: node = IntExpr(value) return node - def parse_str_expr(self) -> Node: + def parse_str_expr(self) -> Union[UnicodeExpr, StrExpr]: # XXX \uxxxx literals token = self.expect_type(StrLit) value = cast(StrLit, token).parsed() @@ -1595,12 +1584,11 @@ def parse_str_expr(self) -> Node: value += token.parsed() is_unicode = True if is_unicode or (self.pyversion[0] == 2 and 'unicode_literals' in self.future_options): - node = UnicodeExpr(value) # type: Node + return UnicodeExpr(value) else: - node = StrExpr(value) - return node + return StrExpr(value) - def parse_bytes_literal(self) -> Node: + def parse_bytes_literal(self) -> Union[BytesExpr, StrExpr]: # XXX \uxxxx literals tok = [self.expect_type(BytesLit)] value = (cast(BytesLit, tok[0])).parsed() @@ -1608,12 +1596,11 @@ def parse_bytes_literal(self) -> Node: t = cast(BytesLit, self.skip()) value += t.parsed() if self.pyversion[0] >= 3: - node = BytesExpr(value) # type: Node + return BytesExpr(value) else: - node = StrExpr(value) - return node + return StrExpr(value) - def parse_unicode_literal(self) -> Node: + def parse_unicode_literal(self) -> Union[StrExpr, UnicodeExpr]: # XXX \uxxxx literals token = self.expect_type(UnicodeLit) value = cast(UnicodeLit, token).parsed() @@ -1622,29 +1609,25 @@ def parse_unicode_literal(self) -> Node: value += token.parsed() if self.pyversion[0] >= 3: # Python 3.3 supports u'...' as an alias of '...'. - node = StrExpr(value) # type: Node + return StrExpr(value) else: - node = UnicodeExpr(value) - return node + return UnicodeExpr(value) def parse_float_expr(self) -> FloatExpr: tok = self.expect_type(FloatLit) - node = FloatExpr(float(tok.string)) - return node + return FloatExpr(float(tok.string)) def parse_complex_expr(self) -> ComplexExpr: tok = self.expect_type(ComplexLit) - node = ComplexExpr(complex(tok.string)) - return node + return ComplexExpr(complex(tok.string)) - def parse_call_expr(self, callee: Any) -> CallExpr: + def parse_call_expr(self, callee: Expression) -> CallExpr: self.expect('(') args, kinds, names = self.parse_arg_expr() self.expect(')') - node = CallExpr(callee, args, kinds, names) - return node + return CallExpr(callee, args, kinds, names) - def parse_arg_expr(self) -> Tuple[List[Node], List[int], List[str]]: + def parse_arg_expr(self) -> Tuple[List[Expression], List[int], List[str]]: """Parse arguments in a call expression (within '(' and ')'). Return a tuple with these items: @@ -1652,7 +1635,7 @@ def parse_arg_expr(self) -> Tuple[List[Node], List[int], List[str]]: argument kinds argument names (for named arguments; None for ordinary args) """ - args = [] # type: List[Node] + args = [] # type: List[Expression] kinds = [] # type: List[int] names = [] # type: List[str] var_arg = False @@ -1690,18 +1673,17 @@ def parse_arg_expr(self) -> Tuple[List[Node], List[int], List[str]]: self.expect(',') return args, kinds, names - def parse_member_expr(self, expr: Any) -> Node: + def parse_member_expr(self, expr: Expression) -> Union[SuperExpr, MemberExpr]: self.expect('.') name = self.expect_type(Name) if (isinstance(expr, CallExpr) and isinstance(expr.callee, NameExpr) and expr.callee.name == 'super'): # super() expression - node = SuperExpr(name.string) # type: Node + return SuperExpr(name.string) else: - node = MemberExpr(expr, name.string) - return node + return MemberExpr(expr, name.string) - def parse_index_expr(self, base: Any) -> IndexExpr: + def parse_index_expr(self, base: Expression) -> IndexExpr: self.expect('[') index = self.parse_slice_item() if self.current_str() == ',': @@ -1713,12 +1695,12 @@ def parse_index_expr(self, base: Any) -> IndexExpr: break items.append(self.parse_slice_item()) index = TupleExpr(items) - index.set_line(items[0].line) + index.set_line(items[0]) self.expect(']') node = IndexExpr(base, index) return node - def parse_slice_item(self) -> Node: + def parse_slice_item(self) -> Expression: if self.current_str() != ':': if self.current_str() == '...': # Ellipsis is valid here even in Python 2 (but not elsewhere). @@ -1743,10 +1725,11 @@ def parse_slice_item(self) -> Node: self.expect(':') if self.current_str() not in (']', ','): stride = self.parse_expression(precedence[',']) - item = SliceExpr(index, end_index, stride).set_line(colon.line) + item = SliceExpr(index, end_index, stride) + item.set_line(colon) return item - def parse_bin_op_expr(self, left: Node, prec: int) -> OpExpr: + def parse_bin_op_expr(self, left: Expression, prec: int) -> OpExpr: op = self.expect_type(Op) op_str = op.string if op_str == '~': @@ -1756,7 +1739,7 @@ def parse_bin_op_expr(self, left: Node, prec: int) -> OpExpr: node = OpExpr(op_str, left, right) return node - def parse_comparison_expr(self, left: Node, prec: int) -> ComparisonExpr: + def parse_comparison_expr(self, left: Expression, prec: int) -> ComparisonExpr: operators_str = [] operands = [left] @@ -1807,13 +1790,15 @@ def parse_lambda_expr(self) -> FuncExpr: # less precise. ret_type = UnboundType('__builtins__.object') typ = self.build_func_annotation(ret_type, args, - lambda_tok.line, is_default_ret=True) + lambda_tok.line, lambda_tok.column, is_default_ret=True) colon = self.expect(':') expr = self.parse_expression(precedence[',']) - nodes = [ReturnStmt(expr).set_line(lambda_tok)] + return_stmt = ReturnStmt(expr) + return_stmt.set_line(lambda_tok) + nodes = [return_stmt] # type: List[Statement] # Potentially insert extra assignment statements to the beginning of the # body, used to decompose Python 2 tuple arguments. nodes[:0] = extra_stmts @@ -1839,11 +1824,11 @@ def expect_indent(self) -> Token: if isinstance(self.current(), Indent): return self.expect_type(Indent) else: - self.fail('Expected an indented block', self.current().line) + self.fail('Expected an indented block', self.current().line, self.current().column) return none - def fail(self, msg: str, line: int) -> None: - self.errors.report(line, msg) + def fail(self, msg: str, line: int, column: int) -> None: + self.errors.report(line, column, msg) def expect_type(self, typ: type) -> Token: current = self.current() @@ -1883,7 +1868,7 @@ def parse_error_at(self, tok: Token, skip: bool = True, reason: Optional[str] = formatted_reason = ": {}".format(reason) if reason else "" msg = 'Parse error before {}{}'.format(token_repr(tok), formatted_reason) - self.errors.report(tok.line, msg) + self.errors.report(tok.line, tok.column, msg) if skip: self.skip_until_next_line() @@ -1936,7 +1921,7 @@ def parse_type_comment(self, token: Token, signature: bool) -> Type: tokens = lex.lex(type_as_str, token.line)[0] if len(tokens) < 2: # Empty annotation (only Eof token) - self.errors.report(token.line, 'Empty type annotation') + self.errors.report(token.line, token.column, 'Empty type annotation') return None try: if not signature: diff --git a/mypy/semanal.py b/mypy/semanal.py index 511e9adbdbf1..60947cc4fbdd 100644 --- a/mypy/semanal.py +++ b/mypy/semanal.py @@ -49,7 +49,7 @@ from mypy.nodes import ( MypyFile, TypeInfo, Node, AssignmentStmt, FuncDef, OverloadedFuncDef, - ClassDef, Var, GDEF, MODULE_REF, FuncItem, Import, + ClassDef, Var, GDEF, MODULE_REF, FuncItem, Import, Expression, Lvalue, ImportFrom, ImportAll, Block, LDEF, NameExpr, MemberExpr, IndexExpr, TupleExpr, ListExpr, ExpressionStmt, ReturnStmt, RaiseStmt, AssertStmt, OperatorAssignmentStmt, WhileStmt, @@ -60,7 +60,7 @@ FuncExpr, MDEF, FuncBase, Decorator, SetExpr, TypeVarExpr, NewTypeExpr, StrExpr, BytesExpr, PrintStmt, ConditionalExpr, PromoteExpr, ComparisonExpr, StarExpr, ARG_POS, ARG_NAMED, MroError, type_aliases, - YieldFromExpr, NamedTupleExpr, NonlocalDecl, + YieldFromExpr, NamedTupleExpr, NonlocalDecl, SymbolNode, SetComprehension, DictionaryComprehension, TYPE_ALIAS, TypeAliasExpr, YieldExpr, ExecStmt, Argument, BackquoteExpr, ImportBase, AwaitExpr, IntExpr, FloatExpr, UnicodeExpr, EllipsisExpr, @@ -167,8 +167,6 @@ class SemanticAnalyzer(NodeVisitor): bound_tvars = None # type: List[SymbolTableNode] # Stack of type variables that were bound by outer classess tvar_stack = None # type: List[List[SymbolTableNode]] - # Do weak type checking in this file - weak_opts = set() # type: Set[str] # Per-file options options = None # type: Options @@ -222,7 +220,6 @@ def visit_file(self, file_node: MypyFile, fnam: str, options: Options) -> None: self.cur_mod_id = file_node.fullname() self.is_stub_file = fnam.lower().endswith('.pyi') self.globals = file_node.names - self.weak_opts = file_node.weak_opts if 'builtins' in self.modules: self.globals['__builtins__'] = SymbolTableNode( @@ -405,7 +402,7 @@ def find_type_variables_in_type( assert False, 'Unsupported type %s' % type return result - def is_defined_type_var(self, tvar: str, context: Node) -> bool: + def is_defined_type_var(self, tvar: str, context: Context) -> bool: return self.lookup_qualified(tvar, context).kind == BOUND_TVAR def visit_overloaded_func_def(self, defn: OverloadedFuncDef) -> None: @@ -606,7 +603,7 @@ def unbind_class_type_vars(self) -> None: if self.bound_tvars: enable_typevars(self.bound_tvars) - def analyze_class_decorator(self, defn: ClassDef, decorator: Node) -> None: + def analyze_class_decorator(self, defn: ClassDef, decorator: Expression) -> None: decorator.accept(self) def setup_is_builtinclass(self, defn: ClassDef) -> None: @@ -801,7 +798,7 @@ def analyze_base_classes(self, defn: ClassDef) -> None: if info.mro and info.mro[-1].fullname() != 'builtins.object': info.mro.append(self.object_type().type) - def expr_to_analyzed_type(self, expr: Node) -> Type: + def expr_to_analyzed_type(self, expr: Expression) -> Type: if isinstance(expr, CallExpr): expr.accept(self) info = self.check_namedtuple(expr) @@ -1135,11 +1132,10 @@ def visit_assignment_stmt(self, s: AssignmentStmt) -> None: isinstance(s.rvalue, (ListExpr, TupleExpr))): self.add_exports(*s.rvalue.items) - def analyze_simple_literal_type(self, rvalue: Node) -> Optional[Type]: + def analyze_simple_literal_type(self, rvalue: Expression) -> Optional[Type]: """Return builtins.int if rvalue is an int literal, etc.""" - if self.weak_opts or self.options.semantic_analysis_only or self.function_stack: - # Skip this if any weak options are set. - # Also skip if we're only doing the semantic analysis pass. + if self.options.semantic_analysis_only or self.function_stack: + # Skip this if we're only doing the semantic analysis pass. # This is mostly to avoid breaking unit tests. # Also skip inside a function; this is to avoid confusing # the code that handles dead code due to isinstance() @@ -1177,7 +1173,7 @@ def check_and_set_up_type_alias(self, s: AssignmentStmt) -> None: # just an alias for the type. self.globals[lvalue.name].node = node - def analyze_lvalue(self, lval: Node, nested: bool = False, + def analyze_lvalue(self, lval: Lvalue, nested: bool = False, add_global: bool = False, explicit_type: bool = False) -> None: """Analyze an lvalue or assignment target. @@ -1300,11 +1296,11 @@ def is_self_member_ref(self, memberexpr: MemberExpr) -> bool: node = memberexpr.expr.node return isinstance(node, Var) and node.is_self - def check_lvalue_validity(self, node: Node, ctx: Context) -> None: + def check_lvalue_validity(self, node: Union[Expression, SymbolNode], ctx: Context) -> None: if isinstance(node, (TypeInfo, TypeVarExpr)): self.fail('Invalid assignment target', ctx) - def store_declared_types(self, lvalue: Node, typ: Type) -> None: + def store_declared_types(self, lvalue: Lvalue, typ: Type) -> None: if isinstance(typ, StarType) and not isinstance(lvalue, StarExpr): self.fail('Star type only allowed for starred expressions', lvalue) if isinstance(lvalue, RefExpr): @@ -1508,7 +1504,7 @@ def get_typevar_declaration(self, s: AssignmentStmt) -> Optional[CallExpr]: return None return call - def process_typevar_parameters(self, args: List[Node], + def process_typevar_parameters(self, args: List[Expression], names: List[Optional[str]], kinds: List[int], has_values: bool, @@ -1585,7 +1581,7 @@ def process_namedtuple_definition(self, s: AssignmentStmt) -> None: # TODO call.analyzed node.node = named_tuple - def check_namedtuple(self, node: Node, var_name: str = None) -> TypeInfo: + def check_namedtuple(self, node: Expression, var_name: str = None) -> TypeInfo: """Check if a call defines a namedtuple. The optional var_name argument is the name of the variable to @@ -1617,7 +1613,8 @@ def check_namedtuple(self, node: Node, var_name: str = None) -> TypeInfo: info = self.build_namedtuple_typeinfo(name, items, types) # Store it as a global just in case it would remain anonymous. self.globals[name] = SymbolTableNode(GDEF, info, self.cur_mod_id) - call.analyzed = NamedTupleExpr(info).set_line(call.line) + call.analyzed = NamedTupleExpr(info) + call.analyzed.set_line(call.line, call.column) return info def parse_namedtuple_args(self, call: CallExpr, @@ -1664,7 +1661,7 @@ def parse_namedtuple_args(self, call: CallExpr, + ', '.join(underscore), call) return items, types, ok - def parse_namedtuple_fields_with_types(self, nodes: List[Node], + def parse_namedtuple_fields_with_types(self, nodes: List[Expression], context: Context) -> Tuple[List[str], List[Type], bool]: items = [] # type: List[str] types = [] # type: List[Type] @@ -1769,7 +1766,7 @@ def add_method(funcname: str, ret: Type, args: List[Argument], name=None, def make_argument(self, name: str, type: Type) -> Argument: return Argument(Var(name), type, None, ARG_POS) - def analyze_types(self, items: List[Node]) -> List[Type]: + def analyze_types(self, items: List[Expression]) -> List[Type]: result = [] # type: List[Type] for node in items: try: @@ -1930,7 +1927,7 @@ def visit_del_stmt(self, s: DelStmt) -> None: if not self.is_valid_del_target(s.expr): self.fail('Invalid delete target', s) - def is_valid_del_target(self, s: Node) -> bool: + def is_valid_del_target(self, s: Expression) -> bool: if isinstance(s, (IndexExpr, NameExpr, MemberExpr)): return True elif isinstance(s, TupleExpr): @@ -2501,7 +2498,7 @@ def add_local(self, node: Union[Var, FuncBase], ctx: Context) -> None: node._fullname = name self.locals[-1][name] = SymbolTableNode(LDEF, node) - def add_exports(self, *exps: Node) -> None: + def add_exports(self, *exps: Expression) -> None: for exp in exps: if isinstance(exp, StrExpr): self.all_exports.add(exp.value) @@ -2536,7 +2533,7 @@ def fail(self, msg: str, ctx: Context, serious: bool = False, *, return # In case it's a bug and we don't really have context assert ctx is not None, msg - self.errors.report(ctx.get_line(), msg, blocker=blocker) + self.errors.report(ctx.get_line(), ctx.get_column(), msg, blocker=blocker) def fail_blocker(self, msg: str, ctx: Context) -> None: self.fail(msg, ctx, blocker=True) @@ -2546,7 +2543,7 @@ def note(self, msg: str, ctx: Context) -> None: self.function_stack and self.function_stack[-1].is_dynamic()): return - self.errors.report(ctx.get_line(), msg, severity='note') + self.errors.report(ctx.get_line(), ctx.get_column(), msg, severity='note') def undefined_name_extra_info(self, fullname: str) -> Optional[str]: if fullname in obsolete_name_mapping: @@ -2558,7 +2555,7 @@ def accept(self, node: Node) -> None: try: node.accept(self) except Exception as err: - report_internal_error(err, self.errors.file, node.line, self.errors) + report_internal_error(err, self.errors.file, node.line, self.errors, self.options) class FirstPass(NodeVisitor): @@ -2678,7 +2675,7 @@ def visit_class_def(self, cdef: ClassDef) -> None: self.sem.check_no_global(cdef.name, cdef) cdef.fullname = self.sem.qualified_name(cdef.name) info = TypeInfo(SymbolTable(), cdef, self.sem.cur_mod_id) - info.set_line(cdef.line) + info.set_line(cdef.line, cdef.column) cdef.info = info self.sem.globals[cdef.name] = SymbolTableNode(GDEF, info, self.sem.cur_mod_id) @@ -2752,7 +2749,7 @@ def visit_if_stmt(self, s: IfStmt) -> None: def visit_try_stmt(self, s: TryStmt) -> None: self.sem.analyze_try_stmt(s, self, add_global=True) - def analyze_lvalue(self, lvalue: Node, explicit_type: bool = False) -> None: + def analyze_lvalue(self, lvalue: Lvalue, explicit_type: bool = False) -> None: self.sem.analyze_lvalue(lvalue, add_global=True, explicit_type=explicit_type) @@ -2767,15 +2764,16 @@ def __init__(self, modules: Dict[str, MypyFile], errors: Errors) -> None: self.modules = modules self.errors = errors - def visit_file(self, file_node: MypyFile, fnam: str) -> None: + def visit_file(self, file_node: MypyFile, fnam: str, options: Options) -> None: self.errors.set_file(fnam) + self.options = options self.accept(file_node) def accept(self, node: Node) -> None: try: node.accept(self) except Exception as err: - report_internal_error(err, self.errors.file, node.line, self.errors) + report_internal_error(err, self.errors.file, node.line, self.errors, self.options) def visit_block(self, b: Block) -> None: if b.is_unreachable: @@ -2873,7 +2871,7 @@ def analyze(self, type: Type) -> None: type.accept(analyzer) def fail(self, msg: str, ctx: Context, *, blocker: bool = False) -> None: - self.errors.report(ctx.get_line(), msg) + self.errors.report(ctx.get_line(), ctx.get_column(), msg) def fail_blocker(self, msg: str, ctx: Context) -> None: self.fail(msg, ctx, blocker=True) @@ -2918,12 +2916,12 @@ def set_callable_name(sig: Type, fdef: FuncDef) -> Type: return sig -def refers_to_fullname(node: Node, fullname: str) -> bool: +def refers_to_fullname(node: Expression, fullname: str) -> bool: """Is node a name or member expression with the given full name?""" return isinstance(node, RefExpr) and node.fullname == fullname -def refers_to_class_or_function(node: Node) -> bool: +def refers_to_class_or_function(node: Expression) -> bool: """Does semantically analyzed node refer to a class?""" return (isinstance(node, RefExpr) and isinstance(node.node, (TypeInfo, FuncDef, OverloadedFuncDef))) @@ -2996,7 +2994,7 @@ def infer_reachability_of_if_statement(s: IfStmt, break -def infer_if_condition_value(expr: Node, pyversion: Tuple[int, int], platform: str) -> int: +def infer_if_condition_value(expr: Expression, pyversion: Tuple[int, int], platform: str) -> int: """Infer whether if condition is always true/false. Return ALWAYS_TRUE if always true, ALWAYS_FALSE if always false, @@ -3025,6 +3023,8 @@ def infer_if_condition_value(expr: Node, pyversion: Tuple[int, int], platform: s result = ALWAYS_TRUE if pyversion[0] == 3 else ALWAYS_FALSE elif name == 'MYPY' or name == 'TYPE_CHECKING': result = ALWAYS_TRUE + elif name == 'TYPE_CHECKING': + result = ALWAYS_TRUE if negated: if result == ALWAYS_TRUE: result = ALWAYS_FALSE @@ -3033,7 +3033,7 @@ def infer_if_condition_value(expr: Node, pyversion: Tuple[int, int], platform: s return result -def consider_sys_version_info(expr: Node, pyversion: Tuple[int, ...]) -> int: +def consider_sys_version_info(expr: Expression, pyversion: Tuple[int, ...]) -> int: """Consider whether expr is a comparison involving sys.version_info. Return ALWAYS_TRUE, ALWAYS_FALSE, or TRUTH_VALUE_UNKNOWN. @@ -3075,7 +3075,7 @@ def consider_sys_version_info(expr: Node, pyversion: Tuple[int, ...]) -> int: return TRUTH_VALUE_UNKNOWN -def consider_sys_platform(expr: Node, platform: str) -> int: +def consider_sys_platform(expr: Expression, platform: str) -> int: """Consider whether expr is a comparison involving sys.platform. Return ALWAYS_TRUE, ALWAYS_FALSE, or TRUTH_VALUE_UNKNOWN. @@ -3134,7 +3134,8 @@ def fixed_comparison(left: Targ, op: str, right: Targ) -> int: return TRUTH_VALUE_UNKNOWN -def contains_int_or_tuple_of_ints(expr: Node) -> Union[None, int, Tuple[int], Tuple[int, ...]]: +def contains_int_or_tuple_of_ints(expr: Expression + ) -> Union[None, int, Tuple[int], Tuple[int, ...]]: if isinstance(expr, IntExpr): return expr.value if isinstance(expr, TupleExpr): @@ -3148,7 +3149,8 @@ def contains_int_or_tuple_of_ints(expr: Node) -> Union[None, int, Tuple[int], Tu return None -def contains_sys_version_info(expr: Node) -> Union[None, int, Tuple[Optional[int], Optional[int]]]: +def contains_sys_version_info(expr: Expression + ) -> Union[None, int, Tuple[Optional[int], Optional[int]]]: if is_sys_attr(expr, 'version_info'): return (None, None) # Same as sys.version_info[:] if isinstance(expr, IndexExpr) and is_sys_attr(expr.base, 'version_info'): @@ -3172,7 +3174,7 @@ def contains_sys_version_info(expr: Node) -> Union[None, int, Tuple[Optional[int return None -def is_sys_attr(expr: Node, name: str) -> bool: +def is_sys_attr(expr: Expression, name: str) -> bool: # TODO: This currently doesn't work with code like this: # - import sys as _sys # - from sys import version_info @@ -3210,7 +3212,7 @@ def is_identity_signature(sig: Type) -> bool: return False -def returns_any_if_called(expr: Node) -> bool: +def returns_any_if_called(expr: Expression) -> bool: """Return True if we can predict that expr will return Any if called. This only uses information available during semantic analysis so this @@ -3233,7 +3235,7 @@ def returns_any_if_called(expr: Node) -> bool: return False -def find_fixed_callable_return(expr: Node) -> Optional[CallableType]: +def find_fixed_callable_return(expr: Expression) -> Optional[CallableType]: if isinstance(expr, RefExpr): if isinstance(expr.node, FuncDef): typ = expr.node.type diff --git a/mypy/stats.py b/mypy/stats.py index ac914f4e0e8a..e6c611139c0a 100644 --- a/mypy/stats.py +++ b/mypy/stats.py @@ -2,7 +2,6 @@ import cgi import os.path -import re from typing import Any, Dict, List, cast, Tuple @@ -13,7 +12,7 @@ ) from mypy import nodes from mypy.nodes import ( - Node, FuncDef, TypeApplication, AssignmentStmt, NameExpr, CallExpr, + Node, FuncDef, TypeApplication, AssignmentStmt, NameExpr, CallExpr, MypyFile, MemberExpr, OpExpr, ComparisonExpr, IndexExpr, UnaryExpr, YieldFromExpr ) @@ -198,7 +197,7 @@ def record_line(self, line: int, precision: int) -> None: self.line_map.get(line, TYPE_PRECISE)) -def dump_type_stats(tree: Node, path: str, inferred: bool = False, +def dump_type_stats(tree: MypyFile, path: str, inferred: bool = False, typemap: Dict[Node, Type] = None) -> None: if is_special_module(path): return @@ -266,7 +265,7 @@ def is_complex(t: Type) -> bool: html_files = [] # type: List[Tuple[str, str, int, int]] -def generate_html_report(tree: Node, path: str, type_map: Dict[Node, Type], +def generate_html_report(tree: MypyFile, path: str, type_map: Dict[Node, Type], output_dir: str) -> None: if is_special_module(path): return diff --git a/mypy/stubgen.py b/mypy/stubgen.py index 6b5f71c8264e..2bf79654fae0 100644 --- a/mypy/stubgen.py +++ b/mypy/stubgen.py @@ -40,6 +40,7 @@ import importlib import json import os.path +import pkgutil import subprocess import sys import textwrap @@ -52,7 +53,7 @@ import mypy.traverser from mypy import defaults from mypy.nodes import ( - Node, IntExpr, UnaryExpr, StrExpr, BytesExpr, NameExpr, FloatExpr, MemberExpr, TupleExpr, + Expression, IntExpr, UnaryExpr, StrExpr, BytesExpr, NameExpr, FloatExpr, MemberExpr, TupleExpr, ListExpr, ComparisonExpr, CallExpr, ClassDef, MypyFile, Decorator, AssignmentStmt, IfStmt, ImportAll, ImportFrom, Import, FuncDef, FuncBase, ARG_STAR, ARG_STAR2, ARG_NAMED ) @@ -66,7 +67,10 @@ ('doc_dir', str), ('search_path', List[str]), ('interpreter', str), - ('modules', List[str])]) + ('modules', List[str]), + ('ignore_errors', bool), + ('recursive', bool), + ]) def generate_stub_for_module(module: str, output_dir: str, quiet: bool = False, @@ -356,7 +360,7 @@ def visit_assignment_stmt(self, o: AssignmentStmt) -> None: if all(foundl): self._state = VAR - def is_namedtuple(self, expr: Node) -> bool: + def is_namedtuple(self, expr: Expression) -> bool: if not isinstance(expr, CallExpr): return False callee = expr.callee @@ -441,7 +445,7 @@ def visit_import(self, o: Import) -> None: self.add_import_line('import %s as %s\n' % (id, target_name)) self.record_name(target_name) - def get_init(self, lvalue: str, rvalue: Node) -> str: + def get_init(self, lvalue: str, rvalue: Expression) -> str: """Return initializer for a variable. Return None if we've generated one already or if the variable is internal. @@ -500,7 +504,7 @@ def is_private_name(self, name: str) -> bool: '__setstate__', '__slots__')) - def get_str_type_of_node(self, rvalue: Node, + def get_str_type_of_node(self, rvalue: Expression, can_infer_optional: bool = False) -> str: if isinstance(rvalue, IntExpr): return 'int' @@ -539,8 +543,8 @@ def is_recorded_name(self, name: str) -> bool: return self.is_top_level() and name in self._toplevel_names -def find_self_initializers(fdef: FuncBase) -> List[Tuple[str, Node]]: - results = [] # type: List[Tuple[str, Node]] +def find_self_initializers(fdef: FuncBase) -> List[Tuple[str, Expression]]: + results = [] # type: List[Tuple[str, Expression]] class SelfTraverser(mypy.traverser.TraverserVisitor): def visit_assignment_stmt(self, o: AssignmentStmt) -> None: @@ -554,7 +558,7 @@ def visit_assignment_stmt(self, o: AssignmentStmt) -> None: return results -def find_classes(node: Node) -> Set[str]: +def find_classes(node: MypyFile) -> Set[str]: results = set() # type: Set[str] class ClassTraverser(mypy.traverser.TraverserVisitor): @@ -565,7 +569,7 @@ def visit_class_def(self, o: ClassDef) -> None: return results -def get_qualified_name(o: Node) -> str: +def get_qualified_name(o: Expression) -> str: if isinstance(o, NameExpr): return o.name elif isinstance(o, MemberExpr): @@ -574,10 +578,22 @@ def get_qualified_name(o: Node) -> str: return '' +def walk_packages(packages: List[str]): + for package_name in packages: + package = __import__(package_name) + yield package.__name__ + for importer, qualified_name, ispkg in pkgutil.walk_packages(package.__path__, + prefix=package.__name__ + ".", + onerror=lambda r: None): + yield qualified_name + + def main() -> None: options = parse_options() if not os.path.isdir('out'): raise SystemExit('Directory "out" does not exist') + if options.recursive and options.no_import: + raise SystemExit('recursive stub generation without importing is not currently supported') sigs = {} # type: Any class_sigs = {} # type: Any if options.doc_dir: @@ -589,21 +605,29 @@ def main() -> None: all_class_sigs += class_sigs sigs = dict(find_unique_signatures(all_sigs)) class_sigs = dict(find_unique_signatures(all_class_sigs)) - for module in options.modules: - generate_stub_for_module(module, 'out', - add_header=True, - sigs=sigs, - class_sigs=class_sigs, - pyversion=options.pyversion, - no_import=options.no_import, - search_path=options.search_path, - interpreter=options.interpreter) + for module in (options.modules if not options.recursive else walk_packages(options.modules)): + try: + generate_stub_for_module(module, 'out', + add_header=True, + sigs=sigs, + class_sigs=class_sigs, + pyversion=options.pyversion, + no_import=options.no_import, + search_path=options.search_path, + interpreter=options.interpreter) + except Exception as e: + if not options.ignore_errors: + raise e + else: + print("Stub generation failed for", module, file=sys.stderr) def parse_options() -> Options: args = sys.argv[1:] pyversion = defaults.PYTHON3_VERSION no_import = False + recursive = False + ignore_errors = False doc_dir = '' search_path = [] # type: List[str] interpreter = '' @@ -619,6 +643,10 @@ def parse_options() -> Options: elif args[0] == '-p': interpreter = args[1] args = args[1:] + elif args[0] == '--recursive': + recursive = True + elif args[0] == '--ignore-errors': + ignore_errors = True elif args[0] == '--py2': pyversion = defaults.PYTHON2_VERSION elif args[0] == '--no-import': @@ -637,7 +665,9 @@ def parse_options() -> Options: doc_dir=doc_dir, search_path=search_path, interpreter=interpreter, - modules=args) + modules=args, + ignore_errors=ignore_errors, + recursive=recursive) def default_python2_interpreter() -> str: @@ -664,6 +694,8 @@ def usage() -> None: Options: --py2 run in Python 2 mode (default: Python 3 mode) + --recursive traverse listed modules to generate inner package modules as well + --ignore-errors ignore errors when trying to generate stubs for modules --no-import don't import the modules, just parse and analyze them (doesn't work with C extension modules and doesn't respect __all__) diff --git a/mypy/test/data.py b/mypy/test/data.py index 7801f6397010..4d34b9dc32e4 100644 --- a/mypy/test/data.py +++ b/mypy/test/data.py @@ -53,11 +53,15 @@ def parse_test_cases( while i < len(p) and p[i].id != 'case': if p[i].id == 'file': # Record an extra file needed for the test case. - files.append((os.path.join(base_path, p[i].arg), + arg = p[i].arg + assert arg is not None + files.append((os.path.join(base_path, arg), '\n'.join(p[i].data))) elif p[i].id in ('builtins', 'builtins_py2'): # Use a custom source file for the std module. - mpath = os.path.join(os.path.dirname(path), p[i].arg) + arg = p[i].arg + assert arg is not None + mpath = os.path.join(os.path.dirname(path), arg) if p[i].id == 'builtins': fnam = 'builtins.pyi' else: @@ -66,15 +70,17 @@ def parse_test_cases( with open(mpath) as f: files.append((os.path.join(base_path, fnam), f.read())) elif p[i].id == 'stale': - if p[i].arg is None: + arg = p[i].arg + if arg is None: stale_modules = set() else: - stale_modules = {item.strip() for item in p[i].arg.split(',')} + stale_modules = {item.strip() for item in arg.split(',')} elif p[i].id == 'rechecked': - if p[i].arg is None: + arg = p[i].arg + if arg is None: rechecked_modules = set() else: - rechecked_modules = {item.strip() for item in p[i].arg.split(',')} + rechecked_modules = {item.strip() for item in arg.split(',')} elif p[i].id == 'out' or p[i].id == 'out1': tcout = p[i].data if native_sep and os.path.sep == '\\': @@ -95,7 +101,9 @@ def parse_test_cases( # If the set of rechecked modules isn't specified, make it the same as the set of # modules with a stale public interface. rechecked_modules = stale_modules - if stale_modules is not None and not stale_modules.issubset(rechecked_modules): + if (stale_modules is not None + and rechecked_modules is not None + and not stale_modules.issubset(rechecked_modules)): raise ValueError( 'Stale modules must be a subset of rechecked modules ({})'.format(path)) @@ -225,7 +233,7 @@ class TestItem: """ id = '' - arg = '' + arg = '' # type: Optional[str] # Text data, array of 8-bit strings data = None # type: List[str] @@ -233,7 +241,7 @@ class TestItem: file = '' line = 0 # Line number in file - def __init__(self, id: str, arg: str, data: List[str], file: str, + def __init__(self, id: str, arg: Optional[str], data: List[str], file: str, line: int) -> None: self.id = id self.arg = arg @@ -248,8 +256,8 @@ def parse_test_data(l: List[str], fnam: str) -> List[TestItem]: ret = [] # type: List[TestItem] data = [] # type: List[str] - id = None # type: str - arg = None # type: str + id = None # type: Optional[str] + arg = None # type: Optional[str] i = 0 i0 = 0 @@ -335,16 +343,22 @@ def expand_includes(a: List[str], base_path: str) -> List[str]: def expand_errors(input: List[str], output: List[str], fnam: str) -> None: - """Transform comments such as '# E: message' in input. + """Transform comments such as '# E: message' or + '# E:3: message' in input. The result is lines like 'fnam:line: error: message'. """ for i in range(len(input)): - m = re.search('# ([EN]): (.*)$', input[i]) + m = re.search('# ([EN]):((?P\d+):)? (?P.*)$', input[i]) if m: severity = 'error' if m.group(1) == 'E' else 'note' - output.append('{}:{}: {}: {}'.format(fnam, i + 1, severity, m.group(2))) + col = m.group('col') + if col is None: + output.append('{}:{}: {}: {}'.format(fnam, i + 1, severity, m.group('message'))) + else: + output.append('{}:{}:{}: {}: {}'.format( + fnam, i + 1, col, severity, m.group('message'))) def fix_win_path(line: str) -> str: diff --git a/mypy/test/testcheck.py b/mypy/test/testcheck.py index e9a326f7fa40..209076ecddf2 100644 --- a/mypy/test/testcheck.py +++ b/mypy/test/testcheck.py @@ -5,8 +5,10 @@ import shutil import sys import time +import typed_ast +import typed_ast.ast35 -from typing import Tuple, List, Dict, Set +from typing import Dict, List, Optional, Set, Tuple from mypy import build, defaults from mypy.main import parse_version, process_options @@ -18,7 +20,7 @@ assert_string_arrays_equal, normalize_error_messages, testcase_pyversion, update_testcase_output, ) -from mypy.errors import CompileError, set_show_tb +from mypy.errors import CompileError from mypy.options import Options from mypy import experiments @@ -32,7 +34,6 @@ 'check-generics.test', 'check-tuples.test', 'check-dynamic-typing.test', - 'check-weak-typing.test', 'check-functions.test', 'check-inference.test', 'check-inference-context.test', @@ -65,8 +66,12 @@ 'check-warnings.test', 'check-async-await.test', 'check-newtype.test', + 'check-columns.test', ] +if 'annotation' in typed_ast.ast35.Assign._fields: + files.append('check-newsyntax.test') + class TypeCheckSuite(DataSuite): def __init__(self, *, update_data=False): @@ -112,7 +117,7 @@ def run_case_once(self, testcase: DataDrivenTestCase, incremental=0) -> None: options = self.parse_options(original_program_text, testcase) options.use_builtins_fixtures = True - set_show_tb(True) # Show traceback on crash. + options.show_traceback = True if 'optional' in testcase.file: options.strict_optional = True @@ -143,15 +148,15 @@ def run_case_once(self, testcase: DataDrivenTestCase, incremental=0) -> None: sources = [] for module_name, program_path, program_text in module_data: # Always set to none so we're forced to reread the module in incremental mode - program_text = None if incremental else program_text - sources.append(BuildSource(program_path, module_name, program_text)) + sources.append(BuildSource(program_path, module_name, + None if incremental else program_text)) + res = None try: res = build.build(sources=sources, options=options, alt_lib_path=test_temp_dir) a = res.errors except CompileError as e: - res = None a = e.messages a = normalize_error_messages(a) @@ -185,7 +190,8 @@ def run_case_once(self, testcase: DataDrivenTestCase, incremental=0) -> None: testcase.expected_stale_modules, res.manager.stale_modules) - def check_module_equivalence(self, name: str, expected: Set[str], actual: Set[str]) -> None: + def check_module_equivalence(self, name: str, + expected: Optional[Set[str]], actual: Set[str]) -> None: if expected is not None: assert_string_arrays_equal( list(sorted(expected)), diff --git a/mypy/test/testcmdline.py b/mypy/test/testcmdline.py index d1ad9e76688e..78cc9ce3ab5c 100644 --- a/mypy/test/testcmdline.py +++ b/mypy/test/testcmdline.py @@ -44,7 +44,7 @@ def test_python_evaluation(testcase: DataDrivenTestCase) -> None: for s in testcase.input: file.write('{}\n'.format(s)) args = parse_args(testcase.input[0]) - args.append('--tb') # Show traceback on crash. + args.append('--show-traceback') # Type check the program. fixed = [python3_path, os.path.join(testcase.old_cwd, 'scripts', 'mypy')] diff --git a/mypy/test/testpythoneval.py b/mypy/test/testpythoneval.py index f5d94a8c8a94..e4dfb0231764 100644 --- a/mypy/test/testpythoneval.py +++ b/mypy/test/testpythoneval.py @@ -62,7 +62,7 @@ def test_python_evaluation(testcase): interpreter = python3_path args = [] py2 = False - args.append('--tb') # Show traceback on crash. + args.append('--show-traceback') # Write the program to a file. program = '_program.py' program_path = os.path.join(test_temp_dir, program) diff --git a/mypy/test/testsemanal.py b/mypy/test/testsemanal.py index c2f7280db8f5..3486d0fdc528 100644 --- a/mypy/test/testsemanal.py +++ b/mypy/test/testsemanal.py @@ -36,6 +36,7 @@ def get_semanal_options(): options = Options() options.use_builtins_fixtures = True options.semantic_analysis_only = True + options.show_traceback = True return options diff --git a/mypy/test/testtransform.py b/mypy/test/testtransform.py index bceee5c15b7e..782b2655126f 100644 --- a/mypy/test/testtransform.py +++ b/mypy/test/testtransform.py @@ -45,6 +45,7 @@ def test_transform(testcase): options = Options() options.use_builtins_fixtures = True options.semantic_analysis_only = True + options.show_traceback = True options.python_version = testfile_pyversion(testcase.file) result = build.build(sources=[BuildSource('main', None, src)], options=options, @@ -67,7 +68,7 @@ def test_transform(testcase): and not os.path.splitext( os.path.basename(f.path))[0].endswith('_')): t = TestTransformVisitor() - f = t.node(f) + f = t.mypyfile(f) a += str(f).split('\n') except CompileError as e: a = e.messages diff --git a/mypy/test/testtypegen.py b/mypy/test/testtypegen.py index e8278e07ef24..b176090fa983 100644 --- a/mypy/test/testtypegen.py +++ b/mypy/test/testtypegen.py @@ -39,6 +39,7 @@ def run_test(self, testcase): src = '\n'.join(testcase.input) options = Options() options.use_builtins_fixtures = True + options.show_traceback = True result = build.build(sources=[BuildSource('main', None, src)], options=options, alt_lib_path=config.test_temp_dir) diff --git a/mypy/treetransform.py b/mypy/treetransform.py index 9c090f94588f..100ff7854a8c 100644 --- a/mypy/treetransform.py +++ b/mypy/treetransform.py @@ -16,12 +16,12 @@ UnicodeExpr, FloatExpr, CallExpr, SuperExpr, MemberExpr, IndexExpr, SliceExpr, OpExpr, UnaryExpr, FuncExpr, TypeApplication, PrintStmt, SymbolTable, RefExpr, TypeVarExpr, NewTypeExpr, PromoteExpr, - ComparisonExpr, TempNode, StarExpr, + ComparisonExpr, TempNode, StarExpr, Statement, Expression, YieldFromExpr, NamedTupleExpr, NonlocalDecl, SetComprehension, DictionaryComprehension, ComplexExpr, TypeAliasExpr, EllipsisExpr, YieldExpr, ExecStmt, Argument, BackquoteExpr, AwaitExpr, ) -from mypy.types import Type, FunctionLike, Instance +from mypy.types import Type, FunctionLike from mypy.traverser import TraverserVisitor from mypy.visitor import NodeVisitor @@ -55,9 +55,9 @@ def __init__(self) -> None: # transformed node). self.func_placeholder_map = {} # type: Dict[FuncDef, FuncDef] - def visit_mypy_file(self, node: MypyFile) -> Node: + def visit_mypy_file(self, node: MypyFile) -> MypyFile: # NOTE: The 'names' and 'imports' instance variables will be empty! - new = MypyFile(self.nodes(node.defs), [], node.is_bom, + new = MypyFile(self.statements(node.defs), [], node.is_bom, ignored_lines=set(node.ignored_lines)) new._name = node._name new._fullname = node._fullname @@ -65,13 +65,13 @@ def visit_mypy_file(self, node: MypyFile) -> Node: new.names = SymbolTable() return new - def visit_import(self, node: Import) -> Node: + def visit_import(self, node: Import) -> Import: return Import(node.ids[:]) - def visit_import_from(self, node: ImportFrom) -> Node: + def visit_import_from(self, node: ImportFrom) -> ImportFrom: return ImportFrom(node.id, node.relative, node.names[:]) - def visit_import_all(self, node: ImportAll) -> Node: + def visit_import_all(self, node: ImportAll) -> ImportAll: return ImportAll(node.id, node.relative) def copy_argument(self, argument: Argument) -> Argument: @@ -80,12 +80,12 @@ def copy_argument(self, argument: Argument) -> Argument: if argument.initialization_statement: init_lvalue = cast( NameExpr, - self.node(argument.initialization_statement.lvalues[0]), + self.expr(argument.initialization_statement.lvalues[0]), ) init_lvalue.set_line(argument.line) init_stmt = AssignmentStmt( [init_lvalue], - self.node(argument.initialization_statement.rvalue), + self.expr(argument.initialization_statement.rvalue), self.optional_type(argument.initialization_statement.type), ) @@ -143,7 +143,7 @@ def visit_func_def(self, node: FuncDef) -> FuncDef: else: return new - def visit_func_expr(self, node: FuncExpr) -> Node: + def visit_func_expr(self, node: FuncExpr) -> FuncExpr: new = FuncExpr([self.copy_argument(arg) for arg in node.arguments], self.block(node.body), cast(FunctionLike, self.optional_type(node.type))) @@ -169,7 +169,7 @@ def duplicate_inits(self, result.append(None) return result - def visit_overloaded_func_def(self, node: OverloadedFuncDef) -> Node: + def visit_overloaded_func_def(self, node: OverloadedFuncDef) -> OverloadedFuncDef: items = [self.visit_decorator(decorator) for decorator in node.items] for newitem, olditem in zip(items, node.items): @@ -180,33 +180,33 @@ def visit_overloaded_func_def(self, node: OverloadedFuncDef) -> Node: new.info = node.info return new - def visit_class_def(self, node: ClassDef) -> Node: + def visit_class_def(self, node: ClassDef) -> ClassDef: new = ClassDef(node.name, self.block(node.defs), node.type_vars, - self.nodes(node.base_type_exprs), + self.expressions(node.base_type_exprs), node.metaclass) new.fullname = node.fullname new.info = node.info - new.decorators = [decorator.accept(self) + new.decorators = [self.expr(decorator) for decorator in node.decorators] new.is_builtinclass = node.is_builtinclass return new - def visit_global_decl(self, node: GlobalDecl) -> Node: + def visit_global_decl(self, node: GlobalDecl) -> GlobalDecl: return GlobalDecl(node.names[:]) - def visit_nonlocal_decl(self, node: NonlocalDecl) -> Node: + def visit_nonlocal_decl(self, node: NonlocalDecl) -> NonlocalDecl: return NonlocalDecl(node.names[:]) def visit_block(self, node: Block) -> Block: - return Block(self.nodes(node.body)) + return Block(self.statements(node.body)) def visit_decorator(self, node: Decorator) -> Decorator: # Note that a Decorator must be transformed to a Decorator. func = self.visit_func_def(node.func) func.line = node.func.line - new = Decorator(func, self.nodes(node.decorators), + new = Decorator(func, self.expressions(node.decorators), self.visit_var(node.var)) new.is_overload = node.is_overload return new @@ -229,111 +229,111 @@ def visit_var(self, node: Var) -> Var: self.var_map[node] = new return new - def visit_expression_stmt(self, node: ExpressionStmt) -> Node: - return ExpressionStmt(self.node(node.expr)) + def visit_expression_stmt(self, node: ExpressionStmt) -> ExpressionStmt: + return ExpressionStmt(self.expr(node.expr)) - def visit_assignment_stmt(self, node: AssignmentStmt) -> Node: + def visit_assignment_stmt(self, node: AssignmentStmt) -> AssignmentStmt: return self.duplicate_assignment(node) def duplicate_assignment(self, node: AssignmentStmt) -> AssignmentStmt: - new = AssignmentStmt(self.nodes(node.lvalues), - self.node(node.rvalue), + new = AssignmentStmt(self.expressions(node.lvalues), + self.expr(node.rvalue), self.optional_type(node.type)) new.line = node.line return new def visit_operator_assignment_stmt(self, - node: OperatorAssignmentStmt) -> Node: + node: OperatorAssignmentStmt) -> OperatorAssignmentStmt: return OperatorAssignmentStmt(node.op, - self.node(node.lvalue), - self.node(node.rvalue)) + self.expr(node.lvalue), + self.expr(node.rvalue)) - def visit_while_stmt(self, node: WhileStmt) -> Node: - return WhileStmt(self.node(node.expr), + def visit_while_stmt(self, node: WhileStmt) -> WhileStmt: + return WhileStmt(self.expr(node.expr), self.block(node.body), self.optional_block(node.else_body)) - def visit_for_stmt(self, node: ForStmt) -> Node: - return ForStmt(self.node(node.index), - self.node(node.expr), + def visit_for_stmt(self, node: ForStmt) -> ForStmt: + return ForStmt(self.expr(node.index), + self.expr(node.expr), self.block(node.body), self.optional_block(node.else_body)) - def visit_return_stmt(self, node: ReturnStmt) -> Node: - return ReturnStmt(self.optional_node(node.expr)) + def visit_return_stmt(self, node: ReturnStmt) -> ReturnStmt: + return ReturnStmt(self.optional_expr(node.expr)) - def visit_assert_stmt(self, node: AssertStmt) -> Node: - return AssertStmt(self.node(node.expr)) + def visit_assert_stmt(self, node: AssertStmt) -> AssertStmt: + return AssertStmt(self.expr(node.expr)) - def visit_del_stmt(self, node: DelStmt) -> Node: - return DelStmt(self.node(node.expr)) + def visit_del_stmt(self, node: DelStmt) -> DelStmt: + return DelStmt(self.expr(node.expr)) - def visit_if_stmt(self, node: IfStmt) -> Node: - return IfStmt(self.nodes(node.expr), + def visit_if_stmt(self, node: IfStmt) -> IfStmt: + return IfStmt(self.expressions(node.expr), self.blocks(node.body), self.optional_block(node.else_body)) - def visit_break_stmt(self, node: BreakStmt) -> Node: + def visit_break_stmt(self, node: BreakStmt) -> BreakStmt: return BreakStmt() - def visit_continue_stmt(self, node: ContinueStmt) -> Node: + def visit_continue_stmt(self, node: ContinueStmt) -> ContinueStmt: return ContinueStmt() - def visit_pass_stmt(self, node: PassStmt) -> Node: + def visit_pass_stmt(self, node: PassStmt) -> PassStmt: return PassStmt() - def visit_raise_stmt(self, node: RaiseStmt) -> Node: - return RaiseStmt(self.optional_node(node.expr), - self.optional_node(node.from_expr)) + def visit_raise_stmt(self, node: RaiseStmt) -> RaiseStmt: + return RaiseStmt(self.optional_expr(node.expr), + self.optional_expr(node.from_expr)) - def visit_try_stmt(self, node: TryStmt) -> Node: + def visit_try_stmt(self, node: TryStmt) -> TryStmt: return TryStmt(self.block(node.body), self.optional_names(node.vars), - self.optional_nodes(node.types), + self.optional_expressions(node.types), self.blocks(node.handlers), self.optional_block(node.else_body), self.optional_block(node.finally_body)) - def visit_with_stmt(self, node: WithStmt) -> Node: - return WithStmt(self.nodes(node.expr), - self.optional_nodes(node.target), + def visit_with_stmt(self, node: WithStmt) -> WithStmt: + return WithStmt(self.expressions(node.expr), + self.optional_expressions(node.target), self.block(node.body)) - def visit_print_stmt(self, node: PrintStmt) -> Node: - return PrintStmt(self.nodes(node.args), + def visit_print_stmt(self, node: PrintStmt) -> PrintStmt: + return PrintStmt(self.expressions(node.args), node.newline, - self.optional_node(node.target)) + self.optional_expr(node.target)) - def visit_exec_stmt(self, node: ExecStmt) -> Node: - return ExecStmt(self.node(node.expr), - self.optional_node(node.variables1), - self.optional_node(node.variables2)) + def visit_exec_stmt(self, node: ExecStmt) -> ExecStmt: + return ExecStmt(self.expr(node.expr), + self.optional_expr(node.variables1), + self.optional_expr(node.variables2)) - def visit_star_expr(self, node: StarExpr) -> Node: + def visit_star_expr(self, node: StarExpr) -> StarExpr: return StarExpr(node.expr) - def visit_int_expr(self, node: IntExpr) -> Node: + def visit_int_expr(self, node: IntExpr) -> IntExpr: return IntExpr(node.value) - def visit_str_expr(self, node: StrExpr) -> Node: + def visit_str_expr(self, node: StrExpr) -> StrExpr: return StrExpr(node.value) - def visit_bytes_expr(self, node: BytesExpr) -> Node: + def visit_bytes_expr(self, node: BytesExpr) -> BytesExpr: return BytesExpr(node.value) - def visit_unicode_expr(self, node: UnicodeExpr) -> Node: + def visit_unicode_expr(self, node: UnicodeExpr) -> UnicodeExpr: return UnicodeExpr(node.value) - def visit_float_expr(self, node: FloatExpr) -> Node: + def visit_float_expr(self, node: FloatExpr) -> FloatExpr: return FloatExpr(node.value) - def visit_complex_expr(self, node: ComplexExpr) -> Node: + def visit_complex_expr(self, node: ComplexExpr) -> ComplexExpr: return ComplexExpr(node.value) - def visit_ellipsis(self, node: EllipsisExpr) -> Node: + def visit_ellipsis(self, node: EllipsisExpr) -> EllipsisExpr: return EllipsisExpr() - def visit_name_expr(self, node: NameExpr) -> Node: + def visit_name_expr(self, node: NameExpr) -> NameExpr: return self.duplicate_name(node) def duplicate_name(self, node: NameExpr) -> NameExpr: @@ -343,8 +343,8 @@ def duplicate_name(self, node: NameExpr) -> NameExpr: self.copy_ref(new, node) return new - def visit_member_expr(self, node: MemberExpr) -> Node: - member = MemberExpr(self.node(node.expr), + def visit_member_expr(self, node: MemberExpr) -> MemberExpr: + member = MemberExpr(self.expr(node.expr), node.name) if node.def_var: member.def_var = self.visit_var(node.def_var) @@ -363,64 +363,64 @@ def copy_ref(self, new: RefExpr, original: RefExpr) -> None: new.node = target new.is_def = original.is_def - def visit_yield_from_expr(self, node: YieldFromExpr) -> Node: - return YieldFromExpr(self.node(node.expr)) + def visit_yield_from_expr(self, node: YieldFromExpr) -> YieldFromExpr: + return YieldFromExpr(self.expr(node.expr)) - def visit_yield_expr(self, node: YieldExpr) -> Node: - return YieldExpr(self.node(node.expr)) + def visit_yield_expr(self, node: YieldExpr) -> YieldExpr: + return YieldExpr(self.expr(node.expr)) - def visit_await_expr(self, node: AwaitExpr) -> Node: - return AwaitExpr(self.node(node.expr)) + def visit_await_expr(self, node: AwaitExpr) -> AwaitExpr: + return AwaitExpr(self.expr(node.expr)) - def visit_call_expr(self, node: CallExpr) -> Node: - return CallExpr(self.node(node.callee), - self.nodes(node.args), + def visit_call_expr(self, node: CallExpr) -> CallExpr: + return CallExpr(self.expr(node.callee), + self.expressions(node.args), node.arg_kinds[:], node.arg_names[:], - self.optional_node(node.analyzed)) + self.optional_expr(node.analyzed)) - def visit_op_expr(self, node: OpExpr) -> Node: - new = OpExpr(node.op, self.node(node.left), self.node(node.right)) + def visit_op_expr(self, node: OpExpr) -> OpExpr: + new = OpExpr(node.op, self.expr(node.left), self.expr(node.right)) new.method_type = self.optional_type(node.method_type) return new - def visit_comparison_expr(self, node: ComparisonExpr) -> Node: - new = ComparisonExpr(node.operators, self.nodes(node.operands)) + def visit_comparison_expr(self, node: ComparisonExpr) -> ComparisonExpr: + new = ComparisonExpr(node.operators, self.expressions(node.operands)) new.method_types = [self.optional_type(t) for t in node.method_types] return new - def visit_cast_expr(self, node: CastExpr) -> Node: - return CastExpr(self.node(node.expr), + def visit_cast_expr(self, node: CastExpr) -> CastExpr: + return CastExpr(self.expr(node.expr), self.type(node.type)) - def visit_reveal_type_expr(self, node: RevealTypeExpr) -> Node: - return RevealTypeExpr(self.node(node.expr)) + def visit_reveal_type_expr(self, node: RevealTypeExpr) -> RevealTypeExpr: + return RevealTypeExpr(self.expr(node.expr)) - def visit_super_expr(self, node: SuperExpr) -> Node: + def visit_super_expr(self, node: SuperExpr) -> SuperExpr: new = SuperExpr(node.name) new.info = node.info return new - def visit_unary_expr(self, node: UnaryExpr) -> Node: - new = UnaryExpr(node.op, self.node(node.expr)) + def visit_unary_expr(self, node: UnaryExpr) -> UnaryExpr: + new = UnaryExpr(node.op, self.expr(node.expr)) new.method_type = self.optional_type(node.method_type) return new - def visit_list_expr(self, node: ListExpr) -> Node: - return ListExpr(self.nodes(node.items)) + def visit_list_expr(self, node: ListExpr) -> ListExpr: + return ListExpr(self.expressions(node.items)) - def visit_dict_expr(self, node: DictExpr) -> Node: - return DictExpr([(self.node(key), self.node(value)) + def visit_dict_expr(self, node: DictExpr) -> DictExpr: + return DictExpr([(self.expr(key), self.expr(value)) for key, value in node.items]) - def visit_tuple_expr(self, node: TupleExpr) -> Node: - return TupleExpr(self.nodes(node.items)) + def visit_tuple_expr(self, node: TupleExpr) -> TupleExpr: + return TupleExpr(self.expressions(node.items)) - def visit_set_expr(self, node: SetExpr) -> Node: - return SetExpr(self.nodes(node.items)) + def visit_set_expr(self, node: SetExpr) -> SetExpr: + return SetExpr(self.expressions(node.items)) - def visit_index_expr(self, node: IndexExpr) -> Node: - new = IndexExpr(self.node(node.base), self.node(node.index)) + def visit_index_expr(self, node: IndexExpr) -> IndexExpr: + new = IndexExpr(self.expr(node.base), self.expr(node.index)) if node.method_type: new.method_type = self.type(node.method_type) if node.analyzed: @@ -432,50 +432,51 @@ def visit_index_expr(self, node: IndexExpr) -> Node: return new def visit_type_application(self, node: TypeApplication) -> TypeApplication: - return TypeApplication(self.node(node.expr), + return TypeApplication(self.expr(node.expr), self.types(node.types)) - def visit_list_comprehension(self, node: ListComprehension) -> Node: + def visit_list_comprehension(self, node: ListComprehension) -> ListComprehension: generator = self.duplicate_generator(node.generator) generator.set_line(node.generator.line) return ListComprehension(generator) - def visit_set_comprehension(self, node: SetComprehension) -> Node: + def visit_set_comprehension(self, node: SetComprehension) -> SetComprehension: generator = self.duplicate_generator(node.generator) generator.set_line(node.generator.line) return SetComprehension(generator) - def visit_dictionary_comprehension(self, node: DictionaryComprehension) -> Node: - return DictionaryComprehension(self.node(node.key), self.node(node.value), - [self.node(index) for index in node.indices], - [self.node(s) for s in node.sequences], - [[self.node(cond) for cond in conditions] + def visit_dictionary_comprehension(self, node: DictionaryComprehension + ) -> DictionaryComprehension: + return DictionaryComprehension(self.expr(node.key), self.expr(node.value), + [self.expr(index) for index in node.indices], + [self.expr(s) for s in node.sequences], + [[self.expr(cond) for cond in conditions] for conditions in node.condlists]) - def visit_generator_expr(self, node: GeneratorExpr) -> Node: + def visit_generator_expr(self, node: GeneratorExpr) -> GeneratorExpr: return self.duplicate_generator(node) def duplicate_generator(self, node: GeneratorExpr) -> GeneratorExpr: - return GeneratorExpr(self.node(node.left_expr), - [self.node(index) for index in node.indices], - [self.node(s) for s in node.sequences], - [[self.node(cond) for cond in conditions] + return GeneratorExpr(self.expr(node.left_expr), + [self.expr(index) for index in node.indices], + [self.expr(s) for s in node.sequences], + [[self.expr(cond) for cond in conditions] for conditions in node.condlists]) - def visit_slice_expr(self, node: SliceExpr) -> Node: - return SliceExpr(self.optional_node(node.begin_index), - self.optional_node(node.end_index), - self.optional_node(node.stride)) + def visit_slice_expr(self, node: SliceExpr) -> SliceExpr: + return SliceExpr(self.optional_expr(node.begin_index), + self.optional_expr(node.end_index), + self.optional_expr(node.stride)) - def visit_conditional_expr(self, node: ConditionalExpr) -> Node: - return ConditionalExpr(self.node(node.cond), - self.node(node.if_expr), - self.node(node.else_expr)) + def visit_conditional_expr(self, node: ConditionalExpr) -> ConditionalExpr: + return ConditionalExpr(self.expr(node.cond), + self.expr(node.if_expr), + self.expr(node.else_expr)) - def visit_backquote_expr(self, node: BackquoteExpr) -> Node: - return BackquoteExpr(self.node(node.expr)) + def visit_backquote_expr(self, node: BackquoteExpr) -> BackquoteExpr: + return BackquoteExpr(self.expr(node.expr)) - def visit_type_var_expr(self, node: TypeVarExpr) -> Node: + def visit_type_var_expr(self, node: TypeVarExpr) -> TypeVarExpr: return TypeVarExpr(node.name(), node.fullname(), self.types(node.values), self.type(node.upper_bound), variance=node.variance) @@ -488,13 +489,13 @@ def visit_newtype_expr(self, node: NewTypeExpr) -> NewTypeExpr: res.info = node.info return res - def visit_namedtuple_expr(self, node: NamedTupleExpr) -> Node: + def visit_namedtuple_expr(self, node: NamedTupleExpr) -> NamedTupleExpr: return NamedTupleExpr(node.info) - def visit__promote_expr(self, node: PromoteExpr) -> Node: + def visit__promote_expr(self, node: PromoteExpr) -> PromoteExpr: return PromoteExpr(node.type) - def visit_temp_node(self, node: TempNode) -> Node: + def visit_temp_node(self, node: TempNode) -> TempNode: return TempNode(self.type(node.type)) def node(self, node: Node) -> Node: @@ -502,13 +503,31 @@ def node(self, node: Node) -> Node: new.set_line(node.line) return new + def mypyfile(self, node: MypyFile) -> MypyFile: + new = node.accept(self) + assert isinstance(new, MypyFile) + new.set_line(node.line) + return new + + def expr(self, expr: Expression) -> Expression: + new = expr.accept(self) + assert isinstance(new, Expression) + new.set_line(expr.line) + return new + + def stmt(self, stmt: Statement) -> Statement: + new = stmt.accept(self) + assert isinstance(new, Statement) + new.set_line(stmt.line) + return new + # Helpers # # All the node helpers also propagate line numbers. - def optional_node(self, node: Node) -> Node: - if node: - return self.node(node) + def optional_expr(self, expr: Expression) -> Expression: + if expr: + return self.expr(expr) else: return None @@ -523,11 +542,14 @@ def optional_block(self, block: Block) -> Block: else: return None - def nodes(self, nodes: List[Node]) -> List[Node]: - return [self.node(node) for node in nodes] + def statements(self, statements: List[Statement]) -> List[Statement]: + return [self.stmt(stmt) for stmt in statements] + + def expressions(self, expressions: List[Expression]) -> List[Expression]: + return [self.expr(expr) for expr in expressions] - def optional_nodes(self, nodes: List[Node]) -> List[Node]: - return [self.optional_node(node) for node in nodes] + def optional_expressions(self, expressions: List[Expression]) -> List[Expression]: + return [self.optional_expr(expr) for expr in expressions] def blocks(self, blocks: List[Block]) -> List[Block]: return [self.block(block) for block in blocks] diff --git a/mypy/typeanal.py b/mypy/typeanal.py index 0fe5c451f2b2..931cf7cf5363 100644 --- a/mypy/typeanal.py +++ b/mypy/typeanal.py @@ -9,7 +9,7 @@ ) from mypy.nodes import ( BOUND_TVAR, TYPE_ALIAS, UNBOUND_IMPORTED, - TypeInfo, Context, SymbolTableNode, Var, Node, + TypeInfo, Context, SymbolTableNode, Var, Expression, IndexExpr, RefExpr ) from mypy.sametypes import is_same_type @@ -28,7 +28,7 @@ } -def analyze_type_alias(node: Node, +def analyze_type_alias(node: Expression, lookup_func: Callable[[str, Context], SymbolTableNode], lookup_fqn_func: Callable[[str], SymbolTableNode], fail_func: Callable[[str, Context], None]) -> Type: @@ -107,6 +107,9 @@ def visit_unbound_type(self, t: UnboundType) -> Type: elif fullname == 'typing.Any': return AnyType() elif fullname == 'typing.Tuple': + if len(t.args) == 0 and not t.empty_tuple_index: + # Bare 'Tuple' is same as 'tuple' + return self.builtin_type('builtins.tuple') if len(t.args) == 2 and isinstance(t.args[1], EllipsisType): # Tuple[T, ...] (uniform, variable-length tuple) node = self.lookup_fqn_func('builtins.tuple') diff --git a/mypy/types.py b/mypy/types.py index 0b6c46970e7f..09e473d213f4 100644 --- a/mypy/types.py +++ b/mypy/types.py @@ -21,15 +21,20 @@ class Type(mypy.nodes.Context): """Abstract base class for all types.""" line = 0 + column = 0 can_be_true = True can_be_false = True - def __init__(self, line: int = -1) -> None: + def __init__(self, line: int = -1, column: int = -1) -> None: self.line = line + self.column = column def get_line(self) -> int: return self.line + def get_column(self) -> int: + return self.column + def accept(self, visitor: 'TypeVisitor[T]') -> T: raise RuntimeError('Not implemented') @@ -110,9 +115,11 @@ class TypeVarDef(mypy.nodes.Context): upper_bound = None # type: Type variance = INVARIANT # type: int line = 0 + column = 0 def __init__(self, name: str, id: Union[TypeVarId, int], values: Optional[List[Type]], - upper_bound: Type, variance: int = INVARIANT, line: int = -1) -> None: + upper_bound: Type, variance: int = INVARIANT, line: int = -1, + column: int = -1) -> None: self.name = name if isinstance(id, int): id = TypeVarId(id) @@ -121,16 +128,20 @@ def __init__(self, name: str, id: Union[TypeVarId, int], values: Optional[List[T self.upper_bound = upper_bound self.variance = variance self.line = line + self.column = column @staticmethod def new_unification_variable(old: 'TypeVarDef') -> 'TypeVarDef': new_id = TypeVarId.new(meta_level=1) return TypeVarDef(old.name, new_id, old.values, - old.upper_bound, old.variance, old.line) + old.upper_bound, old.variance, old.line, old.column) def get_line(self) -> int: return self.line + def get_column(self) -> int: + return self.column + def __repr__(self) -> str: if self.values: return '{} in {}'.format(self.name, tuple(self.values)) @@ -170,20 +181,25 @@ class UnboundType(Type): optional = False # is this type a return type? is_ret_type = False + # special case for X[()] + empty_tuple_index = False def __init__(self, name: str, args: List[Type] = None, line: int = -1, + column: int = -1, optional: bool = False, - is_ret_type: bool = False) -> None: + is_ret_type: bool = False, + empty_tuple_index: bool = False) -> None: if not args: args = [] self.name = name self.args = args self.optional = optional self.is_ret_type = is_ret_type - super().__init__(line) + self.empty_tuple_index = empty_tuple_index + super().__init__(line, column) def accept(self, visitor: 'TypeVisitor[T]') -> T: return visitor.visit_unbound_type(self) @@ -218,8 +234,8 @@ class TypeList(Type): items = None # type: List[Type] - def __init__(self, items: List[Type], line: int = -1) -> None: - super().__init__(line) + def __init__(self, items: List[Type], line: int = -1, column: int = -1) -> None: + super().__init__(line, column) self.items = items def accept(self, visitor: 'TypeVisitor[T]') -> T: @@ -239,8 +255,8 @@ def deserialize(self, data: JsonDict) -> 'TypeList': class AnyType(Type): """The type 'Any'.""" - def __init__(self, implicit: bool = False, line: int = -1) -> None: - super().__init__(line) + def __init__(self, implicit: bool = False, line: int = -1, column: int = -1) -> None: + super().__init__(line, column) self.implicit = implicit def accept(self, visitor: 'TypeVisitor[T]') -> T: @@ -265,15 +281,15 @@ class Void(Type): can_be_true = False source = '' # May be None; function that generated this value - def __init__(self, source: str = None, line: int = -1) -> None: + def __init__(self, source: str = None, line: int = -1, column: int = -1) -> None: self.source = source - super().__init__(line) + super().__init__(line, column) def accept(self, visitor: 'TypeVisitor[T]') -> T: return visitor.visit_void(self) def with_source(self, source: str) -> 'Void': - return Void(source, self.line) + return Void(source, self.line, self.column) def serialize(self) -> JsonDict: return {'.class': 'Void'} @@ -301,8 +317,8 @@ class UninhabitedType(Type): can_be_true = False can_be_false = False - def __init__(self, line: int = -1) -> None: - super().__init__(line) + def __init__(self, line: int = -1, column: int = -1) -> None: + super().__init__(line, column) def accept(self, visitor: 'TypeVisitor[T]') -> T: return visitor.visit_uninhabited_type(self) @@ -336,8 +352,8 @@ class NoneTyp(Type): can_be_true = False - def __init__(self, is_ret_type: bool = False, line: int = -1) -> None: - super().__init__(line) + def __init__(self, is_ret_type: bool = False, line: int = -1, column: int = -1) -> None: + super().__init__(line, column) self.is_ret_type = is_ret_type def accept(self, visitor: 'TypeVisitor[T]') -> T: @@ -373,9 +389,9 @@ class DeletedType(Type): source = '' # May be None; name that generated this value - def __init__(self, source: str = None, line: int = -1) -> None: + def __init__(self, source: str = None, line: int = -1, column: int = -1) -> None: self.source = source - super().__init__(line) + super().__init__(line, column) def accept(self, visitor: 'TypeVisitor[T]') -> T: return visitor.visit_deleted_type(self) @@ -401,11 +417,11 @@ class Instance(Type): erased = False # True if result of type variable substitution def __init__(self, typ: mypy.nodes.TypeInfo, args: List[Type], - line: int = -1, erased: bool = False) -> None: + line: int = -1, column: int = -1, erased: bool = False) -> None: self.type = typ self.args = args self.erased = erased - super().__init__(line) + super().__init__(line, column) def accept(self, visitor: 'TypeVisitor[T]') -> T: return visitor.visit_instance(self) @@ -448,13 +464,13 @@ class TypeVarType(Type): # See comments in TypeVarDef for more about variance. variance = INVARIANT # type: int - def __init__(self, binder: TypeVarDef, line: int = -1) -> None: + def __init__(self, binder: TypeVarDef, line: int = -1, column: int = -1) -> None: self.name = binder.name self.id = binder.id self.values = binder.values self.upper_bound = binder.upper_bound self.variance = binder.variance - super().__init__(line) + super().__init__(line, column) def accept(self, visitor: 'TypeVisitor[T]') -> T: return visitor.visit_type_var(self) @@ -544,13 +560,14 @@ class CallableType(FunctionLike): def __init__(self, arg_types: List[Type], arg_kinds: List[int], - arg_names: List[str], + arg_names: List[Optional[str]], ret_type: Type, fallback: Instance, name: str = None, definition: SymbolNode = None, variables: List[TypeVarDef] = None, line: int = -1, + column: int = -1, is_ellipsis_args: bool = False, implicit: bool = False, is_classmethod_class: bool = False, @@ -572,7 +589,7 @@ def __init__(self, self.is_ellipsis_args = is_ellipsis_args self.implicit = implicit self.special_sig = special_sig - super().__init__(line) + super().__init__(line, column) def copy_modified(self, arg_types: List[Type] = _dummy, @@ -584,6 +601,7 @@ def copy_modified(self, definition: SymbolNode = _dummy, variables: List[TypeVarDef] = _dummy, line: int = _dummy, + column: int = _dummy, is_ellipsis_args: bool = _dummy, special_sig: Optional[str] = _dummy) -> 'CallableType': return CallableType( @@ -596,6 +614,7 @@ def copy_modified(self, definition=definition if definition is not _dummy else self.definition, variables=variables if variables is not _dummy else self.variables, line=line if line is not _dummy else self.line, + column=column if column is not _dummy else self.column, is_ellipsis_args=( is_ellipsis_args if is_ellipsis_args is not _dummy else self.is_ellipsis_args), implicit=self.implicit, @@ -694,7 +713,7 @@ class Overloaded(FunctionLike): def __init__(self, items: List[CallableType]) -> None: self._items = items self.fallback = items[0].fallback - super().__init__(items[0].line) + super().__init__(items[0].line, items[0].column) def items(self) -> List[CallableType]: return self._items @@ -748,13 +767,13 @@ class TupleType(Type): implicit = False def __init__(self, items: List[Type], fallback: Instance, line: int = -1, - implicit: bool = False) -> None: + column: int = -1, implicit: bool = False) -> None: self.items = items self.fallback = fallback self.implicit = implicit self.can_be_true = len(self.items) > 0 self.can_be_false = len(self.items) == 0 - super().__init__(line) + super().__init__(line, column) def length(self) -> int: return len(self.items) @@ -782,11 +801,11 @@ def copy_modified(self, *, fallback: Instance = None, fallback = self.fallback if items is None: items = self.items - return TupleType(items, fallback, self.line) + return TupleType(items, fallback, self.line, self.column) def slice(self, begin: int, stride: int, end: int) -> 'TupleType': return TupleType(self.items[begin:end:stride], self.fallback, - self.line, self.implicit) + self.line, self.column, self.implicit) class StarType(Type): @@ -797,9 +816,9 @@ class StarType(Type): type = None # type: Type - def __init__(self, type: Type, line: int = -1) -> None: + def __init__(self, type: Type, line: int = -1, column: int = -1) -> None: self.type = type - super().__init__(line) + super().__init__(line, column) def accept(self, visitor: 'TypeVisitor[T]') -> T: return visitor.visit_star_type(self) @@ -810,16 +829,16 @@ class UnionType(Type): items = None # type: List[Type] - def __init__(self, items: List[Type], line: int = -1) -> None: + def __init__(self, items: List[Type], line: int = -1, column: int = -1) -> None: self.items = items self.can_be_true = any(item.can_be_true for item in items) self.can_be_false = any(item.can_be_false for item in items) - super().__init__(line) + super().__init__(line, column) @staticmethod - def make_union(items: List[Type], line: int = -1) -> Type: + def make_union(items: List[Type], line: int = -1, column: int = -1) -> Type: if len(items) > 1: - return UnionType(items, line) + return UnionType(items, line, column) elif len(items) == 1: return items[0] else: @@ -829,7 +848,7 @@ def make_union(items: List[Type], line: int = -1) -> Type: return Void() @staticmethod - def make_simplified_union(items: List[Type], line: int = -1) -> Type: + def make_simplified_union(items: List[Type], line: int = -1, column: int = -1) -> Type: while any(isinstance(typ, UnionType) for typ in items): all_items = [] # type: List[Type] for typ in items: @@ -973,8 +992,8 @@ class TypeType(Type): # a generic class instance, a union, Any, a type variable... item = None # type: Type - def __init__(self, item: Type, *, line: int = -1) -> None: - super().__init__(line) + def __init__(self, item: Type, *, line: int = -1, column: int = -1) -> None: + super().__init__(line, column) self.item = item def accept(self, visitor: 'TypeVisitor[T]') -> T: @@ -1112,7 +1131,7 @@ def visit_deleted_type(self, t: DeletedType) -> Type: return t def visit_instance(self, t: Instance) -> Type: - return Instance(t.type, self.translate_types(t.args), t.line) + return Instance(t.type, self.translate_types(t.args), t.line, t.column) def visit_type_var(self, t: TypeVarType) -> Type: return t @@ -1128,13 +1147,13 @@ def visit_callable_type(self, t: CallableType) -> Type: def visit_tuple_type(self, t: TupleType) -> Type: return TupleType(self.translate_types(t.items), cast(Any, t.fallback.accept(self)), - t.line) + t.line, t.column) def visit_star_type(self, t: StarType) -> Type: - return StarType(t.type.accept(self), t.line) + return StarType(t.type.accept(self), t.line, t.column) def visit_union_type(self, t: UnionType) -> Type: - return UnionType(self.translate_types(t.items), t.line) + return UnionType(self.translate_types(t.items), t.line, t.column) def visit_ellipsis_type(self, t: EllipsisType) -> Type: return t @@ -1157,7 +1176,7 @@ def visit_overloaded(self, t: Overloaded) -> Type: return Overloaded(items=items) def visit_type_type(self, t: TypeType) -> Type: - return TypeType(t.item.accept(self), line=t.line) + return TypeType(t.item.accept(self), line=t.line, column=t.column) class TypeStrVisitor(TypeVisitor[str]): @@ -1446,14 +1465,14 @@ def true_only(t: Type) -> Type: """ if not t.can_be_true: # All values of t are False-ish, so there are no true values in it - return UninhabitedType(line=t.line) + return UninhabitedType(line=t.line, column=t.column) elif not t.can_be_false: # All values of t are already True-ish, so true_only is idempotent in this case return t elif isinstance(t, UnionType): # The true version of a union type is the union of the true versions of its components new_items = [true_only(item) for item in t.items] - return UnionType.make_simplified_union(new_items, line=t.line) + return UnionType.make_simplified_union(new_items, line=t.line, column=t.column) else: new_t = copy_type(t) new_t.can_be_false = False @@ -1473,7 +1492,7 @@ def false_only(t: Type) -> Type: elif isinstance(t, UnionType): # The false version of a union type is the union of the false versions of its components new_items = [false_only(item) for item in t.items] - return UnionType.make_simplified_union(new_items, line=t.line) + return UnionType.make_simplified_union(new_items, line=t.line, column=t.column) else: new_t = copy_type(t) new_t.can_be_true = False diff --git a/runtests.py b/runtests.py index 8abc636a0206..08c74d1c4822 100755 --- a/runtests.py +++ b/runtests.py @@ -77,7 +77,7 @@ def add_mypy_cmd(self, name: str, mypy_args: List[str], cwd: Optional[str] = Non if not self.allow(full_name): return args = [sys.executable, self.mypy] + mypy_args - args.append('--tb') # Show traceback on crash. + args.append('--show-traceback') self.waiter.add(LazySubprocess(full_name, args, cwd=cwd, env=self.env)) def add_mypy(self, name: str, *args: str, cwd: Optional[str] = None) -> None: @@ -88,8 +88,8 @@ def add_mypy_modules(self, name: str, modules: Iterable[str], args = list(itertools.chain(*(['-m', mod] for mod in modules))) self.add_mypy_cmd(name, args, cwd=cwd) - def add_mypy_package(self, name: str, packagename: str) -> None: - self.add_mypy_cmd(name, ['-p', packagename]) + def add_mypy_package(self, name: str, packagename: str, *flags: str) -> None: + self.add_mypy_cmd(name, ['-p', packagename] + list(flags)) def add_mypy_string(self, name: str, *args: str, cwd: Optional[str] = None) -> None: self.add_mypy_cmd(name, ['-c'] + list(args), cwd=cwd) @@ -168,6 +168,7 @@ def add_basic(driver: Driver) -> None: def add_selftypecheck(driver: Driver) -> None: driver.add_mypy_package('package mypy', 'mypy') + driver.add_mypy_package('package mypy', 'mypy', '--strict-optional') def find_files(base: str, prefix: str = '', suffix: str = '') -> List[str]: diff --git a/test-data/unit/check-async-await.test b/test-data/unit/check-async-await.test index 3742b2b180dc..e74c9e53b1b1 100644 --- a/test-data/unit/check-async-await.test +++ b/test-data/unit/check-async-await.test @@ -316,7 +316,7 @@ main: note: In function "f": -- ------------------------------------------ [case testFullCoroutineMatrix] -# flags: --fast-parser --suppress-error-context +# flags: --fast-parser --hide-error-context from typing import Any, AsyncIterator, Awaitable, Generator, Iterator from types import coroutine diff --git a/test-data/unit/check-classes.test b/test-data/unit/check-classes.test index 7533c67ffaa9..5792c7c966a6 100644 --- a/test-data/unit/check-classes.test +++ b/test-data/unit/check-classes.test @@ -1510,8 +1510,8 @@ def error(u_c: Type[U]) -> P: [out] main:11: error: Revealed type is '__main__.WizUser*' main: note: In function "error": -main:13: error: Type argument 1 of "new_pro" has incompatible value "U" main:13: error: Incompatible return value type (got "U", expected "P") +main:13: error: Type argument 1 of "new_pro" has incompatible value "U" [case testTypeUsingTypeCCovariance] from typing import Type, TypeVar diff --git a/test-data/unit/check-columns.test b/test-data/unit/check-columns.test new file mode 100644 index 000000000000..e9ded7161389 --- /dev/null +++ b/test-data/unit/check-columns.test @@ -0,0 +1,73 @@ +[case testColumnsSyntaxError] +# flags: --show-column-numbers +1 + +[out] +main:2:3: error: Parse error before end of line + + +[case testColumnsNestedFunctions] +# flags: --show-column-numbers +import typing +def f() -> 'A': + def g() -> 'B': + return A() # fail + return B() # fail +class A: pass +class B: pass +[out] +main: note: In function "g": +main:5:8: error: Incompatible return value type (got "A", expected "B") +main: note: In function "f": +main:6:4: error: Incompatible return value type (got "B", expected "A") + +[case testColumnsNestedFunctionsWithFastParse] +# flags: --show-column-numbers --fast-parser +import typing +def f() -> 'A': + def g() -> 'B': + return A() # fail + return B() # fail +class A: pass +class B: pass +[out] +main: note: In function "g": +main:5:8: error: Incompatible return value type (got "A", expected "B") +main: note: In function "f": +main:6:4: error: Incompatible return value type (got "B", expected "A") + + +[case testColumnsMethodDefaultArgumentsAndSignatureAsComment] +# flags: --show-column-numbers +import typing +class A: + def f(self, x = 1, y = 'hello'): # type: (int, str) -> str + pass +A().f() +A().f(1) +A().f('') # E:5: Argument 1 to "f" of "A" has incompatible type "str"; expected "int" +A().f(1, 1) # E:5: Argument 2 to "f" of "A" has incompatible type "int"; expected "str" +A().f(1, 'hello', 'hi') # E:5: Too many arguments for "f" of "A" + +[case testColumnsMultipleStatementsPerLine] +# flags: --show-column-numbers +x = 1 +y = 'hello' +x = 2; y = x; y += 1 +[out] +main:4:7: error: Incompatible types in assignment (expression has type "int", variable has type "str") +main:4:14: error: Unsupported operand types for + ("str" and "int") + +[case testColumnsSimpleIsinstance] +# flags: --show-column-numbers +import typing +def f(x: object, n: int, s: str) -> None: + n = x # E:4: Incompatible types in assignment (expression has type "object", variable has type "int") + if isinstance(x, int): + n = x + s = x # E:8: Incompatible types in assignment (expression has type "int", variable has type "str") + n = x # E:4: Incompatible types in assignment (expression has type "object", variable has type "int") +[builtins fixtures/isinstance.pyi] +[out] +main: note: In function "f": + + diff --git a/test-data/unit/check-expressions.test b/test-data/unit/check-expressions.test index 2ec8f8b9641d..2878e42ab26f 100644 --- a/test-data/unit/check-expressions.test +++ b/test-data/unit/check-expressions.test @@ -1111,6 +1111,19 @@ main:4: error: Incompatible types in string interpolation (expression has type " [case testStringInterpolationSpaceKey] '%( )s' % {' ': 'foo'} +[case testByteByteInterpolation] +def foo(a: bytes, b: bytes): + b'%s:%s' % (a, b) +foo(b'a', b'b') == b'a:b' + +[case testBytePercentInterpolationSupported] +b'%s' % (b'xyz',) +b'%(name)s' % {'name': 'jane'} +b'%c' % (123) + +[case testUnicodeInterpolation_python2] +u'%s' % (u'abc',) + -- Lambdas -- ------- @@ -1122,8 +1135,8 @@ f = lambda: ''.x f = lambda: '' [out] main:3: error: "str" has no attribute "x" -main:4: error: Incompatible return value type (got "str", expected "int") main:4: error: Incompatible types in assignment (expression has type Callable[[], str], variable has type Callable[[], int]) +main:4: error: Incompatible return value type (got "str", expected "int") [case testVoidLambda] import typing @@ -1246,8 +1259,8 @@ class C: pass def f(b: A) -> C: pass [builtins fixtures/dict.pyi] [out] -main:4: error: Argument 1 to "f" has incompatible type "B"; expected "A" main:4: error: Value expression in dictionary comprehension has incompatible type "C"; expected type "B" +main:4: error: Argument 1 to "f" has incompatible type "B"; expected "A" -- Generator expressions @@ -1565,8 +1578,8 @@ reveal_type(1) # E: Revealed type is 'builtins.int' [case testUndefinedRevealType] reveal_type(x) [out] -main:1: error: Name 'x' is not defined main:1: error: Revealed type is 'Any' +main:1: error: Name 'x' is not defined [case testUserDefinedRevealType] def reveal_type(x: int) -> None: pass diff --git a/test-data/unit/check-incremental.test b/test-data/unit/check-incremental.test index 37e52560f1ed..6dcd613b7453 100644 --- a/test-data/unit/check-incremental.test +++ b/test-data/unit/check-incremental.test @@ -1567,3 +1567,51 @@ class Outer: [out2] main:1: note: In module imported here: tmp/foo.py:3: error: Argument 1 to "accept_int" has incompatible type "str"; expected "int" + +[case testIncrementalPartialSubmoduleUpdate] +# cmd: mypy -m a +# cmd2: mypy -m a a.c +# flags: --silent-imports + +[file a/__init__.py] +from .b import B +from .c import C + +[file a/b.py] +class B: pass + +[file a/c.py] +class C: pass + +[file a/c.py.next] +class C: pass +pass + +[rechecked a, a.c] +[stale a, a.c] +[out] + +[case testIncrementalNestedClassRef] +import top + +[file top.py] +from funcs import callee +from classes import Outer +def caller(a: Outer.Inner) -> None: + callee(a) + +[file funcs.py] +from classes import Outer +def callee(a: Outer.Inner) -> None: + pass + +[file classes.py] +class Outer: + class Inner: + pass + +[file top.py.next] +from funcs import callee +from classes import Outer +def caller(a: Outer.Inner) -> int: + callee(a) diff --git a/test-data/unit/check-inference-context.test b/test-data/unit/check-inference-context.test index 4f2d08079039..9dbbcb9820aa 100644 --- a/test-data/unit/check-inference-context.test +++ b/test-data/unit/check-inference-context.test @@ -590,8 +590,8 @@ f2 = lambda: A() # type: Callable[[A], A] class A: pass [out] main:2: error: Cannot infer type of lambda -main:3: error: Cannot infer type of lambda main:3: error: Incompatible types in assignment (expression has type Callable[[], A], variable has type Callable[[A], A]) +main:3: error: Cannot infer type of lambda [case testEllipsisContextForLambda] from typing import Callable @@ -602,8 +602,8 @@ f4 = lambda x: x # type: Callable[..., int] g = lambda x: 1 # type: Callable[..., str] [builtins fixtures/dict.pyi] [out] -main:6: error: Incompatible return value type (got "int", expected "str") main:6: error: Incompatible types in assignment (expression has type Callable[[Any], int], variable has type Callable[..., str]) +main:6: error: Incompatible return value type (got "int", expected "str") [case testEllipsisContextForLambda2] from typing import TypeVar, Callable @@ -804,8 +804,8 @@ def f(a: T, b: S) -> None: c = lambda x: x # type: Callable[[T], S] [out] main: note: In function "f": -main:5: error: Incompatible return value type (got "T", expected "S") main:5: error: Incompatible types in assignment (expression has type Callable[[T], T], variable has type Callable[[T], S]) +main:5: error: Incompatible return value type (got "T", expected "S") [case testLambdaInGenericClass] from typing import TypeVar, Callable, Generic @@ -816,5 +816,5 @@ class A(Generic[T]): c = lambda x: x # type: Callable[[T], S] [out] main: note: In member "f" of class "A": -main:6: error: Incompatible return value type (got "T", expected "S") main:6: error: Incompatible types in assignment (expression has type Callable[[T], T], variable has type Callable[[T], S]) +main:6: error: Incompatible return value type (got "T", expected "S") diff --git a/test-data/unit/check-newsyntax.test b/test-data/unit/check-newsyntax.test new file mode 100644 index 000000000000..628648bd1d44 --- /dev/null +++ b/test-data/unit/check-newsyntax.test @@ -0,0 +1,90 @@ +[case testNewSyntaxRequire36] +# flags: --fast-parser --python-version 3.5 +x: int = 5 # E: Variable annotation syntax is only suppoted in Python 3.6, use type comment instead +[out] + +[case testNewSyntaxSyntaxError] +# flags: --fast-parser --python-version 3.6 +x: int: int # E: invalid syntax +[out] + +[case testNewSyntaxBasics] +# flags: --fast-parser --python-version 3.6 +x: int +x = 5 +y: int = 5 + +a: str +a = 5 # E: Incompatible types in assignment (expression has type "int", variable has type "str") +b: str = 5 # E: Incompatible types in assignment (expression has type "int", variable has type "str") + +zzz: int +zzz: str # E: Name 'zzz' already defined +[out] + +[case testNewSyntaxWithDict] +# flags: --fast-parser --python-version 3.6 +from typing import Dict, Any + +d: Dict[int, str] = {} +d[42] = 'ab' +d[42] = 42 # E: Incompatible types in assignment (expression has type "int", target has type "str") +d['ab'] = 'ab' # E: Invalid index type "str" for "dict" +[builtins fixtures/dict.pyi] +[out] + +[case testNewSyntaxWithRevealType] +# flags: --fast-parser --python-version 3.6 +from typing import Dict + +def tst_local(dct: Dict[int, T]) -> Dict[T, int]: + ret: Dict[T, int] = {} + return ret + +reveal_type(tst_local({1: 'a'})) # E: Revealed type is 'builtins.dict[builtins.str*, builtins.int]' +[builtins fixtures/dict.pyi] +[out] + +[case testNewSyntaxWithInstanceVars] +# flags: --fast-parser --python-version 3.6 +class TstInstance: + a: str + def __init__(self) -> None: + self.x: int + +TstInstance().x = 5 +TstInstance().x = 'ab' # E: Incompatible types in assignment (expression has type "str", variable has type "int") +TstInstance().a = 5 # E: Incompatible types in assignment (expression has type "int", variable has type "str") +TstInstance().a = 'ab' +[out] + +[case testNewSyntaxWithClassVars] +# flags: --fast-parser --strict-optional --python-version 3.6 +class CCC: + a: str = None # E: Incompatible types in assignment (expression has type None, variable has type "str") +[out] +main: note: In class "CCC": + +[case testNewSyntaxWithStrictOptional] +# flags: --fast-parser --strict-optional --python-version 3.6 +strict: int +strict = None # E: Incompatible types in assignment (expression has type None, variable has type "int") +strict2: int = None # E: Incompatible types in assignment (expression has type None, variable has type "int") +[out] + +[case testNewSyntaxWithStrictOptionalFunctions] +# flags: --fast-parser --strict-optional --python-version 3.6 +def f() -> None: + x: int + x = None # E: Incompatible types in assignment (expression has type None, variable has type "int") +[out] +main: note: In function "f": + +[case testNewSyntaxWithStrictOptionalClasses] +# flags: --fast-parser --strict-optional --python-version 3.6 +class C: + def meth(self) -> None: + x: int = None # E: Incompatible types in assignment (expression has type None, variable has type "int") + self.x: int = None # E: Incompatible types in assignment (expression has type None, variable has type "int") +[out] +main: note: In member "meth" of class "C": diff --git a/test-data/unit/check-python2.test b/test-data/unit/check-python2.test index f8d42f97f5f5..5d4edfd4fb4a 100644 --- a/test-data/unit/check-python2.test +++ b/test-data/unit/check-python2.test @@ -213,3 +213,8 @@ def bar(key: Callable[[Tuple[int, int]], int]) -> int: def foo() -> int: return bar(key=lambda (a, b): a) [out] + +[case testImportBuiltins] +# flags: --fast-parser +import __builtin__ +__builtin__.str diff --git a/test-data/unit/check-tuples.test b/test-data/unit/check-tuples.test index 64165a8f72cd..b4bcbfc8429a 100644 --- a/test-data/unit/check-tuples.test +++ b/test-data/unit/check-tuples.test @@ -820,3 +820,22 @@ from typing import List, Tuple def f() -> Tuple[List[str], ...]: return ([],) [builtins fixtures/tuple.pyi] + +[case testTupleWithoutArgs] +from typing import Tuple +def f(a: Tuple) -> None: pass +f(()) +f((1,)) +f(('', '')) +f(0) # E: Argument 1 to "f" has incompatible type "int"; expected Tuple[Any, ...] +[builtins fixtures/tuple.pyi] + +[case testTupleSingleton] +# flags: --fast-parser +from typing import Tuple +def f(a: Tuple[()]) -> None: pass +f(()) +f((1,)) # E: Argument 1 to "f" has incompatible type "Tuple[int]"; expected "Tuple[]" +f(('', '')) # E: Argument 1 to "f" has incompatible type "Tuple[str, str]"; expected "Tuple[]" +f(0) # E: Argument 1 to "f" has incompatible type "int"; expected "Tuple[]" +[builtins fixtures/tuple.pyi] diff --git a/test-data/unit/check-unions.test b/test-data/unit/check-unions.test index 121413836014..cc396d734de4 100644 --- a/test-data/unit/check-unions.test +++ b/test-data/unit/check-unions.test @@ -129,3 +129,123 @@ class C(Generic[T, U]): a = C() # type: C[int, int] b = a.f('a') a.f(b) # E: Argument 1 to "f" of "C" has incompatible type "int"; expected "str" + +[case testUnionMultiassign1] +from typing import Union, Tuple, Any + +b = None # type: Union[Tuple[int], Tuple[float]] +(b1,) = b +reveal_type(b1) # E: Revealed type is 'builtins.float' + +[case testUnionMultiassign2] +from typing import Union, Tuple + +c = None # type: Union[Tuple[int, int], Tuple[int, str]] +q = None # type: Tuple[int, float] +(c1, c2) = c +reveal_type(c1) # E: Revealed type is 'builtins.int' +reveal_type(c2) # E: Revealed type is 'builtins.object' + +[case testUnionMultiassignAny] +from typing import Union, Tuple, Any + +d = None # type: Union[Any, Tuple[float, float]] +(d1, d2) = d +reveal_type(d1) # E: Revealed type is 'Any' +reveal_type(d2) # E: Revealed type is 'Any' + +e = None # type: Union[Any, Tuple[float, float], int] +(e1, e2) = e # E: 'builtins.int' object is not iterable + +[case testUnionMultiassignJoin] +from typing import Union, List + +class A: pass +class B(A): pass +class C(A): pass +a = None # type: Union[List[B], List[C]] +x, y = a +reveal_type(x) # E: Revealed type is '__main__.A' + +[builtins fixtures/list.pyi] +[case testUnionMultiassignRebind] +from typing import Union, List + +class A: pass +class B(A): pass +class C(A): pass +obj = None # type: object +c = None # type: object +a = None # type: Union[List[B], List[C]] +obj, new = a +reveal_type(obj) # E: Revealed type is '__main__.A' +reveal_type(new) # E: Revealed type is '__main__.A' + +obj = 1 +reveal_type(obj) # E: Revealed type is 'builtins.int' + +[builtins fixtures/list.pyi] + +[case testUnionMultiassignAlreadyDeclared] +from typing import Union, Tuple + +a = None # type: Union[Tuple[int, int], Tuple[int, float]] +a1 = None # type: object +a2 = None # type: int + +(a1, a2) = a # E: Incompatible types in assignment (expression has type "float", variable has type "int") + +b = None # type: Union[Tuple[float, int], Tuple[int, int]] +b1 = None # type: object +b2 = None # type: int + +(b1, b2) = a # E: Incompatible types in assignment (expression has type "float", variable has type "int") + +c = None # type: Union[Tuple[int, int], Tuple[int, int]] +c1 = None # type: object +c2 = None # type: int + +(c1, c2) = c +reveal_type(c1) # E: Revealed type is 'builtins.int' +reveal_type(c2) # E: Revealed type is 'builtins.int' + +d = None # type: Union[Tuple[int, int], Tuple[int, float]] +d1 = None # type: object + +(d1, d2) = d +reveal_type(d1) # E: Revealed type is 'builtins.int' +reveal_type(d2) # E: Revealed type is 'builtins.float' + +[case testUnionMultiassignIndexed] +from typing import Union, Tuple, List + +class B: + x = None # type: object + +x = None # type: List[int] +b = None # type: B + +a = None # type: Union[Tuple[int, int], Tuple[int, object]] +(x[0], b.x) = a + +# I don't know why is it incomplete type +reveal_type(x[0]) # E: Revealed type is 'builtins.int*' +reveal_type(b.x) # E: Revealed type is 'builtins.object' + +[builtins fixtures/list.pyi] + +[case testUnionMultiassignPacked] +from typing import Union, Tuple, List + +a = None # type: Union[Tuple[int, int, int], Tuple[int, int, str]] +a1 = None # type: int +a2 = None # type: object +--FIX: allow proper rebinding of packed +xs = None # type: List[int] +(a1, *xs, a2) = a + +reveal_type(a1) # E: Revealed type is 'builtins.int' +reveal_type(xs) # E: Revealed type is 'builtins.list[builtins.int]' +reveal_type(a2) # E: Revealed type is 'builtins.int' + +[builtins fixtures/list.pyi] \ No newline at end of file diff --git a/test-data/unit/check-varargs.test b/test-data/unit/check-varargs.test index 854d4dfccefb..b2568baf2169 100644 --- a/test-data/unit/check-varargs.test +++ b/test-data/unit/check-varargs.test @@ -544,8 +544,8 @@ class B: pass main:9: error: Incompatible types in assignment (expression has type List[A], variable has type "A") main:9: error: Incompatible types in assignment (expression has type List[None], variable has type List[A]) main:10: error: Incompatible types in assignment (expression has type List[None], variable has type "A") -main:11: error: Argument 1 to "f" of "G" has incompatible type *List[A]; expected "B" main:11: error: Incompatible types in assignment (expression has type List[None], variable has type List[A]) +main:11: error: Argument 1 to "f" of "G" has incompatible type *List[A]; expected "B" -- Comment signatures diff --git a/test-data/unit/check-weak-typing.test b/test-data/unit/check-weak-typing.test deleted file mode 100644 index 0bb94d96d293..000000000000 --- a/test-data/unit/check-weak-typing.test +++ /dev/null @@ -1,229 +0,0 @@ -[case testNonWeakTyping] -x = 17 -y = [] # E: Need type annotation for variable -z = [2] -[builtins fixtures/list.pyi] -[out] - -[case testWeakLocalNotGlobal] -# mypy: weak=local -y = [] # E: Need type annotation for variable -x = 1 -x = 'a' # E: Incompatible types in assignment (expression has type "str", variable has type "int") -[builtins fixtures/list.pyi] -[out] - -[case testWeakMoo] -# mypy: weak=global -x = 7 -x + 'd' # E: Unsupported operand types for + ("int" and "str") - -[builtins fixtures/ops.pyi] - -[case testWeakGlobal] -# mypy: weak=global -y = [] -x = 1 -x() # E: "int" not callable -x = 'a' -x() # E: "str" not callable -x = [1] -x[0]() -[builtins fixtures/list.pyi] -[out] - -[case testWeakTypingList] -# mypy: weak=global -x = 17 -y = [] -z = [2] -[builtins fixtures/list.pyi] -[out] - -[case testWeakTypingRename] -# mypy: weak=global -x = 1 -x = 'a' - -[builtins fixtures/list.pyi] -[out] - -[case testWeakFunctionCall] -# mypy: weak -def g(x: int) -> None: - pass - -def f(): - g(1) - g('a') # E: Argument 1 to "g" has incompatible type "str"; expected "int" -[out] -main: note: In function "f": -[case testNonWeakFunctionCall] -def g(x: int) -> None: - pass - -def f(): - g(1) - g('a') -[out] -[case testWeakListInFunction] -# mypy: weak -def f(): - x = [] - x[0] + 1 - x[0] + 'a' - x.add('a') # E: List[Any] has no attribute "add"; maybe "append" or "extend"? -[builtins fixtures/list.pyi] -[out] -main: note: In function "f": -[case testWeakAssignmentInFunction] -# mypy: weak -w = 1 -def f(): - x = y - z = w -y = 1 -[out] -[case testWeakSubexpression] -# mypy: weak -def g(x: int) -> str: - return '' -def f(): - return g(g(1)) # E: Argument 1 to "g" has incompatible type "str"; expected "int" -[out] -main: note: In function "f": -[case testWeakAnnotatedFunction] -# mypy: weak -def f() -> None: - x = [] # E: Need type annotation for variable -[builtins fixtures/list.pyi] -[out] -main: note: In function "f": - -[case testWeakTupleAssignment] -# mypy: weak -def f(): - x, y = g() # E: 'builtins.int' object is not iterable -def g() -> int: - return 0 -[builtins fixtures/for.pyi] -[out] -main: note: In function "f": - -[case testWeakFunction2] -# mypy: weak -x = 1 -def f(): - x() # E: "int" not callable -[out] -main: note: In function "f": -[case testWeakFunction3] -# mypy: weak -def f(): - 1 + 'a' # E: Unsupported operand types for + ("int" and "str") -[out] -main: note: In function "f": -[case testWeakFunctionCall] -# mypy: weak=global -def f(a: str) -> None: pass -f(1) # E: Argument 1 to "f" has incompatible type "int"; expected "str" -f('a') -[case testWeakImportFunction] -# mypy: weak=global -import m -m.f(object()) # E: Argument 1 to "f" has incompatible type "object"; expected "A" -m.f(m.A()) -x = 1 -x = 'a' -[file m.py] -class A: pass -def f(a: A) -> None: pass -[out] -[case testWeakImportCommentScope] -# mypy: weak=global -import m -x = 1 -x = 'a' -[file m.py] -class A: pass -def f(a: A) -> None: pass -x = 1 -x = 'a' -[out] -main:2: note: In module imported here: -tmp/m.py:4: error: Incompatible types in assignment (expression has type "str", variable has type "int") -[case testWeakReassignment] -# mypy: weak=global -x = 1 -x + 1 -x + 'a' # E: Unsupported operand types for + ("int" and "str") -x = 'a' -x + 1 # E: Unsupported operand types for + ("str" and "int") -x + 'a' -[builtins fixtures/ops.pyi] - -[case testWeakConditionalChanges] -# mypy: weak -def f(): - if 1: - x = 1 - x + 1 - x + 'a' # E: Unsupported operand types for + ("int" and "str") - else: - x = 'a' - x + 1 # E: Unsupported operand types for + ("str" and "int") - x + 'a' - x + 1 - x + 'a' -def g(): - x = 1 - if 1: - x = 1 - else: - x = 'a' - if 1: - return - x + 'a' -[builtins fixtures/ops.pyi] -[out] -main: note: In function "f": -[case testWeakListInFunction2] -# mypy: weak=global -x = [1] -# XXX: not clear what the right result here is -x[0]() # E: "int" not callable -[builtins fixtures/list.pyi] - -[case testWeakArgsKws] -# mypy: weak=global - -def f(x, y): - return x+y - -args = [1] -f(*args, y=2) -args = (1, 2) -f(*args, y=2) # E: "f" gets multiple values for keyword argument "y" -args = (1,) -f(*args, y=2) -[builtins fixtures/dict.pyi] -[out] -[case testWeakBinder] -# mypy: weak=global - -# I couldn't come up with a simple example, but this showed a bug in -# binders. -if 1: - if 1: - pass -if 1: - z = '' - pat = 1 - pat() # E: "int" not callable -pat() -pat = '' -pat() # E: "str" not callable -while 1: - pass -pat() # E: "str" not callable -[out] diff --git a/test-data/unit/cmdline.test b/test-data/unit/cmdline.test index eee05e0ceb55..7d964e51c566 100644 --- a/test-data/unit/cmdline.test +++ b/test-data/unit/cmdline.test @@ -131,7 +131,7 @@ def f(): # cmd: mypy x.py y.py z.py [file mypy.ini] [[mypy] -suppress_error_context = True +hide_error_context = True disallow_untyped_defs = True [[mypy-y*] disallow_untyped_defs = False @@ -161,7 +161,7 @@ x.py:1: error: Function is missing a type annotation # cmd: mypy xx.py xy.py yx.py yy.py [file mypy.ini] [[mypy] -suppress_error_context = True +hide_error_context = True [[mypy-*x*.py] disallow_untyped_defs = True [[mypy-*y*.py] @@ -190,7 +190,7 @@ xx.py:1: error: Function is missing a type annotation # cmd: mypy x.py y.py z.py [file mypy.ini] [[mypy] -suppress_error_context = True +hide_error_context = True [[mypy-x*py,z*py] disallow_untyped_defs = True [file x.py] diff --git a/test-data/unit/fixtures/list.pyi b/test-data/unit/fixtures/list.pyi index 220ab529b818..7b94f25c2b2b 100644 --- a/test-data/unit/fixtures/list.pyi +++ b/test-data/unit/fixtures/list.pyi @@ -19,6 +19,7 @@ class list(Iterable[T], Generic[T]): def __add__(self, x: list[T]) -> list[T]: pass def __mul__(self, x: int) -> list[T]: pass def __getitem__(self, x: int) -> T: pass + def __setitem__(self, x: int, v: T) -> None: pass def append(self, x: T) -> None: pass def extend(self, x: Iterable[T]) -> None: pass diff --git a/test-data/unit/fixtures/tuple.pyi b/test-data/unit/fixtures/tuple.pyi index 1d52c14e6bd4..f6d36cc0d2df 100644 --- a/test-data/unit/fixtures/tuple.pyi +++ b/test-data/unit/fixtures/tuple.pyi @@ -25,5 +25,3 @@ T = TypeVar('T') class list(Sequence[T], Generic[T]): pass def sum(iterable: Iterable[T], start: T = None) -> T: pass - -True = bool() diff --git a/test-data/unit/semanal-errors.test b/test-data/unit/semanal-errors.test index 1c45071268f7..2f3e05ea0a4f 100644 --- a/test-data/unit/semanal-errors.test +++ b/test-data/unit/semanal-errors.test @@ -477,8 +477,8 @@ main:8: error: Two starred expressions in assignment (a for *a, (*b, c) in []) (a for a, (*b, *c) in []) [out] -main:1: error: Two starred expressions in assignment main:1: error: Name 'a' is not defined +main:1: error: Two starred expressions in assignment main:3: error: Two starred expressions in assignment [case testStarExpressionRhs] @@ -1357,5 +1357,5 @@ class A: z = 1 [x for i in z if y] [out] -main:5: error: Name 'y' is not defined main:5: error: Name 'x' is not defined +main:5: error: Name 'y' is not defined diff --git a/typeshed b/typeshed index 1d820d48cfb1..aa549db5e5e5 160000 --- a/typeshed +++ b/typeshed @@ -1 +1 @@ -Subproject commit 1d820d48cfb16e78a2858d7687f129c13ff2111f +Subproject commit aa549db5e5e57ee2702899d1cc660163b52171ed