8000 Pluggable system for generating types from docstrings by chadrik · Pull Request #2240 · python/mypy · GitHub
[go: up one dir, main page]
Skip to content

Pluggable system for generating types from docstrings #2240

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

Pluggable system for generating types from docstrings #2240

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
0a47ddb
Rough draft of pluggable system for producing types from docstrings
chadrik Oct 11, 2016
89e2a12
Revert changes to parse.py since it is deprecated.
chadrik Oct 14, 2016
d44291c
Add basic system for registering hooks and use it for docstring parser.
chadrik Oct 15, 2016
05c66cd
Address docparser review notes.
chadrik Oct 19, 2016
6f56a15
Remove unused import
chadrik Oct 19, 2016
6b0c77b
Remove useless cast.
chadrik Oct 19, 2016
775f575
Use new parser to parse strings into types
chadrik Nov 3, 2016
bbd5964
get tests passing.
chadrik Nov 3, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add basic system for registering hooks and use it for docstring parser.
  • Loading branch information
@chadrik
chadrik committed Nov 3, 2016
commit d44291cf21bbeee453ddd15924622f65d0e18a2a
27 changes: 0 additions & 27 deletions mypy/docstrings.py
View file
Open in desktop

This file was deleted.

49 changes: 40 additions & 9 deletions mypy/fastparse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from functools import wraps
from inspect import cleandoc
import sys
import inspect

from typing import Tuple, Union, TypeVar, Callable, Sequence, Optional, Any, cast, List
from mypy.nodes import (
Expand All @@ -23,7 +23,8 @@
)
from mypy import defaults
from mypy import experiments
from mypy import docstrings
from mypy import hooks
from mypy.parsetype import parse_str_as_type
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This calls into the old parser, which we want to avoid doing from the fast parser, as it'll eventually replace the old one. You can use the fast parser to do this instead: see TypeConverter below.

from mypy.errors import Errors

try:
Expand Down Expand Up @@ -89,6 +90,35 @@ def parse_type_comment(type_comment: str, line: int) -> Type:
return TypeConverter(line=line).visit(typ.body)


def parse_docstring(docstring: str, arg_names: List[str],
line: int) -> Optional[Tuple[List[Type], Type]]:
"""Parse a docstring and return type representations.

Returns a 2-tuple: (list of arguments Types, and return Type).
"""
def pop_and_convert(name):
t = type_map.pop(name, None)
if t is None:
return AnyType()
elif isinstance(t, Type):
return t
else:
return parse_str_as_type(t, line)

docstring_parser = hooks.get_docstring_parser()
if docstring_parser is not None:
type_map = docstring_parser(inspect.cleandoc(docstring), line)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still believe calling cleandoc() is up to the custom parser. They may have some other approach to parsing the contents of the docstring.

if type_map:
arg_types = [pop_and_convert(name) for name in arg_names]
return_type = pop_and_convert('return')
if type_map:
raise TypeCommentParseError(
'Arguments parsed from docstring are not present in '
'function signature: {}'.format(', '.join(type_map)),
line, 0)
return arg_types, return_type


def with_line(f: Callable[['ASTConverter', T], U]) -> Callable[['ASTConverter', T], U]:
@wraps(f)
def wrapper(self: 'ASTConverter', ast: T) -> U:
Expand Down Expand Up @@ -289,15 +319,16 @@ def do_func_def(self, n: Union[ast35.FunctionDef, ast35.AsyncFunctionDef],
else:
arg_types = [a.type_annotation for a in args]
return_type = TypeConverter(line=n.lineno).visit(n.returns)
# docstrings
if not any(arg_types) and return_type is None:
# hooks
if (not any(arg_types) and return_type is None and
hooks.get_docstring_parser()):
doc = ast35.get_docstring(n, clean=False)
if doc:
doc = cleandoc(doc.decode('unicode_escape'))
type_map, rtype = docstrings.parse_docstring(doc, n.lineno)
if type_map is not None:
arg_types = [type_map.get(name) for name in arg_names]
return_type = rtype
doc = doc.decode('unicode_escape')
types = parse_docstring(doc, arg_names, n.lineno)
if types is not None:
arg_types, return_type = types

for arg, arg_type in zip(args, arg_types):
self.set_type_optional(arg_type, arg.initializer)

Expand Down
20 changes: 11 additions & 9 deletions mypy/fastparse2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,9 @@
from mypy import defaults
from mypy import experiments
from mypy.errors import Errors
from mypy.fastparse import TypeConverter, TypeCommentParseError
from mypy import docstrings
from mypy.fastparse import (TypeConverter, TypeCommentParseError,
parse_docstring)
from mypy import hooks

try:
from typed_ast import ast27
Expand Down Expand Up @@ -295,15 +296,16 @@ def visit_FunctionDef(self, n: ast27.FunctionDef) -> Statement:
else:
arg_types = [a.type_annotation for a in args]
return_type = converter.visit(None)
# docstrings
if not any(arg_types) and return_type is None:
# hooks
if (not any(arg_types) and return_type is None and
hooks.get_docstring_parser()):
doc = ast27.get_docstring(n, clean=False)
if doc:
doc = cleandoc(doc.decode('unicode_escape'))
type_map, rtype = docstrings.parse_docstring(doc, n.lineno)
if type_map is not None:
arg_types = [type_map.get(name) for name in arg_names]
return_type = rtype
doc = doc.decode('unicode_escape')
types = parse_docstring(doc, arg_names, n.lineno)
if types is not None:
arg_types, return_type = types

for arg, arg_type in zip(args, arg_types):
self.set_type_optional(arg_type, arg.initializer)

Expand Down
28 changes: 28 additions & 0 deletions mypy/hooks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
from typing import Dict, Optional, Callable, Union
from mypy.types import Type

hooks = {} # type: Dict[str, Callable]

docstring_parser_type = Callable[[str, int], Optional[Dict[str, Union[str, Type]]]]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A type alias like this should have a CapWords name. I would also prefer this to always return a string.



def set_docstring_parser(func: docstring_parser_type) -> None:
"""Enable the docstring parsing hook.

The callable must take a docstring for a function along with its line number
(typically passed to mypy.parsetype.parse_str_as_type), and should return
a mapping of argument name to type. The function's return type, if
specified, is stored in the mapping with the special key 'return'.

The keys of the mapping must be a subset of the arguments of the function
to which the docstring belongs (other than the special 'return'
key); an error will be raised if the mapping contains stray arguments.

The values of the mapping must be either mypy.types.Type or a valid
PEP484-compatible string which can be converted to a Type.
"""
hooks['docstring_parser'] = func


def get_docstring_parser() -> Optional[docstring_parser_type]:
return hooks.get('docstring_parser')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The get and set functions look an afwul lot like Java-style accessor methods. If you want type-safety it's probably better to define a Hooks class whose instance variables are the known hooks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify, are you looking for a single registry for all hooks, like this:

class Hooks:
    # The docstring_parser hook must take a docstring for a function [...etc...]
    docstring_parser = None  # type: Callable[[str], Optional[Dict[str, str]]]

    # another explanation...
    future_hook = None  # type: Callable[whatever]

registry = Hooks()

Where the end user would then override the attribute:

import mypy.hooks
mypy.hooks.registry.docstring_parser = my_parser

If not, can you clarify a bit more.

0