ENH: add type stubs from numpy-stubs #16515
Conversation
Add the type stubs and tests from numpy-stubs. Things this entails:
- Copy over the stubs (numpy/__init__.pyi and
numpy/core/_internal.pyi)
- The only modification made was removing `ndarray.tostring` since
it is deprecated
- Update some setup.py files to include pyi files
- Move the tests from numpy-stubs/tests into numpy/tests
- Skip them if mypy is not installed (planning on setting up CI in a
future PR)
- Add a mypy.ini; use it to configure mypy in the tests
- It tells mypy where to find NumPy in the test env
- It ignores internal NumPy type errors (since we only want to
consider errors from the tests cases)
- Some small edits were made to fix test cases that were emitting
deprecation warnings
- Add numpy/py.typed so that the types are picked up in an
installed version of NumPy
| @@ -0,0 +1,1077 @@ | |||
| import builtins | |||
There was a problem hiding this comment.
Big copy-paste from https://github.com/numpy/numpy-stubs/blob/master/numpy-stubs/__init__.pyi.
| @@ -0,0 +1,18 @@ | |||
| from typing import Any | |||
There was a problem hiding this comment.
numpy/tests/fail/array_like.py
Outdated
| @@ -0,0 +1,22 @@ | |||
| from typing import Any, TYPE_CHECKING | |||
There was a problem hiding this comment.
Tests are a copy paste from https://github.com/numpy/numpy-stubs/tree/master/tests.
| @@ -0,0 +1,127 @@ | |||
| import importlib.util | |||
There was a problem hiding this comment.
Test runner is a copy paste from https://github.com/numpy/numpy-stubs/blob/master/tests/test_stubs.py, but with some modifications:
- In the
numpy-stubsrepo the tests were run against an installed version of the stubs (because you can't point at a*-stubspackage usingMYPYPATH) - Now we can use
MYPYPATHto point at NumPy in the test env, so do that so that the tests can run in the normal way - This does mean that mypy is type checking all of NumPy, so we will get complaints about NumPy internals that we don't want to hear
- To fix that we use a
mypy.inifile that silences them - By default errors from inside installed third-party libraries are silenced, so those internal type errors shouldn't bother end-users.
| ) | ||
|
|
||
|
|
||
| @pytest.mark.skipif(NO_MYPY, reason="Mypy is not installed") |
There was a problem hiding this comment.
Skip if mypy isn't installed, since we want to keep typing purely opt-in.
| @@ -0,0 +1,64 @@ | |||
| import sys | |||
There was a problem hiding this comment.
Copy paste of https://github.com/numpy/numpy-stubs/blob/master/numpy-stubs/typing.pyi.
Next step is to make this available at runtime too.
typing_requirements.txt
Outdated
| @@ -0,0 +1,2 @@ | |||
| mypy==0.770 | |||
There was a problem hiding this comment.
If you install these requirements, then the typing tests should run and pass for you locally (I hope).
There was a problem hiding this comment.
This is nice, thanks. I am not sure about the test directory structure, since it layers the tests into the already-existing numpy/tests/*. Maybe they should be in a separate directory like numpy/tests/typing/* and only the numpy/tests/test_typing.py left in the upper-level directory?
I think we should add this to the regular CI runs. If it takes too long we could mark it whit @slow. All we need to do then is merge the requirements to the current test_requirements.txt file.
I assume this does not slow down import numpy at all, is that correct?
typing_requirements.txt
Outdated
| @@ -0,0 +1,2 @@ | |||
| mypy==0.770 | |||
There was a problem hiding this comment.
mypy uses parts of the C-API that PyPy has not implemented, so this should be (tested)
| mypy==0.770 | |
| mypy==0.770; platform_python_implementation != "PyPy" |
Edit: tested the qualifier.
This will make them get picked up in CI runs.
Done in 3fee17a. Moved the
Done in 744512d.
After the cache is warm they aren't too bad, eventually we can pass the cache along as a CI artifact. They also currently invoke mypy once per file, which slows things down (in the stubs this wasn't an issue because there was much less code for mypy to look at). We can speed that up by running mypy once over the entire
Correct. If you had bunch of annotations not in stubs files then I think it could, though I think that won't be a problem in 3.8 with |
|
Needs a release note. |
It currently splits on ":", which causes problems with drives.
Added in 7c0d99d. < 579F /td> |
|
Ok, getting failures on Python 3.9, but it looks like mypy itself is failing on 3.9 (at least as of writing this): https://travis-ci.org/github/python/mypy so I'm probably going to skip these tests on 3.9 for now. |
Mypy doesn't work with 3.9 yet, and 3.6 doesn't work because it doesn't the py.typed marker.
|
Ah, and Python 3.6 doesn't work because |
|
The travis failure was fixed on master, so restarting this with just the fix for arm64 should get further: no rebase needed. |
|
Ok, CI is all green! |
|
What's the plan for keeping this in sync with numpy-stubs? Is numpy-stubs going to be retired? |
Yes, post-merge I'm planning on updating the numpy-stubs README to point users to the main repo and then archiving numpy-stubs. |
If that's the case, might it be an idea to add a dedicated typing label to the numpy repo? |
|
@BvB93 what is a "dedicated typing label"? |
|
I think what is meant is a Github label along the lines of the one @tylerjereddy made for SciPy: https://github.com/scipy/scipy/pulls?q=label%3A%22static+typing%22+ |
|
Thanks @person142. Looking forward to seeing how this works |
| end-users. The currently available types are | ||
|
|
||
| - ``ArrayLike``: for objects that can be coerced to an array | ||
| - ``DtypeLike``: for objects that can be coerced to a dtype |
There was a problem hiding this comment.
@person142, sorry for being late to the party. Should we make this DTypeLike with a capital T? I have been using DType for the dtype classes (I guess as a short form of the camel-case DataType. I am happy to use this one as well, would be good to keep it in sync).
There was a problem hiding this comment.
That is, unless this actually includes concrete instances, such as strings of specific length. In that case it may actually be better to use different spellings. I would have to review the discussion on that...
There was a problem hiding this comment.
Hm @seberg could you open an issue? I think “DtypeLike” was originally “things that can be passed into np.dtype“, and with the dtype changes we should think carefully about what makes sense...
| @overload | ||
| def __array__(self, dtype: DtypeLike = ...) -> ndarray: ... | ||
|
|
||
| ArrayLike = Union[bool, int, float, complex, _SupportsArray, Sequence] |
There was a problem hiding this comment.
do we support __array_interface__ interface for ArrayLike ?
Add the type stubs and tests from numpy-stubs. See the mailing list discussion about doing this here:
http://numpy-discussion.10968.n7.nabble.com/Put-type-annotations-in-NumPy-proper-tp47996.html
and also the discussion here:
numpy/numpy-stubs#79
Things this entails:
numpy/__init__.pyi,numpy/typing.pyi, andnumpy/core/_internal.pyi)ndarray.tostringsince it is deprecatedsetup.pyfiles to include pyi filesnumpy-stubs/testsintonumpy/testsfuture PR)
mypy.ini; use it to configure mypy in the testsnumpy/py.typedso that the types are picked up in an installed version of NumPyStill to do in follow up PRs:
Install mypy in one of the CI runs so that the typing tests are run at a regular cadenceMake the
typingmodule exist at runtime; see e.g. the discussion here:http://numpy-discussion.10968.n7.nabble.com/Feelings-about-type-aliases-in-NumPy-tp48059.html
and here:
ENH: get more specific about _ArrayLike, make it public numpy-stubs#66