8000 numpy.typing._ufunc is slow when running in mypy · Issue #19719 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

numpy.typing._ufunc is slow when running in mypy #19719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
orsinium opened this issue Aug 20, 2021 · 10 comments
Closed

numpy.typing._ufunc is slow when running in mypy #19719

orsinium opened this issue Aug 20, 2021 · 10 comments
Labels
23 - Wish List 41 - Static typing

Comments

@orsinium
Copy link
orsinium commented Aug 20, 2021 •
edited
Loading

When running mypy on the code using the new numpy, it takes 30 seconds on my machine to produce a result. When running with -v flag, it shows that it freezes when importing numpy.typing._ufunc:

LOG:  Parsing /.../python3.9/site-packages/numpy/typing/_ufunc.pyi (numpy.typing._ufunc)

Reproducing code example:

echo "import numpy.typing._ufunc" > tmp.py
time mypy --no-incremental tmp.py

Error message:

not relevant, performance issue

NumPy/Python version information:

  • mypy 0.910
  • Python 3.9.0
  • numpy 1.21.2
@BvB93
Copy link
Member
BvB93 commented Aug 24, 2021 •
edited
Loading

Since npt._ufunc also imports from numpy proper, I'm not sure it is the actual culprit here,
especially since the main numpy namespace simply contains a lot of annotations.

numpy/numpy/typing/_ufunc.pyi

Line 19 in 94010e3

from numpy import ufunc, _CastingKind, _OrderKACF

In any case, I'm not sure there is anything we can do here from numpy's side.
While this issue might be more suitable for the mypy repo, I'm sure they'll be able to do much either without something more actionable.

@orsinium
Copy link
Author

I'm not sure it is the actual culprit here

I'm also not sure what exactly is the place where exactly it gets slow. If you run the snippet in the issue description with -vvv, all you can see is it freezes at "Processing SCC of size 82". This SCC mostly numpy submodules and, surprisingly, _pytest.

In any case, I'm not sure there is anything we can do here from numpy's side.

Me neither. I'm wondering if there is a possibility to have to make some annotations imports deferred.

The full log message where it gets slow:

LOG: Processing SCC of size 82 (_pytest.config.findpaths _pytest.hookspec _pytest.python_api _pytest.config numpy._pytesttester _pytest.mark.structures numpy.typing._generic_alias numpy.typing._dtype_like numpy.typing._scalars numpy.typing._add_docstring numpy.lib.utils numpy.lib.ufunclike numpy.lib.twodim_base numpy.lib.stride_tricks numpy.lib.shape_base numpy.lib.polynomial numpy.lib.npyio numpy.lib.index_tricks numpy.lib.function_base numpy.lib.arrayterator numpy.core.shape_base numpy.core.numerictypes numpy.core.numeric numpy.core.einsumfunc numpy.core.arrayprint numpy.core._type_aliases numpy.core._asarray numpy.core.fromnumeric numpy.core.function_base numpy.lib numpy.typing._extended_precision numpy.typing._callable numpy.core._internal numpy.typing._array_like numpy.typing numpy numpy.random.bit_generator numpy.polynomial.polynomial numpy.polynomial.legendre numpy.polynomial.laguerre numpy.polynomial.hermite_e numpy.polynomial.hermite numpy.polynomial.chebyshev numpy.ma.core numpy.rec numpy.matrixlib numpy.char numpy.random.mtrand numpy.random._sfc64 numpy.random._philox numpy.random._pcg64 numpy.random._mt19937 numpy.ma.extras numpy.polynomial numpy.random._generator numpy.random numpy.ma _pytest.nodes _pytest.mark _pytest.reports _pytest.assertion.truncate _pytest.fixtures _pytest.pytester_assertions _pytest.runner _pytest.terminal _pytest.recwarn _pytest.monkeypatch _pytest.debugging _pytest.capture _pytest.tmpdir _pytest.main _pytest.assertion.rewrite _pytest.python _pytest.pytester _pytest.logging _pytest.cacheprovider _pytest.assertion pytest.collect pytest _pytest.warnings _pytest.helpconfig numpy.typing._ufunc) as inherently stale with stale deps (future _pytest _pytest._code _pytest._code.code _pytest._io _pytest._io.saferepr _pytest._io.wcwidth _pytest._version _pytest.assertion.util _pytest.compat _pytest.config.argparsing _pytest.config.exceptions _pytest.deprecated _pytest.freeze_support _pytest.mark.expression _pytest.outcomes _pytest.pathlib _pytest.store _pytest.timing _pytest.warning_types abc argparse array ast attr bdb builtins collections collections.abc contextlib copy ctypes datetime decimal doctest enum errno fnmatch functools gc getpass importlib importlib.abc importlib.machinery importlib.util iniconfig inspect io itertools json logging marshal math mmap numbers numpy.core._ufunc_config numpy.core.multiarray numpy.core.umath numpy.ctypeslib numpy.distutils numpy.distutils.cpuinfo numpy.fft numpy.lib._version numpy.lib.arraypad numpy.lib.arraysetops numpy.lib.format numpy.lib.mixins numpy.lib.nanfunctions numpy.lib.scimath numpy.lib.type_check numpy.linalg numpy.polynomial._polybase numpy.polynomial.polyutils numpy.testing numpy.typing._char_codes numpy.typing._nbit numpy.typing._shape numpy.version os packaging.requirements packaging.version pathlib pdb platform pprint py py.path re shlex shutil struct subprocess sys tempfile textwrap threading tokenize toml traceback types typing typing_extensions warnings weakref)

@fbaptista
Copy link

Hello dears, we are facing a similar issue when using numpy on freezed python (using cx-freeze).
When numpy inits, it hangs the entire thread:

 File "/Users/runner/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numpy/__init__.py", line 152, in <module>
 File "/Users/runner/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numpy/random/__init__.py", line 181, in <module>
 File "/Users/runner/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/numpy/random/_pickle.py", line 1, in <module>
 File "mtrand.pyx", line 18, in init numpy.random.mtrand
 File "<frozen importlib._bootstrap>", line 389, in parent
KeyboardInterrupt

Is this the same issue?
Any ideas on how to workaround here?
Thanks!

@BvB93
Copy link
Member
BvB93 commented Sep 30, 2021

Hi @fbaptista,

This issue is about mypy (the static type checker) being slow when analyzing numpy.
What you're describing seems to be purely runtime related, so I'd suggest creating a new issue.

@mparry
Copy link
mparry commented Nov 8, 2021

I see similar problems with numpy 1.21 (but not 1.20 or earlier). It looks like mypy 0.920 addresses this, perhaps thanks to python/mypy#10922.

@ravenexp
Copy link

mypy -m numpy never terminates and hangs on:

LOG:  Processing SCC of size 82 (_pytest.config.findpaths _pytest.hookspec _pytest.python_api _pytest.config numpy._pytesttester _pytest.mark.structures numpy.typing._generic_alias numpy.typing._dtype_like numpy.typing._scalars numpy.typing._add_docstring numpy.lib.utils numpy.lib.ufunclike numpy.lib.twodim_base numpy.lib.stride_tricks numpy.lib.shape_base numpy.lib.polynomial numpy.lib.npyio numpy.lib.index_tricks numpy.lib.function_base numpy.lib.arrayterator numpy.core.shape_base numpy.core.numerictypes numpy.core.numeric numpy.core.einsumfunc numpy.core.arrayprint numpy.core._type_aliases numpy.core._asarray numpy.core.fromnumeric numpy.core.function_base numpy.lib numpy.typing._extended_precision numpy.typing._callable numpy.core._internal numpy.typing._array_like numpy numpy.typing numpy.random.bit_generator numpy.polynomial.polynomial numpy.polynomial.legendre numpy.polynomial.laguerre numpy.polynomial.hermite_e numpy.polynomial.hermite numpy.polynomial.chebyshev numpy.ma.core numpy.rec numpy.matrixlib numpy.char numpy.random.mtrand numpy.random._sfc64 numpy.random._philox numpy.random._pcg64 numpy.random._mt19937 numpy.ma.extras numpy.polynomial numpy.random._generator numpy.random numpy.ma _pytest.nodes _pytest.mark _pytest.reports _pytest.assertion.truncate _pytest.fixtures _pytest.pytester_assertions _pytest.runner _pytest.terminal _pytest.recwarn _pytest.monkeypatch _pytest.debugging _pytest.capture _pytest.tmpdir _pytest.main _pytest.assertion.rewrite _pytest.python _pytest.pytester _pytest.logging _pytest.cacheprovider _pytest.assertion pytest.collect pytest _pytest.warnings _pytest.helpconfig numpy.typing._ufunc) as inherently stale with stale deps (numpy.core.multiarray numpy.core.umath)

python 3.9.7
mypy 0.910
numpy 1.21.2

@orsinium
Copy link
Author

For me, it finished in 36 seconds. mypy 0.910, numpy 1.21.2, python 3.9.0. Try waiting a bit longer?

@ravenexp
Copy link 
________________________________________________________
Executed in  274.70 secs    fish           external
   usr time  273.74 secs  606.00 micros  273.74 secs
   sys time    0.39 secs  267.00 micros    0.39 secs

Oh, you are right, it terminates, but it takes almost 5 minutes on my machine.

@BvB93
Copy link
Member
BvB93 commented Jan 13, 2022

I'm noticing that mypy 0.931 seems to be much faster compared to 0.910 when running locally (i.e. a few seconds compared to ~1 min previously).

Closing as the issue appears to be resolved and there isn't a lot that can be done about it here downstream anyway.

@BvB93 BvB93 closed this as completed Jan 13, 2022
@orsinium
Copy link
Author

Confirmed, now it finishes in 6 seconds on my machine. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
23 - Wish List 41 - Static typing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ravenexp @fbaptista @orsinium @mparry @BvB93
0