8000 ENH: Warn when reloading numpy or using numpy in sub-interpreter by seberg · Pull Request #16241 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content

ENH: Warn when reloading numpy or using numpy in sub-interpreter #16241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 15, 2020

Conversation

seberg
Copy link
Member
@seberg seberg commented May 14, 2020

This adds a warning when the main NumPy module is reloaded
with the assumption that in this case objects such as np.matrix,
np._NoValue or exceptions may be cached internally.

It also gives a warning when NumPy is imported in a sub-interpreter.


I have tested this locally, not sure if adding a tes tmakes sense. The PYPY check was just because I assumed PYPY is different here:

The result for sub-interpreter warning is:

>>> import ctypes
>>> ctypes.pythonapi.Py_NewInterpreter()
30257600
import numpy as np
<stdin>:1: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode.
Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve.

And the other warning:

>>> import numpy as np
>>> import importlib
>>> importlib.reload(np)
/usr/lib/python3.8/importlib/__init__.py:169: UserWarning: The NumPy module was reloaded (imported a second time). This can in some cases result in small but subtle issues and is discouraged.
  _bootstrap._exec(spec, module)
<module 'numpy' from '/home/sebastian/forks/numpy/build/testenv/lib/python3.8/site-packages/numpy/__init__.py'>

Anyway, just a thought, maybe the warnings are too big, or too small I don't know. Marking as draft because its built to be merged after the add_newdocs cleanup.

@@ -12,13 +14,15 @@ def test_numpy_reloading():
VisibleDeprecationWarning = np.VisibleDeprecationWarning
ModuleDeprecationWarning = np.ModuleDeprecationWarning

reload(np)
with assert_warns(UserWarning):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we want to skip this for pypy ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dunno, I think it should be fine on PyPy and probably also applies. The sub-interpreter one I do not know/doubt (but its not tested here).

To be honest, I am fine with not doing this at all (and/or at least leaving the sub-interpreter stuff out). But thought I would propose it. It might be a bit too obnoxious.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I am fine with not doing this at all (and/or at least leaving the sub-interpreter stuff out). But thought I would propose it. It might be a bit too obnoxious.

Skipping the test sounds fine to me if thats what you mean.

I dunno, I think it should be fine on PyPy and probably also applies. The sub-interpreter one I do not know/doubt (but its not tested here).

I tried on pypy, wasn't able to run your ctypes example though since pypy ctypes implementation doesnt have ctypes.pythonapi. But it will probably warn just fine.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ctypes example should do nothing on pypy, since I do not think pypy supports subinterpreters (yet). The reload example should apply just as much to pypy as to cpython (not sure whether it is possible to reload in pypy).

I think this is fairly straight forward, unless there is some unknown failure mode for the sub-interpreter warning being given spuriously. Otherwise mainly the wording matters? I am not even sure how well that ctypes thing works, which is why I did not try to bake it into a test...

@mattip
Copy link
Member
mattip commented May 17, 2020

Windows is unhappy: E AttributeError: Can't pickle local object 'test_full_reimport.<locals>.try_full_reload'

@seberg
Copy link
Member Author
seberg commented May 17, 2020

Yes, same fix as for gh-16239 applies, was hoping to fix that once the other one is merged.

@mattip
Copy link
Member
mattip commented May 17, 2020

You should skip this test on PyPy, you are running into gh-10167 where the call to add_newdocs should be replaced by generated C code that is compiled in and loaded before PyType_Ready

Edit: meant to be on gh-16239

@seberg seberg force-pushed the reimport-warning branch from 38d4415 to 1f97fec Compare May 20, 2020 16:01
@seberg seberg marked this pull request as ready for review May 20, 2020 16:02
@seberg seberg closed this May 21, 2020
@seberg seberg reopened this May 21, 2020
@seberg seberg added the triage review Issue/PR to be discussed at the next triage meeting label May 22, 2020
@seberg
Copy link
Member Author
seberg commented Jun 3, 2020

Hmmm, maybe I will look into just fixing the user warning case. I am still happy to put this in as is, but if nobody else is enthusiastic about it, or thinks it might scare users too much, that is fine.

For the subprocessors, I feel this a good choice to give a warning, to inform users of our policy (which also invites contributions). So as soon as anyone reviews, I am happy to e.g. split it out.

@seberg seberg removed the triage review Issue/PR to be discussed at the next triage meeting label Jun 3, 2020
@charris
Copy link
Member
charris commented Dec 15, 2020

Where are we with this? Needs rebase.

This adds a warning when the main NumPy module is reloaded
with the assumption that in this case objects such as `np.matrix`,
`np._NoValue` or exceptions may be cached internally.

It also gives a warning when NumPy is imported in a sub-interpreter.
@seberg
Copy link
Member Author
seberg commented Dec 15, 2020

I dunno, rebased it. I thought we tended towards just merging this, but it never happened 🤷. The main reason for not doing it would be someone complaining that it is unnecessary noise, but the warnings are probably correct so....

@charris charris merged commit 812b696 into numpy:master Dec 15, 2020
@charris
Copy link
Member
charris commented Dec 15, 2020

Let's give it a shot. Thanks Sebastian.

@charris charris changed the title DOC: Warn when reloading numpy or using numpy in sub-interpreter ENH: Warn when reloading numpy or using numpy in sub-interpreter Dec 15, 2020
@seberg seberg deleted the reimport-warning branch December 15, 2020 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants
0