Description
Describe the issue:
This isn't exactly a bug per se but a surprise after updating to 1.25. So let me know if this should be moved elsewhere.
In 1.25 from numpy import * brings in names like min and max which override the builtin min and max.
The specific case that lead to our failure was in a dependency we use. This is the import and this is the use of min.
To fix I initially tried patching the dependency (warp) in our build system with from builtins import * after the from numpy import *. But, warp also does from gist import * which also does a from numpy import *. I could patch the dependency of a dependency but I’m wondering if there is a better "upstream" fix.
I believe this is the change that caused the bug.
It looks like some thought has already been given to builtin names and from numpy import * (here and here).
I’m wondering if removing builtin names from __all__ or at least removing min and max is something you all are open to considering?
Reproduce the code example:
$ pip install numpy==1.25.1
<snip>
$ python
Python 3.9.15 (main, Jul 15 2023, 14:16:53)
[GCC 12.2.1 20221121 (Red Hat 12.2.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.25.1'
>>> type(min)
<class 'builtin_function_or_method'>
>>> from numpy import *
>>> type(min)
<class 'numpy._ArrayFunctionDispatcher'>
$ pip install numpy==1.24.0
<snip>
>>> numpy.__version__
'1.24.0'
>>> type(min)
<class 'builtin_function_or_method'>
>>> from numpy import *
>>> type(min)
<class 'builtin_function_or_method'>Error message:
One specific example is this
~$ python -c 'import warp'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/vagrant/.pyenv/versions/py3/lib/python3.9/site-packages/warp/__init__.py", line 1, in <module>
from .warp import *
File "/home/vagrant/.pyenv/versions/py3/lib/python3.9/site-packages/warp/warp.py", line 263, in <module>
top.ssn = int(min(sys.maxsize,1./numpy.finfo('d').eps)/npes*me + 1)
File "/home/vagrant/.pyenv/versions/py3/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 2953, in min
return _wrapreduction(a, np.minimum, 'min', axis, None, out,
File "/home/vagrant/.pyenv/versions/py3/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 88, in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
TypeError: 'numpy.float64' object cannot be interpreted as an integerAn example you can more easily run is
~$ python
Python 3.9.15 (main, Jul 15 2023, 14:16:53)
[GCC 12.2.1 20221121 (Red Hat 12.2.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> min(1,1./2)
0.5
>>> from numpy import *
>>> min(1,1./2)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/vagrant/.pyenv/versions/py3/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 2953, in min
return _wrapreduction(a, np.minimum, 'min', axis, None, out,
File "/home/vagrant/.pyenv/versions/py3/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 88, in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
TypeError: 'float' object cannot be interpreted as an integerRuntime information:
1.24.0
~$ python -c 'import sys, numpy; print(numpy.__version__); print(numpy.show_runtime()); print(sys.version)'
1.24.0
[{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3', 'SSE41', 'POPCNT', 'SSE42', 'AVX'],
'not_found': ['F16C',
'FMA3',
'AVX2',
'AVX512F',
'AVX512CD',
'AVX512_KNL',
'AVX512_KNM',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL']}},
{'architecture': 'Sandybridge',
'filepath': '/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-15028c96.3.21.so',
'internal_api': 'openblas',
'num_threads': 4,
'prefix': 'libopenblas',
'threading_layer': 'pthreads',
'user_api': 'blas',
'version': '0.3.21'}]
None
3.9.15 (main, Jul 15 2023, 14:16:53)
[GCC 12.2.1 20221121 (Red Hat 12.2.1-4)]1.25.1
~$ python -c 'import sys, numpy; print(numpy.__version__); print(numpy.show_runtime()); print(sys.version)'
1.25.1
[{'numpy_version': '1.25.1',
'python': '3.9.15 (main, Jul 15 2023, 14:16:53) \n'
'[GCC 12.2.1 20221121 (Red Hat 12.2.1-4)]',
'uname': uname_result(system='Linux', node='v2.radia.run', release='6.2.15-100.fc36.x86_64', version='#1 SMP PREEMPT_DYNAMIC Thu May 11 16:51:53 UTC 2023', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3', 'SSE41', 'POPCNT', 'SSE42', 'AVX'],
'not_found': ['F16C',
'FMA3',
'AVX2',
'AVX512F',
'AVX512CD',
'AVX512_KNL',
'AVX512_KNM',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL']}},
{'architecture': 'Sandybridge',
'filepath': '/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-7a851222.3.23.so',
'internal_api': 'openblas',
'num_threads': 4,
'prefix': 'libopenblas',
'threading_layer': 'pthreads',
'user_api': 'blas',
'version': '0.3.23'}]
None
3.9.15 (main, Jul 15 2023, 14:16:53)
[GCC 12.2.1 20221121 (Red Hat 12.2.1-4)]Context for the issue:
As stated above, dependencies we use do from numpy import * which caused breakage in numpy 1.25. This went undetected until our CD caught it during a release (issue).
A cursory look at our production image reveals a few packages that could have problems:
$ docker run --rm -it radiasoft/sirepo:prod bash -c 'grep --include="*.py" -r "from numpy import \*" ~/.pyenv'
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/Forthon/_Forthon.py:from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/IPython/core/interactiveshell.py: Whether to do `from numpy import *` and `from pylab import *`
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/IPython/core/magics/pylab.py: from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/IPython/core/pylabtools.py: "from numpy import *\n")
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/gist/colorbar.py:from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/gist/gist.py:from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/gist/pl3d.py:from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/gist/shapetest.py:from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/gist/slice3.py:from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/matplotlib/pylab.py:from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/warp/attic/hibeamdefaults.py:from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/warp/attic/namelist.py:from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/warp/data_dumping/PWhdf.py:from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/warp/diagnostics/gistdummy.py:#from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/warp/diagnostics/plarr3d.py:from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/warp/utils/optimizer.py:from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/warp/warp.py:from numpy import *
/home/vagrant/.pyenv/versions/3.9.15/envs/py3/lib/python3.9/site-packages/warp_parallel/__init__.py:from numpy import *A search on github reveals 5.8k results that use from numpy import * followed by a use of min or max. Just from numpy import * has 39.7k results.
I know your docs say not to use from numpy import * and as a general rule of thumb it is frowned on (ex Google python style guide). I personally use qualified imports but, as evidence from the searches, many people don't follow this advice.
I realize those searches are imperfect and won't cause a break in all cases. But, I think the use of from numpy import * could be widespread enough that care needs to be put into what is brought in when doing so.
I don't know what the "best" fix is. It may just be I need to patch the dependencies I use. But, I wanted to float the problem out to get your thoughts.
Thanks for your time.