BUG: f2py treats f90 module strings as variable names for dependent variables #28700
Description
Describe the issue:
While using f2py to generate a Python module from a legacy code base, I
encountered an error which I believe can be traced back to f2py not properly
recognizing strings when determining dependent variables.
The following minimal test case exhibits the encountered behaviour on numpy
2.3.0.dev0+git20241123.5f70dc8
! file: np_bug.f90
module mod1
CHARACTER(6),PUBLIC,PARAMETER :: mkdir='mkdir '
CHARACTER(7),PUBLIC,PARAMETER :: badvar2="badvar2"
end module mod1
module np_bug
contains
subroutine sub1
use mod1
end subroutine sub1
end module np_bug
Executing f2py as follows:
f2py --build-dir output \
-m pot_f2py_bug \
-c \
--backend meson \
np_bug.f90
Triggers the following error:
Reading fortran codes...
Reading file 'np_bug.f90' (format:free)
Post-processing...
Block: pot_f2py_bug
Block: mod1
Block: np_bug
Block: sub1
Traceback (most recent call last):
File "/home/mweigand/.virtualenvs/numpy/bin/f2py", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/f2py2e.py", line 781, in main
run_compile()
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/f2py2e.py", line 727, in run_compile
run_main(f" {' '.join(f2py_flags)} -m {modulename} {' '.join(sources)}".split())
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/f2py2e.py", line 473, in run_main
postlist = callcrackfortran(files, options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/f2py2e.py", line 352, in callcrackfortran
postlist = crackfortran.crackfortran(files)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 3532, in crackfortran
postlist = postcrack(grouplist[0])
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2075, in postcrack
g = postcrack(g, tab=tab + '\t')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2094, in postcrack
block['body'] = analyzebody(block, args, tab=tab)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2266, in analyzebody
b = postcrack(b, as_, tab=tab + '\t')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2094, in postcrack
block['body'] = analyzebody(block, args, tab=tab)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2266, in analyzebody
b = postcrack(b, as_, tab=tab + '\t')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2094, in postcrack
block['body'] = analyzebody(block, args, tab=tab)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2266, in analyzebody
b = postcrack(b, as_, tab=tab + '\t')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2090, in postcrack
block['vars'] = analyzevars(block)
^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2636, in analyzevars
params = get_parameters(vars, get_useparameters(block))
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2015, in get_useparameters
params = get_parameters(mvars)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2487, in get_parameters
for n in get_sorted_names(vars):
^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2415, in get_sorted_names
depend_dict = _calc_depend_dict(vars)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2410, in _calc_depend_dict
_get_depend_dict(n, vars, depend_dict)
8000
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2396, in _get_depend_dict
or _get_depend_dict(word, vars, deps):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2396, in _get_depend_dict
or _get_depend_dict(word, vars, deps):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2396, in _get_depend_dict
or _get_depend_dict(word, vars, deps):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 973 more times]
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/crackfortran.py", line 2388, in _get_depend_dict
if '=' in vars[name] and not isstring(vars[name]):
^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/auxfuncs.py", line 96, in isstring
return isstring_or_stringarray(var) and not isarray(var)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/auxfuncs.py", line 92, in isstring_or_stringarray
return _ischaracter(var) and 'charselector' in var
^^^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/auxfuncs.py", line 71, in _ischaracter
not isexternal(var)
^^^^^^^^^^^^^^^
File "/home/mweigand/.virtualenvs/numpy/lib/python3.11/site-packages/numpy/f2py/auxfuncs.py", line 412, in isexternal
return 'attrspec' in var and 'external' in var['attrspec']
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded in comparison
Analysis of the problem
Running crackfortran.crack2fortran on the file shows the problem in the form of
the two Warnings that are printed:
On numpy 2.3.0.dev0+git20241123.5f70dc8:
In [4]: from numpy.f2py import crackfortran
...: mod = crackfortran.crackfortran('np_bug.f90')
...: crackfortran.crack2fortran(mod)
...:
Reading fortran codes...
Reading file 'np_bug.f90' (format:free)
Post-processing...
Block: mod1
Block: np_bug
Block: sub1 In: np_bug.f90:np_bug:sub1 get_useparameters: no module mod1 info used by sub1 Applying post-processing hooks...
character_backward_compatibility_hook
Post-processing (stage 2)...
vars2fortran: Warning: cross-dependence between variables "mkdir" and "mkdir"
vars2fortran: Warning: cross-dependence between variables "badvar2" and "badvar2"
After applying the (temporary) fix detailed below:
In [1]: from numpy.f2py import crackfortran
...: mod = crackfortran.crackfortran('np_bug.f90')
...: crackfortran.crack2fortran(mod)
Reading fortran codes...
Reading file 'np_bug.f90' (format:free)
Post-processing...
Block: mod1
Block: np_bug
Block: sub1
In: np_bug.f90:np_bug:sub1
get_useparameters: no module mod1 info used by sub1
Applying post-processing hooks...
character_backward_compatibility_hook
Post-processing (stage 2)...
I think the problem boils down to the regex in line 2625:
name_match = re.compile(r'[A-Za-z][\w$]*').match
(link to master:
numpy/numpy/f2py/crackfortran.py
Line 2625 in 67fb349
The regex will also match if the variable name is included as a string (see
example above).
Unfortunately my regex skills are not a match for that challenge, so I just
resolved to removing any strings before checking for dependencies (patch
against 2.3.0.dev0, 5f70dc8):
diff --git a/numpy/f2py/crackfortran.py b/numpy/f2py/crackfortran.py
index 6eea034778..5ae058fcee 100644
--- a/numpy/f2py/crackfortran.py
+++ b/numpy/f2py/crackfortran.py
@@ -2911,7 +2911,18 @@ def compute_deps(v, deps):
vars[n]['depend'] = []
for v, m in list(dep_matches.items()):
if m(vars[n]['=']):
- vars[n]['depend'].append(v)
+ # ignore strings
+ cleaned_content = re.sub(
+ r"('.*?')",
+ '',
+ re.sub(
+ r'(".*?")',
+ '',
+ vars[n]['=']
+ )
+ )
+ if m(cleaned_content):
+ vars[n]['depend'].append(v)
if not vars[n]['depend']:
del vars[n]['depend']
if isscalar(vars[n]):
While I think this "fix" is a little bit too clumsy for a merge request, I
still confirmed that all old tests successfully run (spin test; spin test -m
slow).
Please let me know if I can help in any way with this issue.
Reproduce the code example:
Please see description aboveError message:
Python and NumPy Versions:
import sys, numpy; print(numpy.version); print(sys.version)
2.3.0.dev0+git20241123.5f70dc8
3.11.2 (main, Nov 30 2024, 21:22:50) [GCC 12.2.0]
Runtime Environment:
In [1]: import numpy; numpy.show_runtime()
[{'numpy_version': '2.3.0.dev0+git20241123.5f70dc8',
'python': '3.11.2 (main, Nov 30 2024, 21:22:50) [GCC 12.2.0]',
'uname': uname_result(system='Linux', node='titan', release='6.11.5+bpo-amd64', version='#1 SMP PREEMPT_DYNAMIC Debian 6.11.5-1~bpo12+1 (2024-11-11)', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2'],
'not_found': ['AVX512F',
'AVX512CD',
'AVX512_KNL',
'AVX512_KNM',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL',
'AVX512_SPR']}},
{'architecture': 'Haswell',
'filepath': '/usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.21.so',
'internal_api': 'openblas',
'num_threads': 12,
'prefix': 'libopenblas',
'threading_layer': 'pthreads',
'user_api': 'blas',
'version': '0.3.21'}]
Context for the issue:
Fixing this would allow building python modules for our legacy fortran code.