8000 BUG: f2py (meson): undefined symbol error when building MPI-enabled extension (--dep mpi not working as expected) · Issue #28902 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

BUG: f2py (meson): undefined symbol error when building MPI-enabled extension (--dep mpi not working as expected) #28902

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
elcorto opened this issue May 5, 2025 · 3 comments
Open

BUG: f2py (meson): undefined symbol error when building MPI-enabled extension (--dep mpi not working as expected) #28902

elcorto opened this issue May 5, 2025 · 3 comments
Labels
00 - Bug component: numpy.f2py

Comments

@elcorto
Copy link
elcorto commented May 5, 2025

Describe the issue:

When building an extension, I see the same error as in #25159 (undefined symbol mpi_comm_rank_), but in that issue the OP seems to use the distutils backend, hence a new issue here for the meson backend (and a work around, see below). The code below uses OpenMPI 4.1.4.

Reproduce the code example:

cat > hello_mpi.f90 << EOF
program hello_world_mpi
include 'mpif.h'

integer rnk, wsize, ierr

call mpi_init(ierr)
call mpi_comm_size(mpi_comm_world, wsize, ierr)
call mpi_comm_rank(mpi_comm_world, rnk, ierr)

print *, 'hello world from process: ', rnk, 'of ', wsize

call mpi_finalize(ierr)
end program
EOF

# sanity check, prints
#   hello world from process:            0 of            4
#   hello world from process:            1 of            4
#   hello world from process:            2 of            4
#   hello world from process:            3 of            4
mpifort hello_mpi.f90
mpirun -np 4 ./a.out

FC=mpifort CC=mpicc python -m numpy.f2py --backend meson --build-dir meson_builddir --dep mpi -c hello_mpi.f90 -m hello_mpi 2>&1 | tee log_fail.txt
python -c "import hello_mpi"

Error message:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /path/to/hello_mpi.cpython-311-x86_64-linux-gnu.so: undefined symbol: mpi_comm_rank_

Python and NumPy Versions:

>>> import sys, numpy; print(numpy.__version__); print(sys.version)
2.2.5
3.11.2 (main, Nov 30 2024, 21:22:50) [GCC 12.2.0]
$ pip list
Package    Version
---------- --------
meson      1.8.0
ninja      1.11.1.4
numpy      2.2.5
pip        23.0.1
setuptools 66.1.1

Runtime Environment:

[{'numpy_version': '2.2.5',
  'python': '3.11.2 (main, Nov 30 2024, 21:22:50) [GCC 12.2.0]',
  'uname': uname_result(system='Linux', node='<hostname>', release='6.12.22-amd64', version='#1 SMP PREEMPT_DYNAMIC Debian 6.12.22-1 (2025-04-10)', machine='x86_64')},
 {'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
                      'found': ['SSSE3',
                                'SSE41',
                                'POPCNT',
                                'SSE42',
                                'AVX',
                                'F16C',
                                'FMA3',
                                'AVX2'],
                      'not_found': ['AVX512F',
                                    'AVX512CD',
                                    'AVX512_KNL',
                                    'AVX512_KNM',
                                    'AVX512_SKX',
                                    'AVX512_CLX',
                                    'AVX512_CNL',
                                    'AVX512_ICL']}},
 {'architecture': 'Haswell',
  'filepath': '/path/to/venv/lib/python3.11/site-packages/numpy.libs/libscipy_openblas64_-6bb31eeb.so',
  'internal_api': 'openblas',
  'num_threads': 16,
  'prefix': 'libscipy_openblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.28'}]

Context for the issue:

The error occurs because the extension is not linked against MPI libs. Even though we use FC=mpifort, the compiler wrapper is (maybe?) not used.

ldd hello_mpi.cpython-311-x86_64-linux-gnu.so
    linux-vdso.so.1 (0x00007f56ed409000)
    libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007f56ed000000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f56ece1f000)
    libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f56ed3ad000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f56ed2cd000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f56ecdff000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f56ed40b000)

The meson_builddir/meson.build file looks like this, and says dependency('mpi'):

project('hello_mpi',
        ['c', 'fortran'],
        version : '0.1',
        meson_version: '>= 1.1.0',
        default_options : [
                            'warning_level=1',
                            'buildtype=release'
                          ])
fc = meson.get_compiler('fortran')

py = import('python').find_installation('''/path/to/bin/python3''', pure: false)
py_dep = py.dependency()

incdir_numpy = run_command(py,
  ['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'],
  check : true
).stdout().strip()

incdir_f2py = run_command(py,
    ['-c', 'import os; os.chdir(".."); import numpy.f2py; print(numpy.f2py.get_include())'],
    check : true
).stdout().strip()

inc_np = include_directories(incdir_numpy)
np_dep = declare_dependency(include_directories: inc_np)

incdir_f2py = incdir_numpy / '..' / '..' / 'f2py' / 'src'
inc_f2py = include_directories(incdir_f2py)
fortranobject_c = incdir_f2py / 'fortranobject.c'

inc_np = include_directories(incdir_numpy, incdir_f2py)
# gh-25000
quadmath_dep = fc.find_library('quadmath', required: false)

py.extension_module('hello_mpi',
                     [
                     '''hello_mpi.f90''',
                     '''hello_mpimodule.c''',
                     fortranobject_c
                     ],
                     include_directories: [
                     inc_np,

                     ],
                     dependencies : [
                     py_dep,
                     quadmath_dep,
                     dependency('mpi'),


                     ],

                     install : true)

According to the meson docs, the MPI dependency should be specified as dependency('mpi', language: 'fortran'). With this change, things work:

cd meson_builddir
sed -i -re "s/dependency\('mpi'\)/dependency\('mpi', language: 'fortran')/" meson.build
meson setup --reconfigure bbdir 2>&1 | tee log_ok.txt
meson compile -C bbdir 2>&1 | tee -a log_ok.txt
cd bbdir
ldd hello_mpi.cpython-311-x86_64-linux-gnu.so
    linux-vdso.so.1 (0x00007f0547a0f000)
    libmpi_mpifh.so.40 => /lib/x86_64-linux-gnu/libmpi_mpifh.so.40 (0x00007f0547996000)
    libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007f0547600000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f054741f000)
    libmpi.so.40 => /lib/x86_64-linux-gnu/libmpi.so.40 (0x00007f05472e0000)
    libopen-pal.so.40 => /lib/x86_64-linux-gnu/libopen-pal.so.40 (0x00007f05478dd000)
    libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f0547299000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f05471b9000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0547199000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f0547a11000)
    libopen-rte.so.40 => /lib/x86_64-linux-gnu/libopen-rte.so.40 (0x00007f05470dd000)
    libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007f0547080000)
    libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x00007f054704a000)
    libevent_pthreads-2.1.so.7 =><
8000
/span> /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007f05478d6000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f054702b000)
    libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f0546ffd000)
python -c "import hello_mpi"

Build logs are attatched.

log_fail.txt

log_ok.txt
@rgommers
Copy link
Member
rgommers commented May 5, 2025

Thanks for the clear report @elcorto. This looks like a limitation of the current CLI interface f2py has. The --dep argument only takes a library name so the user cannot indicate any custom arguments to Meson's dependency function:

numpy/numpy/f2py/_backends/_meson.py

Lines 84 to 87 in 00f2733

def deps_substitution(self) -> None:
self.substitutions["dep_list"] = f",\n{self.indent}".join(
[f"{self.indent}dependency('{dep}')," for dep in self.deps]
)

Probably the way to address this is making --dep accept custom syntax, e.g. --dep mpi[language: 'fortran'] (allows for other keyword usage too, e.g. [method: 'config-tool'], [modules : ['thread']] or [static: true])

Of course the CLI interface is going to remain limited compared to writing one's own build files (see https://numpy.org/devdocs/f2py/buildtools/meson.html), but this does feel like too much of a limitation - if we offer --dep it should be able to handle Fortran MPI.

@elcorto
Copy link
Author
elcorto commented May 5, 2025

Thanks for the quick answer.

In the context of f2py, would it be justified to hard-code dependency(<dep>, language: 'fortran') where --dep <dep>? If not then extending the syntax as you suggest might be a way to support the most common use cases, without blowing up the CLI to support complex meson configuration.

In cases that require more customization of the build, the suggested workflow here seems to be writing meson.build from scratch, which requires users to be quite familiar with meson. I think the usefulness of f2py is in part its ability to create meson.build. I haven't found a way to let it do that w/o triggering a build via meson -> ninja. Maybe I have overlooked that in the docs. This would be a useful middle ground -- create meson.build boilerplate with f2py, if needed adapt that file, use meson tooling to finish the build.

@elcorto elcorto mentioned this issue May 5, 2025
Merged
6 tasks
@rgommers
Copy link
Member
rgommers commented May 5, 2025

In the context of f2py, would it be justified to hard-code dependency(<dep>, language: 'fortran') where --dep <dep>?

I don't think so. f2py itself generates C code and it's possible (perhaps even common?) for dependencies to be for C.

This would be a useful middle ground -- create meson.build boilerplate with f2py, if needed adapt that file, use meson tooling to finish the build.

That's a nice suggestion. It's pretty much all implemented already - just requires a new CLI flag I'd think.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug component: numpy.f2py
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rgommers @elcorto
0