8000 TST: Windows `f2py` tests hang indefinitely with msys gcc-fortran on PATH · Issue #9342 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

TST: Windows f2py tests hang indefinitely with msys gcc-fortran on PATH #9342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree t 8000 o our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
matthew-brett opened this issue Jul 1, 2017 · 11 comments
Open

TST: Windows f2py tests hang indefinitely with msys gcc-fortran on PATH #9342

matthew-brett opened this issue Jul 1, 2017 · 11 comments
Labels
05 - Testing component: CI component: numpy.f2py

Comments

@matthew-brett
Copy link
Contributor

With Appveyor file:

# vim ft=yaml

build_script:
  - pip install numpy

test_script:
  # Adding this line causes the tests to hang forever
  # - PATH=%PATH%;c:\msys64\usr\bin
  # Unless you uninstall gcc-fortran like this
  # - pacman -R --noconfirm gcc-fortran
  - pip install nose
  - python -c "import numpy.f2py; numpy.f2py.test(verbose=4)"

Without adding msys64 to the path, install + f2py tests pass in 30 seconds or so:

https://ci.appveyor.com/project/matthew-brett/windows-wheel-builder/build/1.0.53#L29

Adding msys64 to the path - hangs until build timeout (I canceled this one after 90 seconds):

https://ci.appveyor.com/project/matthew-brett/windows-wheel-builder/build/1.0.52#L16

Adding msys64 to the path, but uninstalling gcc-fortran, tests pass again:

https://ci.appveyor.com/project/matthew-brett/windows-wheel-builder/build/1.0.51#L40
matthew-brett added a commit to matthew-brett/windows-wheel-builder that referenced this issue Jul 1, 2017
@matthew-brett
Test without msys2 on the PATH
8fdf590 
See: numpy/numpy#9342
@carlkl
Copy link
Member
carlkl commented Jun 16, 2022

@matthew-brett, is this issue still valid?

@carlkl
Copy link
Member
carlkl commented Jun 16, 2022

@matthew-brett, is this issue still topical?

@carlkl
Copy link
Member
carlkl commented Jun 18, 2022

I successfully tested this with Windows 10 and

Python 3.9.4 (tags/v3.9.4:1f2e308, Apr  6 2021, 13:40:21) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

and the UCRT64 GCC toolchain from msys2:

gcc version 12.1.0 (Rev2, Built by MSYS2 project)

9342.log

However, a lot of tests are skipped right now due to #9673, so I leave this open for now.

@carlkl
Copy link
Member
carlkl commented Jun 19, 2022

With un-skipped test (#9673) I got a lot of errors, but not hang or segfault with python -c "import numpy.f2py; numpy.f2py.test(verbose=4)"

@matthew-brett
Copy link
Contributor Author

@carlkl - I get the same as you, after laboriously editing out the decorators for skipping the tests:

433 passed, 36 deselected, 122 errors in 149.94s (0:02:29)

@matthew-brett
Copy link
Contributor Author 8000

Errors appear to be of form:

ImportError: DLL load failed while importing _test_ext_module_5430: The specified module could not be found.

@carlkl
Copy link
Member
carlkl commented Jun 20, 2022 •
edited
Loading

If I manually add '--compiler=mingw32' to f2py_opts in f2py/tests/utils.py I get rid of most of the errors. I think this is due to the fact, that the '--compiler=mingw32' flag is mandatory if using mingw-w64. This means f2py/tests/utils.py needs some extra logic for the mingw-w64 case.

However, there are 39 errors left, also with i.e. ImportError: DLL load failed while importing _test_ext_module_5403. For this specific case I manually edited part of the log from python -m pytest -l -k "test_callback" --verbosity=4:

_test_ext_module_5403
---------------------

...
INFO: compiling Fortran sources
INFO: Fortran f77 compiler: C:\\Users\\devel\\AppData\\Local\\pool\\msys2\\ucrt64\\bin\\gfortran.exe -Wall -g -ffixed-form -fno-second-underscore -O3 -funroll-loops
Fortran f90 compiler: C:\\Users\\devel\\AppData\\Local\\pool\\msys2\\ucrt64\\bin\\gfortran.exe -Wall -g -fno-second-underscore -O3 -funroll-loops
Fortran fix compiler: C:\\Users\\devel\\AppData\\Local\\pool\\msys2\\ucrt64\\bin\\gfortran.exe -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -O3 -funroll-loops
creating C:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpqakzci6_\\Release\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpxtytyh1g
INFO: compile options: '-IC:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpqakzci6_\\src.win-amd64-3.9 
  -IC:\\Users\\devel\\AppData\\Local\\tmp\\Python39\\Lib\\site-packages\\numpy\\core\\include 
  -IC:\\Users\\devel\\AppData\\Local\\tmp\\Python39\\include 
  -IC:\\Users\\devel\\AppData\\Local\\tmp\\Python39\\Include -cINFO: g"
fortran.exe:f77: C:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpxtytyh1g\\tmp2fg4qn3k.f
INFO: C:\\Users\\devel\\AppData\\Local\\pool\\msys2\\ucrt64\\bin\\gfortran.exe 
  -Wall -g -Wall -g -shared 
  C:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpqakzci6_\\Release\\users\\devel\\appdata\\local\\msys2\\tmp\\tmpqakzci6_\\src.win-amd64-3.9\\_test_ext_module_5403module.o 
  C:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpqakzci6_\\Release\\users\\devel\\appdata\\local\\msys2\\tmp\\tmpqakzci6_\\src.win-amd64-3.9\\fortranobject.o 
  C:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpqakzci6_\\Release\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpxtytyh1g\\tmp2fg4qn3k.o 
  -LC:\\Users\\devel\\AppData\\Local\\pool\\msys2\\ucrt64\\lib\\gcc\\x86_64-w64-mingw32\\12.1.0 
  -LC:\\Users\\devel\\AppData\\Local\\tmp\\Python39\\libs 
  -LC:\\Users\\devel\\AppData\\Local\\tmp\\Python39 
  -LC:\\Users\\devel\\AppData\\Local\\tmp\\Python39\\PCbuild\\amd64 
  -lpython39 -lgfortran 
  -o  .\\_test_ext_module_5403.cp39-win_amd64.pyd

Removing build directory C:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpqakzci6_

        p          = <Popen: returncode: 0 args: ['C:\\Users\\devel\\AppData\\Local\\tmp...>
        skip       = []
        source_files = ['C:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmp2fg4qn3k.f']
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = '_test_ext_module_5403', package = None

    def import_module(name, package=None):
        """Import a module.
    
        The 'package' argument is required when performing a relative import. It
        specifies the package to use as the anchor point from which to resolve the
        relative import to an absolute import.
    
        """
        level = 0
        if name.startswith('.'):
            if not package:
                msg = ("the 'package' argument is required to perform a relative "
                       "import for {!r}")
                raise TypeError(msg.format(name))
            for character in name:
                if character != '.':
                    break
                level += 1
>       return _bootstrap._gcd_import(name[level:], package, level)
E       ImportError: DLL load failed while importing _test_ext_module_5403: Das angegebene Modul wurde nicht gefunden.

It seems, that `.\_test_ext_module_5403.cp39-win_amd64.pyd was successfully build, but the build directory has been removed before running the test?

@HaoZeke
Copy link
Member
HaoZeke commented Nov 11, 2023

@carlkl or @matthew-brett, it'd be really great to see if these errors are still present since the transition to meson (and after #25111).

@HaoZeke HaoZeke changed the title Windows f2py tests hang indefinitely with msys gcc-fortran on PATH TST: Windows f2py tests hang indefinitely with msys gcc-fortran on PATH Nov 11, 2023
@carlkl
Copy link
Member
carlkl commented Nov 12, 2023

I can certainly investigate this, but it will take some time ...

@carlkl
Copy link
Member
carlkl commented Nov 15, 2023 •
edited
Loading

Short test with Windows 10 64-bit

python -c "import numpy.f2py; numpy.f2py.test(verbose=4)"

C:\Users\dev\AppData\Local\home\devel\Python311\Lib\site-packages\numpy\_pytesttester.py:143: DeprecationWarning:

  `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
  of the deprecation of `distutils` itself. It will be removed for
  Python >= 3.12. For older Python versions it will remain present.
  It is recommended to use `setuptools < 60.0` for those Python versions.
  For more details, see:
    https://numpy.org/devdocs/reference/distutils_status_migration.html


  from numpy.distutils import cpuinfo
NumPy version 2.0.0.dev0+git20231114.0f61000
NumPy CPU features:  SSE SSE2 SSE3 SSSE3* SSE41* POPCNT* SSE42* AVX* F16C* FMA3* AVX2* AVX512F* AVX512CD* AVX512_SKX* AVX512_CLX? AVX512_CNL? AVX512_ICL?
================================================= test session starts =================================================
platform win32 -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0 -- C:\Users\dev\AppData\Local\home\devel\Python311\python.exe
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(WindowsPath('C:/Users/dev/AppData/Local/home/tmp3/np_PR25111/numpy_CI_runs_6872272007_job_18690474876/patch/.hypothesis/examples'))
rootdir: C:\Users\dev\AppData\Local\home\tmp3\np_PR25111\numpy_CI_runs_6872272007_job_18690474876\patch
plugins: hypothesis-6.88.1, cov-4.1.0, timeout-2.2.0, xdist-3.3.1
collected 803 items / 36 deselected / 767 selected

=============================================== short test summary info ===============================================
FAILED tests/test_f2py2e.py::test_gen_pyf_no_overwrite - OSError: [WinError 6] Das Handle ist ungültig
================== 1 failed, 59 passed, 704 skipped, 36 deselected, 3 xfailed, 22 warnings in 35.34s ==================

so must of the the tests are skipped, is this intended? test_gen_pyf_no_overwrite fails with an OSError.

@HaoZeke
Copy link
Member
HaoZeke commented Nov 15, 2023

Short test with Windows 10 64-bit

Block (27 lines)

python -c "import numpy.f2py; numpy.f2py.test(verbose=4)"

C:\Users\dev\AppData\Local\home\devel\Python311\Lib\site-packages\numpy\_pytesttester.py:143: DeprecationWarning:

  `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
  of the deprecation of `distutils` itself. It will be removed for
  Python >= 3.12. For older Python versions it will remain present.
  It is recommended to use `setuptools < 60.0` for those Python versions.
  For more details, see:
    https://numpy.org/devdocs/reference/distutils_status_migration.html


  from numpy.distutils import cpuinfo
NumPy version 2.0.0.dev0+git20231114.0f61000
NumPy CPU features:  SSE SSE2 SSE3 SSSE3* SSE41* POPCNT* SSE42* AVX* F16C* FMA3* AVX2* AVX512F* AVX512CD* AVX512_SKX* AVX512_CLX? AVX512_CNL? AVX512_ICL?
================================================= test session starts =================================================
platform win32 -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0 -- C:\Users\dev\AppData\Local\home\devel\Python311\python.exe
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(WindowsPath('C:/Users/dev/AppData/Local/home/tmp3/np_PR25111/numpy_CI_runs_6872272007_job_18690474876/patch/.hypothesis/examples'))
rootdir: C:\Users\dev\AppData\Local\home\tmp3\np_PR25111\numpy_CI_runs_6872272007_job_18690474876\patch
plugins: hypothesis-6.88.1, cov-4.1.0, timeout-2.2.0, xdist-3.3.1
collected 803 items / 36 deselected / 767 selected

=============================================== short test summary info ===============================================
FAILED tests/test_f2py2e.py::test_gen_pyf_no_overwrite - OSError: [WinError 6] Das Handle ist ungültig
================== 1 failed, 59 passed, 704 skipped, 36 deselected, 3 xfailed, 22 warnings in 35.34s ==================

so must of the the tests are skipped, is this intended? test_gen_pyf_no_overwrite fails with an OSError.

Ah, sorry, jumped the gun, testing these should be after #25134..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
05 - Testing component: CI component: numpy.f2py
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@matthew-brett @pearu @HaoZeke @carlkl
0