8000 BLD: default to C17 rather than C99 by rgommers · Pull Request #20515 · scipy/scipy · GitHub
[go: up one dir, main page]

Skip to content

BLD: default to C17 rather than C99 #20515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 19, 2024
Merged

Conversation

rgommers
Copy link
Member
@rgommers rgommers commented Apr 18, 2024

This update should be a safe bump, because C17 is a pure maintenance update of C11 with no new features; and C11 is mostly a maintenance/bugfix update of C99 with a few new features. Our toolchain roadmap, which was updated just days ago, says that we can use C17 given our minimum compiler versions.

I also note that in numpy#25072, the numpy default was updated from C99 to C11 and that went fine (the one change needed was a Cython bug fix, which went into Cython 3.0.6 - and our minimum version is currently 3.0.8).

Finally, another motivation to do this now is that under the --disable-gil build of CPython 3.13, Cython currently generates code that uses static_assert, which is a C11 feature. Hence our current default of -std=c99 doesn't build with that nogil build. (hat tip to @ngoldbaum for pointing out this problem). EDIT: PEP 7 is also relevant here, since it specifies what C standard version is used by CPython (see https://peps.python.org/pep-0007/#c-dialect). To be safe, be should be using >= the version used by CPython.

For a description of changes in C17, see https://gustedt.wordpress.com/2018/04/17/c17/.

Note: opening as draft until I've checked the CI logs for (the absence of) new warnings.

@rgommers rgommers added the Build issues Issues with building from source, including different choices of architecture, compilers and OS label Apr 18, 2024
@github-actions github-actions bot added the Meson Items related to the introduction of Meson as the new build system for SciPy label Apr 18, 2024
@rgommers
Copy link
Member Author

Okay, nothing is ever a one line diff it seems .... the Windows failure is real (only with numpy 1.23.5):

[184/1446] Compiling C object scipy/linalg/_flapack.cp310-win_amd64.pyd.p/meson-generated_..__flapackmodule.c.obj
FAILED: scipy/linalg/_flapack.cp310-win_amd64.pyd.p/meson-generated_..__flapackmodule.c.obj 
"cc" "-Iscipy\linalg\_flapack.cp310-win_amd64.pyd.p" "-Iscipy\linalg" "-I..\scipy\linalg" "-IC:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\numpy\core\include" "-IC:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\numpy\f2py\src" "-IC:/hostedtoolcache/windows/Python/3.10.11/x64/Lib/site-packages/scipy_openblas32/include" "-IC:\hostedtoolcache\windows\Python\3.10.11\x64\Include" "-fvisibility=hidden" "-fdiagnostics-color=always" "-D_FILE_OFFSET_BITS=64" "-Wall" "-Winvalid-pch" "-std=c17" "-O2" "-g" "-Wno-unused-but-set-variable" "-Wno-unused-function" "-Wno-conversion" "-Wno-misleading-indentation" "-mlong-double-64" "-D__USE_MINGW_ANSI_STDIO=1" "-DMS_WIN64=" "-Wno-empty-body" -MD -MQ scipy/linalg/_flapack.cp310-win_amd64.pyd.p/meson-generated_..__flapackmodule.c.obj -MF "scipy\linalg\_flapack.cp310-win_amd64.pyd.p\meson-generated_..__flapackmodule.c.obj.d" -o scipy/linalg/_flapack.cp310-win_amd64.pyd.p/meson-generated_..__flapackmodule.c.obj "-c" scipy/linalg/_flapackmodule.c
In file included from C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\numpy\core\include/numpy/ndarraytypes.h:1948,
                 from C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\numpy\core\include/numpy/ndarrayobject.h:12,
                 from C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\numpy\core\include/numpy/arrayobject.h:5,
                 from C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\numpy\f2py\src/fortranobject.h:13,
                 from scipy/linalg/_flapackmodule.c:23:
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\numpy\core\include/numpy/npy_1_7_deprecated_api.h:14:9: note: '#pragma message: C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\numpy\core\include/numpy/npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION'
   14 | #pragma message(_WARN___LOC__"Using deprecated NumPy API, disable it with " \
      |         ^~~~~~~
scipy/linalg/_flapackmodule.c:101:10: fatal error: threads.h: No such file or directory
  101 | #include <threads.h>
      |          ^~~~~~~~~~~
compilation terminated.

@ev-br
Copy link
Member
ev-br commented Apr 18, 2024

So f2py C17 support in numpy seems to need some work/testing, it seems

@rgommers
Copy link
Member Author

I tested combinations of numpy and C standard versions. Conclusion:

  • for numpy 1.23.5-1.25.2, the threads.h build error on Windows shows up for both C11 and C17, only -std=c99 works
  • for numpy 1.26.4, the problem is gone, and both C11 and C17 work fine

numpy 1.26.x was the first minor version to be built with meson, so we probably fixed the threads.h issue as part of the effort to squash all warnings coming from f2py-generated C code.

I have not yet checked if we can work around the problem somehow.

@rgommers
Copy link
Member Author
rgommers commented Apr 19, 2024

The root cause is a bug in older versions of npy_os.h (numpy/numpy#24761), which was only fixed in numpy 1.26.1. There's a failure to detect MinGW, and after that a problematic usage of threads.h on Windows was guarded by __STDC_VERSION__ >= 201112L (i.e., C11 and up).

This update should be a safe bump, because C17 is a pure maintenance
update of C11 with no new features; and C11 is mostly a
maintenance/bugfix update of C99 with a few new features.
Our [toolchain roadmap](http://scipy.github.io/devdocs/dev/toolchain.html#c-compilers),
which was updated just days ago, says that we can use C17
given our minimum compiler versions.

I also note that in numpy#25072, the numpy default was updated from
C99 to C11 and that went fine (the one change needed was a Cython bug
fix, which went into Cython 3.0.6 - and our minimum version is currently
3.0.8).

Finally, another motivation to do this now is that under the
`--disable-gil` build of CPython 3.13, Cython currently generates code
that uses `static_assert`, which is a C11 feature. Hence our current
default of `-std=c99` doesn't build with that nogil build.
…build]

Due to numpy#24761, which we hit when building against numpy versions
<1.26.1
@lucascolley lucascolley added this to the 1.14.0 milestone Apr 19, 2024
@rgommers rgommers marked this pull request as ready for review April 19, 2024 12:06
@rgommers
Copy link
Member Author

Okay, this is all happy now.

Copy link
Member
@lucascolley lucascolley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks Ralf!

@lucascolley lucascolley merged commit 50274cc into scipy:main Apr 19, 2024
@rgommers rgommers deleted the default-to-c17 branch April 19, 2024 12:58
@rgommers rgommers added the free-threading Items related to supporting free-threaded (a.k.a. "no-GIL") builds of CPython label Apr 19, 2024
@ngoldbaum
Copy link
Contributor

Over in scikit-learn, we hit an issue building under pyiodide after moving to C17: scikit-learn/scikit-learn#29013. From what I can tell, scipy currently doesn't build under pyodide (#15290) due to missing fortran support, so scipy doesn't have a similar issue. However, if pyodide doesn't grow C17 support before that happens, then scipy specifying C17 in meson.build might be an issue.

Just a heads' up!

@rgommers
Copy link
Member Author

Thanks for the heads up. Looking at getting that fixed in Meson (pyodide/pyodide#4762).

Scipy currently doesn't build under pyodide

SciPy does build in Pyodide in-tree, it's at 1.12.0 right now with an update to 1.13.0 in progress. C17 support will become needed for 1.14.0 - we should have this fix included in a new Meson release by then.

@rgommers
Copy link
Member Author

Fix in progress: mesonbuild/meson#13221

@HanNotSoSolo
Copy link
HanNotSoSolo commented Mar 13, 2025

Hello, I was building Scipy in debug mode and got an error related with this thread.
I followed the instructions given to install Scipy, with the exception of the last one where I typed python dev.py build -d. Then I got this message:

💻  meson setup /my/path/to/env/scipy/build --prefix /my/path/to/env/scipy/build-install -Dbuildtype=debug
The Meson build system
Version: 1.7.0
Source dir: /my/path/to/env/scipy
Build dir:/my/path/to/env/scipy/build
Build type: native build
Project name: scipy
Project version: 1.16.0.dev0+git20250312.5ff7297

meson.build:1:0: ERROR: None of values ['c17'] are supported by the C compiler. Possible values for option "C_std" are ['none', 'c89', 'c99', 'c11', 'gnu89', 'gnu99', 'gnu11']

A full log can be found at /my/path/to/env/scipy/build/meson-logs/meson-log.txt
Meson build setup failed!

The full log tells this:

Build started at 2025-03-13T11:29:29.816022
Main binary: /my/path/to/home/miniconda3/envs/femTEST/bin/python3.13
Build Options: -Dbuildtype=debug -Dprefix=/my/path/to/env/scipy/build-install
Python system: Linux
The Meson build system
Version: 1.7.0
Source dir: /my/path/to/env/scipy
Build dir: /my/path/to/env/scipy/build
Build type: native build
Running command:/my/path/to/home/miniconda3/envs/femTEST/bin/python3.13 /my/path/to/env/scipy/tools/gitversion.py
--- stdout ---
1.16.0.dev0+git20250312.5ff7297

--- stderr ---


Project name: scipy
Project version: 1.16.0.dev0+git20250312.5ff7297
-----------
Detecting compiler via: `icc --version` -> 0
stdout:
icc (ICC) 2021.2.0 20210228
Copyright (C) 1985-2021 Intel Corporation.  All rights reserved.
-----------
-----
-----------
Detecting linker via: `icc -Wl,--version` -> 0
stdout:
GNU ld (GNU Binutils) 2.43
Copyright (C) 2024 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
-----------

meson.build:1:0: ERROR: None of values ['c17'] are supported by the C compiler. Possible values for option "C_std" are ['none', 'c89', 'c99', 'c11', 'gnu89', 'gnu99', 'gnu11']

Do you have any idea of how could I solve this?


Edit: I tried changing the meson.build file, asking for c_std=c11 instead of c_std=c17, since it was having problems with it. Now The error that appears is:
meson.build:1:0: ERROR: Compiler icpc cannot compile programs.

@rgommers
Copy link
Member Author

I'm not sure if icc and icpc still work, they were deprecated by Intel a while back. Do you have the most recent icc version at least?

Downgrading to c11 should work I believe. For the icpc error I'd have to see more details. Please open a new issue for that and Cc me, that's more visible than commenting on a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Build issues Issues with building from source, including different choices of architecture, compilers and OS free-threading Items related to supporting free-threaded (a.k.a. "no-GIL") builds of CPython Meson Items related to the introduction of Meson as the new build system for SciPy
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants
0