8000 Merge pull request #26103 from seberg/visibility-hidden-table · numpy/numpy@2234793 · GitHub
[go: up one dir, main page]
Skip to content

Commit 2234793

Browse files
ngoldbaumngoldbaum
authored
Merge pull request #26103 from seberg/visibility-hidden-table
API: Default to hidden visibility for API tables
2 parents 12aa98e + e99f012 commit 2234793

File tree

4 files changed

+171
-25
lines changed

4 files changed

+171
-25
lines changed

‎doc/release/upcoming_changes/26103.c_api.rst

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
API symbols now hidden but customizable
2+
---------------------------------------
3+
NumPy now defaults to hide the API symbols it adds to allow all NumPy API
4+
usage.
5+
This means that by default you cannot dynamically fetch the NumPy API from
6+
another library (this was never possible on windows).
7+
8+
If you are experiencing linking errors related to ``PyArray_API`` or
9+
``PyArray_RUNTIME_VERSION``, you can define the
10+
:c:macro:`NPY_API_SYMBOL_ATTRIBUTE` to opt-out of this change.
11+
12+
If you are experiencing problems due to an upstream header including NumPy,
13+
the solution is to make sure you ``#include "numpy/ndarrayobject.h"`` before
14+
their header and import NumPy yourself based on :ref:`including-the-c-api`.
15+

‎doc/source/reference/c-api/array.rst

Lines changed: 140 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -3815,13 +3815,118 @@ Other conversions
38153815
in the *vals* array. The sequence can be smaller then *maxvals* as
38163816
the number of converted objects is returned.
38173817
3818+
.. _including-the-c-api:
38183819
3819-
Miscellaneous
3820-
-------------
3820+
10000 Including and importing the C API
3821+
---------------------------------
38213822
3823+
To use the NumPy C-API you typically need to include the
3824+
``numpy/ndarrayobject.h`` header and ``numpy/ufuncobject.h`` for some ufunc
3825+
related functionality (``arrayobject.h`` is an alias for ``ndarrayobject.h``).
38223826
3823-
Importing the API
3824-
~~~~~~~~~~~~~~~~~
3827+
These two headers export most relevant functionality. In general any project
3828+
which uses the NumPy API must import NumPy using one of the functions
3829+
``PyArray_ImportNumPyAPI()`` or ``import_array()``.
3830+
In some places, functionality which requires ``import_array()`` is not
3831+
needed, because you only need type definitions. In this case, it is
3832+
sufficient to include ``numpy/ndarratypes.h``.
3833+
3834+
For the typical Python project, multiple C or C++ files will be compiled into
3835+
a single shared object (the Python C-module) and ``PyArray_ImportNumPyAPI()``
3836+
should be called inside it's module initialization.
3837+
3838+
When you have a single C-file, this will consist of:
3839+
3840+
.. code-block:: c
3841+
3842+
#include "numpy/ndarrayobject.h"
3843+
3844+
PyMODINIT_FUNC PyInit_my_module(void)
3845+
{
3846+
if (PyArray_ImportNumPyAPI() < 0) {
3847+
return NULL;
3848+
}
3849+
/* Other initialization code. */
3850+
}
3851+
3852+
However, most projects will have additional C files which are all
3853+
linked together into a single Python module.
3854+
In this case, the helper C files typically do not have a canonical place
3855+
where ``PyArray_ImportNumPyAPI`` should be called (although it is OK and
3856+
fast to call it often).
3857+
3858+
To solve this, NumPy provides the following pattern that the the main
3859+
file is modified to define ``PY_ARRAY_UNIQUE_SYMBOL`` before the include:
3860+
3861+
.. code-block:: c
3862+
3863+
/* Main module file */
3864+
#define PY_ARRAY_UNIQUE_SYMBOL MyModule
3865+
#include "numpy/ndarrayobject.h"
3866+
3867+
PyMODINIT_FUNC PyInit_my_module(void)
3868+
{
3869+
if (PyArray_ImportNumPyAPI() < 0) {
3870+
return NULL;
3871+
}
3872+
/* Other initialization code. */
3873+
}
3874+
3875+
while the other files use:
3876+
3877+
.. code-block:: C
3878+
3879+
/* Second file without any import */
3880+
#define NO_IMPORT_ARRAY
3881+
#define PY_ARRAY_UNIQUE_SYMBOL MyModule
3882+
#include "numpy/ndarrayobject.h"
3883+
3884+
You can of course add the defines to a local header used throughout.
3885+
You just have to make sure that the main file does _not_ define
3886+
``NO_IMPORT_ARRAY``.
3887+
3888+
For ``numpy/ufuncobject.h`` the same logic applies, but the unique symbol
3889+
mechanism is ``#define PY_UFUNC_UNIQUE_SYMBOL`` (both can match).
3890+
3891+
Additionally, you will probably wish to add a
3892+
``#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION``
3893+
to avoid warnings about possible use of old API.
3894+
3895+
.. note::
3896+
If you are experiencing access violations make sure that the NumPy API
3897+
was properly imported and the symbol ``PyArray_API`` is not ``NULL``.
3898+
When in a debugger, this symbols actual name will be
3899+
``PY_ARRAY_UNIQUE_SYMBOL``+``PyArray_API``, so for example
3900+
``MyModulePyArray_API`` in the above.
3901+
(E.g. even a ``printf("%p\n", PyArray_API);`` just before the crash.)
3902+
3903+
3904+
Mechanism details and dynamic linking
3905+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3906+
3907+
The main part of the mechanism is that without NumPy needs to define
3908+
a ``void **PyArray_API`` table for you to look up all functions.
3909+
Depending on your macro setup, this takes different routes depending on
3910+
whether :c:macro:`NO_IMPORT_ARRAY` and :c:macro:`PY_ARRAY_UNIQUE_SYMBOL`
3911+
are defined:
3912+
3913+
* If neither is defined, the C-API is declared to
3914+
``static void **PyArray_API``, so it is only visible within the
3915+
compilation unit/file using ``#includes numpy/arrayobject.h``.
3916+
* If only ``PY_ARRAY_UNIQUE_SYMBOL`` is defined (it could be empty) then
3917+
the it is declared to a non-static ``void **`` allowing it to be used
3918+
by other files which are linked.
3919+
* If ``NO_IMPORT_ARRAY`` is defined, the table is declared as
3920+
``extern void **``, meaning that it must be linked to a file which does not
3921+
use ``NO_IMPORT_ARRAY``.
3922+
3923+
The ``PY_ARRAY_UNIQUE_SYMBOL`` mechanism additionally mangles the names to
3924+
avoid conflicts.
3925+
3926+
.. versionchanged::
3927+
NumPy 2.1 changed the headers to avoid sharing the table outside of a
3928+
single shared object/dll (this was always the case on Windows).
3929+
Please see :c:macro:`NPY_API_SYMBOL_ATTRIBUTE` for details.
38253930
38263931
In order to make use of the C-API from another extension module, the
38273932
:c:func:`import_array` function must be called. If the extension module is
@@ -3845,26 +3950,41 @@ the C-API is needed then some additional steps must be taken.
38453950
module that will make use of the C-API. It imports the module
38463951
where the function-pointer table is stored and points the correct
38473952
variable to it.
3953+
This macro includes a ``return NULL;`` on error, so that
3954+
``PyArray_ImportNumPyAPI()`` is preferable for custom error checking.
3955+
You may also see use of ``_import_array()`` (a function, not
3956+
a macro, but you may want to raise a better error if it fails) and
3957+
the variations ``import_array1(ret)`` which customizes the return value.
38483958
38493959
.. c:macro:: PY_ARRAY_UNIQUE_SYMBOL
38503960
3961+
.. c:macro:: NPY_API_SYMBOL_ATTRIBUTE
3962+
3963+
.. versionadded:: 2.1
3964+
3965+
An additional symbol which can be used to share e.g. visibility beyond
3966+
shared object boundaries.
3967+
By default, NumPy adds the C visibility hidden attribute (if available):
3968+
``void __attribute__((visibility("hidden"))) **PyArray_API;``.
3969+
You can change this by defining ``NPY_API_SYMBOL_ATTRIBUTE``, which will
3970+
make this:
3971+
``void NPY_API_SYMBOL_ATTRIBUTE **PyArray_API;`` (with additional
3972+
name mangling via the unique symbol).
3973+
3974+
Adding an empty ``#define NPY_API_SYMBOL_ATTRIBUTE`` will have the same
3975+
behavior as NumPy 1.x.
3976+
3977+
.. note::
3978+
Windows never had shared visbility although you can use this macro
3979+
to achieve it. We generally discourage sharing beyond shared boundary
3980+
lines since importing the array API includes NumPy version checks.
3981+
38513982
.. c:macro:: NO_IMPORT_ARRAY
38523983
3853-
Using these #defines you can use the C-API in multiple files for a
3854-
single extension module. In each file you must define
3855-
:c:macro:`PY_ARRAY_UNIQUE_SYMBOL` to some name that will hold the
3856-
C-API (*e.g.* myextension_ARRAY_API). This must be done **before**
3857-
including the numpy/arrayobject.h file. In the module
3858-
initialization routine you call :c:func:`import_array`. In addition,
3859-
in the files that do not have the module initialization
3860-
sub_routine define :c:macro:`NO_IMPORT_ARRAY` prior to including
3861-
numpy/arrayobject.h.
3862-
3863-
Suppose I have two files coolmodule.c and coolhelper.c which need
3864-
to be compiled and linked into a single extension module. Suppose
3865-
coolmodule.c contains the required initcool module initialization
3866-
function (with the import_array() function called). Then,
3867-
coolmodule.c would have at the top:
3984+
Defining ``NO_IMPORT_ARRAY`` before the ``ndarrayobject.h`` include
3985+
indicates that the NumPy C API import is handled in a different file
3986+
and the include mechanism will not be added here.
3987+
You must have one file without ``NO_IMPORT_ARRAY`` defined.
38683988
38693989
.. code-block:: c
38703990
@@ -3901,6 +4021,7 @@ the C-API is needed then some additional steps must be taken.
39014021
defaults to ``PyArray_API``, to whatever the macro is
39024022
#defined to.
39034023
4024+
39044025
Checking the API Version
39054026
~~~~~~~~~~~~~~~~~~~~~~~~
39064027

‎numpy/_core/code_generators/generate_numpy_api.py

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -33,13 +33,18 @@
3333
_NPY_VERSION_CONCAT_HELPER(PY_ARRAY_UNIQUE_SYMBOL)
3434
#endif
3535
36+
/* By default do not export API in an .so (was never the case on windows) */
37+
#ifndef NPY_API_SYMBOL_ATTRIBUTE
38+
#define NPY_API_SYMBOL_ATTRIBUTE NPY_VISIBILITY_HIDDEN
39+
#endif
40+
3641
#if defined(NO_IMPORT) || defined(NO_IMPORT_ARRAY)
37-
extern void **PyArray_API;
38-
extern int PyArray_RUNTIME_VERSION;
42+
extern NPY_API_SYMBOL_ATTRIBUTE void **PyArray_API;
43+
extern NPY_API_SYMBOL_ATTRIBUTE int PyArray_RUNTIME_VERSION;
3944
#else
4045
#if defined(PY_ARRAY_UNIQUE_SYMBOL)
41-
void **PyArray_API;
42-
int PyArray_RUNTIME_VERSION;
46+
NPY_API_SYMBOL_ATTRIBUTE void **PyArray_API;
47+
NPY_API_SYMBOL_ATTRIBUTE int PyArray_RUNTIME_VERSION;
4348
#else
4449
static void **PyArray_API = NULL;
4550
static int PyArray_RUNTIME_VERSION = 0;

‎numpy/_core/code_generators/generate_ufunc_api.py

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,16 @@
1818
#define PyUFunc_API PY_UFUNC_UNIQUE_SYMBOL
1919
#endif
2020
21+
/* By default do not export API in an .so (was never the case on windows) */
22+
#ifndef NPY_API_SYMBOL_ATTRIBUTE
23+
#define NPY_API_SYMBOL_ATTRIBUTE NPY_VISIBILITY_HIDDEN
24+
#endif
25+
2126
#if defined(NO_IMPORT) || defined(NO_IMPORT_UFUNC)
22-
extern void **PyUFunc_API;
27+
extern NPY_API_SYMBOL_ATTRIBUTE void **PyUFunc_API;
2328
#else
2429
#if defined(PY_UFUNC_UNIQUE_SYMBOL)
25-
void **PyUFunc_API;
30+
NPY_API_SYMBOL_ATTRIBUTE void **PyUFunc_API;
2631
#else
2732
static void **PyUFunc_API=NULL;
2833
#endif

0 commit comments

Comments
 (0)
0