Test numexpr against pytest-run-parallel on 3.13t #504

andfoy · 2025-02-25T22:35:48Z

Fixes #503

This PR checks that numexpr tests run in free-threaded Python 3.13t, both in non-concurrent as well as concurrent modes (via pytest-run-parallel). There were some trivial issues present in the library that caused major illegal memory accesses, which appeared due to changes on the memory allocation machinery in 3.13, those are fixed as part of this PR.

FrancescAlted · 2025-02-26T11:41:53Z

Thanks @andfoy. As it turns out, tests are not passing because pytest was not a dependency. Currently numexpr just uses unittest, and the command for running the test suite is python -c "import numexpr; numexpr.test()".

But I like pytest, so if you want to add it to the list of dependencies, you are welcome. However, note that we allowed numexpr installations to run tests in the above way, without any additional dependency (but we can pass without that feature, I guess).

FrancescAlted · 2025-02-27T16:33:06Z

Looks pretty good! @andfoy can you add a way for testing that the threaded version is passing? Or just adding cibw_build: ["cp3{..,13t} is testing the thing already? If you need to add a pytest dependency for that in CI, I am fine. Also, a benchmark (in bench/ dir) would be also very welcome.

Finally, @rgommers suggested saying something about the oversubscription issue (i.e. having threads in both the C and Python sides). Would you mind adding a small section in README as well?

Thanks!

rgommers · 2025-02-27T23:11:26Z

.github/workflows/build.yml

@@ -24,12 +24,13 @@ jobs:
      CIBW_BUILD: ${{ matrix.cibw_build }}
      CIBW_ARCHS_LINUX: ${{ matrix.arch }}
      CIBW_ARCHS_MACOS: "x86_64 arm64"
+      CIBW_FREE_THREADED_SUPPORT: true


Note that this recently changed to CIBW_ENABLE, so if its easy to bump cibuildwheel to 2.22.0 then using that may be more future-proof: https://cibuildwheel.pypa.io/en/stable/changelog/#v2220

rgommers · 2025-02-27T23:14:06Z

Nice, thanks for pushing on this @andfoy!

It might be useful to split the whitespace cleanup off to another PR, since that'll reduce the diff of this PR by half - easier to then see what it actually took to add free-threading support.

.github/workflows/build.yml

rgommers · 2025-03-05T19:36:26Z

CI is broken because the upload action v4 refuses to upload multiple zip files named artifact:

This should be easy to fix by using the name: key and setting it to something like ${{ <build-matrix-entry-here> }}-wheels: https://github.com/actions/upload-artifact?tab=readme-ov-file#upload-an-entire-directory

FrancescAlted · 2025-03-06T18:43:41Z

CI is broken because the upload action v4 refuses to upload multiple zip files named artifact:
This should be easy to fix by using the `name: ` key and setting it to something like `${{ }}-wheels`: https://github.com/actions/upload-artifact?tab=readme-ov-file#upload-an-entire-directory

Ok. I am trying to modernize the github build configuration on new PR #505 with @andfoy commits in. As soon as it turns green, I can proceed with the merge. BTW, do you think that you are finished with this one? On my side, I see it on a pretty good shape indeed.

Thanks for your work so far!

FrancescAlted · 2025-03-07T11:54:34Z

@andfoy I don't think you should spent more time fixing the CI builds, as this has been solved in #505, which should be merged soon. What I am missing is a benchmark showing the advantages of the new threaded model. numexpr is about performance, so most of the users will be curious on what advantages they can get, and how. At any rate, this can be added with another PR.

andfoy · 2025-03-07T17:20:33Z

@FrancescAlted just before merging, I need to mark the C modules as free-threaded safe

rgommers

This looks pretty good to me. Testing with ThreadSanitizer (as documented at https://py-free-threading.github.io/debugging/#running-python-under-threadsanitizer) turns up a few failures related to the global_state struct in numexpr/module.cpp|hpp:

% CFLAGS="-fsanitize=thread -O2 -g" CXXFLAGS="-fsanitize=thread -O2 -g" python3.14t -m pip install -e . -v --no-build-isolation

% pytest .
=========================================================================== test session starts ===========================================================================
platform darwin -- Python 3.14.0a5+, pytest-8.3.4, pluggy-1.5.0
rootdir: /Users/rgommers/code/tmp/numexpr
configfile: pyproject.toml
plugins: hypothesis-6.125.3
collected 108 items                                                                                                                                                       

numexpr/tests/test_numexpr.py ..F....................F...................................................x.....................FF.........                          [100%]

================================================================================ FAILURES =================================================================================
_________________________________________________________________ test_numexpr.test_locals_clears_globals _________________________________________________________________

self = <numexpr.tests.test_numexpr.test_numexpr testMethod=test_locals_clears_globals>

    @pytest.mark.thread_unsafe
    def test_locals_clears_globals(self):
        # Check for issue #313, whereby clearing f_locals also clear f_globals
        # if in the top-frame. This cannot be done inside `unittest` as it is always
        # executing code in a child frame.
        script = r';'.join([
                r"import numexpr as ne",
                r"a=10",
                r"ne.evaluate('1')",
                r"a += 1",
                r"ne.evaluate('2', local_dict={})",
                r"a += 1",
                r"ne.evaluate('3', global_dict={})",
                r"a += 1",
                r"ne.evaluate('4', local_dict={}, global_dict={})",
                r"a += 1",
            ])
        # Raises CalledProcessError on a non-normal exit
>       check = subprocess.check_call([sys.executable, '-c', script])

numexpr/tests/test_numexpr.py:355: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['/Users/rgommers/tsanvenv/bin/python', '-c', "import numexpr as ne;a=10;ne.evaluate('1');a += 1;ne.evaluate('2', local_dict={});a += 1;ne.evaluate('3', global_dict={});a += 1;ne.evaluate('4', local_dict={}, global_dict={});a += 1"],)
kwargs = {}, retcode = -6
cmd = ['/Users/rgommers/tsanvenv/bin/python', '-c', "import numexpr as ne;a=10;ne.evaluate('1');a += 1;ne.evaluate('2', local_dict={});a += 1;ne.evaluate('3', global_dict={});a += 1;ne.evaluate('4', local_dict={}, global_dict={});a += 1"]

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.
    
        The arguments are the same as for the call function.  Example:
    
        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['/Users/rgommers/tsanvenv/bin/python', '-c', "import numexpr as ne;a=10;ne.evaluate('1');a += 1;ne.evaluate('2', local_dict={});a += 1;ne.evaluate('3', global_dict={});a += 1;ne.evaluate('4', local_dict={}, global_dict={});a += 1"]' died with <Signals.SIGABRT: 6>.

../cpython/cpython-tsan/lib/python3.14t/subprocess.py:421: CalledProcessError
-------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------
==================
WARNING: ThreadSanitizer: data race (pid=2918)
  Write of size 4 at 0x000149d945e0 by thread T2:
    #0 th_worker(void*) <null>:1244664592 (interpreter.cpython-314t-darwin.so:arm64+0x2753c)

  Previous write of size 4 at 0x000149d945e0 by thread T1:
    #0 th_worker(void*) <null>:1244664592 (interpreter.cpython-314t-darwin.so:arm64+0x2753c)

  Location is global 'gs' at 0x000149d945b8 (interpreter.cpython-314t-darwin.so+0x345e0)

  Thread T2 (tid=5968240, running) created by main thread at:
    #0 pthread_create <null>:73400608 (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x32790)
    #1 init_threads() <null>:1244659952 (interpreter.cpython-314t-darwin.so:arm64+0x27b80)
    #2 numexpr_set_nthreads(int) <null>:1244659952 (interpreter.cpython-314t-darwin.so:arm64+0x27f08)
    #3 Py_set_num_threads(_object*, _object*) <null>:1244659952 (interpreter.cpython-314t-darwin.so:arm64+0x29fbc)
    #4 cfunction_call methodobject.c:562 (python3.14t:arm64+0x10011109c)
    #5 _PyObject_MakeTpCall call.c:242 (python3.14t:arm64+0x100077e30)
    #6 PyObject_Vectorcall call.c:327 (python3.14t:arm64+0x100078a94)
    #7 _PyEval_EvalFrameDefault generated_cases.c.h:1371 (python3.14t:arm64+0x100253314)
    #8 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #9 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #10 builtin_exec bltinmodule.c.h:560 (python3.14t:arm64+0x1002497c4)
    #11 cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c:452 (python3.14t:arm64+0x100110358)
    #12 _PyObject_Call call.c:348 (python3.14t:arm64+0x100078c4c)
    #13 PyObject_Call call.c:373 (python3.14t:arm64+0x100078d00)
    #14 _PyEval_EvalFrameDefault generated_cases.c.h:2421 (python3.14t:arm64+0x1002564a0)
    #15 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #16 _PyFunction_Vectorcall call.c (python3.14t:arm64+0x100078f40)
    #17 object_vacall call.c:819 (python3.14t:arm64+0x10007a828)
    #18 PyObject_CallMethodObjArgs call.c:880 (python3.14t:arm64+0x10007a594)
    #19 PyImport_ImportModuleLevelObject import.c:3764 (python3.14t:arm64+0x1002cc170)
    #20 _PyEval_ImportName ceval.c:2670 (python3.14t:arm64+0x1002728f4)
    #21 _PyEval_EvalFrameDefault generated_cases.c.h:5988 (python3.14t:arm64+0x10025f524)
    #22 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #23 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #24 run_mod pythonrun.c:1389 (python3.14t:arm64+0x10030c8d0)
    #25 _PyRun_SimpleStringFlagsWithName pythonrun.c:548 (python3.14t:arm64+0x100309920)
    #26 Py_RunMain main.c:760 (python3.14t:arm64+0x100342238)
    #27 pymain_main main.c:790 (python3.14t:arm64+0x100342724)
    #28 Py_BytesMain main.c:814 (python3.14t:arm64+0x1003427d0)
    #29 main python.c:15 (python3.14t:arm64+0x1000049a8)

  Thread T1 (tid=5968239, running) created by main thread at:
    #0 pthread_create <null>:73400416 (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x32790)
    #1 init_threads() <null>:1244664592 (interpreter.cpython-314t-darwin.so:arm64+0x27b80)
    #2 numexpr_set_nthreads(int) <null>:1244664592 (interpreter.cpython-314t-darwin.so:arm64+0x27f08)
    #3 Py_set_num_threads(_object*, _object*) <null>:1244664592 (interpreter.cpython-314t-darwin.so:arm64+0x29fbc)
    #4 cfunction_call methodobject.c:562 (python3.14t:arm64+0x10011109c)
    #5 _PyObject_MakeTpCall call.c:242 (python3.14t:arm64+0x100077e30)
    #6 PyObject_Vectorcall call.c:327 (python3.14t:arm64+0x100078a94)
    #7 _PyEval_EvalFrameDefault generated_cases.c.h:1371 (python3.14t:arm64+0x100253314)
    #8 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #9 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #10 builtin_exec bltinmodule.c.h:560 (python3.14t:arm64+0x1002497c4)
    #11 cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c:452 (python3.14t:arm64+0x100110358)
    #12 _PyObject_Call call.c:348 (python3.14t:arm64+0x100078c4c)
    #13 PyObject_Call call.c:373 (python3.14t:arm64+0x100078d00)
    #14 _PyEval_EvalFrameDefault generated_cases.c.h:2421 (python3.14t:arm64+0x1002564a0)
    #15 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #16 _PyFunction_Vectorcall call.c (python3.14t:arm64+0x100078f40)
    #17 object_vacall call.c:819 (python3.14t:arm64+0x10007a828)
    #18 PyObject_CallMethodObjArgs call.c:880 (python3.14t:arm64+0x10007a594)
    #19 PyImport_ImportModuleLevelObject import.c:3764 (python3.14t:arm64+0x1002cc170)
    #20 _PyEval_ImportName ceval.c:2670 (python3.14t:arm64+0x1002728f4)
    #21 _PyEval_EvalFrameDefault generated_cases.c.h:5988 (python3.14t:arm64+0x10025f524)
    #22 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #23 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #24 run_mod pythonrun.c:1389 (python3.14t:arm64+0x10030c8d0)
    #25 _PyRun_SimpleStringFlagsWithName pythonrun.c:548 (python3.14t:arm64+0x100309920)
    #26 Py_RunMain main.c:760 (python3.14t:arm64+0x100342238)
    #27 pymain_main main.c:790 (python3.14t:arm64+0x100342724)
    #28 Py_BytesMain main.c:814 (python3.14t:arm64+0x1003427d0)
    #29 main python.c:15 (python3.14t:arm64+0x1000049a8)

SUMMARY: ThreadSanitizer: data race (interpreter.cpython-314t-darwin.so:arm64+0x2753c) in th_worker(void*)+0x78
==================
ThreadSanitizer: reported 1 warnings
________________________________________________________________ test_numexpr2.test_locals_clears_globals _________________________________________________________________

self = <numexpr.tests.test_numexpr.test_numexpr2 testMethod=test_locals_clears_globals>

    @pytest.mark.thread_unsafe
    def test_locals_clears_globals(self):
        # Check for issue #313, whereby clearing f_locals also clear f_globals
        # if in the top-frame. This cannot be done inside `unittest` as it is always
        # executing code in a child frame.
        script = r';'.join([
                r"import numexpr as ne",
                r"a=10",
                r"ne.evaluate('1')",
                r"a += 1",
                r"ne.evaluate('2', local_dict={})",
                r"a += 1",
                r"ne.evaluate('3', global_dict={})",
                r"a += 1",
                r"ne.evaluate('4', local_dict={}, global_dict={})",
                r"a += 1",
            ])
        # Raises CalledProcessError on a non-normal exit
>       check = subprocess.check_call([sys.executable, '-c', script])

numexpr/tests/test_numexpr.py:355: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['/Users/rgommers/tsanvenv/bin/python', '-c', "import numexpr as ne;a=10;ne.evaluate('1');a += 1;ne.evaluate('2', local_dict={});a += 1;ne.evaluate('3', global_dict={});a += 1;ne.evaluate('4', local_dict={}, global_dict={});a += 1"],)
kwargs = {}, retcode = -6
cmd = ['/Users/rgommers/tsanvenv/bin/python', '-c', "import numexpr as ne;a=10;ne.evaluate('1');a += 1;ne.evaluate('2', local_dict={});a += 1;ne.evaluate('3', global_dict={});a += 1;ne.evaluate('4', local_dict={}, global_dict={});a += 1"]

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.
    
        The arguments are the same as for the call function.  Example:
    
        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['/Users/rgommers/tsanvenv/bin/python', '-c', "import numexpr as ne;a=10;ne.evaluate('1');a += 1;ne.evaluate('2', local_dict={});a += 1;ne.evaluate('3', global_dict={});a += 1;ne.evaluate('4', local_dict={}, global_dict={});a += 1"]' died with <Signals.SIGABRT: 6>.

../cpython/cpython-tsan/lib/python3.14t/subprocess.py:421: CalledProcessError
-------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------
==================
WARNING: ThreadSanitizer: data race (pid=2942)
  Write of size 4 at 0x00014d0d45e0 by thread T2:
    #0 th_worker(void*) <null>:1298141968 (interpreter.cpython-314t-darwin.so:arm64+0x2753c)

  Previous write of size 4 at 0x00014d0d45e0 by thread T1:
    #0 th_worker(void*) <null>:1298141968 (interpreter.cpython-314t-darwin.so:arm64+0x2753c)

  Location is global 'gs' at 0x00014d0d45b8 (interpreter.cpython-314t-darwin.so+0x345e0)

  Thread T2 (tid=5968414, running) created by main thread at:
    #0 pthread_create <null>:126877984 (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x32790)
    #1 init_threads() <null>:1298137328 (interpreter.cpython-314t-darwin.so:arm64+0x27b80)
    #2 numexpr_set_nthreads(int) <null>:1298137328 (interpreter.cpython-314t-darwin.so:arm64+0x27f08)
    #3 Py_set_num_threads(_object*, _object*) <null>:1298137328 (interpreter.cpython-314t-darwin.so:arm64+0x29fbc)
    #4 cfunction_call methodobject.c:562 (python3.14t:arm64+0x10011109c)
    #5 _PyObject_MakeTpCall call.c:242 (python3.14t:arm64+0x100077e30)
    #6 PyObject_Vectorcall call.c:327 (python3.14t:arm64+0x100078a94)
    #7 _PyEval_EvalFrameDefault generated_cases.c.h:1371 (python3.14t:arm64+0x100253314)
    #8 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #9 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #10 builtin_exec bltinmodule.c.h:560 (python3.14t:arm64+0x1002497c4)
    #11 cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c:452 (python3.14t:arm64+0x100110358)
    #12 _PyObject_Call call.c:348 (python3.14t:arm64+0x100078c4c)
    #13 PyObject_Call call.c:373 (python3.14t:arm64+0x100078d00)
    #14 _PyEval_EvalFrameDefault generated_cases.c.h:2421 (python3.14t:arm64+0x1002564a0)
    #15 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #16 _PyFunction_Vectorcall call.c (python3.14t:arm64+0x100078f40)
    #17 object_vacall call.c:819 (python3.14t:arm64+0x10007a828)
    #18 PyObject_CallMethodObjArgs call.c:880 (python3.14t:arm64+0x10007a594)
    #19 PyImport_ImportModuleLevelObject import.c:3764 (python3.14t:arm64+0x1002cc170)
    #20 _PyEval_ImportName ceval.c:2670 (python3.14t:arm64+0x1002728f4)
    #21 _PyEval_EvalFrameDefault generated_cases.c.h:5988 (python3.14t:arm64+0x10025f524)
    #22 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #23 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #24 run_mod pythonrun.c:1389 (python3.14t:arm64+0x10030c8d0)
    #25 _PyRun_SimpleStringFlagsWithName pythonrun.c:548 (python3.14t:arm64+0x100309920)
    #26 Py_RunMain main.c:760 (python3.14t:arm64+0x100342238)
    #27 pymain_main main.c:790 (python3.14t:arm64+0x100342724)
    #28 Py_BytesMain main.c:814 (python3.14t:arm64+0x1003427d0)
    #29 main python.c:15 (python3.14t:arm64+0x1000049a8)

  Thread T1 (tid=5968413, running) created by main thread at:
    #0 pthread_create <null>:126877792 (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x32790)
    #1 init_threads() <null>:1298141968 (interpreter.cpython-314t-darwin.so:arm64+0x27b80)
    #2 numexpr_set_nthreads(int) <null>:1298141968 (interpreter.cpython-314t-darwin.so:arm64+0x27f08)
    #3 Py_set_num_threads(_object*, _object*) <null>:1298141968 (interpreter.cpython-314t-darwin.so:arm64+0x29fbc)
    #4 cfunction_call methodobject.c:562 (python3.14t:arm64+0x10011109c)
    #5 _PyObject_MakeTpCall call.c:242 (python3.14t:arm64+0x100077e30)
    #6 PyObject_Vectorcall call.c:327 (python3.14t:arm64+0x100078a94)
    #7 _PyEval_EvalFrameDefault generated_cases.c.h:1371 (python3.14t:arm64+0x100253314)
    #8 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #9 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #10 builtin_exec bltinmodule.c.h:560 (python3.14t:arm64+0x1002497c4)
    #11 cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c:452 (python3.14t:arm64+0x100110358)
    #12 _PyObject_Call call.c:348 (python3.14t:arm64+0x100078c4c)
    #13 PyObject_Call call.c:373 (python3.14t:arm64+0x100078d00)
    #14 _PyEval_EvalFrameDefault generated_cases.c.h:2421 (python3.14t:arm64+0x1002564a0)
    #15 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #16 _PyFunction_Vectorcall call.c (python3.14t:arm64+0x100078f40)
    #17 object_vacall call.c:819 (python3.14t:arm64+0x10007a828)
    #18 PyObject_CallMethodObjArgs call.c:880 (python3.14t:arm64+0x10007a594)
    #19 PyImport_ImportModuleLevelObject import.c:3764 (python3.14t:arm64+0x1002cc170)
    #20 _PyEval_ImportName ceval.c:2670 (python3.14t:arm64+0x1002728f4)
    #21 _PyEval_EvalFrameDefault generated_cases.c.h:5988 (python3.14t:arm64+0x10025f524)
    #22 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #23 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #24 run_mod pythonrun.c:1389 (python3.14t:arm64+0x10030c8d0)
    #25 _PyRun_SimpleStringFlagsWithName pythonrun.c:548 (python3.14t:arm64+0x100309920)
    #26 Py_RunMain main.c:760 (python3.14t:arm64+0x100342238)
    #27 pymain_main main.c:790 (python3.14t:arm64+0x100342724)
    #28 Py_BytesMain main.c:814 (python3.14t:arm64+0x1003427d0)
    #29 main python.c:15 (python3.14t:arm64+0x1000049a8)

SUMMARY: ThreadSanitizer: data race (interpreter.cpython-314t-darwin.so:arm64+0x2753c) in th_worker(void*)+0x78
==================
ThreadSanitizer: reported 1 warnings
_______________________________________________________________ test_threading_config.test_max_threads_set ________________________________________________________________

self = <numexpr.tests.test_numexpr.test_threading_config testMethod=test_max_threads_set>

    def test_max_threads_set(self):
        # Has to be done in a subprocess as `importlib.reload` doesn't let us
        # re-initialize the threadpool
        script = '\n'.join([
                "import os",
                "os.environ['NUMEXPR_MAX_THREADS'] = '4'",
                "import numexpr",
                "assert(numexpr.MAX_THREADS == 4)",
                "exit(0)"])
>       subprocess.check_call([sys.executable, '-c', script])

numexpr/tests/test_numexpr.py:1170: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['/Users/rgommers/tsanvenv/bin/python', '-c', "import os\nos.environ['NUMEXPR_MAX_THREADS'] = '4'\nimport numexpr\nassert(numexpr.MAX_THREADS == 4)\nexit(0)"],)
kwargs = {}, retcode = -6
cmd = ['/Users/rgommers/tsanvenv/bin/python', '-c', "import os\nos.environ['NUMEXPR_MAX_THREADS'] = '4'\nimport numexpr\nassert(numexpr.MAX_THREADS == 4)\nexit(0)"]

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.
    
        The arguments are the same as for the call function.  Example:
    
        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['/Users/rgommers/tsanvenv/bin/python', '-c', "import os\nos.environ['NUMEXPR_MAX_THREADS'] = '4'\nimport numexpr\nassert(numexpr.MAX_THREADS == 4)\nexit(0)"]' died with <Signals.SIGABRT: 6>.

../cpython/cpython-tsan/lib/python3.14t/subprocess.py:421: CalledProcessError
-------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------
==================
WARNING: ThreadSanitizer: data race (pid=2953)
  Write of size 4 at 0x000149e945e0 by thread T4:
    #0 th_worker(void*) <null>:1245713168 (interpreter.cpython-314t-darwin.so:arm64+0x2753c)

  Previous write of size 4 at 0x000149e945e0 by thread T3:
    #0 th_worker(void*) <null>:1245713168 (interpreter.cpython-314t-darwin.so:arm64+0x2753c)

  Location is global 'gs' at 0x000149e945b8 (interpreter.cpython-314t-darwin.so+0x345e0)

  Thread T4 (tid=5968833, running) created by main thread at:
    #0 pthread_create <null>:73400608 (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x32790)
    #1 init_threads() <null>:1245708528 (interpreter.cpython-314t-darwin.so:arm64+0x27b80)
    #2 numexpr_set_nthreads(int) <null>:1245708528 (interpreter.cpython-314t-darwin.so:arm64+0x27f08)
    #3 Py_set_num_threads(_object*, _object*) <null>:1245708528 (interpreter.cpython-314t-darwin.so:arm64+0x29fbc)
    #4 cfunction_call methodobject.c:562 (python3.14t:arm64+0x10011109c)
    #5 _PyObject_MakeTpCall call.c:242 (python3.14t:arm64+0x100077e30)
    #6 PyObject_Vectorcall call.c:327 (python3.14t:arm64+0x100078a94)
    #7 _PyEval_EvalFrameDefault generated_cases.c.h:1371 (python3.14t:arm64+0x100253314)
    #8 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #9 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #10 builtin_exec bltinmodule.c.h:560 (python3.14t:arm64+0x1002497c4)
    #11 cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c:452 (python3.14t:arm64+0x100110358)
    #12 _PyObject_Call call.c:348 (python3.14t:arm64+0x100078c4c)
    #13 PyObject_Call call.c:373 (python3.14t:arm64+0x100078d00)
    #14 _PyEval_EvalFrameDefault generated_cases.c.h:2421 (python3.14t:arm64+0x1002564a0)
    #15 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #16 _PyFunction_Vectorcall call.c (python3.14t:arm64+0x100078f40)
    #17 object_vacall call.c:819 (python3.14t:arm64+0x10007a828)
    #18 PyObject_CallMethodObjArgs call.c:880 (python3.14t:arm64+0x10007a594)
    #19 PyImport_ImportModuleLevelObject import.c:3764 (python3.14t:arm64+0x1002cc170)
    #20 _PyEval_ImportName ceval.c:2670 (python3.14t:arm64+0x1002728f4)
    #21 _PyEval_EvalFrameDefault generated_cases.c.h:5988 (python3.14t:arm64+0x10025f524)
    #22 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #23 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #24 run_mod pythonrun.c:1389 (python3.14t:arm64+0x10030c8d0)
    #25 _PyRun_SimpleStringFlagsWithName pythonrun.c:548 (python3.14t:arm64+0x100309920)
    #26 Py_RunMain main.c:760 (python3.14t:arm64+0x100342238)
    #27 pymain_main main.c:790 (python3.14t:arm64+0x100342724)
    #28 Py_BytesMain main.c:814 (python3.14t:arm64+0x1003427d0)
    #29 main python.c:15 (python3.14t:arm64+0x1000049a8)

  Thread T3 (tid=5968832, running) created by main thread at:
    #0 pthread_create <null>:73400416 (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x32790)
    #1 init_threads() <null>:1245713168 (interpreter.cpython-314t-darwin.so:arm64+0x27b80)
    #2 numexpr_set_nthreads(int) <null>:1245713168 (interpreter.cpython-314t-darwin.so:arm64+0x27f08)
    #3 Py_set_num_threads(_object*, _object*) <null>:1245713168 (interpreter.cpython-314t-darwin.so:arm64+0x29fbc)
    #4 cfunction_call methodobject.c:562 (python3.14t:arm64+0x10011109c)
    #5 _PyObject_MakeTpCall call.c:242 (python3.14t:arm64+0x100077e30)
    #6 PyObject_Vectorcall call.c:327 (python3.14t:arm64+0x100078a94)
    #7 _PyEval_EvalFrameDefault generated_cases.c.h:1371 (python3.14t:arm64+0x100253314)
    #8 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #9 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #10 builtin_exec bltinmodule.c.h:560 (python3.14t:arm64+0x1002497c4)
    #11 cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c:452 (python3.14t:arm64+0x100110358)
    #12 _PyObject_Call call.c:348 (python3.14t:arm64+0x100078c4c)
    #13 PyObject_Call call.c:373 (python3.14t:arm64+0x100078d00)
    #14 _PyEval_EvalFrameDefault generated_cases.c.h:2421 (python3.14t:arm64+0x1002564a0)
    #15 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #16 _PyFunction_Vectorcall call.c (python3.14t:arm64+0x100078f40)
    #17 object_vacall call.c:819 (python3.14t:arm64+0x10007a828)
    #18 PyObject_CallMethodObjArgs call.c:880 (python3.14t:arm64+0x10007a594)
    #19 PyImport_ImportModuleLevelObject import.c:3764 (python3.14t:arm64+0x1002cc170)
    #20 _PyEval_ImportName ceval.c:2670 (python3.14t:arm64+0x1002728f4)
    #21 _PyEval_EvalFrameDefault generated_cases.c.h:5988 (python3.14t:arm64+0x10025f524)
    #22 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #23 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #24 run_mod pythonrun.c:1389 (python3.14t:arm64+0x10030c8d0)
    #25 _PyRun_SimpleStringFlagsWithName pythonrun.c:548 (python3.14t:arm64+0x100309920)
    #26 Py_RunMain main.c:760 (python3.14t:arm64+0x100342238)
    #27 pymain_main main.c:790 (python3.14t:arm64+0x100342724)
    #28 Py_BytesMain main.c:814 (python3.14t:arm64+0x1003427d0)
    #29 main python.c:15 (python3.14t:arm64+0x1000049a8)

SUMMARY: ThreadSanitizer: data race (interpreter.cpython-314t-darwin.so:arm64+0x2753c) in th_worker(void*)+0x78
==================
ThreadSanitizer: reported 1 warnings
______________________________________________________________ test_threading_config.test_max_threads_unset _______________________________________________________________

self = <numexpr.tests.test_numexpr.test_threading_config testMethod=test_max_threads_unset>

    def test_max_threads_unset(self):
        # Has to be done in a subprocess as `importlib.reload` doesn't let us
        # re-initialize the threadpool
        script = '\n'.join([
                "import os",
                "if 'NUMEXPR_MAX_THREADS' in os.environ: os.environ.pop('NUMEXPR_MAX_THREADS')",
                "if 'OMP_NUM_THREADS' in os.environ: os.environ.pop('OMP_NUM_THREADS')",
                "import n
6D40
umexpr",
                f"assert(numexpr.nthreads <= {MAX_THREADS})",
                "exit(0)"])
>       subprocess.check_call([sys.executable, '-c', script])

numexpr/tests/test_numexpr.py:1159: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['/Users/rgommers/tsanvenv/bin/python', '-c', "import os\nif 'NUMEXPR_MAX_THREADS' in os.environ: os.environ.pop('NUM..._THREADS' in os.environ: os.environ.pop('OMP_NUM_THREADS')\nimport numexpr\nassert(numexpr.nthreads <= 16)\nexit(0)"],)
kwargs = {}, retcode = -6
cmd = ['/Users/rgommers/tsanvenv/bin/python', '-c', "import os\nif 'NUMEXPR_MAX_THREADS' in os.environ: os.environ.pop('NUME...UM_THREADS' in os.environ: os.environ.pop('OMP_NUM_THREADS')\nimport numexpr\nassert(numexpr.nthreads <= 16)\nexit(0)"]

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.
    
        The arguments are the same as for the call function.  Example:
    
        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['/Users/rgommers/tsanvenv/bin/python', '-c', "import os\nif 'NUMEXPR_MAX_THREADS' in os.environ: os.environ.pop('NUMEXPR_MAX_THREADS')\nif 'OMP_NUM_THREADS' in os.environ: os.environ.pop('OMP_NUM_THREADS')\nimport numexpr\nassert(numexpr.nthreads <= 16)\nexit(0)"]' died with <Signals.SIGABRT: 6>.

../cpython/cpython-tsan/lib/python3.14t/subprocess.py:421: CalledProcessError
-------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------
==================
WARNING: ThreadSanitizer: data race (pid=2961)
  Write of size 4 at 0x00014b4945e0 by thread T2:
    #0 th_worker(void*) <null>:1268781840 (interpreter.cpython-314t-darwin.so:arm64+0x2753c)

  Previous write of size 4 at 0x00014b4945e0 by thread T1:
    #0 th_worker(void*) <null>:1268781840 (interpreter.cpython-314t-darwin.so:arm64+0x2753c)

  Location is global 'gs' at 0x00014b4945b8 (interpreter.cpython-314t-darwin.so+0x345e0)

  Thread T2 (tid=5968901, running) created by main thread at:
    #0 pthread_create <null>:96469280 (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x32790)
    #1 init_threads() <null>:1268777200 (interpreter.cpython-314t-darwin.so:arm64+0x27b80)
    #2 numexpr_set_nthreads(int) <null>:1268777200 (interpreter.cpython-314t-darwin.so:arm64+0x27f08)
    #3 Py_set_num_threads(_object*, _object*) <null>:1268777200 (interpreter.cpython-314t-darwin.so:arm64+0x29fbc)
    #4 cfunction_call methodobject.c:562 (python3.14t:arm64+0x10011109c)
    #5 _PyObject_MakeTpCall call.c:242 (python3.14t:arm64+0x100077e30)
    #6 PyObject_Vectorcall call.c:327 (python3.14t:arm64+0x100078a94)
    #7 _PyEval_EvalFrameDefault generated_cases.c.h:1371 (python3.14t:arm64+0x100253314)
    #8 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #9 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #10 builtin_exec bltinmodule.c.h:560 (python3.14t:arm64+0x1002497c4)
    #11 cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c:452 (python3.14t:arm64+0x100110358)
    #12 _PyObject_Call call.c:348 (python3.14t:arm64+0x100078c4c)
    #13 PyObject_Call call.c:373 (python3.14t:arm64+0x100078d00)
    #14 _PyEval_EvalFrameDefault generated_cases.c.h:2421 (python3.14t:arm64+0x1002564a0)
    #15 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #16 _PyFunction_Vectorcall call.c (python3.14t:arm64+0x100078f40)
    #17 object_vacall call.c:819 (python3.14t:arm64+0x10007a828)
    #18 PyObject_CallMethodObjArgs call.c:880 (python3.14t:arm64+0x10007a594)
    #19 PyImport_ImportModuleLevelObject import.c:3764 (python3.14t:arm64+0x1002cc170)
    #20 _PyEval_ImportName ceval.c:2670 (python3.14t:arm64+0x1002728f4)
    #21 _PyEval_EvalFrameDefault generated_cases.c.h:5988 (python3.14t:arm64+0x10025f524)
    #22 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #23 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #24 run_mod pythonrun.c:1389 (python3.14t:arm64+0x10030c8d0)
    #25 _PyRun_SimpleStringFlagsWithName pythonrun.c:548 (python3.14t:arm64+0x100309920)
    #26 Py_RunMain main.c:760 (python3.14t:arm64+0x100342238)
    #27 pymain_main main.c:790 (python3.14t:arm64+0x100342724)
    #28 Py_BytesMain main.c:814 (python3.14t:arm64+0x1003427d0)
    #29 main python.c:15 (python3.14t:arm64+0x1000049a8)

  Thread T1 (tid=5968900, running) created by main thread at:
    #0 pthread_create <null>:96469088 (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x32790)
    #1 init_threads() <null>:1268781840 (interpreter.cpython-314t-darwin.so:arm64+0x27b80)
    #2 numexpr_set_nthreads(int) <null>:1268781840 (interpreter.cpython-314t-darwin.so:arm64+0x27f08)
    #3 Py_set_num_threads(_object*, _object*) <null>:1268781840 (interpreter.cpython-314t-darwin.so:arm64+0x29fbc)
    #4 cfunction_call methodobject.c:562 (python3.14t:arm64+0x10011109c)
    #5 _PyObject_MakeTpCall call.c:242 (python3.14t:arm64+0x100077e30)
    #6 PyObject_Vectorcall call.c:327 (python3.14t:arm64+0x100078a94)
    #7 _PyEval_EvalFrameDefault generated_cases.c.h:1371 (python3.14t:arm64+0x100253314)
    #8 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #9 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #10 builtin_exec bltinmodule.c.h:560 (python3.14t:arm64+0x1002497c4)
    #11 cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c:452 (python3.14t:arm64+0x100110358)
    #12 _PyObject_Call call.c:348 (python3.14t:arm64+0x100078c4c)
    #13 PyObject_Call call.c:373 (python3.14t:arm64+0x100078d00)
    #14 _PyEval_EvalFrameDefault generated_cases.c.h:2421 (python3.14t:arm64+0x1002564a0)
    #15 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #16 _PyFunction_Vectorcall call.c (python3.14t:arm64+0x100078f40)
    #17 object_vacall call.c:819 (python3.14t:arm64+0x10007a828)
    #18 PyObject_CallMethodObjArgs call.c:880 (python3.14t:arm64+0x10007a594)
    #19 PyImport_ImportModuleLevelObject import.c:3764 (python3.14t:arm64+0x1002cc170)
    #20 _PyEval_ImportName ceval.c:2670 (python3.14t:arm64+0x1002728f4)
    #21 _PyEval_EvalFrameDefault generated_cases.c.h:5988 (python3.14t:arm64+0x10025f524)
    #22 _PyEval_Vector ceval.c:1745 (python3.14t:arm64+0x10024f874)
    #23 PyEval_EvalCode ceval.c:660 (python3.14t:arm64+0x10024f468)
    #24 run_mod pythonrun.c:1389 (python3.14t:arm64+0x10030c8d0)
    #25 _PyRun_SimpleStringFlagsWithName pythonrun.c:548 (python3.14t:arm64+0x100309920)
    #26 Py_RunMain main.c:760 (python3.14t:arm64+0x100342238)
    #27 pymain_main main.c:790 (python3.14t:arm64+0x100342724)
    #28 Py_BytesMain main.c:814 (python3.14t:arm64+0x1003427d0)
    #29 main python.c:15 (python3.14t:arm64+0x1000049a8)

SUMMARY: ThreadSanitizer: data race (interpreter.cpython-314t-darwin.so:arm64+0x2753c) in th_worker(void*)+0x78
========================================================================= short test summary info =========================================================================
FAILED numexpr/tests/test_numexpr.py::test_numexpr::test_locals_clears_globals - subprocess.CalledProcessError: Command '['/Users/rgommers/tsanvenv/bin/python', '-c', "import numexpr as ne;a=10;ne.evaluate('1');a += 1;ne.evaluate('2', local_dict={...
FAILED numexpr/tests/test_numexpr.py::test_numexpr2::test_locals_clears_globals - subprocess.CalledProcessError: Command '['/Users/rgommers/tsanvenv/bin/python', '-c', "import numexpr as ne;a=10;ne.evaluate('1');a += 1;ne.evaluate('2', local_dict={...
FAILED numexpr/tests/test_numexpr.py::test_threading_config::test_max_threads_set - subprocess.CalledProcessError: Command '['/Users/rgommers/tsanvenv/bin/python', '-c', "import os\nos.environ['NUMEXPR_MAX_THREADS'] = '4'\nimport numexpr\nassert(nume...
FAILED numexpr/tests/test_numexpr.py::test_threading_config::test_max_threads_unset - subprocess.CalledProcessError: Command '['/Users/rgommers/tsanvenv/bin/python', '-c', "import os\nif 'NUMEXPR_MAX_THREADS' in os.environ: os.environ.pop('NUMEXPR_MAX_...
========================================================== 4 failed, 103 passed, 1 xfailed, 9 warnings in 43.65s ==========================================================
ThreadSanitizer: reported 1 warnings
ThreadSanitizer: reported 1 warnings

The result with pytest . and pytest . --parallel-threads=2 is the same four failures.

Sorry about the missing line numbers, that output would be easier to interpret with line numbers - I'm traveling and my macOS setup is a bit broken. Using Docker on Linux as in the link above should yield better results - maybe you want to try @andfoy? Anyway, it's these lines:

numexpr/numexpr/module.cpp

Lines 233 to 237 in 36aa11b

    
           /* Now create the threads */ 
        
           for (tid = 0; tid < gs.nthreads; tid++) { 
        
               gs.tids[tid] = tid; 
        
               rc = pthread_create(&gs.threads[tid], NULL, th_worker, 
        
                                   (void *)&gs.tids[tid]);

rgommers · 2025-03-10T00:00:45Z

numexpr/tests/test_numexpr.py

+
+    pytest = MagicMock()
+    pytest.mark = MagicMock()
+    pytest.mark.thread_unsafe = identity


It'd be better to register thread_unsafe also if pytest is installed but pytest-run-parallel isn't. That way the test suite can be run with pytest also without pytest-run-parallel

rgommers · 2025-03-10T00:38:59Z

numexpr is about performance, so most of the users will be curious on what advantages they can get, and how. At any rate, this can be added with another PR.

Given that numexpr is already using threading internally, I suspect that a micro-benchmark isn't going to show much of interest. What will be better is to show this in a larger program at some point. Supporting free-threading allows installing it with projects like pandas that can use numexpr under the hood.

FrancescAlted · 2025-03-10T09:15:39Z

numexpr is about performance, so most of the users will be curious on what advantages they can get, and how. At any rate, this can be added with another PR.

Given that numexpr is already using threading internally, I suspect that a micro-benchmark isn't going to show much of interest. What will be better is to show this in a larger program at some point. Supporting free-threading allows installing it with projects like pandas that can use numexpr under the hood.

Ok. If a micro-benchmark as such would not be useful, at least some small script using threading in Python code in combination with the new numexpr would be useful for users. If, for demonstration purposes, you want to add something that uses pandas, that would be fine too.

andfoy · 2025-03-12T19:08:47Z

cc2842d should fix the concurrency issue reported by TSAN, as there are no more errors reported

FrancescAlted · 2025-03-17T11:49:08Z

numexpr/module.cpp

@@ -51,7 +51,9 @@ void *th_worker(void *tidptr)
    while (1) {

        /* Sentinels have to be initialised yet */
-        gs.init_sentinels_done = 0;
+        if(tid == 0) {


Just for consistency, can you add a space after the if?

FrancescAlted · 2025-03-17T12:02:44Z

CI tests seem to stall with aarch64. I disabled those in the andfoy-support_free_threading branch, and this is why it is green. Do you think these are caused by a qemu deficiency, or they are real? If you think that it is a qemu issue, we can merge when you think this complete (the example for the new mode threading mode is my only pending suggestion, but this can be done on another PR).

rgommers · 2025-03-17T15:05:47Z

Hard to know for sure if it's qemu or not, but often it is. Since GitHub Actions now offers native Linux aarch64 runners, how about migrating to those in a follow-up PR (or merge that change first and then retrigger CI here)? That should be a generally useful thing to do.

andfoy · 2025-03-17T15:25:11Z

at least some small script using threading in Python code in combination with the new numexpr would be useful for users

@FrancescAlted let me take care of that in a follow up PR!

FrancescAlted · 2025-03-17T16:18:50Z

Ok. So I can cherry-pick the latest commits related with the threading support in this PR to andfoy-support_free_threading branch, merge that, and you can proceed with further enhancements on a new PR. Sounds good?

andfoy · 2025-03-17T16:41:52Z

Sounds great! Let me add the whitespace change on the if, and it should be ready to go

@andfoy

This is an extension of #504 to fix some issues with tests. The tests for aarch64 are currently commented out, because CI (via qemu) gets stalled. However, my arm64 box (MacMini M4) does not have a single issue, so I suppose it is an issue with qemu. This overrides #504. @andfoy @rgommers

FrancescAlted · 2025-03-18T12:44:09Z

The work here has been included in #505 and merged in master. Feel free to push additional refinements in another PR. Thanks!

FrancescAlted · 2025-03-18T16:49:18Z

I forgot to mention that I had to disable a commit of yours with: 0c42b7d . I guess that init_sentinels_done cannot be shared among different threads.

andfoy marked this pull request as ready for review February 26, 2025 19:40

rgommers reviewed Feb 27, 2025

View reviewed changes

rgommers reviewed Mar 3, 2025

View reviewed changes

.github/workflows/build.yml Outdated Show resolved Hide resolved

andfoy force-pushed the support_free_threading branch from 2ff0da0 to 6fe5a14 Compare March 3, 2025 16:25

andfoy added 6 commits March 3, 2025 11:37

Test numexpr against pytest-run-parallel on 3.13t

33ee71b

Mock pytest in case is not available

8680084

Build free-threaded wheels

8af34da

Use CIBW_ENABLE

706cb9d

Use pytest for testing

3181455

Update env variable value

61076a2

andfoy force-pushed the support_free_threading branch from 6fe5a14 to 61076a2 Compare March 3, 2025 16:38

andfoy added 4 commits March 4, 2025 18:00

Move free-threaded builds to an indindependent job

1d15ad4

Set free-threading variables only under free-threaded conditions

40f04d2

Execute pytest with --pyargs

0fb95ec

Add section in README regarding free-threading

e75d15f

FrancescAlted mentioned this pull request Mar 6, 2025

Andfoy support free threading #505

Merged

andfoy added 2 commits March 6, 2025 13:57

Address duplicate artifact uploads

8fc5991

Use an independent label to signal freethreaded artifacts

d412bd6

Mark numexpr interpreter as free-threaded safe

cf0aba6

rgommers reviewed Mar 10, 2025

View reviewed changes

Ensure single thread write to gs.init_sentinels_done

cc2842d

FrancescAlted reviewed Mar 17, 2025

View reviewed changes

Address review comments

7be1ca9

FrancescAlted closed this Mar 18, 2025

rgommers mentioned this pull request Mar 19, 2025

Support free-threaded Python 3.13 #503

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Test numexpr against pytest-run-parallel on 3.13t #504

Test numexpr against pytest-run-parallel on 3.13t #504

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

	/* Now create the threads */
	for (tid = 0; tid < gs.nthreads; tid++) {
	gs.tids[tid] = tid;
	rc = pthread_create(&gs.threads[tid], NULL, th_worker,
	(void *)&gs.tids[tid]);

Uh oh!

Test numexpr against pytest-run-parallel on 3.13t #504

Test numexpr against pytest-run-parallel on 3.13t #504

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!