8000 Integration testing: Meson SIMD + vendored meson-python by rgommers · Pull Request #24395 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

Integration testing: Meson SIMD + vendored meson-python #24395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 32 commits into from
Closed

Integration testing: Meson SIMD + vendored meson-python #24395

wants to merge 32 commits into from

Conversation

rgommers
Copy link
Member

Not for merging! Testing the needed build system changes on top of gh-23096.

seiko2plus and others added 29 commits August 10, 2023 14:40
@seiko2plus @rgommers
ENH, SIMD: The meson CPU dispatcher implementation
ab496b3 
   Almost gives the same functionality as Distutils/CCompiler Opt,
   with a few changes to the way we specify the targets, also its
   abend idea of wrap the dispatchable sources, instead it counts
   on static libraries to enable different paths and flags.
@seiko2plus @rgommers
Provide compatibility with distutils
9a139d3 
  Several renames have been made to certain compiler definitions
  that were used to determine the current target, as well as the macros
  in the dispatch configuration headers.

  These changes were necessary to allow for the modification of the prefix 'NPY_'
  in case this work is used outside of NumPy.
@seiko2plus @rgommers
Extend test_requirements/pyproject to cover meson module
166477d
@seiko2plus @rgommers
BUG: Fix SSE build on meson
9421795
@seiko2plus @rgommers
fix the build when it disabled
0c3111c
@seiko2plus @rgommers
enable AVX512_SPR for quicksort
a78ef6b
@seiko2plus @rgommers
Add support for build option --test-simd
a3058a9
@seiko2plus @rgommers
Fix sdist build
e4b3d72
@seiko2plus @rgommers
Pass Opt level 3 to all dispach-able sources
4203a5b
@seiko2plus @rgommers
Disable SIMD kernels of log/exp/sin/cos on clang-cl
04b2e2a
@seiko2plus @rgommers
CI: Transition x86 specialized tests to Meson from Distutils
1917470
@seiko2plus @rgommers
Cleanup the main configration header and improves docs
0963fe0
@seiko2plus @rgommers
Detect global architecture args
481114a
@seiko2plus @rgommers
update the meson module name
58239f3
@seiko2plus @rgommers
fix SSE41 flag on Intel-cl
2271d02
@seiko2plus @rgommers
rename method multi_target to multi_targets
080e19c
@seiko2plus @rgommers
Disables mmx when AVX512 is enabled similar to distutils
35292aa
@seiko2plus @rgommers
Add build target AVX512_ICL for simd_qsort
077a09f
@seiko2plus @rgommers
CI: Allow noblas for SIMD tests
7ec6933
@seiko2plus @rgommers
Bybass sort validation for _simd module
0beab65
@seiko2plus @rgommers
Removes build option boolean warning
f580462
@seiko2plus @rgommers
removes py_dep from _simd extention
de71d9b
@seiko2plus @rgommers
fix Initlize typo
dedd413
@seiko2plus @rgommers
Minimize the log of CPU optimization
a751c20
@seiko2plus @rgommers
Remove debug log and count on multi_targets() debug
76327b8 
  This simplifies the implementation and improves code readability.
@seiko2plus @rgommers
update multi_targets to reduce the number of objects
e13ce41
@rgommers
BLD: updates to build and test dependencies
9509508
@rgommers
BLD: add Meson version check, to catch older installed versions early
371bb76
@rgommers
CI: fix doc refguide check failure on CircleCI
bfb504f
@rgommers rgommers added the 36 - Build Build related PR label Aug 11, 2023
@rgommers rgommers marked this pull request as draft August 11, 2023 13:16
@rgommers
BLD: use numpy's friendly fork of meson-python
34ec848
@rgommers
DEV: vendor spin's meson.py, modified to use our vendored Meson
7a40779 
Note that this is a bit messy. I tried to vendor spin's `meson.py`
separately, but it's not possible to do so cleanly as far as I can
tell. T
8000
he import machinery is unhappy bouncing around between
`bin/spin`, `.spin/cmds.py and `.spin/meson.py`. So it was either
folding the `spin/cmds/meson.py` content all into cmds.py, or vendor all
of the spin package. This seems to work.
@rgommers
CI: fix Windows GHA jobs for vendored meson-python and spin
a594207
@rgommers rgommers force-pushed the vendor-mesonpython branch from 924294b to a594207 Compare August 11, 2023 13:22
@rgommers
Copy link
Member Author

Finally Windows seems happy, builds are completing with the vendored meson/meson-python, and:

  Test features "SSE SSE2 SSE3" : Supported
  Test features "SSSE3" : Supported
  Test features "SSE41" : Supported
  Test features "POPCNT" : Supported
  Test features "SSE42" : Supported
  Test features "AVX" : Supported
  Test features "F16C" : Supported
  Test features "FMA3" : Supported
  Test features "AVX2" : Supported (cached)
  Test features "AVX512F" : Supported
  Test features "AVX512CD" : Supported (cached)
  Test features "AVX512_KNL" : disabled due to AVX512_KNL is disabled due to "msvc compiler does not support it"
  Test features "AVX512_KNM" : disabled due to AVX512_KNM is disabled due to "msvc compiler does not support it"
  Test features "AVX512_SKX" : Supported (cached)
  Test features "AVX512_CLX" : Supported
  Test features "AVX512_CNL" : Supported
  Test features "AVX512_ICL" : Supported
  Test features "AVX512_SPR" : disabled due to AVX512_SPR is disabled due to "msvc compiler does not support it"
  Configuring npy_cpu_dispatch_config.h using configuration
  Message:
  CPU Optimiztion Options
    baseline:
      Requested : min
      Enabled   : SSE SSE2 SSE3
    dispatch:
      Requested : max -xop -fma4
      Enabled   : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL

@rgommers
Copy link
Member Author

The failure on the Python 3.12 wheels is probably unrelated (and otherwise caused by the SIMD changes, not the build system ones):

  ____________________ TestAsArray.test_struct_array_pointer ____________________
  
  self = <numpy.tests.test_ctypeslib.TestAsArray object at 0x000001A2F2AB37A0>
  
      def test_struct_array_pointer(self):
          from ctypes import c_int16, Structure, pointer
      
          class Struct(Structure):
              _fields_ = [('a', c_int16)]
      
          Struct3 = 3 * Struct
      
          c_array = (2 * Struct3)(
              Struct3(Struct(a=1), Struct(a=2), Struct(a=3)),
              Struct3(Struct(a=4), Struct(a=5), Struct(a=6))
          )
      
          expected = np.array([
              [(1,), (2,), (3,)],
              [(4,), (5,), (6,)],
          ], dtype=[('a', np.int16)])
      
          def check(x):
              assert_equal(x.dtype, expected.dtype)
              assert_equal(x, expected)
      
          # all of these should be equivalent
          check(as_array(c_array))
          check(as_array(pointer(c_array), shape=()))
  >       check(as_array(pointer(c_array[0]), shape=(2,)))
  
  Struct     = <class 'numpy.tests.test_ctypeslib.TestAsArray.test_struct_array_pointer.<locals>.Struct'>
  Struct3    = <class 'numpy.tests.test_ctypeslib.Struct_Array_3'>
  Structure  = <class '_ctypes.Structure'>
  c_array    = <Struct_Array_3_Array_2 object at 0x000001A28A7820D0>
  c_int16    = <class 'ctypes.c_short'>
  check      = <function TestAsArray.test_struct_array_pointer.<locals>.check at 0x000001A28A77E340>
  expected   = array([[(1,), (2,), (3,)],
         [(4,), (5,), (6,)]], dtype=[('a', '<i2')])
  pointer    = <built-in function pointer>
  self       = <numpy.tests.test_ctypeslib.TestAsArray object at 0x000001A2F2AB37A0>
  
  ..\venv-test\Lib\site-packages\numpy\tests\test_ctypeslib.py:241: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  ..\venv-test\Lib\site-packages\numpy\tests\test_ctypeslib.py:236: in check
      assert_equal(x, expected)
          expected   = array([[(1,), (2,), (3,)],
         [(4,), (5,), (6,)]], dtype=[('a', '<i2')])
          x          = array([(1,), (2,), (3,)], dtype=[('a', '<i2')])
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  
  args = (<built-in function eq>, array([(1,), (2,), (3,)], dtype=[('a', '<i2')]), array([[(1,), (2,), (3,)],
         [(4,), (5,), (6,)]], dtype=[('a', '<i2')]))
  kwds = {'err_msg': '', 'header': 'Arrays are not equal', 'strict': False, 'verbose': True}
  
      @wraps(func)
      def inner(*args, **kwds):
          with self._recreate_cm():
  >           return func(*args, **kwds)
  E           AssertionError: 
  E           Arrays are not equal
  E           
  E           (shapes (3,), (2, 3) mismatch)
  E            x: array([(1,), (2,), (3,)], dtype=[('a', '<i2')])
  E            y: array([[(1,), (2,), (3,)],
  E                  [(4,), (5,), (6,)]], dtype=[('a', '<i2')])
  
  args       = (<built-in function eq>, array([(1,), (2,), (3,)], dtype=[('a', '<i2')]), array([[(1,), (2,), (3,)],
         [(4,), (5,), (6,)]], dtype=[('a', '<i2')]))
  func       = <function assert_array_compare at 0x000001A2E78184A0>
  kwds       = {'err_msg': '', 'header': 'Arrays are not equal', 'strict': False, 'verbose': True}
  self       = <contextlib._GeneratorContextManager object at 0x000001A2E7888830>

Is this a known issue?

@rgommers rgommers mentioned this pull request Aug 11, 2023
Merged
@rgommers rgommers closed this Aug 11, 2023
rgommers added a commit to rgommers/numpy that referenced this pull request Aug 11, 2023
@rgommers
Check if TestAsArray.test_struct_array_pointer fails [wheel build]
eec6725 
Failed on Python 3.12 in numpygh-24395 testing. So this is an empty commit
on top of `main` to check whether it's related to that PR.

[skip azp] [skip circle] [skip cirrus] [skip travis]
@rgommers rgommers mentioned this pull request Aug 11, 2023
Closed
rgommers added a commit to rgommers/numpy that referenced this pull request Aug 11, 2023
@rgommers
Check if TestAsArray.test_struct_array_pointer fails [wheel build]
5ab3e3d 
Failed on Python 3.12 in numpygh-24395 testing. So this is an empty commit
on top of `main` to check whether it's related to that PR.

[skip azp] [skip circle] [skip cirrus] [skip travis]
@charris charris mentioned this pull request Aug 12, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
36 - Build Build related PR component: CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rgommers @seiko2plus
0