More improvements to test_linalg #101

asmeurer · 2022-02-18T01:41:23Z

So far just vector_norm, but more are incoming.

Note that NumPy will fail this test without numpy/numpy#21084.

I also realized that the linalg tests should be making better use of the helper functions. I've done so for this test, but still need to update the other tests.

honno

Looks good.

Note that NumPy will fail this test without numpy/numpy#21084.

Ah, we should skip this in the workflow anywho because we're pinning against NumPy releases now.

array_api_tests/test_linalg.py

Co-authored-by: Matthew Barber <quitesimplymatt@gmail.com>

The upstream has bugs that will be fixed in the next release.

Fixes data-apis#44. Related to data-apis#117.

asmeurer · 2022-04-25T23:22:47Z

@honno see what you think of the allclose helpers I've added here. They can likely use some improvement, but they should work for now.

honno · 2022-04-26T09:34:18Z

array_api_tests/array_helpers.py

+def allclose(x, y, rel_tol=0.25, abs_tol=1, return_indices=False):
+    """
+    Return True all elements of x and y are within tolerance
+
+    If return_indices=True, returns (False, (i, j)) when the arrays are not
+    close, where i and j are the indices into x and y of corresponding
+    non-close elements.
+    """
+    for i, j in iter_indices(x.shape, y.shape):
+        i, j = i.raw, j.raw
+        a = x[i]
+        b = y[j]
+        if not (math.isfinite(a) and math.isfinite(b)):
+            # TODO: If a and b are both infinite, require the same type of infinity
+            continue
+        close = math.isclose(a, b, rel_tol=rel_tol, abs_tol=abs_tol)
+        if not close:
+            if return_indices:
+                return (False, (i, j))
+            return False
+    return True


Yep looks good, this is really how we have to do it if we can't use masking.

Maybe move this and assert_allclose to test_linalg.py for now, as ideally what we'd be doing is standardising these utils across function groups then, and scoping these utils for each function group makes it easier to distinguish where they're used and their subtle differences. (standardising seems quite doable, I just need to get round to it)

One problem here is represents a pretty significant performance hit. Before this commit (with the exact equality test), the linalg tests take 15 seconds on my computer. After, they take 44 seconds. Performance isn't our top priority, but maybe we should try array operations and only fallback to this when there are nonfinite values.

Maybe move this and assert_allclose to test_linalg.py for now, as ideally what we'd be doing is standardising these utils across function groups then, and scoping these utils for each function group makes it easier to distinguish where they're used and their subtle differences. (standardising seems quite doable, I just need to get round to it)

Think I'd still like this for now, esp as I'm generally rethinking the pytest helpers for #200/general look at vectorisation. Not blocking tho.

Actually I'm just going to delete them. I'm not using them anymore, because testing float arrays from linalg functions like this turned out to be too difficult.

asmeurer · 2022-04-27T20:37:22Z

It looks like it isn't as straightforward as this. CuPy has several failures:

_________________________________________________________________________________________________________ test_eigh __________________________________________________________________________________________________________

    @pytest.mark.xp_extension('linalg')
>   @given(x=symmetric_matrices(finite=True))

array_api_tests/test_linalg.py:239: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
array_api_tests/test_linalg.py:256: in test_eigh
    _test_stacks(lambda x: linalg.eigh(x).eigenvectors, x,
array_api_tests/test_linalg.py:90: in _test_stacks
    assert_equal(res_stack, decomp_res_stack)
array_api_tests/test_linalg.py:49: in assert_equal
    assert_allclose(x, y)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

x = Array([[-0.40824828,  0.70710677,  0.57735026],
       [-0.40824828, -0.70710677,  0.57735026],
       [ 0.81649655,  0.        ,  0.57735026]], dtype=float32)
y = Array([[ 0.8164966 ,  0.        , -0.57735026],
       [-0.4082483 , -0.70710677, -0.5773504 ],
       [-0.40824828,  0.7071068 , -0.5773503 ]], dtype=float32), rel_tol = 0.25, abs_tol = 1

    def assert_allclose(x, y, rel_tol=0.25, abs_tol=1):
        """
        Test that x and y are approximately equal to each other.
    
        Also asserts that x and y have the same shape and dtype.
        """
        assert x.shape == y.shape, f"The input arrays do not have the same shapes ({x.shape} != {y.shape})"
    
        assert x.dtype == y.dtype, f"The input arrays do not have the same dtype ({x.dtype} != {y.dtype})"
    
        c = allclose(x, y, rel_tol=rel_tol, abs_tol=abs_tol, return_indices=True)
        if c is not True:
            _, (i, j) = c
>           raise AssertionError(f"The input arrays are not close with {rel_tol = } and {abs_tol = } at indices {i = } and {j = }")
E           AssertionError: The input arrays are not close with rel_tol = 0.25 and abs_tol = 1 at indices i = (0, 0) and j = (0, 0)

array_api_tests/array_helpers.py:187: AssertionError
--------------------------------------------------------------------------------------------------------- Hypothesis ---------------------------------------------------------------------------------------------------------
Falsifying example: test_eigh(
    x=Array([[[1., 1., 1.],
            [1., 1., 1.],
            [1., 1., 1.]]], dtype=float32),
)
___

Note that the sort order of eigenvalues is unspecified in the spec.

Also svd and svdvals fail with errors like

________________________________________________________________________________________________________ test_svdvals ________________________________________________________________________________________________________

    @pytest.mark.xp_extension('linalg')
>   @given(
        x=finite_matrices(),
    )

array_api_tests/test_linalg.py:562: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
array_api_tests/test_linalg.py:577: in test_svdvals
    _test_stacks(linalg.svdvals, x, dims=1, res=res)
array_api_tests/test_linalg.py:90: in _test_stacks
    assert_equal(res_stack, decomp_res_stack)
array_api_tests/test_linalg.py:49: in assert_equal
    assert_allclose(x, y)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

x = Array([34741248.,        0.], dtype=float32), y = Array([3.4741252e+07, 1.3871019e+00], dtype=float32), rel_tol = 0.25, abs_tol = 1

    def assert_allclose(x, y, rel_tol=0.25, abs_tol=1):
        """
        Test that x and y are approximately equal to each other.
    
        Also asserts that x and y have the same shape and dtype.
        """
        assert x.shape == y.shape, f"The input arrays do not have the same shapes ({x.shape} != {y.shape})"
    
        assert x.dtype == y.dtype, f"The input arrays do not have the same dtype ({x.dtype} != {y.dtype})"
    
        c = allclose(x, y, rel_tol=rel_tol, abs_tol=abs_tol, return_indices=True)
        if c is not True:
            _, (i, j) = c
>           raise AssertionError(f"The input arrays are not close with {rel_tol = } and {abs_tol = } at indices {i = } and {j = }")
E           AssertionError: The input arrays are not close with rel_tol = 0.25 and abs_tol = 1 at indices i = (1,) and j = (1,)

array_api_tests/array_helpers.py:187: AssertionError

which shows that using an absolute tolerance is not going to work.

There are also some test failures which I think are due to bugs in cupy. This one I'm not sure about

>>> import cupy.array_api
>>> x1 = cupy.array_api.asarray([51307], dtype=cupy.array_api.uint16)
>>> x2 = cupy.array_api.asarray([[[327]]], dtype=cupy.array_api.uint16)
>>> cupy.array_api.linalg.matmul(x1, x2)
Array([[172]], dtype=uint16)
>>> cupy.array_api.linalg.matmul(x1, x2[0, ...])
Array([173], dtype=uint16)
>>> x1
Array([51307], dtype=uint16)
>>> import numpy as np
>>> np.asarray([51307*327], dtype=np.uint16)
array([173], dtype=uint16)

CuPy is clearly doing something weird here, but should this overflow behavior be something that we should be avoiding in the test suite? If so then I guess we need to be more careful about how we generate integer array inputs to functions like matmul.

honno · 2022-04-28T14:31:53Z

Unfortunately I'm not familiar with linalg, so not super sure on expectations here. (I'll be out until 9th May, where I'll go through linalg and these tests more so I might be more useful in the future heh.)

should this overflow behavior be something that we should be avoiding in the test suite?

Ah interesting catch. Personally I think it's fine to just relatively assert integer operations here (internal dtype-representation-shenanigans like numpy/numpy#20226 maybe?), as such edge cases probably won't affect array-consuming code.

There are different equivalent ways of representing eigenspaces in general, and the same algorithm may choose different ways for stacked vs. non-stacked matrices (e.g., they differ for cupy).

This still is going to require some tweaking.

…to more-linalg2

Floating-point dtypes now only test shape and dtype in the stacking test. Trying to make the allclose test work is too difficult for linear algebra functions.

honno · 2022-06-13T08:44:43Z

array_api_tests/test_linalg.py

+        # It's too difficult to do an approximately equal test here because
+        # different routines can give completely different answers, and even
+        # when it does work, the elementwise comparisons are too slow. So for
+        # floating-point dtypes only test the shape and dtypes.
+
+        # assert_allclose(x, y)
+
+        assert x.shape == y.shape, f"The input arrays do not have the same shapes ({x.shape} != {y.shape})"
+        assert x.dtype == y.dtype, f"The input arrays do not have the same dtype ({x.dtype} != {y.dtype})"


Yep this seems reasonable.

asmeurer · 2024-02-16T01:02:31Z

I've released ndindex 1.8 with the changes needed here, and updated the pin in requirements.txt.

honno · 2024-02-16T10:40:39Z

Can ignore flaky statistical tests (skipped in master now since #233). Looks good, just need resolution on the failing test_cross.

asmeurer · 2024-02-16T21:49:26Z

The test failure is because numpy.array_api is wrong for cross (I never added broadcasting support from the 2022 spec). array-api-strict has already been fixed data-apis/array-api-strict#10. We will need to figure out what we want to do going forward for CI tests, given that numpy.array_api is going to be removed and array-api-strict is in its own repo that has its own array-api-tests CI run.

At any rate, I just remembered that I still wanted to check this with numpy, torch, and cupy. Although if you want to merge anyway, you can, and I'll just make any follow up fixes in another PR.

asmeurer · 2024-02-16T22:04:00Z

Looks like the following causes NumPy main to hang

>>> import numpy as np
>>> a = np.empty((3, 3, 0, 0), dtype=np.float32)
>>> np.linalg.cholesky(a, upper=True)

The issue is only there with upper=True, which is new in 2.0.

asmeurer · 2024-02-16T22:10:53Z

Submitted upstream numpy/numpy#25840. What's the easiest way to skip this test for now? (EDIT: looks like -k 'not cholesky' works)

asmeurer · 2024-02-17T00:10:36Z

There are some other test failures with NumPy:

trace fails because of Reconsider sum/prod/trace upcasting for floating-point dtypes array-api#731. I propose we fix that after this is merged (I have factored out the sum/prod/trace logic into a separate function here)
test_solve fails with

E           ValueError: solve: Input operand 1 does not have enough dimensions (has 1, gufunc core with signature (m,m),(m,n)->(m,n) requires 2)
E           Falsifying example: test_solve(
E               x1=array([], shape=(0, 0, 0), dtype=float32),
E               x2=array([], dtype=float32),
E           )

(need to investigate)

test_matrix_rank fails with

    | Traceback (most recent call last):
    |   File "/Users/aaronmeurer/Documents/array-api-tests/array_api_tests/test_linalg.py", line 429, in test_matrix_rank
    |     linalg.matrix_rank(x, **kw)
    |   File "/Users/aaronmeurer/Documents/numpy/numpy/linalg/_linalg.py", line 2083, in matrix_rank
    |     return count_nonzero(S > tol, axis=-1)
    |                          ^^^^^^^
    | ValueError: operands could not be broadcast together with shapes (3,2) (3,3)
    | Falsifying example: test_matrix_rank(
    |     x=array([[[0., 0.],
    |             [0., 0.]],
    |
    |            [[0., 0.],
    |             [0., 0.]],
    |
    |            [[0., 0.],
    |             [0., 0.]]], dtype=float32),
    |     kw={'rtol': array([0., 0., 0.], dtype=float32)},
    | )
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/Users/aaronmeurer/Documents/array-api-tests/array_api_tests/test_linalg.py", line 429, in test_matrix_rank
    |     linalg.matrix_rank(x, **kw)
    |   File "/Users/aaronmeurer/Documents/numpy/numpy/linalg/_linalg.py", line 2080, in matrix_rank
    |     tol = S.max(axis=-1, keepdims=True) * rtol
    |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~
    | ValueError: operands could not be broadcast together with shapes (0,2,1) (0,2)
    | Falsifying example: test_matrix_rank(
    |     x=array([], shape=(0, 2, 1, 1), dtype=float32),
    |     kw={'rtol': array([], shape=(0, 2), dtype=float32)},
    | )

(also need to investigate)

I get a health check error for filtering too much with invertable_matrices from test_inv sometimes.

honno · 2024-02-20T17:45:10Z

Submitted upstream numpy/numpy#25840. What's the easiest way to skip this test for now? (EDIT: looks like -k 'not cholesky' works)

Looks like it was fixed already 🎉

We will need to figure out what we want to do going forward for CI tests, given that numpy.array_api is going to be removed and array-api-strict is in its own repo that has its own array-api-tests CI run.

I wrote #235 to track this and other musings on our CI. Probably for this PR we're sticking with numpy.array_api.

trace fails because of Reconsider sum/prod/trace upcasting for floating-point dtypes array-api#731. I propose we fix that after this is merged (I have factored out the sum/prod/trace logic into a separate function here)

Yep agreed can fix after, wrote #234 for this.

Regarding test_solve and test_matrix_rank failures (+ test_inv flakiness), yeah if this week you can identify the issues and either fix or slap a pytest.mark.skip if non-trivial, otherwise I think it'd be nice to just merge this PR and look at later.

asmeurer · 2024-02-22T22:08:11Z

Looks like the numpy solve failures are due to numpy/numpy#15349, which I guess was never addressed for 2.0. This is worked around in array-api-strict, but I never added a workaround to array-api-compat.

asmeurer · 2024-02-22T22:33:23Z

The np solve issue in array-api-compat is fixed here data-apis/array-api-compat#93. Unfortunately, I don't think we can fix cupy as it doesn't seem to actually support stacking in its solve().

asmeurer · 2024-02-22T22:38:18Z

Here's the invertable_matrices strategy that is giving the health error for too much filtering: array-api-tests/array_api_tests/hypothesis_helpers.py Lines 306 to 318 in 4f83bb3
	@composite
	def invertible_matrices(draw, dtypes=xps.floating_dtypes(), stack_shapes=shapes()):
	# For now, just generate stacks of diagonal matrices.
	n = draw(integers(0, SQRT_MAX_ARRAY_SIZE),)
	stack_shape = draw(stack_shapes)
	d = draw(arrays(dtypes, shape=(*stack_shape, 1, n),
	elements=dict(allow_nan=False, allow_infinity=False)))
	# Functions that require invertible matrices may do anything when it is
	# singular, including raising an exception, so we make sure the diagonals
	# are sufficiently nonzero to avoid any numerical issues.
	assume(xp.all(xp.abs(d) > 0.5))
	diag_mask = xp.arange(n) == xp.reshape(xp.arange(n), (n, 1))
	return xp.where(diag_mask, d, xp.zeros_like(d))

asmeurer · 2024-02-22T23:03:37Z

Looks like the matrix_rank thing is a legitimate bug in the numpy implementation https://github.com/numpy/numpy/pull/25437/files#r1500043325.

… linalg namespaces separately

asmeurer · 2024-02-23T00:17:38Z

@honno let me know what you think of the latest commit as a way to test functions like matmul which should be in both the main and linalg namespace separately. Previously only the linalg one was tested, but I want to make sure that the compat library properly wraps things in both namespaces.

…to more-linalg2

honno

let me know what you think of the latest commit as a way to test functions like matmul which should be in both the main and linalg namespace separately. Previously only the linalg one was tested, but I want to make sure that the compat library properly wraps things in both namespaces.

Yeah looks good, nice catch 👍

Is there a more direct way to generate arrays d s.t. all(abs(d) > 0.5)?

Pushed a change which does this by passing a customer elements strategy.

Lets merge as-is and resolve everything else in separate PRs 🎉

asmeurer added 3 commits February 17, 2022 15:39

Add shape testing for vector_norm()

360df9f

Test the dtype and stacks in the vector_norm() test

e34f492

Remove an ununsed variable

111c237

honno reviewed Feb 18, 2022

View reviewed changes

array_api_tests/test_linalg.py Outdated Show resolved Hide resolved

asmeurer and others added 6 commits February 18, 2022 19:11

Use a simpler strategy for ord in test_vector_norm

4f3aa54

Co-authored-by: Matthew Barber <quitesimplymatt@gmail.com>

Skip the test_vector_norm test on the NumPy CI

979b81b

The upstream has bugs that will be fixed in the next release.

Fix syntax error

8df237a

Fix the input strategies for test_tensordot()

d11a685

Add a test for the tensordot result shape

a776cd4

Test stacking for tensordot

45b36d6

honno mentioned this pull request Apr 19, 2022

test_matrix_norm incorrect sampling #114

Closed

asmeurer added 2 commits April 25, 2022 17:20

Add allclose() and assert_allclose() helper functions

414b322

Use assert_allclose() in the linalg tests for float inputs

9bb8c7a

Fixes data-apis#44. Related to data-apis#117.

honno reviewed Apr 26, 2022

View reviewed changes

Remove skip from test_eigh

b3fb4ec

asmeurer mentioned this pull request Apr 28, 2022

matrix_power issues #117

Closed

asmeurer added 6 commits May 5, 2022 19:30

Disable eigenvectors stack test

241220e

There are different equivalent ways of representing eigenspaces in general, and the same algorithm may choose different ways for stacked vs. non-stacked matrices (e.g., they differ for cupy).

Reduce the relative tolerance in assert_allclose

ca70fbe

This still is going to require some tweaking.

Sort the eigenvalues when testing stacks

720b309

Merge branch 'more-linalg2' of github.com:asmeurer/array-api-tests in…

17d93bf

…to more-linalg2

Sort the results in eigvalsh before comparing

f439259

Remove the allclose testing in linalg

75ca73a

Floating-point dtypes now only test shape and dtype in the stacking test. Trying to make the allclose test work is too difficult for linear algebra functions.

honno reviewed Jun 13, 2022

View reviewed changes

asmeurer added 3 commits June 16, 2022 15:28

Add (commented out) stacking tests for solve()

d86a0a1

Remove unused none standin in the linalg tests

9bccfa5

Don't compare float elements in test_tensordot

f494b45

asmeurer added 2 commits February 7, 2024 15:23

Remove unused allclose and assert_allclose helpers

5aa9083

Update ndindex version requirement

938f086

Fix linting issue

3856b8f

asmeurer mentioned this pull request Feb 16, 2024

BUG: cholesky(empty((0, 0)), upper=True) hangs numpy/numpy#25840

Closed

Skip test_cross in CI

ccc6ca3

This was referenced Feb 20, 2024

Update dtype assertions in sum-like tests for >2023.12 versions #234

Open

Update libraries we test on CI #235

Open

asmeurer mentioned this pull request Feb 22, 2024

Some fixes for torch.linalg wrappers data-apis/array-api-compat#94

Merged

Test matmul, matrix_transpose, tensordot, and vecdot for the main and…

3092422

… linalg namespaces separately

asmeurer and others added 2 commits February 22, 2024 18:13

Merge branch 'more-linalg2' of github.com:asmeurer/array-api-tests in…

2d918e4

…to more-linalg2

Remove need for filtering in invertible_matrices()

3fefd20

honno approved these changes Feb 26, 2024

View reviewed changes

honno added 3 commits February 26, 2024 13:11

Merge branch 'master' into more-linalg2

a76e051

Skip flaky test_reshape

268682d

Less filtering in positive_definitive_matrices

0ddb0cd

honno merged commit a1d7701 into data-apis:master Feb 26, 2024

vfdev-5 mentioned this pull request May 29, 2024

[array api] update test suite to most recent commit jax-ml/jax#21469

Merged

asmeurer mentioned this pull request Nov 26, 2024

ENH: test vecdot values, incl complex conj #314

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

More improvements to test_linalg #101

More improvements to test_linalg #101

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

More improvements to test_linalg #101

More improvements to test_linalg #101

Uh oh!

Conversation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this 8000 comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!