Incorrect invalid device error introduced in #25956 #29107

jph00 · 2024-05-25T05:03:16Z

Describe the bug

#25956 introduced a new sklearn.utils._array_api._check_device_cpu function to test whether a tensor is on CPU. However, the implementation of the test, which is device not in {"cpu", None}, is incorrect -- the device will actually not be a string, but device(type='cpu'). Therefore, you should attempt to get the type attr, and use that if available.

Steps/Code to Reproduce

You can view a sample error here:
https://github.com/fastai/fastai/actions/runs/9232979440/job/25404873935

Expected Results

ValueError: Unsupported device for NumPy: device(type='cpu') should not be thrown.

Actual Results

File /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sklearn/utils/_array_api.py:308, in _check_device_cpu(device)
    306 def _check_device_cpu(device):  # noqa
    307     if device not in {"cpu", None}:
--> 308         raise ValueError(f"Unsupported device for NumPy: {device!r}")

ValueError: Unsupported device for NumPy: device(type='cpu')

Versions

I've seen this on multiple Linux and Mac versions. My current Mac version:


System:
    python: 3.11.8 (main, Feb 26 2024, 15:36:12) [Clang 14.0.6 ]
executable: /Users/jhoward/miniconda3/bin/python
   machine: macOS-14.3.1-arm64-arm-64bit

Python dependencies:
      sklearn: 1.4.2
          pip: 23.3.1
   setuptools: 68.2.2
        numpy: 1.26.4
        scipy: 1.13.0
       Cython: None
       pandas: 2.2.1
   matplotlib: 3.8.4
       joblib: 1.4.0
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       filepath: /Users/jhoward/miniconda3/lib/libopenblasp-r0.3.21.dylib
         prefix: libopenblas
       user_api: blas
   internal_api: openblas
        version: 0.3.21
    num_threads: 8
threading_layer: pthreads
   architecture: armv8

       filepath: /Users/jhoward/miniconda3/lib/libomp.dylib
         prefix: libomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 8

The text was updated successfully, but these errors were encountered:

Avoids incorrect `ValueError: Unsupported device for NumPy: device(type='cpu')` by using `type` attr if available.

glemaitre · 2024-05-27T10:09:57Z

Do you have a minimal reproducer.

Looking at our code, I think that we expect to have a call to the sklearn.utils._array_api.device that is going to return the string "cpu". So it would be interesting to understand what is generating the device object to have a proper regression test.

I'm not the most familiar with the array API work so, ping @betatim and @ogrisel.

ogrisel · 2024-05-27T14:27:25Z

Looking at the code, this function _check_device_cpu seems to only be used for _NumPyAPIWrapper which is not meant to be called in torch inputs I think. So we need to understand how you ended up with an xp module that is an instance of the _NumPyAPIWrapper but with a device variable that holds a torch device (instead of None).

ogrisel · 2024-05-27T14:30:45Z

I tried to craft a minimal reproducer by mixing numpy and torch inputs as follows:

>>> import numpy as np
>>> import array_api_compat.torch as xp
>>> import sklearn
>>> sklearn.set_config(array_api_dispatch=True)
>>> from sklearn.metrics import r2_score
>>> r2_score(xp.arange(10, device="mps"), xp.arange(10, device="mps"), sample_weight=xp.asarray(range(10), device="mps"))
1.0
>>> r2_score(np.arange(10), xp.arange(10, device="mps"), sample_weight=xp.arange(10, device="mps"))
Traceback (most recent call last):
  Cell In[16], line 1
    r2_score(np.arange(10), xp.arange(10, device="mps"), sample_weight=xp.arange(10, device="mps"))
  File ~/code/scikit-learn/sklearn/utils/_param_validation.py:213 in wrapper
    return func(*args, **kwargs)
  File ~/code/scikit-learn/sklearn/metrics/_regression.py:1214 in r2_score
    xp, _, device_ = get_namespace_and_device(
  File ~/code/scikit-learn/sklearn/utils/_array_api.py:572 in get_namespace_and_device
    *get_namespace(*array_list, **skip_remove_kwargs),
  File ~/code/scikit-learn/sklearn/utils/_array_api.py:553 in get_namespace
    namespace, is_array_api_compliant = array_api_compat.get_namespace(*arrays), True
  File ~/miniforge3/envs/dev/lib/python3.11/site-packages/array_api_compat/common/_helpers.py:372 in array_namespace
    raise TypeError(f"Multiple namespaces for array inputs: {namespaces}")
TypeError: Multiple namespaces for array inputs: {<module 'array_api_compat.numpy' from '/Users/ogrisel/miniforge3/envs/dev/lib/python3.11/site-packages/array_api_compat/numpy/__init__.py'>, <module 'array_api_compat.torch' from '/Users/ogrisel/miniforge3/envs/dev/lib/python3.11/site-packages/array_api_compat/torch/__init__.py'>}

but it's raising the expected error message that states that r2_score should not be called with mixed-type inputs (as of now, we might relax this in the future if we have a good reason to do so).

ogrisel · 2024-05-27T14:36:52Z

From the traceback I also tried:

>>> import numpy as np
>>> import sklearn
>>> sklearn.set_config(array_api_dispatch=True)
>>> from sklearn.metrics import r2_score
>>> import torch
>>> x1, x2 = torch.randn(20, 5), torch.randn(20, 5)
>>> r2_score(x2.view(-1), x1.view(-1))
-0.8340672254562378

but I cannot reproduce either.

ogrisel · 2024-05-27T14:38:45Z

Ok I understand, it's happening when array API dispatch is disabled and regular numpy code is expected:

>>> import numpy as np
>>> import sklearn
>>> sklearn.set_config(array_api_dispatch=False)
>>> from sklearn.metrics import r2_score
>>> import torch
>>> x1, x2 = torch.randn(20, 5), torch.randn(20, 5)
>>> r2_score(x2.view(-1), x1.view(-1))
Traceback (most recent call last):
  Cell In[27], line 7
    r2_score(x2.view(-1), x1.view(-1))
  File ~/code/scikit-learn/sklearn/utils/_param_validation.py:213 in wrapper
    return func(*args, **kwargs)
  File ~/code/scikit-learn/sklearn/metrics/_regression.py:1242 in r2_score
    return _assemble_r2_explained_variance(
  File ~/code/scikit-learn/sklearn/metrics/_regression.py:897 in _assemble_r2_explained_variance
    output_scores = xp.ones([n_outputs], device=device, dtype=dtype)
  File ~/code/scikit-learn/sklearn/utils/_array_api.py:314 in wrapped_func
    _check_device_cpu(kwargs.pop("device", None))
  File ~/code/scikit-learn/sklearn/utils/_array_api.py:308 in _check_device_cpu
    raise ValueError(f"Unsupported device for NumPy: {device!r}")
ValueError: Unsupported device for NumPy: device(type='cpu')

so this is indeed a regression: when array API dispatch is disabled, the usual implicit conversion from torch CPU tensors to NumPy arrays should still happen and for some reason, we get a non-None device where we don't expect it. I need to investigate.

ogrisel · 2024-05-27T14:46:20Z

I think I get it. It's get_namespace_and_device that is wrong when array API dispatch is disabled. I will open a PR with a non-regression test.

jph00 · 2024-12-10T07:35:23Z

@ogrisel is it possible that this bug has reappeared? I'm getting the same error again here:
https://github.com/AnswerDotAI/fastcore/actions/runs/12251169586/job/34175356676

lesteve · 2024-12-10T12:54:32Z

@jph00, indeed it seems like a regression in scikit-learn 1.6. I am taking a closer look.

ogrisel · 2024-12-10T13:26:51Z

Reopening. Indeed, we made some changes in #29476 (including in the non-regression test named test_get_namespace_and_device) without realizing that it would break calls like r2_score(torch.ones(3), torch.ones(3)) when array API dispatch is disabled (default). In this case want to rely on the regular np.asarray implicit conversion instead.

We need to fix it (either in the sklearn.utils._array_api.device function or in the sklearn.utils._array_api.get_namespace_and_device function.

This time, add a non-regression test that directly calls r2_score(torch.ones(3), torch.ones(3)) (or other metrics functions) with array API dispatch disabled and check that it works. The intent of the test will be clearer and we won't risk changing it without understanding the consequence...

jph00 · 2024-12-18T07:12:39Z

@OmarManzoor just to confirm -- have you checked that your PR resolves this issue too?

lesteve · 2024-12-18T07:47:38Z

Yes, we added a non-regression test so that hopefully this does not happen again.

I double-checked that the snippet from #29107 (comment) works fine on main:

In [1]: >>> import numpy as np
   ...: >>> import sklearn
   ...: >>> sklearn.set_config(array_api_dispatch=False)
   ...: >>> from sklearn.metrics import r2_score
   ...: >>> import torch
   ...: >>> x1, x2 = torch.randn(20, 5), torch.randn(20, 5)
   ...: >>> r2_score(x2.view(-1), x1.view(-1))
Out[1]: -1.0773991838970147

lesteve · 2024-12-18T07:49:39Z

For completeness, the plan is probably to do a scikit-learn 1.6.1 bug-fix release in January that includes the fix from #30454.

girwin1 · 2025-01-31T13:34:56Z

Was this done? also getting this error

lesteve · 2025-02-03T07:33:47Z

@girwin1 check your scikit-learn version. This should be fixed in scikit-learn 1.6.1. If you are still getting an error with scikit-learn 1.6.1, please open a separate issue with a stand-alone snippet that reproduces the issue.

jph00 added Bug Needs Triage Issue requires triage labels May 25, 2024

jph00 added a commit to jph00/scikit-learn that referenced this issue May 25, 2024

Fixes scikit-learn#29107

adc3a0a

Avoids incorrect `ValueError: Unsupported device for NumPy: device(type='cpu')` by using `type` attr if available.

jph00 mentioned this issue May 25, 2024

Avoid incorrect "Unsupported device for NumPy"; fixes #29107 #29108

Closed

ogrisel added Regression Array API and removed Needs Triage Issue requires triage labels May 27, 2024

ogrisel added this to the 1.5.1 milestone May 27, 2024

ogrisel mentioned this issue May 27, 2024

FIX always return None as device when array API dispatch is disabled #29119

Merged

OmarManzoor closed this as completed in #29119 May 28, 2024

ogrisel reopened this Dec 10, 2024

ogrisel modified the milestones: 1.5.1, 1.6.1 Dec 10, 2024

lesteve mentioned this issue Dec 10, 2024

8000 FIX Fix device detection when array API dispatch is disabled #30454

Merged

thomasjpfan mentioned this issue Dec 16, 2024

Make skorch work with sklearn 1.6.0, attempt 2 skorch-dev/skorch#1078

Merged

luisfpereira mentioned this issue Dec 16, 2024

Update adapters for sklearn geomstats/geomstats#2038

Merged

OmarManzoor closed this as completed in #30454 Dec 18, 2024

snarayan21 mentioned this issue Jan 6, 2025

Address sklearn device issues mosaicml/composer#3748

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Incorrect invalid device error introduced in #25956 #29107

Incorrect invalid device error introduced in #25956 #29107

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!