8000 TYP: `np.ndarray.tolist` return type seems broken in numpy 2.2.0 · Issue #27944 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

TYP: np.ndarray.tolist return type seems broken in numpy 2.2.0 #27944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
neutrinoceros opened this issue Dec 9, 2024 · 8 comments · Fixed by #28644
Closed

TYP: np.ndarray.tolist return type seems broken in numpy 2.2.0 #27944

neutrinoceros opened this issue Dec 9, 2024 · 8 comments · Fixed by #28644
Assignees
@jorenham
Labels
41 - Static typing
Milestone
2.2.5 release

Comments

@neutrinoceros
Copy link
Contributor

Describe the issue:

While upgrading CMasher's CI to numpy 2.2.0, I got a new typechecking error that seems spurious to me. No error is raised with numpy 2.1.3

Reproduce the code example:

import numpy as np

def to_rgb(in_) -> tuple[float, float, float]:
    # minimal mock for matplotlib.colors.to_rgb
    return (1. ,2., 3.)

def foo(name: str, data: list[tuple[float, float, float]]) -> None:
    data_arr = np.array(data)

    if data_arr.dtype.kind == "S":
        cm_data = [str(x) for x in data_arr]
        colorlist = [to_rgb(_) for _ in cm_data]
    else:
        data_arr = np.atleast_2d(data_arr)
        colorlist = data_arr.tolist()

Error message:

cmasher.py:15: error: Incompatible types in assignment (expression has type "str | list[str] | list[list[str]] | list[list[list[Any]]]", variable has type "list[tuple[float, float, float]]")  [assignment]
Found 1 error in 1 file (checked 1 source file)

Python and NumPy Versions:

2.2.0
3.13.1 (main, Dec  4 2024, 16:25:14) [Clang 16.0.0 (clang-1600.0.26.4)]

Type-checker version and settings:

mypy==1.13.0

no custom settings

Additional typing packages.

Included with mypy, but not used here (as far as I'm aware)

typing-extensions==4.12.2
@neutrinoceros neutrinoceros mentioned this issue Dec 9, 2024
Merged
@neutrinoceros neutrinoceros changed the title TYP: np.ndarray.tolist return type seems broken in numpy 2.2 TYP: np.ndarray.tolist return type seems broken in numpy 2.2.0 Dec 9, 2024
@charris charris added this to the 2.2.1 release milestone Dec 9, 2024
@cgohlke cgohlke mentioned this issue Dec 9, 2024
Open
@jorenham
Copy link
Member
jorenham commented Dec 10, 2024 •
edited
Loading

data_arr = np.array(data) is inferred as NDArray[Any], which is technically correct, but far from ideal.

To give a bit more context, this is how ndarray.tolist is annotated:

    @overload
    def tolist(self: _HasShapeAndSupportsItem[tuple[()], _T], /) -> _T: ...
    @overload
    def tolist(self: _HasShapeAndSupportsItem[tuple[int], _T], /) -> list[_T]: ...
    @overload
    def tolist(self: _HasShapeAndSupportsItem[tuple[int, int], _T], /) -> list[list[_T]]: ...
    @overload
    def tolist(self: _HasShapeAndSupportsItem[tuple[int, int, int], _T], /) -> list[list[list[_T]]]: ...
    @overload
    def tolist(self: _HasShapeAndSupportsItem[Any, _T], /) -> _T | list[_T] | list[list[_T]] | list[list[list[Any]]]: ...

So without a know shape-type, the last overload should be used. And without a known dtype (bound to _T here), the correct return type should be

Any | list[Any] | list[list[Any]] | list[list[list[Any]]]

And pyright actually infers this correctly, and accepts your example, as it should.
But you're using mypy, and mypy is very broken, which this case is an example of, and causes it to infer _T as str. That makes no sense at all, which can only mean that it's one of the >1200 mypy bugs that aren't in pyright.

So I'm afraid that I can't help you here, and that this is a bug within mypy.

@jorenham jorenham added the 57 - Close? Issues which may be closable unless discussion continued label Dec 10, 2024
@neutrinoceros
Copy link
Contributor Author

Understood,. I may try to migrate to pyright, but for now I'll just put a lid on it. Thanks for your swift reply !

@jorenham jorenham removed the 57 - Close? Issues which may be closable unless discussion continued label Dec 10, 2024
@chrisjkuch chrisjkuch mentioned this issue Dec 18, 2024
Merged
@jorenham jorenham mentioned this issue Dec 19, 2024
Closed
@neutrinoceros neutrinoceros mentioned this issue Dec 24, 2024
Open
@seberg seberg mentioned this issue Dec 30, 2024
Open
@jorenham jorenham mentioned this issue Jan 6, 2025
Closed
@jorenham jorenham mentioned this issue Jan 28, 2025
Closed
@alexander-held alexander-held mentioned this issue Mar 11, 2025
Merged
@ishaan-mehta
Copy link

Is anyone else seeing this with pyright too? I am.

@jorenham
Copy link
Member
jorenham commented Apr 2, 2025

Is anyone else seeing this with pyright too? I am.

Could you share the exact code?

@ishaan-mehta
Copy link
ishaan-mehta commented Apr 3, 2025 •
edited
Loading

Is anyone else seeing this with pyright too? I am.

Could you share the exact code?

Sure, here's an MRE:

import pandas as pd

df = pd.DataFrame(
    ["2024-01-01", "2022-11-12", "2019-08-27", "2024-05-11", "2022-02-28"],
    columns=["date"],
)

df["date"] = pd.to_datetime(df["date"])
unique_years = df["date"].dt.year.unique()
print(f"unique_years type: {type(unique_years)}")
unique_years_list: list[int] = unique_years.tolist()
print(f"unique_years_list type: {type(unique_years_list)}")
print(f"unique_years_list[0] type: {type(unique_years_list[0])}")

This outputs:

unique_years type: <class 'numpy.ndarray'>
unique_years_list type: <class 'list'>
unique_years_list[0] type: <class 'int'>

But reveal_type(unique_years.tolist()) with Pylance says that it is a str. Same with raw pyright instead of Pylance:

error: Type "str" is not assignable to declared type "list[int]"
    "str" is not assignable to "list[int]" (reportAssignmentType)

Versions:

  • numpy==2.2.4
  • pyright==1.1.398
  • Pylance: 2025.4.1

@jorenham jorenham removed this from the 2.2.1 release milestone Apr 4, 2025
@jorenham
Copy link
Member
jorenham commented Apr 4, 2025

In your example @ishaan-mehta, unique_years is inferred as ndarray[Unknown, Unknown]. This is effectively the same as ndarray[Any, Any]. In this case, both pyright and mypy will indeed infer str as the return type of .tolist(). Here's a minimal .pyi example that shows this:

from typing import Any, reveal_type
import numpy as np

a: np.ndarray[Any, Any]
reveal_type(a.tolist())

mypy: Revealed type is "builtins.str"
pyright: Type of "a.tolist()" is "str"

The original post was about NDArray[Any], which reduces to ndarray[Any, dtype[Any]]. It's a subtle difference, but it seems to leads to different behavior in pyright. Mypy's behaviour is the same on both situations. So technically, this is a different issue.

But that being said, solving one issue will probably solve the other. And to be honest, I was too quick in dismissing this as a "mypy-only" issue. So I'm reopening this, and I'll try my best to find a solution in the coming days.

@jorenham jorenham reopened this Apr 4, 2025
@jorenham jorenham self-assigned this Apr 4, 2025
@jorenham jorenham mentioned this issue Apr 4, 2025
Merged
@jorenham jorenham added this to the 2.2.5 release milestone Apr 4, 2025
@ishaan-mehta
Copy link

Thanks for isolating the issue and looking into it @jorenham! (And I appreciate you accommodating the semi-related topic in the same issue.)

@charris charris mentioned this issue Apr 4, 2025
Merged
@jorenham
Copy link
Member

Thanks @neutrinoceros and @ishaan-mehta for reporting this! The fix is included in the latest 2.2.5 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jorenham jorenham

Labels
41 - Static typing
Projects
None yet
Milestone
2.2.5 release
Development

Successfully merging a pull request may close this issue.

TYP: fix ndarray.tolist() and .item() for unknown dtype jorenham/numpy
4 participants
@charris @jorenham @neutrinoceros @ishaan-mehta
0