8000 Pydantic v2 and numpy.typing conflict when using npt.DtypeLike in pydantic.BaseModel with python <=3.11 · Issue #25206 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

Pydantic v2 and numpy.typing conflict when using npt.DtypeLike in pydantic.BaseModel with python <=3.11 #25206

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
rbavery opened this issue Nov 20, 2023 · 4 comments
Closed

Pydantic v2 and numpy.typing conflict when using npt.DtypeLike in pydantic.BaseModel with python <=3.11 #25206

rbavery opened this issue Nov 20, 2023 · 4 comments
Labels
41 - Static typing

Comments

@rbavery
Copy link
rbavery commented Nov 20, 2023 •
edited
Loading

Describe the issue:

When I try to use npt.DTypeLike in a pydantic BaseModel, I get an error from pydantic that won't be fixed by pydantic

Please use `typing_extensions.TypedDict` instead of `typing.TypedDict` on Python < 3.12.

For further information visit https://errors.pydantic.dev/2.5/u/typed-dict-version
  File "/opt/workspace/src/sedonaai/tasks/ml_model_schemas.py", line 46, in <module>
    class TensorInfo(BaseModel):
pydantic.errors.PydanticUserError: Please use `typing_extensions.TypedDict` instead of `typing.TypedDict` on Python < 3.12.

For further information visit https://errors.pydantic.dev/2.5/u/typed-dict-version

this issue was first reported here:
pydantic/pydantic#6645

and this is where TypedDict is used from the typing module: https://github.com/numpy/numpy/blob/main/numpy/_typing/_dtype_like.py#L8

Reproduce the code example:

from pydantic import BaseModel, Field, FilePath, field_validator
from typing import List, Tuple, Dict, Optional, Literal, Union
import numpy.typing as npt

class TensorInfo(BaseModel):
    dtype: npt.DTypeLike = Field(...)
    shape: Union[Tuple[int, ...], List[int]] = Field(...)

    class Config:
        arbitrary_types_allowed = True

Error message:

Please use `typing_extensions.TypedDict` instead of `typing.TypedDict` on Python < 3.12.

For further information visit https://errors.pydantic.dev/2.5/u/typed-dict-version
  File "/opt/workspace/src/sedonaai/tasks/ml_model_schemas.py", line 46, in <module>
    class TensorInfo(BaseModel):
pydantic.errors.PydanticUserError: Please use `typing_extensions.TypedDict` instead of `typing.TypedDict` on Python < 3.12.

For further information visit https://errors.pydantic.dev/2.5/u/typed-dict-version

Runtime information:

1.26.2
3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]

[{'numpy_version': '1.26.2',
  'python': '3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]',
  'uname': uname_result(system='Linux', node='0591b98d73a9', release='6.2.0-36-generic', version='#37~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Oct  9 15:34:04 UTC 2', machine='x86_64')},
 {'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
                      'found': ['SSSE3',
                                'SSE41',
                                'POPCNT',
                                'SSE42',
                                'AVX',
                                'F16C',
                                'FMA3',
                                'AVX2'],
                      'not_found': ['AVX512F',
                                    'AVX512CD',
                                    'AVX512_KNL',
                                    'AVX512_KNM',
                                    'AVX512_SKX',
                                    'AVX512_CLX',
                                    'AVX512_CNL',
                                    'AVX512_ICL']}},
 {'architecture': 'Haswell',
  'filepath': '/usr/local/lib/python3.10/dist-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so',
  'internal_api': 'openblas',
  'num_threads': 12,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.23.dev'}]

Context for the issue:

I'm trying to define pydantic models for a metadata standard for computer vision models that operate on satellite images. validating numpy types would be very useful so that users of the standard can easily check if their model inputs match the correct numpy type and dimension shape. Supporting this functionality with numpy.typing is desirable across python versions and so I don't need to reinvent the wheel on how to serialize and deserialize numpy types.
@rgommers
Copy link
Member

Thanks for the report @rbavery. We're already relying on typing_extensions, so I think we can make this change in principle.

It can't really be considered a numpy bug of course, all we're doing is from typing import TypedDict. I 8000 t's quite aggressive from Pydantic to raise an error for that.

@rbavery rbavery changed the title BUG: Pydantic v2 and numpy.typing conflict when using npt.DtypeLike in pydantic.BaseModel with python <=3.11 Pydantic v2 and numpy.typing conflict when using npt.DtypeLike in pydantic.BaseModel with python <=3.11 Dec 9, 2023
@rbavery rbavery mentioned this issue Dec 9, 2023
Closed
@rbavery
Copy link
Author
rbavery commented Dec 9, 2023

I put up a PR to address the imports, am I missing anything besides changing the imports?

#25352

@jorenham
Copy link
Member
jorenham commented Apr 4, 2025 •
edited
Loading

This is the current DTypeLike definition:

numpy/numpy/_typing/_dtype_like.py

Lines 96 to 128 in 5825d67

8000
# Would create a dtype[np.void]
_VoidDTypeLike: TypeAlias = (
# (flexible_dtype, itemsize)
tuple[_DTypeLikeNested, int]
# (fixed_dtype, shape)
| tuple[_DTypeLikeNested, _ShapeLike]
# [(field_name, field_dtype, field_shape), ...]
#
# The type here is quite broad because NumPy accepts quite a wide
# range of inputs inside the list; see the tests for some
# examples.
| list[Any]
# {'names': ..., 'formats': ..., 'offsets': ..., 'titles': ...,
# 'itemsize': ...}
| _DTypeDict
# (base_dtype, new_dtype)
| tuple[_DTypeLikeNested, _DTypeLikeNested]
)
# Anything that can be coerced into numpy.dtype.
# Reference: https://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html
DTypeLike: TypeAlias = (
np.dtype[Any]
# default data type (float64)
| None
# array-scalar types and generic types
| type[Any] # NOTE: We're stuck with `type[Any]` due to object dtypes
# anything with a dtype attribute
| _SupportsDType[np.dtype[Any]]
# character codes, type strings or comma-separated fields, e.g., 'float64'
| str
| _VoidDTypeLike
)

The _DTypeDict is the only TypedDict type here.

Wouldn't it be easiest to create a custom alias like that for use in pydantic, that doesn't require the TypedDict? If you don't require void support, you could write that as something like

PydanticDTypeLike TypeAlias = np.dtype[Any] | type | str | None

which would match the most frequently used (non-void) dtype-likes

@jorenham jorenham linked a pull request Apr 4, 2025 that will close this issue
MAINT: update all TypedDict imports [skip ci] #25352
Closed
@jorenham
Copy link
Member
jorenham commented Apr 4, 2025

I'm closing this for the reasons explained in #25352 (review).

@jorenham jorenham closed this as not planned Won't fix, can't repro, duplicate, stale Apr 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
41 - Static typing
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

MAINT: update all TypedDict imports [skip ci] rbavery/numpy
3 participants
@rgommers @jorenham @rbavery
0