ENH: Virtually implement __buffer__ for Python < 3.12 #26784
Conversation
|
@BvB93 can you please take a look? |
There was a problem hiding this comment.
Thanks for the PR!
I totally agree that it's annoying that opn Python <3.12 static type-checkers mark e.g. memoryview(my_ndarray) as an error.
And PEP 688 was (in part) proposed to solve this issue, which it did, but that requires at least Python 3.12.
In typeshed they have "backported" this by adding a __buffer__ to the builtin typing stubs, ignoring the fact that this isn't the case at runtime. But static stub-checkers (e.g. mypy's stubtest or pyright --verifytypes) report this as a "hard" error, making this fall into the category of bugs.
An although I fully understand why this is practical, and that it could make static typing easier in several situations, I don't think it's a good idea to (knowingly) have the typing stubs deviate from the (true) runtime.
And that's not only because it's wrong, or because "bugs are bad". It's mostly because I know from experience that things like these can (and will) lead to very tricky and significant problems , and many confused users (myself included).
I could list several major problems such deviations have caused if you want, but that'll probably become a huge rant 😅 (and is unrelated to this PR).
In this case, I think that especially for ndarray subclasses this could cause many typing issues, e.g. in 3rd party libraries, most of which are unforeseeable "unknown unknowns".
So to make a long story short: As far as I'm concerned, I don't think this is worth the risk.
ELet me conclude by quoting Tim Peters wise words from PEP 20 (even though this probably makes me sound like a total wise-ass):
Special cases aren't special enough to break the rules.
Although practicality beats purity.
| dtype[Any], | ||
| Union[bool, int, float, complex, str, bytes], | ||
| ] | ||
| from typing_extensions import Buffer |
There was a problem hiding this comment.
typing_extensions isn't a required dependency; it can only be used in stubs or if typing.TYPE_CHECKING
|
Sad but makes sense, though it is inconsistent with the decision made in typeshed. I guess will be stuck with those warnings for a couple years more since VFX Platform CY2025 is also currently planning to use Python 3.11 :\ |
|
For what it's worth; perhaps you could work around it with a custom from typing import cast
from typing_extensions import Buffer
import numpy as np
import numpy.typing as npt
def better_memoryview(x: Buffer | npt.NDArray[np.generic], /) -> memoryview:
# alternatively, you could also use `# type: ignore[arg-type]` (with mypy)
return memoryview(cast(Buffer, x))You could (but probably shouldn't) even go as far as replace import builtins
builtins.memoryview = better_memoryview... but that'll probably get you fired 😅 |
See #26783 for the details
Note that there is also
__release_buffer__part of the buffer protocol -numpy/numpy/__init__.pyi
Lines 2843 to 2844 in 7687245
I've kept it Python 3.12+ only as it's optional part of the protocol and it's
Bufferis using just__buffer__to detect whether class supports the protocol. So, there is no need to virtually implement it for <3.12.