8000 `ParamSpec` substitution: difference in behaviour between `collections.abc.Callable` and `typing.Callable` · Issue #102723 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

ParamSpec substitution: difference in behaviour between collections.abc.Callable and typing.Callable #102723

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
AlexWaygood opened this issue Mar 15, 2023 · 6 comments
Closed

ParamSpec substitution: difference in behaviour between collections.abc.Callable and typing.Callable #102723

AlexWaygood opened this issue Mar 15, 2023 · 6 comments
Assignees
@sobolevn
Labels
3.12 only security fixes stdlib Python modules in the Lib dir topic-typing type-bug An unexpected behavior, bug, or error

Comments

@AlexWaygood
Copy link
Member
AlexWaygood commented Mar 15, 2023 •
edited
Loading

There is currently a difference in behaviour between collections.abc.Callable and typing.Callable for the following edge case involving ParamSpec substitution:

Running PGUpdate|x64 interpreter...
Python 3.12.0a6+ (heads/main:12226bec25, Mar 10 2023, 17:32:23) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import typing as t, collections.abc as c
>>> P, T = t.ParamSpec("P"), t.TypeVar("T")
>>> t.Callable[P, T][[P, str], bool][int]
typing.Callable[[int, str], bool]
>>> c.Callable[P, T][[P, str], bool][int]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen _collections_abc>", line 501, in __getitem__
  File "<frozen _collections_abc>", line 456, in __new__
TypeError: Callable must be used as Callable[[arg, ...], result].

According to PEP-612, this is an invalid substitution, for two reasons:

  1. Parameters can only be prepended to a ParamSpec in a parameters list using typing(_extensions).Concatenate.
  2. Appending parameters to a ParamSpec in a parameters list is disallowed.

As such, the behaviour of collections.abc.Callable is more correct here, so ideally we'd change the behaviour of typing.Callable to match collections.abc.Callable.

However, this error should hopefully be caught by static type checkers anyway, and this is a false negative rather than a false positive. The runtime makes no promises that it will raise TypeError on all invalid substitutions, so fixing this should be low priority, in my opinion. We should first concentrate on fixing substitutions where the runtime raises exceptions, even though it shouldn't. For example:

This discrepancy in behaviour was first uncovered as part of a broader discussion in:

Cc. @sobolevn
@AlexWaygood AlexWaygood added type-bug An unexpected behavior, bug, or error stdlib Python modules in the Lib dir topic-typing 3.12 only security fixes labels Mar 15, 2023
@AlexWaygood AlexWaygood mentioned this issue Mar 15, 2023
Closed
@sobolevn sobolevn self-assigned this Mar 16, 2023
@sobolevn
Copy link
Member

I am going to research, how hard it is to change.

If this is a rather simple patch, I am going to apply it.
If this is a complex one, I will close this issue as "won't fix".

Because parameters and their substitution is a very hard thing to get right.

@sobolevn
Copy link
Member

Wait, I am confused again. Why c.Callable[P, T][[P, str], bool] is allowed?

It seems like P can only be substituted with:

  1. Some other ParamSpec variable: P -> P2
  2. ...
  3. Arguments list without any ParamSpec variables: [int, str]
  4. Concatenate with ParamSpec: Concatenate[int, P]

Am I missing something?

@AlexWaygood
Copy link
Member Author
AlexWaygood commented Mar 16, 2023 •
edited
Loading

You're correct, c.Callable[P, T][[P, str], bool] should also, strictly speaking, not be allowed.

Honestly, I think we should probably leave this one. The runtime allows all sorts of crazy things that should be rejected by type checkers. My personal favourite (we should make this an alias for Any):

>>> list[r"¯\_(ツ)_/¯"]
list['¯\\_(ツ)_/¯']

We've never promised that the runtime would disallow all things that should be rejected by type checkers, and I don't want us to start making that promise now.

The fact that c.Callable[P, T][[P, str], bool][int] raises a TypeError is, strictly speaking, correct -- but I think it's almost certainly just an accident of the current implementation rather than something that was designed. So we shouldn't spend time trying to change the behaviour (since the behaviour is not incorrect), but we also probably shouldn't spend time trying to make typing.Callable match it.

#88965 seems like a much more important issue to me, since that involves the runtime incorrectly raising a TypeError for valid substitutions.

@sobolevn
Copy link
Member
sobolevn commented Mar 16, 2023 •
edited
Loading

Yes, I agree. We should close this as won't fix. Because the correct behaviour is very complex to achieve. And will probably break other things.

I am going to research, how hard it is to change.

There's no point to address a minor issue like this one by the cost of increasing code complexity.

@sobolevn sobolevn closed this as not planned Won't fix, can't repro, duplicate, stale Mar 16, 2023
@AlexWaygood
Copy link
Member Author

Would you be interested in taking a look at #88965? Serhiy assigned it to himself several months ago, but there's been no activity on the issue since, and it would be great to have it fixed.

@sobolevn
Copy link
Member

I will tomorrow!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sobolevn sobolevn

Labels
3.12 only security fixes stdlib Python modules in the Lib dir topic-typing type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sobolevn @AlexWaygood
0