Description
We've had some useful, long discussions about writing tests for stubs (#754, #917). Most of the discussion has been around having tests for the bulk of the functionality in stubs and the libraries they describe. I think we're all broadly in agreement that
- such comprehensive tests would be valuable; but
- just writing out a test for each function and class stubbed isn't helpful, because it amounts to repeating what we're already saying in the stubs; and
- a higher-tech approach to testing the bulk of the stubs' content won't be trivial to get to work with a good signal-to-noise ratio. (It'll be great if it can be done; some folks are working on it, and others are skeptical it can.)
This issue is for a narrower proposal: sometimes when a library's real signature is something complex or hard to express in the type system, our signature ends up pretty tricky or we end up telling sort of a white lie to the type-checkers. For these, even when looking right at the signature in one window and the library's docs (or an experiment in the REPL) in the other, it can be hard to tell whether they match. In these cases, I think a hand-written test would be useful -- probably looking a lot like a mypy unit test, with some code to be type-checked and an indication that certain lines should have errors and the rest shouldn't, and then we could e.g. run mypy and pytype against it.
@JukkaL suggested just this in one bullet point of a comment on #754, and gave some examples:
Add a test that exercises particularly complex signatures. Some recent examples where this could have been helpful are dict.get (a recent change to it broke it on mypy) and requests.api (there was a bad signature).
A new example today is #1331: auto doesn't really inherit from IntFlag, and we may find a new way of refining how we describe that interaction in the stubs. When we do it'll be useful to have a test for the functionality that that little lie in the stubs is intended to provide.