Closed
Description
Reviewing stubs for correctness is hard, and the existing stubs have many bugs (that are gradually being fixed). We could perhaps improve both of these by having better tests for stubs. Here's a concrete idea for that (inspired by a proposal I remember having seen at Gitter):
- Write a tester that checks definitions a stub conform to what happens at runtime by importing the target module, parsing the stub and running some checks.
- We'd find if a stub defines a module-level thing that's not actually there at runtime.
- We can verify that module attributes have the right types (at least for
int,str,booland other simple types). - Check that a class in a stub is a class at runtime (there probably are some exceptions which we can whitelist).
- Check that a function in a stub is a callable at runtime.
- Check for the existence of methods.
- Check that static and class methods are declared as such (assuming that we can reliably introspect this).
- Check that argument names of functions agree (when we can introspect them).
- Check that arguments with a
Nonedefault value have typeOptional[x].
- For 3rd party modules,
pip installthe package before running the tests. We can have a config file that specifies the versions of 3rd party modules to use to make the tests repeatable.
We could blacklist the modules that currently don't pass the test and gradually burn down the blacklist.
This wouldn't perfect and several things couldn't be checked automatically:
- Instance attributes are hard to check for using introspection.
- Things missing from stubs are hard to check for, since it's not clear how to know whether something is an internal thing or an imported things that shouldn't be part of the public interface.
- Types can't generally be tested for correctness, except for some instances of missing
Optional[...]and maybe a few other things. - Internal things accidentally included in stubs also probably can't be automatically detected.
However, this could still be pretty valuable, and this shouldn't be too hard to implement. We could start with very rudimentary checks and gradually improve the testing tool as we encounter new bugs in the stubs that we could automatically guard against.