8000 Initial check-in of type specification conformance suite. by erictraut · Pull Request #1552 · python/typing · GitHub
[go: up one dir, main page]

Skip to content

Initial check-in of type specification conformance suite. #1552

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Dec 28, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
a9a94ef
Initial check-in of type specification conformance suite.
erictraut Dec 26, 2023
2f565e5
Fixed syntax error in f-string that affects 3.11 and older.
erictraut Dec 26, 2023
815ec19
Addressed PR feedback. Addressed linter issues.
erictraut Dec 26, 2023
b35bfa9
Updated test results for pyright 1.1.343, which addresses all conform…
erictraut Dec 26, 2023
89d41d3
Added support for pytest.
erictraut Dec 26, 2023
448fffe
Improved report visuals. Added timing stats.
erictraut Dec 26, 2023
4b9ec56
Incorporated PR feedback: use `{sys.executable} -m` for `run` invocat…
erictraut Dec 26, 2023
f89c985
Switched from "Yes" to "Pass" for clarity. Started to add dataclass t…
erictraut Dec 26, 2023
e057706
Added more dataclass tests.
erictraut Dec 27, 2023
e696bed
Added more dataclass tests.
erictraut Dec 27, 2023
3f4eb6d
Added tests for dataclass_transform.
erictraut Dec 27, 2023
c797202
Added some tests for callables.
erictraut Dec 27, 2023
3ec7eff
Added test for `**kwargs` with Unpack TypedDict annotation.
erictraut Dec 27, 2023
f4a0c96
Started to add tests for type aliases.
erictraut Dec 27, 2023
31985ba
Renamed "traditional" to "implicit" for clarity.
erictraut Dec 27, 2023
560791c
Added tests for explicit type aliases, recursive type aliases, and Ne…
erictraut Dec 27, 2023
5e10e0b
Added tests for Python 3.12 `type` statement to define type aliases.
erictraut Dec 28, 2023
ca01411
Added tests for TypeAliasType.
erictraut Dec 28, 2023
cc3fb5a
Added code to skip tests for a type checker if it cannot be installed…
erictraut Dec 28, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8000
Prev Previous commit
Added code to skip tests for a type checker if it cannot be installed…
… for some reason.
  • Loading branch information
erictraut committed Dec 28, 2023
commit cc3fb5a6eeff60fd1acc31b57eb8aa35d7262f43
2 changes: 2 additions & 0 deletions conformance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ To run the conformance test suite:
* Switch to the `conformance` subdirectory and install all dependencies (`pip install -r requirements.txt`).
* Switch to the `src` subdirectory and run `python main.py`.

Note that some type checkers may not run on some platforms. For example, pytype cannot be installed on Windows. If a type checker fails to install, tests will be skipped for that type checker.

## Reporting Conformance Results

Different type checkers report errors in different ways (with different wording in error messages and different line numbers or character ranges for errors). This variation makes it difficult to fully automate test validation given that tests will want to check for both false positive and false negative type errors. Some level of manual inspection will therefore be needed to determine whether a type checker is fully conformant with all tests in any given test file. This "scoring" process is required only when the output of a test changes — e.g. when a new version of that type checker is released and the tests are rerun. We assume that the output of a type checker will be the same from one run to the next unless/until a new version is released that fixes or introduces a bug. In this case, the output will need to be manually inspected and the conformance results re-scored for those tests whose output has changed.
Expand Down
3 changes: 2 additions & 1 deletion conformance/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,5 @@ tomlkit
pyright
mypy
pyre-check
pytype
pytype; platform_system != "Windows"

6 changes: 4 additions & 2 deletions conformance/src/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,8 +116,10 @@ def main():

# Run each test case with each type checker.
for type_checker in TYPE_CHECKERS:
type_checker.install()
run_tests(root_dir, type_checker, test_cases)
if not type_checker.install():
print(f'Skipping tests for {type_checker.name}')
else:
run_tests(root_dir, type_checker, test_cases)

# Generate a summary report.
generate_summary(root_dir)
Expand Down
60 changes: 41 additions & 19 deletions conformance/src/type_checker.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,10 @@ def name(self) -> str:
raise NotImplementedError

@abstractmethod
def install(self) -> None:
def install(self) -> bool:
"""
Ensures that the latest version of the type checker is installed.
Returns False if installation fails.
"""
raise NotImplementedError

Expand All @@ -48,8 +49,13 @@ class MypyTypeChecker(TypeChecker):
def name(self) -> str:
return "mypy"

def install(self) -> None:
run(f"{sys.executable} -m pip install mypy --upgrade", shell=True)
def install(self) -> bool:
try:
run(f"{sys.executable} -m pip install mypy --upgrade", check=True, shell=True)
return True
except:
print('Unable to install mypy')
return False

def get_version(self) -> str:
proc = run(
Expand Down Expand Up @@ -80,13 +86,18 @@ class PyrightTypeChecker(TypeChecker):
def name(self) -> str:
return "pyright"

def install(self) -> None:
# Install the Python wrapper if it's not installed.
run(f"{sys.executable} -m pip install pyright --upgrade", shell=True)
def install(self) -> bool:
try:
# Install the Python wrapper if it's not installed.
run(f"{sys.executable} -m pip install pyright --upgrade", check=True, shell=True)

# Force the Python wrapper to install node if needed
# and download the latest version of pyright.
self.get_version()
# Force the Python wrapper to install node if needed
# and download the latest version of pyright.
self.get_version()
return True
except:
print('Unable to install pyright')
return False

def get_version(self) -> str:
proc = run(
Expand Down Expand Up @@ -122,15 +133,21 @@ class PyreTypeChecker(TypeChecker):
def name(self) -> str:
return "pyre"

def install(self) -> None:
run(f"{sys.executable} -m pip install pyre-check --upgrade", shell=True)
def install(self) -> bool:
try:
run(f"{sys.executable} -m pip install pyre-check --upgrade", check=True, shell=True)

# Generate a default config file.
pyre_config = (
'{"site_package_search_strategy": "pep561", "source_directories": ["."]}\n'
)
with open(".pyre_configuration", "w") as f:
f.write(pyre_config)
# Generate a default config file.
pyre_config = (
'{"site_package_search_strategy": "pep561", "source_directories": ["."]}\n'
)
with open(".pyre_configuration", "w") as f:
f.write(pyre_config)

return True
except:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoid raw except

print('Unable to install pyre')
return False

def get_version(self) -> str:
proc = run("pyre --version", stdout=PIPE, text=True, shell=True)
Expand All @@ -156,8 +173,13 @@ class PytypeTypeChecker(TypeChecker):
def name(self) -> str:
return "pytype"

def install(self) -> None:
run(f"{sys.executable} -m pip install pytype --upgrade", shell=True)
def install(self) -> bool:
try:
run(f"{sys.executable} -m pip install pytype --upgrade", check=True, shell=True)
return True
except:
print('Unable to install pytype on this platform')
return False

def get_version(self) -> str:
proc = run(
Expand Down
0