Description
Why Mypy re-enable
For a few weeks MyPy checks have been disabled after the switch to Python 3.7 (per #19317).
We should, however, re-enable it back as it is very useful in catching a number of mistakes.
How does it work
We 've re-added the mypy pre-commit now - with mypy bumped to 0.910. This version detects far more errors and we should fix them all before we switch the CI check back.
-
mypy will be running for incremental changes in pre-commit, same as before. This will enable incremental fixes of the code changed by committers who use pre-commits locally
-
mypy on CI runs in non-failing mode. When the main pre-commit check is run, mypy is disabled, but then it is run as a separate step (which does not fail but will show the result of running mypy on all our code). This will enable us to track the progress of fixes
Can I help with the effort, you ask?
We started concerted effort now and incrementally fix all the mypy incompatibilities - ideally package/by/package to avoid huge code reviews. We'd really appreciate a number of people to contribute, so that we can re-enable mypy back fully and quickly :).
How can I help?
What you need is:
- checkout
main ./breeeze build-imagepip install pre-commitpre-commit install
This will enable automated checks for when you do a regular contribution. When you make your change, any MyPy issues will be reporteed and you need to fix them all to commit. You can also commit with --no-verify flag to skip that, bu, well, if you can improve airlfow a little - why not?
How can I help more ?
You can add PRs that are fixing whole packages, without contributing features or bugfixes. Please refer to this issue #19891 and ideally comment below in the issue that you want to take care of a package (to avoid duplicate work).
An easy way to run MyPy check for package can be done either from the host:
find DIRECTORY -name "*.py" | xargs pre-commit run mypy --files
or from ./breeze shell:
mypy --namespace-packages DIRECTORY
Current list of mypy PRs:
https://github.com/apache/airflow/pulls?q=is%3Aopen+is%3Apr+label%3Amypy
Remaining packages
Here is the list of remaining packages to be "mypy compliant" generated with:
pre-commit run mypy --all-files 2>&1 | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" | grep "error:" | sort | awk 'FS=":" { print $1 }' | xargs dirname | sort | uniq -c | xargs -n 2 printf "* [ ] (%4d) %s\n"
- ( 1) airflow/api/common/experimental
- ( 1) airflow/contrib/sensors
- ( 1) airflow/example_dags
- ( 1) airflow/jobs
- ( 4) airflow/models
- ( 1) airflow/providers/microsoft/winrm/hooks
- ( 1) airflow/providers/ssh/hooks
- ( 1) tests/providers/amazon/aws/hooks
- ( 1) tests/providers/google/cloud/hooks
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.