CI result inspector util #37976
CI result inspector util #37976
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
cc @ydshieh! |
|
Oh yeah you're welcome to have a look, but I don't think of this PR as done yet. Hence the long draft status :) |
|
I am also thinking to improve these kind of things too. Can take a look once I get the time (not too far way but not tomorrow or the next few days neither) |
|
Nice. The changes in this PR is also what I am looking for, including the name changes in At this moment, could we go #38230 first, and I will try to incorporate the changes in this PR progressively 🙏 (starting with the name changes) ? |
This util lets us compare different CI run results. Can be used to find tests that are failing on one specific architecture, or find tests that are failing on multiple/all architectures (in which case perhaps the test isn't working at all).
Example usage:
python3 utils/ci_results_inspector.py -a 14837567374 -b 14850580148 --output_dir ~/transformers-ci --token <...>This will print all the tests that are failing in
14837567374that are not failing in14837567374.You can also specify the
operatorused, where you can either use|for OR, or&for AND operations.python3 utils/ci_results_inspector.py -op '&' -a 14837567374 -b 14850580148 --output_dir ~/transformers-ci --token <...>This will print all tests that are failing in both runs.
Bring your own token 🫡