FEAT SLEP6: scorers #22757
Merged
FEAT SLEP6: scorers #22757
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
CI is green. This is ready for review. |
adrinjalali
commented
Mar 11, 2022
jnothman
reviewed
Mar 12, 2022
There was a problem hiding this comment.
Should we add:
- a test for
_passthrough_scorerwhere the estimator requests a score param? (Can we use a mock estimator and a generic test to avoid reinventing such tests for every simple router?) - documentation for implementing a custom scorer with a request here?
| @@ -1140,3 +1144,34 @@ def test_scorer_no_op_multiclass_select_proba(): | |||
| labels=lr.classes_, | |||
| ) | |||
| scorer(lr, X_test, y_test) | |||
|
|
|||
|
|
|||
| @pytest.mark.parametrize("name, scorer", SCORERS.items()) | |||
There was a problem hiding this comment.
do we really need to test them all? I suppose it's not expensive?
There was a problem hiding this comment.
yeah:
$ time pytest -x sklearn/metrics/tests/test_score_objects.py -k test_scorer_metadata_request
============================================================================ test session starts ============================================================================
platform linux -- Python 3.10.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/adrin/Projects/sklearn/scikit-learn, configfile: setup.cfg
plugins: xdist-2.5.0, typeguard-2.13.3, forked-1.4.0
collected 187 items / 135 deselected / 52 selected
sklearn/metrics/tests/test_score_objects.py .................................................... [100%]
==================================================================== 52 passed, 135 deselected in 0.17s =====================================================================
pytest -x sklearn/metrics/tests/test_score_objects.py -k 1.84s user 1.38s system 329% cpu 0.974 totaland most of that time is spent collecting the tests.
28 tasks
Thanks for that, it made me realize I think this is now ready for hopefully a final review. |
|
@thomasjpfan shall we merge? |
thomasjpfan
reviewed
Mar 23, 2022
jnothman
approved these changes
Mar 23, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds metadata routing methods to scorers.
It is a small and contained PR since it only touches scorers, and doesn't touch objects which use scorers. It only tests that putting a scorer which requests a metadata is correctly in routing.
Note: This PR is to be merged into the
sample-propsand not themainbranch.cc @jnothman @lorentzenchr @thomasjpfan