8000 test_sample_order_invariance in common metrics tests applied to threshold metrics · Issue #12308 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

test_sample_order_invariance in common metrics tests applied to threshold metrics #12308

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
amueller opened this issue Oct 5, 2018 · 5 comments
Open

test_sample_order_invariance in common metrics tests applied to threshold metrics #12308

amueller opened this issue Oct 5, 2018 · 5 comments
Labels
module:test-suite everything related to our tests

Comments

@amueller
Copy link
Member
amueller commented Oct 5, 2018

This sample invariance test is currently applied to the ranking tests, and that seems strange.
For ranking the scores should have a different shape and be integer (right now y_true and y_pred are binary so it's valid scores but it's not a great test).

Also, the same should be tested for multi-class.
Via #10481.
@amueller
Copy link
Member Author
amueller commented Oct 5, 2018

The same kinda applies to test_format_invariance_with_1d_vectors though that test might make less sense for multiclass.

@amueller
Copy link
Member Author
amueller commented Oct 5, 2018

Similarly test_thresholded_invariance_string_vs_numbers_labels uses binary outcomes instead of scores and doesn't use multiclass data.

@amueller amu 8000 eller changed the title test_sample_order_invariance in common metrics tests applied to ranking test_sample_order_invariance in common metrics tests applied to threshold metrics Oct 5, 2018
@amueller amueller mentioned this issue Oct 5, 2018
Closed
@amueller
Copy link
Member Author

@thomasjpfan you implemented this, right?

@thomasjpfan
Copy link
Member

@thomasjpfan you implemented this, right?

Nope. Every test that uses METRIC_UNDEFINED_BINARY_MULTICLASS is mentioned in this issue. Looking into this.

@jnothman
Copy link
Member

I'm confused by your notion of ranking. I'm pretty sure ranking as far as our metrics are concerned is about having the estimator assign a score to each candidate label, and for the ranking to then be implied, not for the rank to be provided necessarily as an integer.

Not sure exactly what you're looking for here. But yes, it can be hard to know if a particular metric is being fully covered by the common tests.

@thomasjpfan thomasjpfan added the module:test-suite everything related to our tests label Feb 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
module:test-suite everything related to our tests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jnothman @amueller @thomasjpfan
0