Function to get scorers for task #12385
Comments
|
@jnothman Trying to understand. Can this be implemented by maintaining separate dict of scorers as in scikit-learn/sklearn/metrics/scorer.py Lines 501 to 521 in a1d0e96 for different tasks such as classification , regression, clustering etc. |
|
No, while related to that:
|
|
Not sure if I really get what you want but can't we do a class wrapper for the scorers? For example a scorer wrapper class would have something like class Scorer:
is_binary_scorer = True
needs_probability = True
def __init__(self, *args, **kwargs):
pass
def compute(self, y_true, y_pred):
passThen to take binary scorers you can do a list comprehension. params = {
'weights': 10,
'something': 'pass'
}
binary_scorers = [s(**params) for s in ALL_SCORERS where s.is_binary_scorer] If a scorer needs a predict_proba when doing multi metrics you can also skip them with a warning if the model used doesn't have it. Just my 2 cents. You can also do something like this for the multi metrics for s in scorers:
if s.needs_probability:
if proba is None:
try:
proba = estimator.predict_proba(X_test)
score = s.compute(y_true, proba)
except AttributeError:
pass
else:
score = s.compute(y_true, proba)
else:
if y_pred is None:
estimator.predict(X_test)
score = s.compute(y_true, y_pred)Oops seems like something like this was already posted in the referenced issue. |
|
This is not about computing various metrics efficiently, but about helping users find metrics appropriate for a task, and making sure they do not get caught in some traps. For example, the scorers available do not return any per-class metrics in the case of multiclass, multilabel, etc. They also make some sometimes-poor assumptions, including that:
|
|
If that's the case won't sub classes or separate metrics module do fine? If I'm working on a classification problem and |
|
It still doesn't satisfy the need for specifying labels, nor does it readily create scorers for search. |
I suppose that most likely you will need the model too. |
|
Yes, you're right, the estimator might be useful to determine |
|
Bit stale, but still: the sklearn.utils.discovery module could be the best place for a function all_scorers([task_filter]) to live in. |
I would like to see a utility which would construct a set of applicable scorers for a particular task, returning a Mapping from string to callable scorer. It will be hard to design the API of this right the first time. [Maybe this should be initially developed outside this project and contributed to scikit-learn-contrib, but I think it reduces risk of mis-specifying scorers, so it's of benefit to this project.]
The user will be able to select a subset of the scorers, either with a dict comprehension or with some specialised methods or function parameters. Initially it wouldn't be efficient to run all these scorers, but hopefully we can do something to fix #10802 :|.
Let's take for instance a binary classification task. The function
get_applicable_scorers(y, pos_label='yes')for binaryymight produce something like:Doing the same for multiclass classification would pass
labelsas appropriate, and would optionally would get per-class binary metrics, as well as overall multiclass metrics.I'm not sure how
sample_weightfits in here, but ha! we still don't support weighted scoring in cross validation (#1574), so let's not worry about that.The text was updated successfully, but these errors were encountered: