[MRG] API Replace scorer brier_score_loss with neg_brier_score #14898
Conversation
…into pr/13918 # Conflicts: # sklearn/metrics/tests/test_score_objects.py
This reverts commit 17a6886.
- change neg_brier_score to neg_brier_score_loss
This reverts commit 17a6886.
- change neg_brier_score to neg_brier_score_loss
…-learn into pr/13918 � Conflicts: � sklearn/metrics/tests/test_score_objects.py
|
What do you mean by
The only way I see this happening is by checking for all the scorer objects declared in scorer.py or by creating a new private dict containing deprecated scorers What if we go with the first solution proposed by @jnothman?
The error message remains the same meaning that the user should check SCORERS dict for valid scorers. The way I implemented now |
Let's remove 'brier_score_loss' from SCORERS
we only need to handle brier_score_loss seperately at the beginning. |
doc/modules/model_evaluation.rst
Outdated
| @@ -61,7 +61,7 @@ Scoring Function | |||
| 'accuracy' :func:`metrics.accuracy_score` | |||
| 'balanced_accuracy' :func:`metrics.balanced_accuracy_score` | |||
| 'average_precision' :func:`metrics.average_precision_score` | |||
| 'brier_score_loss' :func:`metrics.brier_score_loss` | |||
| 'neg_brier_score_loss' :func:`metrics.brier_score_loss` | |||
There was a problem hiding this comment.
I think 'neg_brier_score` might be a more friendly name. neg already implies that the score would otherwise decrease with improvement, and "Brier score loss" is not something known in the literature: it appears 70 times on Google not associated with scikit-learn.
There was a problem hiding this comment.
I think 'neg_brier_score` might be a more friendly name. neg already implies that the score would otherwise decrease with improvement,
Maybe neg_brier_score_loss is a better name? because the name of the function is brier_score_loss and we also have other similar scorers like "neg_log_loss"? @jnothman
and "Brier score loss" is not something known in the literature
If so, we should change the name of the function.
There was a problem hiding this comment.
Can you explain the definition of this neg_brier_score? look at my post please: h 8000 ttps://stackoverflow.com/questions/62853440/neg-brier-score-returns-negative-score
doc/modules/model_evaluation.rst
Outdated
| @@ -61,7 +61,7 @@ Scoring Function | |||
| 'accuracy' :func:`metrics.accuracy_score` | |||
| 'balanced_accuracy' :func:`metrics.balanced_accuracy_score` | |||
| 'average_precision' :func:`metrics.average_precision_score` | |||
| 'brier_score_loss' :func:`metrics.brier_score_loss` | |||
| 'neg_brier_score_loss' :func:`metrics.brier_score_loss` | |||
There was a problem hiding this comment.
I think 'neg_brier_score` might be a more friendly name. neg already implies that the score would otherwise decrease with improvement,
Maybe neg_brier_score_loss is a better name? because the name of the function is brier_score_loss and we also have other similar scorers like "neg_log_loss"? @jnothman
and "Brier score loss" is not something known in the literature
If so, we should change the name of the function.
|
We have the _loss suffix to clarify that smaller is better. But unlike
this, log loss is called "log loss".
|
The problem is that we add the _loss suffix to the name of the function, so maybe we should add it to the name of the scorer. |
|
We drop _score from other metric names. I don't think there's any way to
make this consistent, only readable.
|
OK, I don't think I'll argue with you about this. Let's use @stefan-matcovici please take some time to update the PR, thanks. |
|
Do you have time to finish this one? @stefan-matcovici thanks. |
|
yes I do |
Why is it more consistent? According to google, "brier score" is a term but "brier" is not. (Similarly, "precision" is a term but "precision score" is not.) |
|
Yeah, I'm okay with that. But people actually do say "F1 score" and "roc
score"
|
|
tests are failing @stefan-matcovici , otherwise LGTM |
|
thanks a lot, guys, my first pr 😁 |
|
thanks for your patience :) |
|
Thanks for helping us finally close this, @stefan-matcovici, and congrats on the merge!! |
Reference Issues/PRs
Fixes #13887
What does this implement/fix? Explain your changes.
Introduced new scorer neg_brier_score to reflect the real behavior of the scorer
Added deprecation of brier_loss_score (not sure about the versions mentioned in the deprecation message 😕)
Updated documentation to reflect changes
Any other comments?
cloned from #14123