Documentation section 3.3.1.1 has incorrect description of brier_score_loss #13887
Comments
|
Indeed this is probably the right course of action. Please feel free to open a PR if your wish. |
|
@Sycor4x I'll gladly work on it if you're not already doing it |
|
@qdeffense Thank you. I had planned to start these revisions if this suggestion were well-received; however, I've just come down with a cold and won't be able to write coherent code at the moment. If you want to take a stab at this, I support your diligence. It occurred to me after I wrote this that it is possible for the verbal description in 3.3.1.1 to be incorrect while the behavior of the scorer objects called via the strings in 3.3.1.1 might work correctly in the sense that internally,
If this is the case, then the documentation is the only thing that needs to be tweaked: just make it explicit that some kind of reversal is applied to I haven't been able to check -- I'm basically incapacitated right now. |
|
No, we should be using |
|
Apologies the mistake was introduced by me. |
|
Hi! I would want to start working on this. |
|
Thanks @stefan-matcovici |
|
I'll work on this. |
|
@nityamd I think @stefan-matcovici is working on that already in #14123. |
Uh oh!
There was an error while loading. Please reload this page.
In the documentation, section 3.3.1.1. "Common cases: predefined values" includes the remark
As far as I can tell, this is true for all of the listed metrics, except the
brier_score_loss. In the case ofbrier_score_loss, a lower loss value is better. This is becausebrier_score_lossmeasures the mean-square difference between a predicted probability and a categorical outcome; the Brier score is minimized at 0.0 because all summands are either(0 - 0) ^ 2=0or(1 -1) ^ 2=0when the model is making perfect predictions. On the other hand, the Brier score is maximized at 1.0 when all predictions are opposite the correct label, as all summands are either(0 - 1)^2=1or(1 - 0)^2=1.Therefore, the definition of the
brier_score_lossis not consistent with the quotation from section 3.3.1.1.I suggest making 2 changes to relieve this confusion.
Implement a function
neg_brier_score_losswhich simply negates the value ofbrier_score_loss; this is a direct analogy to what is done in the case ofneg_log_loss. A better model has a lower value of log-loss (categorical cross-entropy loss), therefore a larger value of the negative log-loss implies a better model. Naturally, the same is true for Brier score, where it is also the case that a better model is assigned a lower loss.Remove reference to
brier_score_lossfrom section 3.3.1.1. Brier score is useful in lots of ways; however, because it does not have the property that a larger value implies a better model, it seems confusing to mention it in the context of section 3.3.1.1. References tobrier_score_losscan be replaced withneg_brier_score_loss, which has the property that better models have large values, just like accuracy, ROC AUC and the rest of the listed metrics.The text was updated successfully, but these errors were encountered: