@@ -311,15 +311,15 @@ Point forecasts and consistent scoring functions
311
311
Let's assume that the target variable :math: `Y` is a random variable, that
312
312
we have observations/realizations :math: `y` and that we make predictions
313
313
:math: `\hat {y}`.
314
- Scoring functions :math: `S(\hat {y}, y )` then rank the prediction :math: `\hat {y}`
314
+ Scoring functions :math: `S(y, \hat {y})` then rank the prediction :math: `\hat {y}`
315
315
of different models, given the observation :math: `y`.
316
- The higher the score the better the correponding model.
316
+ The higher the score the better the corresponding model.
317
317
For a test or validation set :math: `y_i`, one usually uses
318
- :math: `\bar {S} = \frac {1 }{n_\text {samples}}\sum _{i=0 }^{n_\text {samples}-1 } S(\hat {y}_i, y_i )`.
318
+ :math: `\bar {S} = \frac {1 }{n_\text {samples}}\sum _{i=0 }^{n_\text {samples}-1 } S(y_i, \hat {y}_i)`.
319
319
The prediction :math: `\hat {y}` is said to be a point forecast.
320
320
The optimal point forecast under :math: `S` is the Bayes Rule
321
- :math: `\hat {y} = \operatorname {argmin }_x \mathbb {E}[S(x,Y )]` (to get an
322
- unbiased estimate of :math: `\mathbb {E}[S(x,Y )]` for model evaluation is one
321
+ :math: `\hat {y} = \operatorname {argmax }_x \mathbb {E}[S(Y,x )]` (to get an
322
+ unbiased estimate of :math: `\mathbb {E}[S(Y,x )]` for model evaluation is one
323
323
reason to use a test set independent of the training set).
324
324
Instead of a point forecast, one could try to issue the whole probability
325
325
distribution :math: `F(y)` of the target variable :math: `Y`.
@@ -335,21 +335,21 @@ for the functional at interest, this functional is the (unique) optimal point
335
335
forecast under this scoring function.
336
336
337
337
338
- ================== ================ ===============================================
339
- functional scoring function property
340
- ================== ================ ===============================================
338
+ ================== ======================== ===============================================
339
+ functional scoring or loss function property
340
+ ================== ======================== ===============================================
341
341
**Classification **
342
- mean brier score strictly consistent
343
- mean log loss strictly consistent
344
- median absolute error strictly consistent
345
- mode zero-one loss consistent (for binary classification)
342
+ mean Brier score strictly consistent
343
+ mean log loss strictly consistent
344
+ median absolute error strictly consistent
345
+ mode zero-one loss consistent (for binary classification)
346
346
**Regression **
347
- mean squared error strictly consistent (if finite 2nd moment)
348
- mean Poisson deviance strictly consistent (for non-negative target y)
349
- mean Gamma deviance strictly consistent (for positive target y)
350
- median absolute error strictly consistent
351
- mode zero-one loss asymptotically consistent
352
- ================== ================ ===============================================
347
+ mean squared error strictly consistent (if finite 2nd moment)
348
+ mean Poisson deviance strictly consistent (for non-negative target y)
349
+ mean Gamma deviance strictly consistent (for positive target y)
350
+ median absolute error strictly consistent
351
+ mode zero-one loss asymptotically consistent
352
+ ================== ======================== ===============================================
353
353
354
354
The zero-one loss is equivalent to the accuracy score, meaning it gives
355
355
different score values but the same ranking.
0 commit comments