RFC Metric pos_label handling for multiclass (and multilabel) data #33143
Description
Warning
This is a request for comment (RFC), please do not open a pull request for this issue. If you wish to contribute to scikit-learn please have a look at our contributing doc and in particular the section New contributors.
Noticed in #32755 (comment)
Here are the metrics that have a pos_label:
scikit-learn/sklearn/metrics/tests/test_common.py
Lines 364 to 380 in 7fbe0c2
Of these, some only support binary input (e.g., roc_curve, confusion_matrix_at_thresholds).
Of the metrics that also support multi-class input, most silently ignore the pos_label parameter if data is multiclass (or multilabel) but pos_label is also set. average_precision_score is the only one that raises an error:
scikit-learn/sklearn/metrics/_ranking.py
Lines 264 to 274 in 7fbe0c2
I am not sure whether it is better to error or silently ignore (or third option, warn) - but some consistency is probably ideal.
How should pos_label be handled?
cc @StefanieSenger @AnneBeyer @ogrisel ?
Edit:
Side note, brier_score_loss and d2_brier_score do not explicitly state that pos_label is ignored, but probably should. Maybe you'd be interested in tackling this @AnneBeyer ?