Open
Description
Moving the discussion with @amueller from pydata/patsy#77 (comment).
Proposing to:
- add an optional 'labels' argument to
log_loss - add an argument to
make_scorerto enable passing labels argument to the custom loss function
Related to this - perhaps also allow make_scorer's need_threshold = True to be used for non-binary classification problems. Not sure why it's limited to binary. Totally meaningful for multi-class situations, where the metric might use the decision function to evaluate based on the rank order or something (for example, SVC doesn't have a cheap predict_proba, but has a cheap decision_function, so it would allow it to be used with rank-based metris). From make_scorer perspective, it just seems like a completely arbitrary restriction because it shouldn't care about what the custom loss function does with the decision_function output.