API proposal for losses #5044

arogozhnikov · 2015-07-28T21:30:16Z

I was working on different modifications of gradient boosting with specific loss functions, and unfortunately I was not able to reuse scikit-learn's gradient boosting for my purposes.

My problems are not a reason to change anything in sklearn, but the following concept also resolves other issues (see below).

Loss function is an estimator

Loss function has parameters.
For instance, this is useful for different kind or regularization
Loss function should be fitted: loss.fit(X, y, sample_weight=sample_weight)
This means, that loss function has state (and may keep some useful information), or do some heavy precomputations.
there are methods for negative gradient and for updating tree leaves and, etc.
Since loss is estimator, it can be cloned.

Possible benefits

Ranking algorithms implemented as loss functions
- they require some initial work with ranks, which is done during fit
- also name of special column with ranks is passed as parameter to loss function, but the ranks are obtained during fitting.
My algorithm requires computing neighbors in some variables, this is done during fitting of loss function.
Loss-specific regularizations as parameters.
Flexibility: loss functions that can be reused by other algorithms (i.e. I'm using them for pruning, but one can use them to build rankers over logistic regression with modified loss function, which seems nice)

Loss function will become the main logic of algorithm, but this is probably fine, because there are different losses for different problems, which actually makes GB so universal.

Implementation

If you're interested in some details of implementation, you're welcome to see hep_ml.losses sources, there is already an example of ranking loss function.

amueller · 2015-07-28T22:16:53Z

What do you understand as a loss function?
What I think when I hear loss function is just a formula, which is stateless and not estimated from data.

I don't understand 1, 2 and 4.

arogozhnikov · 2015-07-28T22:42:13Z

Hi, Andreas.

By loss function I mean target of optimization, and in many cases this is indeed a formula (mse, logloss).

However, the idea of stepwise optimization of GB was many times generalized, probably ranking is good demonstration of this:

in most cases loss involves some information from dataset (are samples from same query?)
in many cases we cannot even give the expression for target, but we still use GB scheme by providing some pseudoresidual and building tree on that.
in some cases we use updating tree leaves, sometimes this are very specific (see DirectRank as a limit case)

glouppe · 2015-07-29T07:38:00Z

Thanks for raising the issue @arogozhnikov ! In general, I agree we should improve the API on the loss functions used within our GBRT implementation. This is something I have been thinking about lately as well and I agree we should make it easier for people to plugin new functions if they need to. Currently, doing that is quite cumbersome... I have no opinion yet on how to best do that however -- I'll dig into what you have implemented for hep_ml.

CC: @pprett @ndawe @jmschrei who might have opinion on this as well.

arjoly · 2015-07-29T08:11:41Z

I agree that loss should be an estimator.

I have a private boosting implementation with loss being regressor. The fit and predict method correspond to the initial estimate of the gradient boosting algorithm. It allows to fix a lot api issue due to the fact that initial estimator is a mix of inconsistent and non-api compliant estimators.

However, @pprett raised previously that fixing that would break the backward compatibility with serialization. This is not worth it if there is no benefit for direct users.

jmschrei · 2015-07-29T11:08:43Z

My thoughts are similar to @arjoly . I am unsure how to handle the backwards compatibility issue, however. I may have a more informative response when I get to that section of the code (still in _tree.pyx).

arjoly · 2015-07-29T14:41:50Z

For the backward compatibility with serialisation, it's better to have it. But in practice, we make modifications if there are some benefits. Preserving serialisation between versions is a lot of trouble and huge maintenance burden.

For the backward compatibility in term of interface (functions, classes, ...), we have the rules to be backward compatible for 2 versions to allow users to adapt. Deprecation warning are raised in the meantime.

glouppe · 2015-09-08T18:15:01Z

Ping @jmschrei @arjoly : This is something to keep in mind with respect to #5212

lorentzenchr · 2021-11-27T11:58:08Z

Closing in favor of #15123 which is now implemented in #20567.

arogozhnikov · 2021-11-29T17:30:12Z

Thanks for working on deduplication of losses in sklearn @lorentzenchr.

However cases discussed in this issue are not covered by #15123

I have implemented generic loss for GB in hep_ml a while ago, it covers case of ranking and other losses that use additional information in their formulation.

lorentzenchr · 2021-12-15T17:32:38Z

You can use

scikit-learn/sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py

Line 31 in 845b1fa

class BaseHistGradientBoosting(BaseEstimator, ABC):

and pass your own loss function in init. Note, however, that this is not part of the public API and may change at any time.

< 8000 div class="avatar-parent-child TimelineItem-avatar d-none d-md-block">

arjoly mentioned this issue Sep 9, 2015

[MRG] Support arbitrary init estimators for Gradient Boosting #5221

Closed

jmschrei mentioned this issue Sep 21, 2015

[MRG+1] Add Huber Estimator to sklearn linear models #5291

Merged

7 tasks

rth mentioned this issue Oct 3, 2019

A common private module for differentiable loss functions used as objective functions in estimators #15123

Open

lorentzenchr closed this as completed Nov 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API proposal for losses #5044

API proposal for losses #5044

API proposal for losses #5044

API proposal for losses #5044

Comments

Loss function is an estimator

Possible benefits

Implementation