8000 [Feature Request] Custom loss functions for GradientBoostingRegressor/Classifier and HistGradientBoostingRegressor/Classifier · Issue #17659 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

[Feature Request] Custom loss functions for GradientBoostingRegressor/Classifier and HistGradientBoostingRegressor/Classifier #17659

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zachmayer opened this issue Jun 22, 2020 · 5 comments
Closed

[Feature Request] Custom loss functions for GradientBoostingRegressor/Classifier and HistGradientBoostingRegressor/Classifier #17659

zachmayer opened this issue Jun 22, 2020 · 5 comments
Labels
New Feature

Comments

@zachmayer
Copy link
Contributor

Describe the workflow you want to enable

In Xgboost and LightGBM, you can specify a custom objective function (as well as gradients for that objective): https://xgboost.readthedocs.io/en/latest/tutorials/custom_metric_obj.html

It would be pretty cool if Sklearn supported that too

Describe your proposed solution

Use Xgboost / LightGBM instead of sklearn, but who wants to do that 😁

Describe alternatives you've considered, if relevant

Xgboost / LightGBM

Additional context

If you have a problem-specific objective function, it can be really useful to boost directly on that function.
@zachmayer zachmayer changed the title Custom loss functions for GradientBoostingRegressor/Classifier and HistGradientBoostingRegressor/Classifier [Feature Request] Custom loss functions for GradientBoostingRegressor/Classifier and HistGradientBoostingRegressor/Classifier Jun 22, 2020
@thomasjpfan
Copy link
Member
thomasjpfan commented Jun 22, 2020 •
edited
Loading

We opened using custom loss functions in for HistGradientBoosting in #16908

Support for this feature is summarized here: #15841 (comment)

@zachmayer
Copy link
Contributor Author

Oh awesome! I'm sorry I missed that, that's very cool!

@Sandy4321
Copy link

is it possible to add custom loss function to classification tree as well?
https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html?highlight=tree#sklearn.tree.DecisionTreeClassifier

@Sandy4321
Copy link

people really need it
https://stackoverflow.com/questions/54267745/implementing-custom-loss-function-in-scikit-learn

It's tricky, but you can do it...

  1. Open up your classifier. Let's use an RFC for example: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

  2. click [source]

  3. See how it's inheriting from ForestClassifier? Right there in the class definition. Click that word to jump to it's parent definition.

  4. See how this new object is inheriting from ClassifierMixin? Click that.

  5. See how the bottom of that ClassifierMixin class says this?

from .metrics import accuracy_score
return accuracy_score(y, self.predict(X), sample_weight=sample_weight)
That's your model being trained on accuracy. You need to inject at this point if you want to train your model to be a "recall model" or a "precision model" or whatever model. This accuracy metric is baked into SKlearn. Some day, a better man than I will make this a parameter which models accept, however in the mean time, you gotta go into your sklearn installation, and tweak this accuracy_score to be whatever you want.

@NicolasHug NicolasHug mentioned this issue Jun 28, 2021
Closed
@NicolasHug
Copy link
Member

@Sandy4321 please see #15841 (comment). The instructions copied pasted from SO above are incorrect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
New Feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@zachmayer @NicolasHug @thomasjpfan @Sandy4321
0