ENH replace loss module Gradient boosting #26278
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Make it similar to HGBT
* Make it more similar to HGBT
Note that in the case raw_prediction == y, we now have a different sign in AbsoluteError.gradient
|
A very simple benchmark for binary classification gives: import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
n_samples, n = 10, 10_000
y = np.tile(np.arange(n_samples) % 2, n)
x1 = np.minimum(y, n_samples / 2)
x2 = np.minimum(-y, -n_samples / 2)
X = np.c_[x1, x2]
%timeit GradientBoostingClassifier(n_estimators=100).fit(X, y)Same for multiclass classification (10 classes) gives: import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
n_samples, n = 10, 10_000
y = np.tile(np.arange(n_samples), n)
x1 = np.minimum(y, n_samples / 2)
x2 = np.minimum(-y, -n_samples / 2)
X = np.c_[x1, x2]
%timeit GradientBoostingClassifier(n_estimators=100).fit(X, y) |
7 tasks
thomasjpfan
reviewed
Apr 28, 2023
Therefore, I provided the different |
The exact origin for the necessity of this change is unclear. The train_score_ of the GBT inside the pipeline is exactly the same for this branch and 1.2.2.
- This test is not necessary anymore. Already tested in the _loss module.
thomasjpfan
approved these changes
Aug 27, 2023
|
Dear future 2nd reviewer |
OmarManzoor
reviewed
Sep 6, 2023
OmarManzoor
reviewed
Sep 7, 2023
There was a problem hiding this comment.
Thank you for the updates @lorentzenchr! Looks good now. Just a few minor comments.
OmarManzoor
approved these changes
Sep 7, 2023
|
It looks like one of the example broke after merging this PR, namely From build log: |
|
@lesteve has been faster to me to report. |
|
OK so it is due to the In [5]: with np.errstate(divide="raise"):
...: np.float64(0.0) / np.float64(0.0)
...:
<ipython-input-5-941bf0f2d1fa>:2: RuntimeWarning: invalid value encountered in scalar divide
np.float64(0.0) / np.float64(0.0)
In [6]: with np.errstate(divide="raise"):
...: np.array([1.0, 2.0]) / np.float64(0.0)
...:
---------------------------------------------------------------------------
FloatingPointError Traceback (most recent call last)
Cell In[6], line 2
1 with np.errstate(divide="raise"):
----> 2 np.array([1.0, 2.0]) / np.float64(0.0)
FloatingPointError: divide by zero encountered in divideLooking at the code, it seems that we expect to have the second case. If we are sure to always have some scalar, then we could convert them to a Python scalar that would always raise an error. |
|
Actually, the only case that go side ways is when both the numerator and denominator goes to 0.0. I assume that it can happen when at the previous iteration we return 0.0 because of the |
REDVM
pushed a commit
to REDVM/scikit-learn
that referenced
this pull request
Nov 16, 2023
iamDecode
added a commit
to iamDecode/sklearn-pmml-model
that referenced
this pull request
Apr 14, 2024
iamDecode
added a commit
to iamDecode/sklearn-pmml-model
that referenced
this pull request
Apr 14, 2024
iamDecode
added a commit
to iamDecode/sklearn-pmml-model
that referenced
this pull request
Apr 14, 2024
valeriy42
added a commit
to elastic/eland
that referenced
this pull request
Nov 11, 2024
Introduce a warning indicating that exporting data frame analytics models as ESGradientBoostingModel subclasses is deprecated and will be removed in version 9.0.0. The implementation of ESGradientBoostingModel relies on importing undocumented private classes that were changed in 1.4 to scikit-learn/scikit-learn#26278. This dependency makes the code difficult to maintain, while the functionality is not widely used by users. Therefore, we will deprecate this functionality in 8.16 and remove it completely in 9.0.0. --------- Co-authored-by: Quentin Pradet <quentin.pradet@elastic.co>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reference Issues/PRs
Closes #25964.
What does this implement/fix? Exp 8000 lain your changes.
This replaces the losses from
_gb_losses.pywith the once of the common private loss submodulesklearn._lossin a very backward compatible way. Some factors of the 2 loss implementations differ by a factor of 2. Those factors are here accounted for.Any other comments?
oob_improvement_andtrain_score_wrt the above mentioned constant factor of the loss? Asoob_scores_andoob_score_is about to be introduced with 1.3, we could still change them.