8000 FIX _safe_divide should handle zero-division with numpy scalar by glemaitre · Pull Request #27312 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

FIX _safe_divide should handle zero-division with numpy scalar #27312

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
Sep 10, 2023
Merged

FIX _safe_divide should handle zero-division with numpy scalar #27312

merged 13 commits into from
Sep 10, 2023

Conversation

glemaitre
Copy link
Member
@glemaitre glemaitre commented Sep 7, 2023 •
edited by lorentzenchr
Loading

Fixing the main branch.

This PR should handle the case of zero-division with two numpy scalar.

ping @lorentzenchr @thomasjpfan @OmarManzoor @lesteve

Edit: The CI failure popped up after merging #26278 in the example examples/ensemble/plot_gradient_boosting_regularization.py

@glemaitre glemaitre marked this pull request as draft September 7, 2023 17:02
@github-actions
Copy link
github-actions bot commented Sep 7, 2023 •
edited
Loading

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

Generated for commit: ad04d77. Link to the linter CI: here

@glemaitre
Copy link
Member Author

Given the failure in the example, it seems that the provided fix is not enough. We can hand up returning a very large value because the denominator is extremely small

numerator=-0.2857142857142857
denominator=3.18528276243611e-217
numerator=-8.969824879715574e+215

The next iteration, the numerator and denominator will nan.

@glemaitre
Copy link
Member Author

I am not sure what it the right fix here to handle the numerical instability: either clip in the safe division because we should not assign such a large value in raw_predictions or handle the nan values of the next iteration?

@glemaitre
Copy link
Member Author

Apparently, once we get some nan, it seems that this is too late:

Figure_1

vs. previously

image

@glemaitre
Copy link
Member Author

I also see that we had potentially another strategy in the past: https://github.com/scikit-learn/scikit-learn/pull/26278/files#diff-dac0eef4868535102a7b9aeb319e1501dfbd10f1256f91d532927e12d30bfb15L732

@glemaitre
Copy link
Member Author

So this is not enough. The negative gradient vector has some nan.

In the previous loss it seems that np.nan_to_num was used. Here I tried but the algorithm is still diverging.

@lorentzenchr
Copy link
Member

I‘ll also have a look, but I need some time. Fortunately, we‘ve plenty of time to fix it.

@lorentzenchr lorentzenchr added this to the 1.4 milestone Sep 7, 2023
@glemaitre glemaitre mentioned this pull request Sep 7, 2023
Merged
@lorentzenchr
Copy link
Member
lorentzenchr commented Sep 8, 2023 •
edited
Loading

Which version of numpy do you use?
With numpy 1.24.1 on macos, I can't reproduce this failure. My blue curve is like the one at the bottom plot of #27312 (comment).
I was mistaken, now I see the error.

@OmarManzoor
Copy link
Contributor
OmarManzoor commented Sep 8, 2023 •
edited
Loading

@lorentzenchr I have numpy 1.25.1 on mac M1 and I can reproduce the error when I use the main branch.

@lesteve
Copy link
Member
lesteve commented Sep 8, 2023

As an aside (I don't know how much we care to be perfectly honest): code based on np.seterr + catching an exception is not going to work in Pyodide, i.e. an exception will never be raised. Floating point exceptions are not supported in WebAssembly and this is unlikely to change in the near future, see numpy/numpy#21895 (comment) for more details.

@OmarManzoor
Copy link
Contributor
OmarManzoor commented Sep 8, 2023 •
edited
Loading

@lorentzenchr @glemaitre @lesteve

scikit-learn/sklearn/ensemble/_gb.py

Lines 237 to 244 in c634b8a

for leaf in np.nonzero(tree.children_left == TREE_LEAF)[0]:
indices = np.nonzero(terminal_regions == leaf)[0] # of terminal regions
y_ = y.take(indices, axis=0)
sw = None if sample_weight is None else sample_weight[indices]
update = compute_update(y_, indices, neg_gradient, raw_prediction, k)
# TODO: Multiply here by learning rate instead of everywhere else.
tree.value[leaf, 0, 0] = update

I think we should have

indices = np.nonzero(masked_terminal_regions == leaf)[0]  # of terminal regions

if we compare with the original code.

@OmarManzoor
Copy link
Contributor

Plot if I use on main

indices = np.nonzero(masked_terminal_regions == leaf)[0]

Figure_1

@glemaitre
Copy link
Member Author

Yep this look much what we had. It makes sense also if we where computing gradient on data that we should not have :)

@glemaitre
Copy link
Member Author

@lesteve @lorentzenchr Instead of the exception catching, we could use the previous trick that return 0.0 if smaller than smaller than a really small values. This would be OK for Pyodide?

@lesteve
Copy link
Member
lesteve commented Sep 8, 2023
8000

we could use the previous trick that return 0.0 if smaller than smaller than a really small values. This would be OK for Pyodide?

I think this would work with Pyodide.

I am not too sure to which extent we want to support Pyodide quirkiness, as I mentioned above. It feels like not getting some warnings in Pyodide would acceptable but having a gradient boosting algorithm that behave weirdly because the loss becomes NaN or inf is maybe not that great.

@glemaitre
Copy link
Member Author

So I made the fix of @OmarManzoor and change the error catching as it was previously.

@glemaitre glemaitre marked this pull request as ready for review September 8, 2023 12:57
@glemaitre
Copy link
Member Author

I also added a comment to remember why we are not using np.errstate and who knows, we could potentially change it in the future if Pyodide handles it.

@lesteve
Copy link
Member
lesteve commented Sep 8, 2023 •
edited
Loading

I also added a comment to remember why we are not using np.errstate and who knows, we could potentially change it in the future if Pyodide handles it.

Don't hold your breath too much though, the link above says there is no plan to support it right now, so that's at least 3 years away.

lorentzenchr
lorentzenchr reviewed Sep 8, 2023
View reviewed changes
lorentzenchr
lorentzenchr reviewed Sep 8, 2023
View reviewed changes
glemaitre
glemaitre commented Sep 8, 2023
View reviewed changes
lorentzenchr
lorentzenchr approved these changes Sep 8, 2023
View reviewed changes
Copy link
Member
@lorentzenchr lorentzenchr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM just some nits.

@glemaitre @OmarManzoor @lesteve Thanks for fixing my bugs.

glemaitre and others added 3 commits September 8, 2023 23:27
@glemaitre @lorentzenchr
Apply suggestions from code review
9b437be 
Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>
@glemaitre @lorentzenchr
Apply suggestions from code review
ffb605a 
Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>
@glemaitre
iter
ad04d77
OmarManzoor
OmarManzoor approved these changes Sep 10, 2023
View reviewed changes
Copy link
Contributor
@OmarManzoor OmarManzoor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @glemaitre , @lesteve and @lorentzenchr

@OmarManzoor OmarManzoor merged commit bbc73cf into scikit-learn:main Sep 10, 2023
REDVM pushed a commit to REDVM/scikit-learn that referenced this pull request Nov 16, 2023
@glemaitre @lorentzenchr @REDVM
FIX _safe_divide should handle zero-division with numpy scalar (sciki…
3178b39 
…t-learn#27312)

Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>
@lorentzenchr lorentzenchr mentioned this pull request Jan 10, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lorentzenchr lorentzenchr lorentzenchr approved these changes

@OmarManzoor OmarManzoor OmarManzoor approved these changes

Assignees
No one assigned
Labels
module:ensemble No Changelog Needed
Projects
None yet
Milestone
1.4
Development

Successfully merging this pull request may close these issues.
4 participants
@glemaitre @lorentzenchr @OmarManzoor @lesteve
0