[MRG+1] FIX out of bounds array access in SAGA #9376
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
did you spot this by setting boundscheck=True? |
|
I did not. I found it by narrowing in using old-school diffs over printed debug output. |
|
Is there a way to disable the nogil feature for a compilation so that exceptions will be propagated? |
|
I commented on the boundscheck at #9351 (comment) |
|
I confirm that I don't get the error in the branch after running 10 times the test. It fails on master on my mac. |
|
I've tried it well over 10 times, if that adds any weight. :) And even disabling the nogil, I'm not getting an exception raised with boundscheck=True :\ |
|
To get the exceptions to propagate:
to: and update the implementation to
with gil:
raise ValueError(the_msg) |
|
I also confirm that fixes the issue on my mac. The diff looks good, however I am not familiar enough with the algorithm to tell whether the fix is correct. @arthurmensch can you confirm? |
|
Well, I'm sure that the successful runs are those when |
|
If there's reason to believe @arthurmensch will take more than a day to
review, I'd consider merging this as a temporary fix and building the RC.
It does, after all, make saga equal liblinear.
|
|
I will be reviewing this in one hour
…On Mon, Jul 17, 2017, 1:40 AM Joel Nothman ***@***.***> wrote:
If there's reason to believe @arthurmensch will take more than a day to
review, I'd consider merging this as a temporary fix and building the RC.
It does, after all, make saga equal liblinear.
On 17 Jul 2017 5:37 am, "Olivier Grisel" ***@***.***> wrote:
I also confirm that fixes the issue on my mac. The diff looks good, however
I am not familiar enough with the algorithm to tell whether the fix is
correct. @arthurmensch <https://github.com/arthurmensch> can you confirm?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<
#9376 (comment)
>,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AAEz6yS7EsHt_SWILOwNqhE5ask1PCb3ks5sOmaLgaJpZM4OZTV_
>
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#9376 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AD48YyXK3fsSKaGFbXLF9xvqow4hlPqoks5sOp91gaJpZM4OZTV_>
.
|
|
:D
…On 17 Jul 2017 3:14 pm, "Arthur Mensch" ***@***.***> wrote:
I will be reviewing this in one hour
On Mon, Jul 17, 2017, 1:40 AM Joel Nothman ***@***.***>
wrote:
> If there's reason to believe @arthurmensch will take more than a day to
> review, I'd consider merging this as a temporary fix and building the RC.
> It does, after all, make saga equal liblinear.
>
> On 17 Jul 2017 5:37 am, "Olivier Grisel" ***@***.***>
wrote:
>
> I also confirm that fixes the issue on my mac. The diff looks good,
however
> I am not familiar enough with the algorithm to tell whether the fix is
> correct. @arthurmensch <https://github.com/arthurmensch> can you
confirm?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <
> #9376#
issuecomment-315631765
> >,
> or mute the thread
> <
> https://github.com/notifications/unsubscribe-auth/AAEz6yS7EsHt_
SWILOwNqhE5ask1PCb3ks5sOmaLgaJpZM4OZTV_
> >
> .
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#9376#
issuecomment-315646390>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/
AD48YyXK3fsSKaGFbXLF9xvqow4hlPqoks5sOp91gaJpZM4OZTV_>
> .
>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#9376 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAEz62WY3X8aREydHX9GdtZ52AsqXLCBks5sOu2lgaJpZM4OZTV_>
.
|
sklearn/linear_model/sag_fast.pyx
Outdated
| - cumulative_sums[lagged_ind - 1]) | ||
| prox_step = (cumulative_sums_prox[lagged_ind] | ||
| - cumulative_sums_prox[lagged_ind - 1]) | ||
| if lagged_ind > 1: |
There was a problem hiding this comment.
I think it should read >= 1
There was a problem hiding this comment.
Yes! Whoops. I think we were all a bit too enthusiastic to fix this.
sklearn/linear_model/sag_fast.pyx
Outdated
| - cumulative_sums[lagged_ind - 1]) | ||
| prox_step = (cumulative_sums_prox[lagged_ind] | ||
| - cumulative_sums_prox[lagged_ind - 1]) | ||
| if lagged_ind > 1: |
There was a problem hiding this comment.
Otherwise there was indeed a problem there, I wonder how it went silently although wraparound = False, any idea ?
There was a problem hiding this comment.
Right: those are pointers.
There was a problem hiding this comment.
The wraparound, etc, flags do not apply at all to pointers.
But the basic issue is that O/Ses allocate memory on the heap differently. Most of it tends to be zero (if only because we use 64 bit int arrays to store small numbers). So this just worked when the array happened to be allocated next to a zero ... which always happened on Travis and AppVeyor.
There was a problem hiding this comment.
Yes. We choose pointers because apparently they're a little more efficient than memoryviews, and much better where we want to handle read-only memmaps. But they're much less safe!
|
Fixed. Good for you? |
|
LGTM. I hope there is no other errors like this... I wondered why we used pointer there in the first place, but it's true that it allows to handle read-only memmap. Thanks ! |
|
Thanks very much for the fix, merging. |
|
I will do the backport to 0.19.X. |
ogrisel
pushed a commit
that referenced
this pull request
Jul 17, 2017
* FIX out of bounds array access in SAGA * FIX Fix boundary for condition
|
Done. |
yarikoptic
added a commit
to yarikoptic/scikit-learn
that referenced
this pull request
Jul 27, 2017
Release 0.19b2 * tag '0.19b2': (808 commits) Preparing 0.19b2 [MRG+1] FIX out of bounds array access in SAGA (scikit-learn#9376) FIX make test_importances pass on 32 bit linux Release 0.19b1 DOC remove 'in dev' header in whats_new.rst DOC typos in whats_news.rst [ci skip] [MRG] DOC cleaning up what's new for 0.19 (scikit-learn#9252) FIX t-SNE memory usage and many other optimizer issues (scikit-learn#9032) FIX broken link in gallery and bad title rendering [MRG] DOC Replace \acute by prime (scikit-learn#9332) Fix typos (scikit-learn#9320) [MRG + 1 (rv) + 1 (alex) + 1] Add a check to test the docstring params and their order (scikit-learn#9206) DOC Residual sum vs. regression sum (scikit-learn#9314) [MRG] [HOTFIX] Fix capitalization in test and hence fix failing travis at master (scikit-learn#9317) More informative error message for classification metrics given regression output (scikit-learn#9275) [MRG] COSMIT Remove unused parameters in private functions (scikit-learn#9310) [MRG+1] Ridgecv normalize (scikit-learn#9302) [MRG + 2] ENH Allow `cross_val_score`, `GridSearchCV` et al. to evaluate on multiple metrics (scikit-learn#7388) Add data_home parameter to fetch_kddcup99 (scikit-learn#9289) FIX makedirs(..., exists_ok) not available in Python 2 (scikit-learn#9284) ...
yarikoptic
added a commit
to yarikoptic/scikit-learn
that referenced
this pull request
Jul 27, 2017
* releases: (808 commits) Preparing 0.19b2 [MRG+1] FIX out of bounds array access in SAGA (scikit-learn#9376) FIX make test_importances pass on 32 bit linux Release 0.19b1 DOC remove 'in dev' header in whats_new.rst DOC typos in whats_news.rst [ci skip] [MRG] DOC cleaning up what's new for 0.19 (scikit-learn#9252) FIX t-SNE memory usage and many other optimizer issues (scikit-learn#9032) FIX broken link in gallery and bad title rendering [MRG] DOC Replace \acute by prime (scikit-learn#9332) Fix typos (scikit-learn#9320) [MRG + 1 (rv) + 1 (alex) + 1] Add a check to test the docstring params and their order (scikit-learn#9206) DOC Residual sum vs. regression sum (scikit-learn#9314) [MRG] [HOTFIX] Fix capitalization in test and hence fix failing travis at master (scikit-learn#9317) More informative error message for classification metrics given regression output (scikit-learn#9275) [MRG] COSMIT Remove unused parameters in private functions (scikit-learn#9310) [MRG+1] Ridgecv normalize (scikit-learn#9302) [MRG + 2] ENH Allow `cross_val_score`, `GridSearchCV` et al. to evaluate on multiple metrics (scikit-learn#7388) Add data_home parameter to fetch_kddcup99 (scikit-learn#9289) FIX makedirs(..., exists_ok) not available in Python 2 (scikit-learn#9284) ...
yarikoptic
added a commit
to yarikoptic/scikit-learn
that referenced
this pull request
Jul 27, 2017
* dfsg: (808 commits) Preparing 0.19b2 [MRG+1] FIX out of bounds array access in SAGA (scikit-learn#9376) FIX make test_importances pass on 32 bit linux Release 0.19b1 DOC remove 'in dev' header in whats_new.rst DOC typos in whats_news.rst [ci skip] [MRG] DOC cleaning up what's new for 0.19 (scikit-learn#9252) FIX t-SNE memory usage and many other optimizer issues (scikit-learn#9032) FIX broken link in gallery and bad title rendering [MRG] DOC Replace \acute by prime (scikit-learn#9332) Fix typos (scikit-learn#9320) [MRG + 1 (rv) + 1 (alex) + 1] Add a check to test the docstring params and their order (scikit-learn#9206) DOC Residual sum vs. regression sum (scikit-learn#9314) [MRG] [HOTFIX] Fix capitalization in test and hence fix failing travis at master (scikit-learn#9317) More informative error message for classification metrics given regression output (scikit-learn#9275) [MRG] COSMIT Remove unused parameters in private functions (scikit-learn#9310) [MRG+1] Ridgecv normalize (scikit-learn#9302) [MRG + 2] ENH Allow `cross_val_score`, `GridSearchCV` et al. to evaluate on multiple metrics (scikit-learn#7388) Add data_home parameter to fetch_kddcup99 (scikit-learn#9289) FIX makedirs(..., exists_ok) not available in Python 2 (scikit-learn#9284) ...
dmohns
pushed a commit
to dmohns/scikit-learn
that referenced
this pull request
Aug 7, 2017
* FIX out of bounds array access in SAGA * FIX Fix boundary for condition
dmohns
pushed a commit
to dmohns/scikit-learn
that referenced
this pull request
Aug 7, 2017
* FIX out of bounds array access in SAGA * FIX Fix boundary for condition
NelleV
pushed a commit
to NelleV/scikit-learn
that referenced
this pull request
Aug 11, 2017
* FIX out of bounds array access in SAGA * FIX Fix boundary for condition
paulha
pushed a commit
to paulha/scikit-learn
that referenced
this pull request
Aug 19, 2017
* FIX out of bounds array access in SAGA * FIX Fix boundary for condition
AishwaryaRK
pushed a commit
to AishwaryaRK/scikit-learn
that referenced
this pull request
Aug 29, 2017
* FIX out of bounds array access in SAGA * FIX Fix boundary for condition
maskani-moh
pushed a commit
to maskani-moh/scikit-learn
that referenced
this pull request
Nov 15, 2017
* FIX out of bounds array access in SAGA * FIX Fix boundary for condition
jwjohnson314
pushed a commit
to jwjohnson314/scikit-learn
that referenced
this pull request
Dec 18, 2017
* FIX out of bounds array access in SAGA * FIX Fix boundary for condition
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #9351 (random inconsistencies between SAGA and liblinear LogisticRegression on OS X).
To hedge my bets that this is the right fix, I'll ping @TomDLT, @arthurmensch, @dsullivan7.