Convergence issues in l1 logistic regression path example #15903

amueller · 2019-12-16T15:47:41Z

This example:
https://scikit-learn.org/dev/auto_examples/linear_model/plot_logistic_path.html

Shows wrong results according to
https://papers.nips.cc/paper/8491-geno-generic-optimization-for-classical-machine-learning.pdf

I'm surprised the data is not scaled, I wouldn't expect saga to work without scaling. That paper has some other interesting graphs also.

cc @agramfort who might now actually see this ping [you did unfollow the repo, right?]

agramfort · 2019-12-16T21:23:26Z

I will look tomorrow.

agramfort · 2019-12-17T08:21:55Z

I have liblinear match GENO in figure 1 when setting intercept_scaling=10000. so intercept is not regularized much.

Now regarding SAGA I agree that it's weird. SAGA should converge to machine precision so graphs should match... bug? cc @TomDLT @arthurmensch

TomDLT · 2019-12-17T20:12:23Z

SAGA is bad on small dataset, since the gradient estimate is very noisy. This is even worst when C is large, since the soft-thresholding (1 / C) is too small to kill even the smallest noisy coefficients variations.

In the example, if you increase the number of iterations and decrease the tolerance (tol=1e-10, max_iter=int(1e8)), you get almost the same results. It is very slow (2230 sec), but I don't think we should use SAGA on such small dataset anyway, especially with large C.

agramfort · 2019-12-17T20:34:02Z

great. thanks for checking @TomDLT I think we're good then although I would have expected to have SAGA be able to better leverage warm starts

…

TomDLT · 2019-12-17T20:57:47Z

Note that directly calling _logistic_regression_path with the entire list of C does not change the computational time nor the results, despite using more advanced SAGA warm starting.

scikit-learn/sklearn/linear_model/_sag.py

Lines 335 to 338 in 078e3ef

    
           warm_start_mem = {'coef': coef_init, 'sum_gradient': sum_gradient_init, 
        
                             'intercept_sum_gradient': intercept_sum_gradient, 
        
                             'gradient_memory': gradient_memory_init, 
        
                             'seen': seen_init, 'num_seen': num_seen}