8000 BENCH newton-lsmr solver for GLMs - inner stopping criterion by lorentzenchr · Pull Request #26707 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

BENCH newton-lsmr solver for GLMs - inner stopping criterion #26707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 40 commits into
base: main
Choose a base branch
from
Draft

BENCH newton-lsmr solver for GLMs - inner stopping criterion #26707

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
bfb29a0
ENH add NewtonLSMRSolver
lorentzenchr Nov 4, 2022
9c3fd7f
TST add test_solver_on_ill_conditioned_X
lorentzenchr Nov 6, 2022
3874810
ENH add multinomial to LSMR
lorentzenchr Nov 19, 2022
86da909
ENH add newton-lsmr to LogisticRegression
lorentzenchr Jan 23, 2023
1d13089
CLN fix dtype and tests
lorentzenchr Jan 24, 2023
4e1a696
ENH speed up LDL by better handling q==0
lorentzenchr Jan 24, 2023
7ef5877
ENH speed up LDL by using einsum and q_inv
lorentzenchr Jan 25, 2023
65d7a87
Merge branch 'main' into glm_newton_lsmr_only
lorentzenchr Feb 13, 2023
74dab94
TST fix atol in test_multinomial_identifiability_properties
lorentzenchr Feb 13, 2023
0dc87cf
DOC add whatsnew
lorentzenchr Feb 13, 2023
8e78465
TST reduce tolerances
lorentzenchr Feb 23, 2023
3da89d5
TST skip LinearOperator transpose for scipy<1.4
lorentzenchr Feb 23, 2023
298e63e
TST fix skipif
lorentzenchr Feb 24, 2023
27ddf56
TST loosen rtol a bit
lorentzenchr Feb 24, 2023
d26004d
TST make tests pass for all random seeds
lorentzenchr Feb 24, 2023
59c1322
DOC add comment about initial A_norm
lorentzenchr Feb 24, 2023
a84b938
Merge branch 'main' into glm_newton_lsmr_only
lorentzenchr Jun 2, 2023
c50c859
FIX warning condition and cleaner for loop
lorentzenchr Jun 3, 2023
cf9facc
ENH improve Multinomial_LDL_Decomposition by precomputation
lorentzenchr Jun 3, 2023
fc1cb24
DOC improve docstrings and comments
lorentzenchr Jun 4, 2023
e7368e7
ENH inner stopping criterion for LSMR with forcing sequence
lorentzenchr Jun 4, 2023
7295409
DOC improve docs and comments of Newton solvers
lorentzenchr Jun 6, 2023
3ea7d98
ENH set initial A_norm based on n_samples, n_features and l2_reg_stre…
lorentzenchr Jun 11, 2023
99a9508
CLN address review comments
lorentzenchr Jun 11, 2023
d108ffc
TST set lsmr to xfail in test_solver_on_ill_conditioned_X like other …
lorentzenchr Jun 11, 2023
ab075fe
Doc docstring solver arg in LogisticRegression
lorentzenchr Jun 12, 2023
e4c0ee3
ENH set btol back to self.tol
lorentzenchr Jun 14, 2023
39030c4
TST higher tol in test_glm_regression_hstacked_X
lorentzenchr Jun 14, 2023
83ce34f
ENH increase conlim to make more tests pass
lorentzenchr Jun 15, 2023
8846903
Merge branch 'main' into glm_newton_lsmr_only
lorentzenchr Jun 16, 2023
b887390
ENH verbose print total number of LSMR iterations
lorentzenchr Jun 17, 2023
b9bd70e
ENH vendor LSMR from scipy + blackify
lorentzenchr Jun 17, 2023
7e2a404
ENH add artol to LSMR
lorentzenchr Jun 17, 2023
f4a70f5
DOC enhance Taylor series and comments
lorentzenchr Jun 22, 2023
f5ec2d2
ENH add qtol to LSMR
lorentzenchr Jun 24, 2023
d7ff7aa
ENH add convergence_report and inner_stopping to GLMs
lorentzenchr Jun 26, 2023
9f12183
ENH avoid to large reductions in atol
lorentzenchr Jun 26, 2023
edb7c34
FIX qtol criterion
lorentzenchr Jun 26, 2023
1c9257d
ENH regularize atol a bit less
lorentzenchr Jun 26, 2023
56e4953
BENCH add notebook and result files
lorentzenchr Jun 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8,213 changes: 8,213 additions & 0 deletions Bench_Newton_LSMR.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

27 changes: 27 additions & 0 deletions bench_multinomial_logistic_regression_20newsgroup.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
,solver,tol,train_loss,train_time,train_score,test_score,n_iter,converged
0,lbfgs,0.1,0.00024072038402909673,31.939832549999892,1.0,0.8404966571155683,89,True
1,newton-cg,0.1,0.00039439618415178706,36.08560347899993,1.0,0.8750928579008809,11,True
2,newton-lsmr,0.1,0.0003298564847793821,9.701729847000024,0.9997877533694153,0.8997134670487106,1,True
3,lbfgs,0.01,6.820164844196484e-06,35.74219161899987,1.0,0.8365700944497506,102,True
4,newton-cg,0.01,5.1106853523666066e-05,43.272124965999865,1.0,0.8782765573596519,14,True
5,newton-lsmr,0.01,1.853908790183233e-05,18.595135632999927,1.0,0.8930276981852913,1,True
6,lbfgs,0.001,1.445298576353388e-06,38.01809711300007,1.0,0.8333863949909796,108,True
7,newton-cg,0.001,6.4550818986236005e-06,49.913992363000034,1.0,0.8779581874137748,17,True
8,newton-lsmr,0.001,4.748370640334657e-06,107.08343836600011,1.0,0.8936644380770455,2,True
9,lbfgs,0.0001,1.283201425047801e-07,39.29532392099986,1.0,0.8315822986310092,113,True
10,newton-cg,0.0001,4.986584375450593e-07,57.387584808000156,1.0,0.8781704340443596,20,True
11,newton-lsmr,0.0001,4.57493348172439e-06,108.11839779599995,1.0,0.8934521914464608,2,True
12,lbfgs,1e-05,1.509632286295713e-08,40.61666484399984,1.0,0.8290353390639924,117,True
13,newton-cg,1e-05,7.038655793315655e-08,63.683267528000215,1.0,0.8779581874137748,22,True
14,newton-lsmr,1e-05,4.4478587835592996e-06,107.01624073999983,1.0,0.8934521914464608,2,True
15,lbfgs,1e-06,3.951707067206023e-09,42.12825341200005,1.0,0.8281863525416534,122,True
16,newton-cg,1e-06,2.52865032001237e-08,151.0573313330001,1.0,0.8784888039902367,300,False
17,newton-lsmr,1e-06,1.6281669984467293e-06,121.50261462300023,1.0,0.8929215748699989,3,True
18,lbfgs,1e-07,2.1974730924494106e-09,49.3890974699998,1.0,0.8337047649368566,135,True
19,newton-lsmr,1e-07,8.629130620416836e-08,120.83360728600019,1.0,0.8965297675899395,6,True
20,lbfgs,1e-08,7.216431400566044e-10,85.10779890499998,1.0,0.8548233046800382,251,True
21,newton-lsmr,1e-08,1.2504659735610402e-08,141.8642732440003,1.0,0.9000318369945877,8,True
22,lbfgs,1e-09,4.1981133498751296e-10,136.35830511800032,1.0,0.8711662952350632,406,True
23,newton-lsmr,1e-09,2.2785341213774982e-09,175.95382772199991,1.0,0.901835933354558,10,True
24,lbfgs,1e-10,4.1981133498751296e-10,148.50081864899994,1.0,0.8711662952350632,406,True
25,newton-lsmr,1e-10,6.019492979924515e-10,514.3074960230001,1.0,0.8863419293218721,12,True
31 changes: 31 additions & 0 deletions bench_multinomial_logistic_regression_20newsgroup_penalized.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
,solver,tol,train_loss,train_time,train_score,test_score,n_iter,converged
0,lbfgs,0.1,1.618697770712165,38.590389550000054,0.8797622837737451,0.775549188156638,96,True
1,newton-cg,0.1,1.6186975767450917,9.165175215999625,0.8796561604584527,0.7757614347872227,9,True
2,newton-lsmr,0.1,1.7290700379429031,2.6405239509999774,0.8736071314867877,0.7901942056669851,2,True
3,lbfgs,0.01,1.618697272311927,63.72527494999986,0.8797622837737451,0.7757614347872227,117,True
4,newton-cg,0.01,1.6186972594116371,15.079571959000532,0.8797622837737451,0.7757614347872227,11,True
5,newton-lsmr,0.01,1.6187919996512297,8.853888564000044,0.8797622837737451,0.776398174678977,4,True
6,lbfgs,0.001,1.618697272311927,51.045178975000454,0.8797622837737451,0.7757614347872227,117,True
7,newton-cg,0.001,1.6186972504549746,17.63947204499982,0.8797622837737451,0.7757614347872227,14,True
8,newton-lsmr,0.001,1.618697674042506,10.79128873100035,0.8796561604584527,0.7756553114719303,5,True
9,lbfgs,0.0001,1.618697272311927,46.366761603999294,0.8797622837737451,0.7757614347872227,117,True
10,newton-cg,0.0001,1.6186972504137835,22.722229132999928,0.8797622837737451,0.7757614347872227,18,True
11,newton-lsmr,0.0001,1.618697674042506,10.542509142999734,0.8796561604584527,0.7756553114719303,5,True
12,lbfgs,1e-05,1.618697272311927,60.463799537999876,0.8797622837737451,0.7757614347872227,117,True
13,newton-cg,1e-05,1.6186972504137835,22.564616426999237,0.8797622837737451,0.7757614347872227,18,True
14,newton-lsmr,1e-05,1.6186972504742156,15.169568252999852,0.8797622837737451,0.7757614347872227,6,True
15,lbfgs,1e-06,1.618697272311927,49.37727712099968,0.8797622837737451,0.7757614347872227,117,True
16,newton-cg,1e-06,1.6186972504137822,24.66753051700016,0.8797622837737451,0.7757614347872227,19,True
17,newton-lsmr,1e-06,1.6186972504742156,14.978434931000265,0.8797622837737451,0.7757614347872227,6,True
18,lbfgs,1e-07,1.618697272311927,49.034414286000356,0.8797622837737451,0.7757614347872227,117,True
19,newton-cg,1e-07,1.6186972504137822,25.25996708799994,0.8797622837737451,0.7757614347872227,19,True
20,newton-lsmr,1e-07,1.6186972504137838,20.164864198000032,0.8797622837737451,0.7757614347872227,7,True
21,lbfgs,1e-08,1.618697272311927,48.73946950899972,0.8797622837737451,0.7757614347872227,117,True
22,newton-cg,1e-08,1.6186972504137822,34.41269656199984,0.8797622837737451,0.7757614347872227,19,True
23,newton-lsmr,1e-08,1.6186972504137838,20.05967276299998,0.8797622837737451,0.7757614347872227,7,True
24,lbfgs,1e-09,1.618697272311927,48.923003240999606,0.8797622837737451,0.7757614347872227,117,True
25,newton-cg,1e-09,1.6186972504137822,36.1835074239998,0.8797622837737451,0.7757614347872227,19,True
26,newton-lsmr,1e-09,1.6186972504137838,17.595025589000215,0.8797622837737451,0.7757614347872227,7,True
27,lbfgs,1e-10,1.618697272311927,46.683688672000244,0.8797622837737451,0.7757614347872227,117,True
28,newton-cg,1e-10,1.6186972504137822,32.766633301000184,0.8797622837737451,0.7757614347872227,19,True
29,newton-lsmr,1e-10,1.618697250413784,21.522521755000525,0.8797622837737451,0.7757614347872227,8,True
Loading
0