TST / DEBUG: better warnings and tests when facing singular hessian problems #6
Conversation
| f"Line search of Newton solver {self.__class__.__name__} did not " | ||
| "converge after 21 line search refinement iterations.", | ||
| f"Line search of Newton solver {self.__class__.__name__} at iteration" | ||
| f" #{self.iteration} did not converge after 21 line search refinement" |
There was a problem hiding this comment.
I think it can be helpful to include the iteration number in the warning but I am not sure if we should use the one-based convention "#{self.iteration}" or the 0-based convention used elsewhere in the code with #{self.iteration - 1}.
| @@ -31,37 +31,47 @@ | |||
| from .._linear_loss import LinearModelLoss | |||
|
|
|||
|
|
|||
| def _solve_singular(H, g): | |||
| def _solve_singular(H, g, method="lstsq"): | |||
There was a problem hiding this comment.
I decided to privately expose the method param to quickly switch between the 2 approaches to make it easier to investigate the singular Hessian problem but as I found that neither seems to be helpful I do not know which to keep. If we can actually make use of the indefinite_factorization variant, I think we should unittest it individually.
|
Nocedal & Wright write in Chapter 3.4:
Let's see, if we find a satisfying solution. |
|
I moved to a new place and my copy of N & W is still somewhere buried in a cardboard box :) |
|
I finally went with option a) simple gradient steps, see scikit-learn#23314 (comment). |
Here is an attempt (failed or successful depending on the viewpoint) at improving the tests for the fallback inner solver for the singular Hessian case in the context of the
"newton-cholesky"solver introduced in scikit-learn#23314.I think in it's current state, it's pretty useless:
lstsqorindefinite_factorization, the model never converges, all line searches fail, the deviance stays very high and we issue a ton of useless warnings.lstsqseems to be a bit faster but it's not important since neither method is helpful at making the model converge.Possible things to try:
Note that the suggestion to slightly increase the regularization works as shown in the last section of the test.