FEAT - Implement SmoothQuantileRegression #312
Conversation
examples/plot_smooth_quantile.py
Outdated
| return np.mean(residuals * (quantile - (residuals < 0))) | ||
|
|
||
|
|
||
| def create_data(n_samples=1000, n_features=10, noise=0.1): |
There was a problem hiding this comment.
avoid this: this is literally just wrapping make_regression
examples/plot_smooth_quantile.py
Outdated
| plt.tight_layout() | ||
| plt.show() | ||
|
|
||
|
|
There was a problem hiding this comment.
no need to wrap in if name == main for example plots
skglm/experimental/quantile_huber.py
Outdated
| res += self._loss_scalar(residual) | ||
| return res / n_samples | ||
|
|
||
| def _loss_scalar(self, residual): |
There was a problem hiding this comment.
loss_sample may be a clearer name
skglm/experimental/quantile_huber.py
Outdated
| grad_j += -X[i, j] * self._grad_scalar(residual) | ||
| return grad_j / n_samples | ||
|
|
||
| def _grad_scalar(self, residual): |
There was a problem hiding this comment.
having gradient_scalar and _grad_scalar is a massive risk of confusion in the future; _grad_per_sample ?
skglm/experimental/quantile_huber.py
Outdated
| return grad_j / n_samples | ||
|
|
||
| def _grad_scalar(self, residual): | ||
| """Calculate gradient for a single residual.""" |
skglm/experimental/quantile_huber.py
Outdated
|
|
||
| def fit(self, X, y): | ||
| """Fit using progressive smoothing: delta_init --> delta_final.""" | ||
| X, y = check_X_y(X, y) |
There was a problem hiding this comment.
no need to check: GeneralizedLinearEstimator will do it
skglm/experimental/quantile_huber.py
Outdated
|
|
||
| for i, delta in enumerate(deltas): | ||
| datafit = QuantileHuber(quantile=self.quantile, delta=delta) | ||
| penalty = L1(alpha=self.alpha) |
There was a problem hiding this comment.
those can be taken out of the for loop
There was a problem hiding this comment.
(initialize datafit, penalty, solver and est outside of the loop; then in the loop only update the delta parameter of GLE.datafit)
skglm/experimental/quantile_huber.py
Outdated
| solver=solver | ||
| ) | ||
|
|
||
| if i > 0: |
There was a problem hiding this comment.
this way you won't need this (if est is fixed outside the loop and uses warm_start=True)
|
Ok as discussed separately, you need to implement the maths computation to make the solver work with Fista solver and AndersonCD solver; then it should be easy to support the intercept as these solvers rely on |
skglm/experimental/quantile_huber.py
Outdated
| f"n_iter={est.n_iter_}" | ||
| ) | ||
|
|
||
| self.est = est |
There was a problem hiding this comment.
since this is an attribute that exists only after fitting, call it est_ with a trailing underscore, in the sklearn convention
There was a problem hiding this comment.
Thanks @floriankozikowski
As a sanity check I wanted to see if, with delta going to 0, we get the same solution as sklearn. However the solver seems to get stuck, in debug_quantile.py, if I use delta_final less than its current value of 0.01. Can you investigate ? Setting the inner solver to verbose mode may hep
skglm/experimental/quantile_huber.py
Outdated
|
|
||
| print( | ||
| f" Stage {i+1:2d}: delta={delta:.4f}, " | ||
| f"coverage={coverage:.3f}, pinball_loss={pinball_loss:.6f}, " |
There was a problem hiding this comment.
no need to print coverage here, it's a statistical value, we're more interested in the value of delta and of the loss
There was a problem hiding this comment.
okay, I understand as its not an optimization metric and might make things too verbose, so I removed it. However, coverage is still really important to see if the quantile regression is well-calibrated. I think it might make sense to include it in the unit test or the example script to ensure the model is predicting the correct quantile level. What do you think?
There was a problem hiding this comment.
I'm -1 on this, we focus more on the convergence part ; if the model is not adapted to the data, it's not skglm's fault
What we want to check is only that we solve the optimization problem correctly (same for LienarRegression, we do not check if the residuals are homoscedastic)
skglm/experimental/quantile_huber.py
Outdated
| pinball_loss = np.mean(residuals * (self.quantile - (residuals < 0))) | ||
|
|
||
| print( | ||
| f" Stage {i+1:2d}: delta={delta:.4f}, " |
There was a problem hiding this comment.
since delta goes to zero use scientific notation: delta:.2e
otherwise for delta less than 1e-4 this prints 0.0000
skglm/experimental/quantile_huber.py
Outdated
|
|
||
| def predict(self, X): | ||
| """Predict using the fitted model.""" | ||
| if not hasattr(self, "est"): |
There was a problem hiding this comment.
change to est_ here too
skglm/experimental/quantile_huber.py
Outdated
| def predict(self, X): | ||
| """Predict using the fitted model.""" | ||
| if not hasattr(self, "est"): | ||
| raise ValueError("Call 'fit' before 'predict'.") |
There was a problem hiding this comment.
copy the error message that is printed when you predict with LinearRegression() is sklearn without fitting yet
and raise a NotFittedError
…skglm into quantilehuber
| sk_est = QuantileRegressor(quantile=quantile, alpha=0.1, solver='highs').fit(X, y) | ||
| smooth_est = SmoothQuantileRegressor( | ||
| quantile=quantile, | ||
| alpha=0.1, |
There was a problem hiding this comment.
@floriankozikowski alpha is chosen by hand, just check that it's not too high (= it does not give you coefficients all equal to 0)
| delta_init=0.5, | ||
| delta_final=0.00001, | ||
| n_deltas=15, | ||
| verbose=True, |
There was a problem hiding this comment.
no verbose in test usually
| delta_final=0.00001, | ||
| n_deltas=15, | ||
| verbose=True, | ||
| fit_intercept=True, |
There was a problem hiding this comment.
make this a parameter of the test
skglm/experimental/quantile_huber.py
Outdated
| # Use AndersonCD solver | ||
| solver = AndersonCD(max_iter=self.max_iter, tol=self.tol, | ||
| warm_start=True, fit_intercept=self.fit_intercept, | ||
| verbose=3) |
There was a problem hiding this comment.
this should be verbose=max(0,verbose - 1) (so that we pass verbose >= 2 corresponds to inner solver being verbose)
|
Only some minor changes needed, thanks @floriankozikowski ! |
doc/changes/0.5.rst
Outdated
| @@ -3,3 +3,4 @@ | |||
| Version 0.5 (in progress) | |||
| ------------------------- | |||
| - Add support for fitting an intercept in :ref:`SqrtLasso <skglm.experimental.sqrt_lasso.SqrtLasso>` (PR: :gh:`298`) | |||
| - Add experimental QuantileHuber and SmoothQuantileRegressor for quantile regression, and an example script (PR: :gh:`312`). | |||
There was a problem hiding this comment.
Add link like in line above
debug_quantile.py
Outdated
| @@ -0,0 +1,78 @@ | |||
| """ | |||
| @@ -0,0 +1,72 @@ | |||
| """ | |||
There was a problem hiding this comment.
look at some other plotting file format
There was a problem hiding this comment.
- write some short text to explain what each 3 or 4 blocks of code does
|
|
||
| __all__ = [ | ||
| IterativeReweightedL1, | ||
| PDCD_WS, | ||
| Pinball, | ||
| SqrtQuadratic, | ||
| SqrtLasso, | ||
| QuantileHuber, |
| @@ -2,11 +2,14 @@ | |||
| from .sqrt_lasso import SqrtLasso, SqrtQuadratic | |||
| from numba import float64 | ||
| from skglm.datafits.base import BaseDatafit | ||
| from sklearn.base import BaseEstimator, RegressorMixin | ||
| from skglm.solvers import AndersonCD |
There was a problem hiding this comment.
put all skglm imports together in their own block
skglm/experimental/quantile_huber.py
Outdated
| quantile : float, default=0.5 | ||
| Desired quantile level between 0 and 1. | ||
| delta : float, default=1.0 | ||
| Width of quadratic region. |
There was a problem hiding this comment.
| Width of quadratic region. | |
| Smoothing parameter (0 mean no smoothing). |
|
@Badr-MOUFAD LGTM, merge if happy |
|
@mathurinm as discussed just pushed sparsity support. Please one of you two shortly review before merging! |
Context of the PR
This PR implements a smooth quantile regression estimator using a Huberized loss with progressive smoothing. The goal is to provide a faster alternative to scikit-learn's QuantileRegressor while maintaining similar accuracy.
(closes #276 )
(Also it aims to simplify earlier approaches done in PR #306 )
Contributions of the PR
Added QuantileHuber loss in skglm/experimental/quantile_huber.py
Added SmoothQuantileRegressor class in skglm/experimental/smooth_quantile_regressor.py:
Added example in examples/plot_smooth_quantile.py
Checks before merging PR