Instability in test_ridge.py::test_ridge_sample_weights #11200
Comments
|
Just a stand-alone snippet reproducing the problem, which is a small variation around from itertools import product
import numpy as np
from sklearn.utils.testing import (assert_array_almost_equal,
assert_almost_equal)
from sklearn.linear_model import Ridge
from scipy import linalg
def test_ridge_sample_weights(rng):
param_grid = product((1.0, 1e-2), (True, False),
('svd', 'cholesky', 'lsqr', 'sparse_cg'))
for n_samples, n_features in ((6, 5), (5, 10)):
y =
8000
rng.randn(n_samples)
X = rng.randn(n_samples, n_features)
sample_weight = 1.0 + rng.rand(n_samples)
for (alpha, intercept, solver) in param_grid:
# Ridge with explicit sample_weight
est = Ridge(alpha=alpha, fit_intercept=intercept, solver=solver)
est.fit(X, y, sample_weight=sample_weight)
coefs = est.coef_
inter = est.intercept_
# Closed form of the weighted regularized least square
# theta = (X^T W X + alpha I)^(-1) * X^T W y
W = np.diag(sample_weight)
if intercept is False:
X_aug = X
I = np.eye(n_features)
else:
dummy_column = np.ones(shape=(n_samples, 1))
X_aug = np.concatenate((dummy_column, X), axis=1)
I = np.eye(n_features + 1)
I[0, 0] = 0
cf_coefs = linalg.solve(X_aug.T.dot(W).dot(X_aug) + alpha * I,
X_aug.T.dot(W).dot(y))
if intercept is False:
assert_array_almost_equal(coefs, cf_coefs)
else:
assert_array_almost_equal(coefs, cf_coefs[1:])
assert_almost_equal(inter, cf_coefs[0])
rng = np.random.RandomState(0)
for i in range(100):
try:
test_ridge_sample_weights(rng)
except AssertionError:
print('failed')On my machine I get 26 failures out of 100 runs. |
|
could you instead print the absolute and relative differences or a
histogram of them?
|
|
I generated a histogram of mismatch percentages for
|
|
So the problem was the default tolerance in
I am creating a PR to fix this in the test. |
The test
sklearn/linear_model/tests/test_ridge.py::test_ridge_sample_weightspasses on master, however, when it was parametrized as part of #11074 failures were observed (#11074 (comment)).The relevant diff can be found in master...rth:test_ridge_sample_weights-parametrization (whitespaces are ignored in this diff), where the following runs fail,
(out of a total of 32 runs).
This means that this test is brittle and depends on the RNG state. Increasing numerical tolerance might be a solution or possibly increasing the number of samples?
The text was updated successfully, but these errors were encountered: