[MRG+2] Adding return_std options for models in linear_model/bayes.py #7838
Conversation
| X : {array-like, sparse matrix}, shape = (n_samples, n_features) | ||
| Samples. | ||
|
|
||
| predict_std : boolean, optional |
There was a problem hiding this comment.
Please use return_std like GaussianProcessRegressor, and check its docstring for consistency.
There was a problem hiding this comment.
Oh my bad - I'll change that to return_std.
|
@jnothman I changed all instances of |
|
Previous example renderings:
I wonder if there's a way to visually emphasise the difference in uncertainties between the centre and edges of these plots. |
|
Could you put the new figure in the fourth quadrant of the existing plots? |
|
The problem is that it the uncertainty grows linearly. I tried zooming out further, but the plot looks qualitatively the same. One idea is to have a non-linear function f(x), and then use a polynomial kernel to estimate it. I'll give this a shot. And yes, I'll put the figures in the fourth quadrant. Thanks. |
|
@jnothman I tried the polynomial regression thing. Check it out. It works better with BayesianRidge than with ARD, but the idea is there in both I think. |
|
ARD: Inserted with: |
| y_mean, y_std = clf_poly.predict(np.vander(X_plot, degree), return_std=True) | ||
| plt.figure(figsize=(6, 5)) | ||
| plt.errorbar(X_plot, y_mean, y_std, color='navy', | ||
| label="Polynomial Bayesian Ridge Regression", linewidth=2) |
|
I agree those test failures don't appear to be your problem. |
|
@sergeyf you still have flake8 errors in Travis. My advice is to take some time to have on-the-fly flake8 linting in your editor of choice, it will save you a lot of time in the long run. |
|
@lesteve Thanks for the suggestion. I use Spyder - I think it has |
| @@ -144,6 +157,8 @@ def fit(self, X, y): | |||
| X, y = check_X_y(X, y, dtype=np.float64, y_numeric=True) | |||
| X, y, X_offset, y_offset, X_scale = self._preprocess_data( | |||
| X, y, self.fit_intercept, self.normalize, self.copy_X) | |||
| self.X_offset = X_offset | |||
There was a problem hiding this comment.
Should be X_offset_ and X_scale_
| @@ -216,10 +231,43 @@ def fit(self, X, y): | |||
| self.alpha_ = alpha_ | |||
| self.lambda_ = lambda_ | |||
| self.coef_ = coef_ | |||
| sigma_ = np.dot(Vh.T, | |||
| Vh / (eigen_vals_ + lambda_ / alpha_)[:, None]) | |||
|
|
||
| self._set_intercept(X_offset, y_offset, X_scale) | ||
| return self | ||
|
|
||
| def predict(self, X, return_std=False): | ||
| """Predict using the linear model. In addition to the mean of the |
There was a problem hiding this comment.
This is not pep 257: https://www.python.org/dev/peps/pep-0257/
Please add an empty line after the first sentence.
| @@ -434,3 +495,34 @@ def fit(self, X, y): | |||
| self.lambda_ = lambda_ | |||
| self._set_intercept(X_offset, y_offset, X_scale) | |||
| return self | |||
|
|
|||
| def predict(self, X, return_std=False): | |||
| """Predict using the linear model. In addition to the mean of the | |||
There was a problem hiding this comment.
Same comment about docstring pep.
| return np.dot(X, w) + b | ||
|
|
||
| def f_noise(X, noise_mult): | ||
| return f(X) + np.random.randn(X.shape[0])*noise_mult |
There was a problem hiding this comment.
is that pep8 without space around *? hm
There was a problem hiding this comment.
Looks ok, check out: https://www.python.org/dev/peps/pep-0008/#id28
| n_train = 50 | ||
| n_test = 10 | ||
|
|
||
| noise_mult = 0.1 |
There was a problem hiding this comment.
does this work if you do a for-loop over multiple noise_mult?
There was a problem hiding this comment.
Works fine for Bayesian Ridge, but unfortunately, ARD behaves oddly because it gets rid of a bunch of dimensions. It gets it MOSTLY right. If you set noise_mult = 1.0, ARD will get 1.2 or 1.1, which is good enough for the estimation of standard deviation of noise, but I'd have to bump up decimal=0 for it to still pass the tests for such a large noise. I'll do that for the next commit unless you have another suggestion.
| @@ -56,3 +56,55 @@ def test_toy_ard_object(): | |||
| # Check that the model could approximately learn the identity function | |||
| test = [[1], [3], [4]] | |||
| assert_array_almost_equal(clf.predict(test), [1, 3, 4], 2) | |||
|
|
|||
|
|
|||
| def test_return_std_bayesian(): | |||
There was a problem hiding this comment.
If the tests are the same (might have overlooked something?), why not do them both in the same test?
|
@amueller I've made the requested fixes. Thanks! |
|
can you fix pep8? Integration is failing. |
|
Other than that LGTM |
|
Thanks @amueller, just fixed that last one. |
|
|
||
| R. Salakhutdinov, Lecture notes on Statistical Machine Learning, | ||
| http://www.utstat.toronto.edu/~rsalakhu/sta4273/notes/Lecture2.pdf#page=15 | ||
| Their beta is our self.beta_ |
There was a problem hiding this comment.
beta_ is not listed in the attributes, did you mean another attribute maybe?
There was a problem hiding this comment.
Thanks for the catch. I meant self.alpha_.
|
|
||
| R. Salakhutdinov, Lecture notes on Statistical Machine Learning, | ||
| http://www.utstat.toronto.edu/~rsalakhu/sta4273/notes/Lecture2.pdf#page=15 | ||
| Their beta is our self.beta_ |
There was a problem hiding this comment.
Same remark about beta_ not being listed in the attributes.
|
thanks! |
…scikit-learn#7838) * initial commit for return_std * initial commit for return_std * adding tests, examples, ARD predict_std * adding tests, examples, ARD predict_std * a smidge more documentation * a smidge more documentation * Missed a few PEP8 issues * Changing predict_std to return_std #1 * Changing predict_std to return_std scikit-learn#2 * Changing predict_std to return_std scikit-learn#3 * Changing predict_std to return_std final * adding better plots via polynomial regression * trying to fix flake error * fix to ARD plotting issue * fixing some flakes * Two blank lines part 1 * Two blank lines part 2 * More newlines! * Even more newlines * adding info to the doc string for the two plot files * Rephrasing "polynomial" for Bayesian Ridge Regression * Updating "polynomia" for ARD * Adding more formal references * Another asked-for improvement to doc string. * Fixing flake8 errors * Cleaning up the tests a smidge. * A few more flakes * requested fixes from Andy * Mini bug fix * Final pep8 fix * pep8 fix round 2 * Fix beta_ to alpha_ in the comments
…scikit-learn#7838) * initial commit for return_std * initial commit for return_std * adding tests, examples, ARD predict_std * adding tests, examples, ARD predict_std * a smidge more documentation * a smidge more documentation * Missed a few PEP8 issues * Changing predict_std to return_std scikit-learn#1 * Changing predict_std to return_std scikit-learn#2 * Changing predict_std to return_std scikit-learn#3 * Changing predict_std to return_std final * adding better plots via polynomial regression * trying to fix flake error * fix to ARD plotting issue * fixing some flakes * Two blank lines part 1 * Two blank lines part 2 * More newlines! * Even more newlines * adding info to the doc string for the two plot files * Rephrasing "polynomial" for Bayesian Ridge Regression * Updating "polynomia" for ARD * Adding more formal references * Another asked-for improvement to doc string. * Fixing flake8 errors * Cleaning up the tests a smidge. * A few more flakes * requested fixes from Andy * Mini bug fix * Final pep8 fix * pep8 fix round 2 * Fix beta_ to alpha_ in the comments
…scikit-learn#7838) * initial commit for return_std * initial commit for return_std * adding tests, examples, ARD predict_std * adding tests, examples, ARD predict_std * a smidge more documentation * a smidge more documentation * Missed a few PEP8 issues * Changing predict_std to return_std #1 * Changing predict_std to return_std #2 * Changing predict_std to return_std #3 * Changing predict_std to return_std final * adding better plots via polynomial regression * trying to fix flake error * fix to ARD plotting issue * fixing some flakes * Two blank lines part 1 * Two blank lines part 2 * More newlines! * Even more newlines * adding info to the doc string for the two plot files * Rephrasing "polynomial" for Bayesian Ridge Regression * Updating "polynomia" for ARD * Adding more formal references * Another asked-for improvement to doc string. * Fixing flake8 errors * Cleaning up the tests a smidge. * A few more flakes * requested fixes from Andy * Mini bug fix * Final pep8 fix * pep8 fix round 2 * Fix beta_ to alpha_ in the comments
…scikit-learn#7838) * initial commit for return_std * initial commit for return_std * adding tests, examples, ARD predict_std * adding tests, examples, ARD predict_std * a smidge more documentation * a smidge more documentation * Missed a few PEP8 issues * Changing predict_std to return_std #1 * Changing predict_std to return_std scikit-learn#2 * Changing predict_std to return_std scikit-learn#3 * Changing predict_std to return_std final * adding better plots via polynomial regression * trying to fix flake error * fix to ARD plotting issue * fixing some flakes * Two blank lines part 1 * Two blank lines part 2 * More newlines! * Even more newlines * adding info to the doc string for the two plot files * Rephrasing "polynomial" for Bayesian Ridge Regression * Updating "polynomia" for ARD * Adding more formal references * Another asked-for improvement to doc string. * Fixing flake8 errors * Cleaning up the tests a smidge. * A few more flakes * requested fixes from Andy * Mini bug fix * Final pep8 fix * pep8 fix round 2 * Fix beta_ to alpha_ in the comments
…scikit-learn#7838) * initial commit for return_std * initial commit for return_std * adding tests, examples, ARD predict_std * adding tests, examples, ARD predict_std * a smidge more documentation * a smidge more documentation * Missed a few PEP8 issues * Changing predict_std to return_std scikit-learn#1 * Changing predict_std to return_std scikit-learn#2 * Changing predict_std to return_std scikit-learn#3 * Changing predict_std to return_std final * adding better plots via polynomial regression * trying to fix flake error * fix to ARD plotting issue * fixing some flakes * Two blank lines part 1 * Two blank lines part 2 * More newlines! * Even more newlines * adding info to the doc string for the two plot files * Rephrasing "polynomial" for Bayesian Ridge Regression * Updating "polynomia" for ARD * Adding more formal references * Another asked-for improvement to doc string. * Fixing flake8 errors * Cleaning up the tests a smidge. * A few more flakes * requested fixes from Andy * Mini bug fix * Final pep8 fix * pep8 fix round 2 * Fix beta_ to alpha_ in the comments
Reference Issue
The reason for this pull request appears in a conversation for #4844
What does this implement/fix? Explain your changes.
This is the first of two pull requests. The ultimate goal is to add the MICE imputation algorithm to scikit-learn. To do so, we need sklearn's Bayesian regression algorithms to be able to return standard deviations as well as predictions.
This pull requests adds the option
return_stdto thepredictmethods of bothBayesianRidgeandARDRegression.Any other comments?
Once this is accepted, I will make a pull request that implements MICE using
BayesianRidgeby default (which seems more robust to small sample sizes than ARD in my limited experience).