[MRG+2] Gaussian process by @jmetzen #5466

glouppe · 2015-10-19T16:14:06Z

This supersedes #4270. @jmetzen branch has been rebased. Waiting for Travis and then I'll merge.

…gressor

…imizer parameter

…rnels

…sifier

….__init__

…if no theta vector is provided

glouppe · 2015-10-19T16:23:40Z

Tests have passed on my local Travis. See https://travis-ci.org/glouppe/scikit-learn/builds/86218126

Merging this beast! and adding a what's new entry.

Thanks @jmetzen for the wonderful work! 🍻

@jmetzen

[MRG+2] Gaussian process by @jmetzen

raghavrv · 2015-10-19T16:24:48Z

Awesome!! 🍻

amueller · 2015-10-19T16:28:41Z

🍻 💃 🍷 Awesome! thanks @jmetzen !

GaelVaroquaux · 2015-10-20T05:57:49Z

Looks like the example are dependent on a very recent version of matplotlib, and that breaks CircleCI (our new CI infrastructure that is meant for testing the examples:):

https://circleci.com/gh/scikit-learn/scikit-learn/51

Specifically, in plot_gpc.py, there is, on line 95, plt.colorbar(label="Log-marginal Likelihood")

And the version of matplotlib that we have doesn't like the 'label' argument to colorbar.

jnothman · 2015-10-20T06:04:46Z

Ve 8000 ry exciting! (Never mind the teething errors...) Well done to @jmetzen
and everyone involved. I hope someone's writing a What's New entry?

On 20 October 2015 at 16:57, Gael Varoquaux notifications@github.com
wrote:

Looks like the example are dependent on a very recent version of
matplotlib, and that breaks CircleCI (our new CI infrastructure that is
meant for testing the examples:):

https://circleci.com/gh/scikit-learn/scikit-learn/51

Specifically, in plot_gpc.py, there is, on line 95,
plt.colorbar(label="Log-marginal Likelihood")

And the version of matplotlib that we have doesn't like the 'label'
argument to colorbar.

—
Reply to this email directly or view it on GitHub
#5466 (comment)
.

glouppe · 2015-10-20T06:41:07Z

@jnothman Yes, it has been added at 2f15969

arjoly · 2015-10-20T07:40:58Z

Amazing! Thanks!

Squashed commit messages - (For reference) Major ----- * ENH p --> n_labels * FIX *ShuffleSplit: all float/invalid type errors at init and int error at split * FIX make PredefinedSplit accept test_folds in constructor; Cleanup docstrings * ENH+TST KFold: make rng to be generated at every split call for reproducibility * FIX/MAINT KFold: make shuffle a public attr * FIX Make CVIterableWrapper private. * FIX reuse len_cv instead of recalculating it * FIX Prevent adding *SearchCV estimators from the old grid_search module * re-FIX In all_estimators: the sorting to use only the 1st item (name) To avoid collision between the old and the new GridSearch classes. * FIX test_validate.py: Use 2D X (1D X is being detected as a single sample) * MAINT validate.py --> validation.py * MAINT make the submodules private * MAINT Support old cv/gs/lc until 0.19 * FIX/MAINT n_splits --> get_n_splits * FIX/TST test_logistic.py/test_ovr_multinomial_iris: pass predefined folds as an iterable * MAINT expose BaseCrossValidator * Update the model_selection module with changes from master - From scikit-learn#5161 - - MAINT remove redundant p variable - - Add check for sparse prediction in cross_val_predict - From scikit-learn#5201 - DOC improve random_state param doc - From scikit-learn#5190 - LabelKFold and test - From scikit-learn#4583 - LabelShuffleSplit and tests - From scikit-learn#5300 - shuffle the `labels` not the `indxs` in LabelKFold + tests - From scikit-learn#5378 - Make the GridSearchCV docs more accurate. - From scikit-learn#5458 - Remove shuffle from LabelKFold - From scikit-learn#5466(scikit-learn#4270) - Gaussian Process by Jan Metzen Minor ----- * ENH Make the KFold shuffling test stronger * FIX/DOC Use the higher level model_selection module as ref * DOC in check_cv "y : array-like, optional" * DOC a supervised learning problem --> supervised learning problems * DOC cross-validators --> cross-validation strategies * DOC Correct Olivier Grisel's name ;) * MINOR/FIX cv_indices --> kfold * FIX/DOC Align the 'See also' section of the new KFold, LeaveOneOut * TST/FIX imports on separate lines * FIX use __class__ instead of classmethod * TST/FIX import directly from model_selection * COSMIT Relocate the random_state documentation * COSMIT remove pass * MAINT Remove deprecation warnings from old tests * FIX correct import at test_split * FIX/MAINT Move P_sparse, X, y defns to top; rm unused W_sparse, X_sparse * FIX random state to avoid doctest failure * TST n_splits and split wrapping of _CVIterableWrapper * FIX/MAINT Use multilabel indicator matrix directly * TST/DOC clarify why we conflate classes 0 and 1 * DOC add comment that this was taken from BaseEstimator * FIX use of labels is not needed in stratified k fold * Fix cross_validation reference * Fix the labels param doc

jmetzen · 2015-10-20T13:15:42Z

Thanks for the rebasing @glouppe !

Squashed commit messages - (For reference) Major ----- * ENH p --> n_labels * FIX *ShuffleSplit: all float/invalid type errors at init and int error at split * FIX make PredefinedSplit accept test_folds in constructor; Cleanup docstrings * ENH+TST KFold: make rng to be generated at every split call for reproducibility * FIX/MAINT KFold: make shuffle a public attr * FIX Make CVIterableWrapper private. * FIX reuse len_cv instead of recalculating it * FIX Prevent adding *SearchCV estimators from the old grid_search module * re-FIX In all_estimators: the sorting to use only the 1st item (name) To avoid collision between the old and the new GridSearch classes. * FIX test_validate.py: Use 2D X (1D X is being detected as a single sample) * MAINT validate.py --> validation.py * MAINT make the submodules private * MAINT Support old cv/gs/lc until 0.19 * FIX/MAINT n_splits --> get_n_splits * FIX/TST test_logistic.py/test_ovr_multinomial_iris: pass predefined folds as an iterable * MAINT expose BaseCrossValidator * Update the model_selection module with changes from master - From scikit-learn#5161 - - MAINT remove redundant p variable - - Add check for sparse prediction in cross_val_predict - From scikit-learn#5201 - DOC improve random_state param doc - From scikit-learn#5190 - LabelKFold and test - From scikit-learn#4583 - LabelShuffleSplit and tests - From scikit-learn#5300 - shuffle the `labels` not the `indxs` in LabelKFold + tests - From scikit-learn#5378 - Make the GridSearchCV docs more accurate. - From scikit-learn#5458 - Remove shuffle from LabelKFold - From scikit-learn#5466(scikit-learn#4270) - Gaussian Process by Jan Metzen - From scikit-learn#4826 - Move custom error / warnings into sklearn.exception Minor ----- * ENH Make the KFold shuffling test stronger * FIX/DOC Use the higher level model_selection module as ref * DOC in check_cv "y : array-like, optional" * DOC a supervised learning problem --> supervised learning problems * DOC cross-validators --> cross-validation strategies * DOC Correct Olivier Grisel's name ;) * MINOR/FIX cv_indices --> kfold * FIX/DOC Align the 'See also' section of the new KFold, LeaveOneOut * TST/FIX imports on separate lines * FIX use __class__ instead of classmethod * TST/FIX import directly from model_selection * COSMIT Relocate the random_state documentation * COSMIT remove pass * MAINT Remove deprecation warnings from old tests * FIX correct import at test_split * FIX/MAINT Move P_sparse, X, y defns to top; rm unused W_sparse, X_sparse * FIX random state to avoid doctest failure * TST n_splits and split wrapping of _CVIterableWrapper * FIX/MAINT Use multilabel indicator matrix directly * TST/DOC clarify why we conflate classes 0 and 1 * DOC add comment that this was taken from BaseEstimator * FIX use of labels is not needed in stratified k fold * Fix cross_validation reference * Fix the labels param doc

-------------------- * ENH Reogranize classes/fn from grid_search into search.py * ENH Reogranize classes/fn from cross_validation into split.py * ENH Reogranize cls/fn from cross_validation/learning_curve into validate.py * MAINT Merge _check_cv into check_cv inside the model_selection module * MAINT Update all the imports to point to the model_selection module * FIX use iter_cv to iterate throught the new style/old style cv objs * TST Add tests for the new model_selection members * ENH Wrap the old-style cv obj/iterables instead of using iter_cv * ENH Use scipy's binomial coefficient function comb for calucation of nCk * ENH Few enhancements to the split module * ENH Improve check_cv input validation and docstring * MAINT _get_test_folds(X, y, labels) --> _get_test_folds(labels) * TST if 1d arrays for X introduce any errors * ENH use 1d X arrays for all tests; * ENH X_10 --> X (global var) Minor ----- * ENH _PartitionIterator --> _BaseCrossValidator; * ENH CVIterator --> CVIterableWrapper * TST Import the old SKF locally * FIX/TST Clean up the split module's tests. * DOC Improve documentation of the cv parameter * COSMIT consistently hyphenate cross-validation/cross-validator * TST Calculate n_samples from X * COSMIT Use separate lines for each import. * COSMIT cross_validation_generator --> cross_validator Commits merged manually ----------------------- * FIX Document the random_state attribute in RandomSearchCV * MAINT Use check_cv instead of _check_cv * ENH refactor OVO decision function, use it in SVC for sklearn-like decision_function shape * FIX avoid memory cost when sampling from large parameter grids ENH Major to Minor incremental enhancements to the model_selection Squashed commit messages - (For reference) Major ----- * ENH p --> n_labels * FIX *ShuffleSplit: all float/invalid type errors at init and int error at split * FIX make PredefinedSplit accept test_folds in constructor; Cleanup docstrings * ENH+TST KFold: make rng to be generated at every split call for reproducibility * FIX/MAINT KFold: make shuffle a public attr * FIX Make CVIterableWrapper private. * FIX reuse len_cv instead of recalculating it * FIX Prevent adding *SearchCV estimators from the old grid_search module * re-FIX In all_estimators: the sorting to use only the 1st item (name) To avoid collision between the old and the new GridSearch classes. * FIX test_validate.py: Use 2D X (1D X is being detected as a single sample) * MAINT validate.py --> validation.py * MAINT make the submodules private * MAINT Support old cv/gs/lc until 0.19 * FIX/MAINT n_splits --> get_n_splits * FIX/TST test_logistic.py/test_ovr_multinomial_iris: pass predefined folds as an iterable * MAINT expose BaseCrossValidator * Update the model_selection module with changes from master - From #5161 - - MAINT remove redundant p variable - - Add check for sparse prediction in cross_val_predict - From #5201 - DOC improve random_state param doc - From #5190 - LabelKFold and test - From #4583 - LabelShuffleSplit and tests - From #5300 - shuffle the `labels` not the `indxs` in LabelKFold + tests - From #5378 - Make the GridSearchCV docs more accurate. - From #5458 - Remove shuffle from LabelKFold - From #5466(#4270) - Gaussian Process by Jan Metzen - From #4826 - Move custom error / warnings into sklearn.exception Minor ----- * ENH Make the KFold shuffling test stronger * FIX/DOC Use the higher level model_selection module as ref * DOC in check_cv "y : array-like, optional" * DOC a supervised learning problem --> supervised learning problems * DOC cross-validators --> cross-validation strategies * DOC Correct Olivier Grisel's name ;) * MINOR/FIX cv_indices --> kfold * FIX/DOC Align the 'See also' section of the new KFold, LeaveOneOut * TST/FIX imports on separate lines * FIX use __class__ instead of classmethod * TST/FIX import directly from model_selection * COSMIT Relocate the random_state documentation * COSMIT remove pass * MAINT Remove deprecation warnings from old tests * FIX correct import at test_split * FIX/MAINT Move P_sparse, X, y defns to top; rm unused W_sparse, X_sparse * FIX random state to avoid doctest failure * TST n_splits and split wrapping of _CVIterableWrapper * FIX/MAINT Use multilabel indicator matrix directly * TST/DOC clarify why we conflate classes 0 and 1 * DOC add comment that this was taken from BaseEstimator * FIX use of labels is not needed in stratified k fold * Fix cross_validation reference * Fix the labels param doc FIX/DOC/MAINT Addressing the review comments by Arnaud and Andy COSMIT Sort the members alphabetically COSMIT len_cv --> n_splits COSMIT Merge 2 if; FIX Use kwargs DOC Add my name to the authors :D DOC make labels parameter consistent FIX Remove hack for boolean indices; + COSMIT idx --> indices; DOC Add Returns COSMIT preds --> predictions DOC Add Returns and neatly arrange X, y, labels FIX idx(s)/ind(s)--> indice(s) COSMIT Merge if and else to elif COSMIT n --> n_samples COSMIT Use bincount only once COSMIT cls --> class_i / class_i (ith class indices) --> perm_indices_class_i FIX/ENH/TST Addressing the final reviews COSMIT c --> count FIX/TST make check_cv raise ValueError for string cv value TST nested cv (gs inside cross_val_score) works for diff cvs FIX/ENH Raise ValueError when labels is None for label based cvs; TST if labels is being passed correctly to the cv and that the ValueError is being propagated to the cross_val_score/predict and grid search FIX pass labels to cross_val_score FIX use make_classification DOC Add Returns; COSMIT Remove scaffolding TST add a test to check the _build_repr helper REVERT the old GS/RS should also be tested by the common tests. ENH Add a tuple of all/label based CVS FIX raise VE even at get_n_splits if labels is None FIX Fabian's comments PEP8

Jan Hendrik Metzen added 30 commits October 19, 2015 18:04

ADD New kernel module for GPs which supports kernel engineering

ca89b34

ADD New implementation of GaussianProcessRegression

ca03d52

ADD Gaussian Process classification based on Laplace approximation

b76ca72

TST Tests for GP regression and classification

804c1d9

ADD Examples for GP classification

e548d56

ADD Examples for GP regression

1c5db5f

FIX cross_correlation of anisotropic RBF kernel computed correctly

b4f5463

FIX Fix some bugs in kernel module discovered during writing unit test

e9ab6b1

TST Test for GP kernels

9dade31

REFACTOR auto_correlation() -> auto(), cross_correlation() -> cross()

515defe

DOC GP's kernel module documented

7261032

DOC Revised GP examples (added doc etc.)

7b4d935

ENH GPR.predict allows returning standard-deviations of predictions

1aa49b7

DOC Revised examples based on @eickenberg's comments

226bb3a

MISC Revised gaussian_process package based on @eickenberg's suggestions

a5dc349

TST Dealing with failing tests from test_common

2ab097f

REFACTOR Merging kernel methods auto() and cross() in __call__()

03e3575

ENH Adding PairwiseKernel for GPs (wraps kernels from metrics.pairwise)

5eaa8cf

REFACTOR GaussianProcessClassification/Regression -> ...Classifier/Re…

e165fed

…gressor

ENH More efficient approximation of gradient in PairwiseKernel

4dcd706

ENH Avoid to create square matrix W_sr in all but one place (TODO)

e05daae

ENH Adding RationalQuadratic and ExpSineSquared kernels

6b3df1b

FIX Dealing with singular kernel matrix (-inf log-marginal-likelihood)

722ddf1

ENH Adding DotProduct kernel and fixing ExpSine2 kernel

eb44a42

DOC Initial version of Mauna Loa CO2 example

f534a97

REFACTOR Hyperparameter optimization in GPs enabled explicitly by opt…

d8405c2

…imizer parameter

DOC Revising CO2 example

98f9774

TST Cover more kernels in tests

4d3eb96

ENH Analytic gradients for ExpSine2, RationalQuadratic, DotProduct ke…

209b202

…rnels

DOC Extended documentation of GP kernels module

a13f765

Jan Hendrik Metzen added 9 commits October 19, 2015 18:07

FIX Fix message of ValueError for wrong shape of alpha in GPR

213a6be

PEP8 Making pep8 and pyflakes happy

2dd1f28

REF More meaningful names for hyperparameters of GP kernels

e2a340a

MISC Changing default for copy_X_train to True in GaussianProcessClas…

8f77ced

…sifier

FIX String comparison via equality and not identity in Hyperparameter… 8000

c3a41a3

….__init__

ENH GPR.log_marginal_likelihood() returns the current log-likelihood …

3957fc3

…if no theta vector is provided

FIX Enforcing y to be numeric in GPR and fixing python3 issue with maps

2649598

ADD Mixins for normalized and stationary kernels

edc4a65

TST test_lml_precomputed() checks only for equality in first 7 digits

cea3f29

glouppe added a commit that referenced this pull request Oct 19, 2015

Merge pull request #5466 from glouppe/gp_new

5c697ba

[MRG+2] Gaussian process by @jmetzen

glouppe merged commit < 8000 code class="Link--primary text-bold">5c697ba into scikit-learn:master Oct 19, 2015

glouppe mentioned this pull request Oct 19, 2015

[MRG+2] Gaussian process revisited #4270

Closed

13 tasks

glouppe deleted the gp_new branch October 20, 2015 07:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MRG+2] Gaussian process by @jmetzen #5466

[MRG+2] Gaussian process by @jmetzen #5466

[MRG+2] Gaussian process by @jmetzen #5466

[MRG+2] Gaussian process by @jmetzen #5466

Conversation