MRG Random sampling hyper parameters #1194

amueller · 2012-09-30T20:26:30Z

This is another shot at implementing randomized hyper parameter search in sklearn.
In contrast to #455, I use scipy.stats distributions for each parameter.

If I understand James' Paper correctly, one of the important points is not to use a grid.
Imagine having one totally irrelevant parameter. Using a grid basically makes a large part of the samples redundant, while using a continuous distribution is not affected by the presence of the irrelevant parameter at all.

For discrete parameters, the user can also specify a list of names i.e. kernel=['rbf', 'linear', 'poly'], which is sampled from using a uniform distribution.

I don't like passing objects as parameters, but this seems to be the best way to specify a distribution.
A user can put in any object that supports .rvs() and scipy.stats.distributions already has a solid amount of predefined ones.

amueller · 2012-10-06T11:34:48Z

This draft does not support tree-like parameter spaces btw.
These are not so uncommon, even in simple standard models. For example, if you use a rbf kernel in an SVM, you need a gamma and a C parameter. If you use a linear kernel, you only need C.

alextp · 2012-10-06T12:01:53Z

About tree-structured spaces I think it's ok to do without them because you're doing random search, so having extra parameters which are ignored doesn't really cost you anything in statistical efficiency.

In general I'm +1 for merging this code, as I think it's useful and we should steer people towards doing more random search and less explicit grid search (specially as adding more hyperparameters can often improve performance at the expense of more tuning).

jaberg · 2012-10-06T14:32:04Z

sklearn/grid_search.py

+                    params[k] = v[rnd.randint(len(v))]
+            yield params
+
+


Minor point -- code using yield cannot be serialized, but it is often helpful to serialize long-running code such as hyper-parameter optimization.

Consider making __iter__ return self and adding a next method that returns a params sample?

I'm happy enough to do this as a PR of my own later if you like, so that any discussion can be separate from this PR. Anyway there may be other serialization problems, this might be necessary but insufficient.

Will change it as the draws are independent any way...

I'm not sure how to do this right.
It should only sample n_iter times.
But I want to be able to iterate over it more than once.
I guess that means I have to distinguish between container and iterator.
Not sure how to do this without creating another class. Any help?

@amueller AFAIK you're right, you can't get independent iterators like yield gives you without another class. If this is, in your judgement, a pain, then leave it for later / never.

Creating custom iterators is easy enough (I just did it), but pickling them is a pain because scipy.stats distributions can't be pickled.

zaxtax · 2012-11-09T18:13:14Z

What needs to be done to get this merged?

amueller · 2012-11-09T18:29:21Z

At least example, documentation and testing. Sorry this stalled. Not sure if I have time for it on the weekend, I have some other PRs.

There are some changes I made in GridSearchCV in this PR that needs to be reverted. Generally, it needs to be rebased but that should be easy.

Not sure if everyone is happy with the API, none of the core-devs voiced an opinion afaik.
There was not much response to my mail on the list either.
Maybe @GaelVaroquaux, @ogrisel, @larsmans or @mblondel (sorry for random non-inclusive pings) have an opinion?

amueller · 2012-11-09T20:25:28Z

ok, rebased. Code should be fine now.

zaxtax · 2012-11-09T20:30:42Z

Cool. If I push to your repo, will the changes appear here?
On Nov 9, 2012 12:25 PM, "Andreas Mueller" notifications@github.com wrote:

ok, rebased. Code should be fine now.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1194#issuecomment-10243384.

amueller · 2012-11-09T21:01:14Z

they are here I think.

amueller · 2012-11-18T00:03:59Z

I did some refactoring and added an example. Apart from the docs I think this should be good to go. Will try to do the user guide tomorrow.

amueller · 2012-11-18T18:06:13Z

Added docs, renamed PR to MRG.
Still have to fill in a citation, though.

amueller · 2012-11-26T13:20:11Z

Any reviews please?

ogrisel · 2012-11-26T13:44:17Z

I'll add that one to my todo list. Unfortunately don't have time today.

amueller · 2012-11-26T13:56:09Z

Thanks :) don't worry, you did so much reviewing for me yesterday already! I just didn't want it to get lost ;)

zaxtax · 2012-11-29T08:35:54Z

+1 LGTM

larsmans · 2012-11-29T10:32:12Z

May I lazily ask a question, without having read the code or the paper? This just takes random samples of the hyperparameters and remembers the best sample, right?

The reason I'm asking is because I wonder what will happen to random_state hyperparameters in fancy meta-optimizers. Not all estimators that take such a parameter can do random restarts internally.

amueller · 2012-11-29T18:16:17Z

for each sampled set of parameters, cross validation is run. that hopefully makes the score robust to the seed. does that answer your question?

Lars Buitinck notifications@github.com schrieb:

May I lazily ask a question, without having read the code or the paper?
This just takes random samples of the hyperparameters and remembers the
best sample, right?

The reason I'm asking is because I wonder what will happen to
random_state hyperparameters in fancy meta-optimizers. Not all
estimators that take such a parameter can do random restarts
internally.

Reply to this email directly or view it on GitHub:
#1194 (comment)

Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

amueller · 2012-12-23T11:23:53Z

Rebased and stuff, should be mergable again.

arjoly · 2012-12-27T08:24:58Z

sklearn/grid_search.py

+        self.estimator_scores_ = [
+            (clf_params, score, all_scores)
+            for clf_params, (score, _), all_scores
+            in zip(parameter_iterator, scores, cv_scores)]


Is the comment still relevant?

no. Thanks :)

arjoly · 2012-12-27T08:51:05Z

In BaseSearchCV._fit`, there is some names that seems to be more "grid" specific.

amueller · 2012-12-27T11:36:59Z

@arjoly thanks for your comments, I'll address them shortly :)

…ndomizedSearchCV

amueller · 2013-03-02T17:56:05Z

Ok, should be good now. If there are no more comments, I'll squash and merge.

ogrisel · 2013-03-02T18:17:49Z

On a related note @ogrisel have you looked at #1626 and the discussion in #1674?

Not yet but I need to focus on my PyCon tutorial in the short term :)

ogrisel · 2013-03-02T18:26:57Z

sklearn/grid_search.py

@@ -72,11 +75,141 @@ def __iter__(self):
                yield params


+class IterGrid(ParameterGrid):
+    """Generators on the combination of the various parameter lists given.
+


You should add a note on the deprecation either here or at the end of the docstring.

ogrisel · 2013-03-02T18:28:10Z

Code coverage is good:

sklearn.grid_search                                239     10    96%   258, 347-352, 366, 405, 634-636, 667

ogrisel · 2013-03-02T19:00:49Z

In the example, rather than just reporting the top parameters, it would be better to report the, say, top 3 models along with standard of the mean validation score:

diff --git a/examples/randomized_search.py b/examples/randomized_search.py
index c416508..359a4eb 100644
--- a/examples/randomized_search.py
+++ b/examples/randomized_search.py
@@ -21,6 +21,8 @@ simultaneously using grid search, but pick only the ones deemed most important.
 print __doc__

 from time import time
+from operator import itemgetter
+from scipy.stats import sem
 from scipy.stats import randint as sp_randint

 from sklearn.grid_search import GridSearchCV, RandomizedSearchCV
@@ -44,12 +46,23 @@ param_dist = {"max_depth": sp_randint(1, 5), "max_features": sp_randint(1, 4),
 random_search = RandomizedSearchCV(clf, param_distributions=param_dist,
                                    n_iter=20)

+# Utility function to report best scores
+def report(cv_scores, n_top=3):
+    top_scores = sorted(random_search.cv_scores_, key=itemgetter(1),
+                 reverse=True)[:n_top]
+    for i, score in enumerate(top_scores):
+        print("Model with rank: {0}".format(i + 1))
+        print("Mean validation score: {0:.3f} (+/-{1:.3f})".format(
+              score.mean_validation_score,
+              sem(score.cv_validation_scores)))
+        print("Parameters: {0}".format(score.parameters))
+        print("")
+
 start = time()
 random_search.fit(X, y)
 print("RandomizedSearchCV took %.2f seconds for 20 iterations."
       % (time() - start))
-print("Best score: %f" % random_search.best_score_)
-print("Best parameters: %s" % repr(random_search.best_params_))
+report(random_search.cv_scores_)

 # use a full grid over all parameters
 param_grid = {"max_depth": range(1, 5), "max_features": range(1, 4)
F438
,
@@ -63,5 +76,4 @@ grid_search.fit(X, y)

 print("GridSearchCV took %.2f seconds for %d parameter settings."
       % (time() - start, len(grid_search.cv_scores_)))
-print("Best score: %f" % grid_search.best_score_)
-print("Best parameters: %s" % repr(grid_search.best_params_))
+report(grid_search.cv_scores_)

that outputs:

RandomizedSearchCV took 0.86 seconds for 20 iterations.
Model with rank: 1
Mean validation score: 0.960 (+/-0.020)
Parameters: {'bootstrap': False, 'min_samples_leaf': 4, 'min_samples_split': 4, 'criterion': 'entropy', 'max_features': 1, 'max_depth': 2}

Model with rank: 2
Mean validation score: 0.960 (+/-0.020)
Parameters: {'bootstrap': False, 'min_samples_leaf': 2, 'min_samples_split': 3, 'criterion': 'entropy', 'max_features': 1, 'max_depth': 2}

Model with rank: 3
Mean validation score: 0.960 (+/-0.023)
Parameters: {'bootstrap': False, 'min_samples_leaf': 4, 'min_samples_split': 4, 'criterion': 'entropy', 'max_features': 2, 'max_depth': 4}

GridSearchCV took 33.51 seconds for 768 parameter settings.
Model with rank: 1
Mean validation score: 0.960 (+/-0.020)
Parameters: {'bootstrap': False, 'min_samples_leaf': 4, 'min_samples_split': 4, 'criterion': 'entropy', 'max_features': 1, 'max_depth': 2}

Model with rank: 2
Mean validation score: 0.960 (+/-0.020)
Parameters: {'bootstrap': False, 'min_samples_leaf': 2, 'min_samples_split': 3, 'criterion': 'entropy', 'max_features': 1, 'max_depth': 2}

Model with rank: 3
Mean validation score: 0.960 (+/-0.023)
Parameters: {'bootstrap': False, 'min_samples_leaf': 4, 'min_samples_split': 4, 'criterion': 'entropy', 'max_features': 2, 'max_depth': 4}

We could even add a report(n_top=3, format_params=True) method to the search object directly and use pprint.pformat for the formatting of the parameters in the report.

ogrisel · 2013-03-02T19:31:27Z

sklearn/grid_search.py

+    >>> list(ParameterSampler(param_grid, n_iter=4))
+    ...  #doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
+    [{'a': ..., 'b': ...}, {'a': ..., 'b': ...},
+     {'a': ..., 'b': ...}, {'a': ..., 'b': ...}]


Wouldn't it be possible to fix the seed of the sampler and display the values of a and the first 3 digits of b before ellipsis in the output?

ogrisel · 2013-03-02T19:35:46Z

The doctest failure on travis is real:

======================================================================
FAIL: Doctest: model_selection.rst
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/doctest.py", line 2201, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for model_selection.rst
  File "/home/travis/build/scikit-learn/scikit-learn/doc/tutorial/statistical_inference/model_selection.rst", line 0

----------------------------------------------------------------------
File "/home/travis/build/scikit-learn/scikit-learn/doc/tutorial/statistical_inference/model_selection.rst", line 148, in l_selection.rst
Failed example:
    clf.best_score_
Expected:
    0.988991985997974
Got:
    0.98899999999999999

missing an ellipsis.

amueller · 2013-03-03T17:30:41Z

Not sure why that changed... but ellipsis seem like a good idea any way.

amueller · 2013-03-03T17:32:22Z

Oh, my last commit fixes an unrelated bug in master. I should probably add a test. n_test_sampels was wrong in fit_grid_point as far as I can tell.

amueller · 2013-03-03T17:33:23Z

So I guess I shouldn't use ellipsis because this caught a bug ^^

amueller · 2013-03-03T17:37:44Z

should I put the bugfix + test in a separate PR?

amueller · 2013-03-03T17:40:38Z

I would rather not add a function for reporting just now. I started something at some point but it was not generic enough. Once this is in, I'd like to revisit my ideas and create helper functions or an interface for GridSearchCV and RandomizedSearchCV that is more powerful and easier to use. I'll take your code for the example, though ;)

ogrisel · 2013-03-03T17:48:32Z

should I put the bugfix + test in a separate PR?

That would be great. We can get a quick merge for the bugfix.

I would rather not add a function for reporting just now. I started something at some point but it was not generic enough. Once this is in, I'd like to revisit my ideas and create helper functions or an interface for GridSearchCV and RandomizedSearchCV that is more powerful and easier to use. I'll take your code for the example, though ;)

Alright. Let's add the reporting function only in the example for now. It's likely to change if we add train time + train score anyway.

amueller · 2013-03-03T17:53:52Z

there was no test for the iid parameter... great....

amueller · 2013-03-03T18:36:43Z

See #1731 for the bugfix / regression test

amueller · 2013-03-03T18:45:09Z

Ok, I think I addressed your comments.

ogrisel · 2013-03-03T19:30:20Z

Looks good to me :) +1 for merging.

amueller · 2013-03-03T19:37:45Z

Ok, I'll squash and merge.

amueller · 2013-03-03T19:46:54Z

@larsmans is there a good way to squash commits if I merged master in-between? I could drop all the history for a clean commit, not sure that is a great idea, though.

amueller · 2013-03-03T20:01:01Z

Ok, I merged without history. Everything else would be way to messy and / or to much work. Thanks for the reviews, guys :)

ogrisel · 2013-03-03T20:02:44Z

You definitely deserve a 🍺 for this one :)

ogrisel · 2013-03-03T20:03:41Z

Or maybe a caol ila :)

amueller · 2013-03-03T20:05:12Z

Not sure I deserve one, will take it any way ;)

arjoly · 2013-03-04T07:39:42Z

Cool !!! Congratz @amueller !!!

jaberg reviewed Oct 6, 2012
View reviewed changes

arjoly reviewed Dec 27, 2012
View reviewed changes

amueller added 7 commits December 27, 2012 15:29

ENH first draft of random sampling hyper parameters

835c38f

ENH slight cleanup, addressing some comments.

838472b

ENH refactoring: added BaseSearchCV baseclass for GridSearchCV and Ra…

e192d31

…ndomizedSearchCV

DOC add example for randomized search CV

34d3ceb

DOC Documentation of RandomizedSearchCV, cosmit in GridSearchCV docs.

86dbb3a

DOC added reference to James' JMLR paper, added heading to example

65d18e2

FIX missing dot

5c01ab7

ogrisel reviewed Mar 2, 2013
View reviewed changes

DOC fix doctest broken by bugfix

fd026a8

DOC fixed random seed in doctest

db9f1be

amueller added 2 commits March 3, 2013 19:42

DOC apply @ogrisel's patch for example: show best couple of settings

7e3af7f

DOC add note on deprecation to IterGrid docstring

935fce8

amueller closed this Mar 3, 2013

amueller mentioned this pull request Mar 3, 2013

WIP: First draft of a random search in GridSearchCV #455

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MRG Random sampling hyper parameters #1194

MRG Random sampling hyper parameters #1194

MRG Random sampling hyper parameters #1194

MRG Random sampling hyper parameters #1194

Conversation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment