scikit-learn · jnothman · Jan 24, 2017 · Jan 24, 2017 · Jan 24, 2017 · Jan 31, 2017
diff --git a/doc/modules/grid_search.rst b/doc/modules/grid_search.rst
@@ -651,6 +651,24 @@ fold independently. Computations can be run in parallel by using the keyword
 ``n_jobs=-1``. See function signature for more details, and also the Glossary
 entry for :term:`n_jobs`.
 
+Avoiding repeated work
+----------------------
+
+Ordinarily, the model is fit anew for each parameter setting.  However, some
+estimators provide a ``warm_start`` parameter which allows different parameter
+settings to be evaluated without clearing the model.  This can be exploited
+in :class:`GridSearchCV` by using its ``use_warm_start`` parameter.  Users
+should take care to specify the parameter values in an appropriate order for
+greatest efficiency, e.g. in order of increasing regularization for a linear
+model; increasing the number of estimators for an ensemble. Note that
+not all parameters can be varied sensibly with ``warm_start``; it can be used
+to search over ``n_estimators`` in :class:`sklearn.ensemble.GradientBoostingClassifier`,
+but not ``max_depth``, ``min_samples_split``, etc.
+
+.. topic:: Example
+
+    :ref:`sphx_glr_auto_examples_model_selection_plot_grid_search_use_warm_start.py`
+
 Robustness to failure
 ---------------------
 
@@ -669,7 +687,6 @@ Alternatives to brute force parameter search
 Model specific cross-validation
 -------------------------------
 
-
 Some models can fit data for a range of values of some parameter almost
 as efficiently as fitting the estimator for a single value of the
 parameter. This feature can be leveraged to perform a more efficient
@@ -696,6 +713,8 @@ Here is the list of such models:
    linear_model.RidgeCV
    linear_model.RidgeClassifierCV
 
+Similar efficiency may be obtained in some cases by using
+:class:`model_selection.GridSearchCV` with its ``use_warm_start`` parameter.
 
 Information Criterion
 ---------------------

diff --git a/doc/whats_new/v1.5.rst b/doc/whats_new/v1.5.rst
@@ -25,6 +25,13 @@ Changelog
     :pr:`123456` by :user:`Joe Bloggs <joeongithub>`.
     where 123455 is the *pull request* number, not the issue number.
 
+:mod:`sklearn.model_selection`
+..............................
+
+- |Feature| The new ``use_warm_start`` parameter in :class:`~model_selection.GridSearchCV`
+  allows for more efficient grid search over some parameter spaces, utilizing estimators'
+  :term:`warm_start` capabilities. :pr:`8230` by :user:`Joel Nothman <jnothman>`.
+
 Code and Documentation Contributors
 -----------------------------------
 

diff --git a/examples/model_selection/plot_grid_search_use_warm_start.py b/examples/model_selection/plot_grid_search_use_warm_start.py
@@ -0,0 +1,80 @@
+"""
+===========================================
+Efficienct GridSearchCV with use_warm_start
+===========================================
+
+A number of estimators are able to reuse a previously fit model as certain
+parameters change.  This is facilitated by a ``warm_start`` parameter.  For
+:class:`ensemble.GradientBoostingClassifier`, for instance, with
+``warm_start=True``, fit can be called repeatedly with the same data while
+increasing its ``n_estimators`` parameter.
+
+:class:`model_selection.GridSearchCV` can efficiently search over such
+warm-startable parameters through its ``use_warm_start`` parameter.  This
+example compares ``GridSearchCV`` performance for searching over
+``n_estimators`` in :class:`ensemble.GradientBoostingClassifier` with
+and without ``use_warm_start='n_estimators'``.  """
+
+# Authors: Vighnesh Birodkar <vighneshbirodkar@nyu.edu>
+#          Raghav RV <rvraghav93@gmail.com>
+#          Joel Nothman <joel.nothman@gmail.com>
+# License: BSD 3 clause
+
+import matplotlib.pyplot as plt
+import numpy as np
+
+from sklearn import datasets
+from sklearn.ensemble import GradientBoostingClassifier
+from sklearn.model_selection import GridSearchCV
+
+print(__doc__)
+
+data_list = [datasets.load_iris(return_X_y=True), datasets.make_hastie_10_2()]
+names = ["Iris Data", "Hastie Data"]
+
+search_n_estimators = range(1, 20)
+
+times = []
+
+for use_warm_start in [None, "n_estimators"]:
+    for X, y in data_list:
+        gb_gs = GridSearchCV(
+            GradientBoostingClassifier(random_state=42, warm_start=True),
+            param_grid={
+                "n_estimators": search_n_estimators,
+                "min_samples_leaf": [1, 5],
+            },
+            scoring="f1_micro",
+            cv=3,
+            refit=True,
+            verbose=True,
+            use_warm_start=use_warm_start,
+        ).fit(X, y)
+        times.append(gb_gs.cv_results_["mean_fit_time"].sum())
+
+
+plt.figure(figsize=(9, 5))
+bar_width = 0.2
+n_datasets = len(data_list)
+index = np.arange(0, n_datasets * bar_width, bar_width) * 2.5
+index = index[0:n_datasets]
+
+true_times = times[len(times) // 2 :]
+false_times = times[: len(times) // 2]
+
+
+plt.bar(
+    index, true_times, bar_width, label='use_warm_start="n_estimators"', color="green"
+)
+plt.bar(
+    index + bar_width, false_times, bar_width, label="use_warm_start=None", color="red"
+)
+
+plt.xticks(index + bar_width, names)
+
+plt.legend(loc="best")
+plt.grid(True)
+
+plt.xlabel("Datasets")
+plt.ylabel("Mean fit time")
+plt.show()