MRG: Functionality to change scoring method of searchlight #3502

kaichogami · 2016-08-08T19:14:24Z

Enhancement as discussed in #3475. Aim to support

sl = SearchLight(LogisticRegression(), scoring=roc_auc_score)
score = sl.fit(X, y).score(X, y)

score = cross_val_score(SearchLight(LogisticRegression()), X, y, scoring='roc_auc')

kaichogami · 2016-08-08T19:17:38Z

mne/decoding/tests/test_search_light.py

+    assert_array_equal(score.shape, [n_time])
+    assert_true(score.dtype == float)
+    sl = SearchLight(LogisticRegression())
+    assert_array_equal(cross_val_score(sl, X, y, scoring='roc_auc'), score)


cross_val_score fails with an error.

File "/home/kaichogami/codes/dev_mne/scikit-learn/sklearn/utils/validation.py", line 561, in column_or_1d raise ValueError("bad input shape {0}".format(shape)) ValueError: bad input shape (17, 10)

kingjr · 2016-08-09T14:18:26Z

I think the current design won't work, notably because we need to change the type of prediction as a function of the scoring. e.g. if you pass a roc_auc_score, the prediction should be continuous (e.g. predict_proba or decision_function) and not discrete (this is probably the cause of your error). Check how this is handled in sklearn cross_val_score.

Additionally, could you add tests to make sure that this is working:

SearchLight(Ridge(), scoring=my_scorer())
cross_val_score(SearchLight(Ridge()), X, y, scoring=my_scorer())

And once it passes, update the GeneralizationLight too

kaichogami · 2016-08-10T16:34:29Z

mne/decoding/search_light.py

@@ -338,7 +358,10 @@ def _sl_score(estimators, X, y):
    """
    n_iter = X.shape[-1]
    for ii, est in enumerate(estimators):
-        _score = est.score(X[..., ii], y)
+        if scoring is not None:
+            _score = scoring(est, X[..., ii], y)


@kingjr I made it with the signature score_func(estimator, y, y_pred) by using make_scorer so as to take care of the prediction type which make_scorer handles internally. However cross_val_score doesn't still pass. Let me know if you have any idea about it.

I'm afraid I don't know. Do you manage to make cross_val_score(base_estimator, scoring=scoring) pass?

your y is 2d and roc_auc requires y to be 1d or 1 column. You need to look over the columns of y.

@agramfort this is a single/multi class scoring problem. Isn't there a way to handle this by sklearn scorers?

_score = scoring(est, X[..., ii], y), shouldn't this fail as well in case when scoring parameter is changed to roc_auc_score in the test?

Ignore my last comment. Going through the code, I think error occurs because X is not 2D before being passed to decision_function. I can see two solutions.

Make the input to be a 2D matrix or at least convert X to 2D before passing to decision_function

Follow the other API @kingjr mentioned cross_val_score(make_pipeline(SearchLight(Ridge(), scoring=my_scorer()), X, y)

I hope I make sense..

A sklearn score returns a float per fold. Look maybe at multi_output scorers

On Thu, Aug 11, 2016 at 3:32 PM +0200, "Jean-Rémi KING" notifications@github.com wrote:

In mne/decoding/search_light.py:

@@ -338,7 +358,10 @@ def _sl_score(estimators, X, y):
"""
n_iter = X.shape[-1]
for ii, est in enumerate(estimators):

_score = est.score(X[..., ii], y)

if scoring is not None:

_score = scoring(est, X[..., ii], y)

@agramfort this is a single/multi class scoring problem. Isn't there a way to handle thus by sklearn scorers?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

kaichogami · 2016-08-22T09:22:02Z

@kingjr I am not sure what should be done now.

agramfort · 2016-08-23T13:41:20Z

@kaichogami make CIs happy?

kaichogami · 2016-08-24T14:16:11Z

@agramfort yes, however I don't know how to solve the issue with cross_val_score(clf, X, y, scorer='roc_auc').

agramfort · 2016-08-27T10:56:37Z

you cannot obtain more than one value per fold with a sklearn scorer. If you have multiple targets you need a custom scorer that return more than a float

so

cross_val_score(clf, X, y, scorer='roc_auc')

simple cannot work as is.

kingjr · 2016-08-27T13:52:17Z

@agramfort I think there is a confusion between multiple issues:

SearchLight involves multiple scores, so cross_val_score(SearchLight(), scoring='roc_auc') is not viable with the current sklearn API. IMO scoring metrics need not output a float by principle. Confusion matrices are a form of scoring metrics. In any case this debate is not to be discussed here.
We want to be able to parametrize the SearchLight scoring e.g. SearchLight(clf, scoring='roc_auc'). However this involves
i) internally adapting the predict function: if you want to score with an AUC, you need to ensure that the predict function outputs an array of floats (e.g. distance to hyperplan, proba etc) and not of integers.
ii) identifying or handling 2 or multi class problem: i.e. predict_proba generates a y_pred, of shape (n_samples, n_classes), which can therefore not be passed to the sklearn roc_auc scorer. This is an API problem, not an intrinsic one, because we could in principle identify that there are 2 classes and compute the AUC. In case of multiclass, we could compute each auc and return the average AUC. All this should be solved/able at the sklearn level IMO

agramfort · 2016-08-29T12:53:17Z

does using check_scoring function from sklearn help? see also implementation of make_scorer function. a sklearn scorer made from a string knows if it needs proba or decision function or just predict.

kingjr · 2016-09-12T22:05:48Z

@kaichogami how are we doing over here?

kaichogami · 2016-09-13T11:31:06Z

@kingjr I haven't touched it after my last comment. I'll look over it again and let you know if I can solve it, in few days.

kaichogami · 2016-09-15T08:19:22Z

@kingjr as Alex mentioned we need to design custom scorer to make cross_val_score(clf, X, y, scorer='custom_scorer') work. Or we will have to use it the other way you mentioned i.e, cross_val_score(make_pipeline(SearchLight(Ridge(), scoring=roc_auc_score), X, y). I think using the second option is more viable than creating many custom scorers.
Also as we have used make_scorer it should know whether it requires proba or decision
function or just predict.
Let me know what you think.

kingjr · 2016-09-15T08:34:44Z

kaichogami · 2016-09-15T19:16:50Z

Apologies @kingjr, the above design wouldn't work as well. We are now returning one value per fold, however we still have an array of score, one float for each estimator and this line does not allow score to be anything other than a float, which makes sense. Error output

Traceback (most recent call last):
  File "/home/kaichogami/codes/dev_mne/env/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/kaichogami/codes/dev_mne/mne-python/mne/utils.py", line 791, in dec
    return function(*args, **kwargs)
  File "/home/kaichogami/codes/dev_mne/mne-python/mne/decoding/tests/test_search_light.py", line 66, in test_searchlight
    assert_array_equal(cross_val_score(make_pipeline(sl), X, y), score)
  File "/home/kaichogami/codes/dev_mne/scikit-learn/sklearn/cross_validation.py", line 1480, in cross_val_score
    for train, test in cv)
  File "/home/kaichogami/codes/dev_mne/scikit-learn/sklearn/externals/joblib/parallel.py", line 800, in __call__
    while self.dispatch_one_batch(iterator):
  File "/home/kaichogami/codes/dev_mne/scikit-learn/sklearn/externals/joblib/parallel.py", line 658, in dispatch_one_batch
    self._dispatch(tasks)
  File "/home/kaichogami/codes/dev_mne/scikit-learn/sklearn/externals/joblib/parallel.py", line 566, in _dispatch
    job = ImmediateComputeBatch(batch)
  File "/home/kaichogami/codes/dev_mne/scikit-learn/sklearn/externals/joblib/parallel.py", line 180, in __init__
    self.results = batch()
  File "/home/kaichogami/codes/dev_mne/scikit-learn/sklearn/externals/joblib/parallel.py", line 72, in __call__
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "/home/kaichogami/codes/dev_mne/scikit-learn/sklearn/cross_validation.py", line 1593, in _fit_and_score
    test_score = _score(estimator, X_test, y_test, scorer)
  File "/home/kaichogami/codes/dev_mne/scikit-learn/sklearn/cross_validation.py", line 1653, in _score
    % (str(score), type(score)))
ValueError: scoring must return a number, got [ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.] (<type 'numpy.ndarray'>) instead.

The only solution I can think of is to take average of scores, which could be a parameter in constructor. This should be made true while using with cross_val_score.
`cross_val_score(SearchLight(Ridge(), scoring=roc_auc_score, average=True), X, y)

kingjr · 2016-09-16T13:51:21Z

I think we have to dissociate three problems:

parametrize the scoring within the SearchLight (independently of the cross val score) i.e; SearchLight(LogisiticRegression(), scorer=roc_auc_score); this should be close to what you already wrote
Deal with multiclass AUC: this is outside this PR, and will be addressed in sklearn (Support for multi-class roc_auc scores scikit-learn/scikit-learn#3298)
make SearchLight compatible with cross_val_score, which expects float scores, not array. I think that may be a broader issue though, as it's similar to i.e. get a confusion matrix in a cross_val_score.

I recommend to solve 1, and leave 2 and 3 for now.

kaichogami · 2016-09-17T16:38:52Z

@kingjr I have removed the cross_val_score example and only solved the case 1 as mentioned in your previous comment.
I am not sure what is causing the travis and appveyor error.

kingjr · 2016-09-19T13:24:41Z

Not sure what s going on with Travis. Can you repush with a forced set-upstream?

kaichogami · 2016-09-20T09:33:38Z

Pushing again didn't help. Perhaps @Eric89GXL has some idea?

larsoner · 2016-09-20T16:11:29Z

mne/decoding/search_light.py

@@ -4,6 +4,8 @@

 import numpy as np

+from sklearn.metrics import make_scorer


The Travis error says:

Found un-nested sklearn import")

In other words, this needs to be nested

@Eric89GXL Thanks for your help!

codecov-io · 2016-09-21T06:47:42Z

Current coverage is 87.35% (diff: 96.55%)

No coverage report found for master at 577d5fa.

Powered by Codecov. Last update 577d5fa...f164fa1

kaichogami · 2016-09-21T07:32:16Z

I added the import in the relevant methods rather than using global imports, which solved the nested import issue.

kingjr · 2016-09-21T09:57:23Z

mne/decoding/search_light.py

@@ -19,6 +19,9 @@ class SearchLight(BaseEstimator, TransformerMixin):
    ----------
    base_estimator : object
        The base estimator to iteratively fit on a subset of the dataset.
+    scoring : callable, defaults to None


can you make it accept strings as well, similar to sklearn cross_val_score

kingjr · 2016-09-21T09:58:08Z

mne/decoding/search_light.py

+
+        if not (scoring is None or hasattr(scoring, '__call__')):
+            raise ValueError("scoring must be None or a callable type")
+


remove extra line

kingjr · 2016-09-21T09:59:50Z

mne/decoding/search_light.py

+        # generated by estimator.score(). Else, we must first get the
+        # predictions based on the scorer.
+        self.scoring = (make_scorer(self.scoring) if self.scoring is not None
+                        else self.scoring)


object parameters should not be change.

You could store the scorer in self.scorer_, but I think we need not store the scorer, WDYT @agramfort ?

I am now making all scorer checks in __init__ which also handles string.

kingjr · 2016-09-21T10:01:23Z

mne/decoding/search_light.py

+        self.n_jobs = n_jobs
+
+        if not isinstance(self.n_jobs, int):
+            raise ValueError('n_jobs must be int, got %s' % n_jobs)


Why did you add the init? GeneralizationLight inherits from Searchlight and should have the same init params no?

It would otherwise result in error for the doc string for the scorer parameter, as in GeneralizationLight we have not modified it to change scorer.

You shoudlnt change this init, but also add a scoring parameter to GeneralizationLight

kingjr · 2016-09-21T10:02:30Z

mne/decoding/tests/test_search_light.py

@@ -55,6 +57,13 @@ def test_searchlight():
    assert_true(np.sum(np.abs(score)) != 0)
    assert_true(score.dtype == float)

+    # change score method


add test where

scoring='foo'

check that sl.scoring == default

kingjr · 2016-09-21T10:04:24Z

mne/decoding/tests/test_search_light.py

+    # change score method
+    sl = SearchLight(LogisticRegression(), scoring=roc_auc_score)
+    sl.fit(X, y)
+    score = sl.score(X, y)


add explicit test that check the score

assert_equal(score[0], roc_auc_score(y, LogisticRegression().fit(X,y).predict_proba(X)[:, 1))

kaichogami · 2016-09-23T04:47:18Z

mne/decoding/search_light.py

@@ -30,10 +33,24 @@ def __repr__(self):
            repr_str += ', fitted with %i estimators' % len(self.estimators_)
        return repr_str + '>'

-    def __init__(self, base_estimator, n_jobs=1):
+    def __init__(self, base_estimator, scoring=None, n_jobs=1):
+        from sklearn.metrics import make_scorer, get_scorer


@kingjr Is it alright to add imports in __init__ of class?

No see below

kaichogami · 2016-09-23T07:08:22Z

@kingjr please have a look at the changes.
The travis error looks unrelated to the PR changes.

kingjr · 2016-09-23T10:53:54Z

mne/decoding/search_light.py

+        self.n_jobs = n_jobs
+
+        if not isinstance(self.n_jobs, int):
+            raise ValueError('n_jobs must be int, got %s' % n_jobs)


You shoudlnt change this init, but also add a scoring parameter to GeneralizationLight

kingjr · 2016-09-23T10:55:18Z

mne/decoding/search_light.py

@@ -30,10 +33,24 @@ def __repr__(self):
            repr_str += ', fitted with %i estimators' % len(self.estimators_)
        return repr_str + '>'

-    def __init__(self, base_estimator, n_jobs=1):
+    def __init__(self, base_estimator, scoring=None, n_jobs=1):
+        from sklearn.metrics import make_scorer, get_scorer


No see below

kingjr · 2016-09-23T10:57:05Z

mne/decoding/search_light.py

+
+        elif self.scoring is not None:
+            self.scoring = get_scorer(self.scoring)
+


Don't change the param at init, this will break the cloning behavior.

get_scorer and make_scorer should be called at .score() not init

kingjr

thanks @kaichogami , only some docstring fixes, and I'll merge if @agramfort is ok.

Note that we will temporarily privatize this enhancement next week to ensure that SearchLight isn't in the next release.

kingjr · 2016-09-25T13:52:56Z

mne/decoding/search_light.py

@@ -328,6 +345,10 @@ def _sl_score(estimators, X, y):
    X : array, shape (n_samples, nd_features, n_estimators)
        The target data. The feature dimension can be multidimensional e.g.
        X.shape = (n_samples, n_features_1, n_features_2, n_estimators)
+    scoring : callable or None


kingjr · 2016-09-25T13:53:33Z

mne/decoding/search_light.py

@@ -375,11 +399,13 @@ class GeneralizationLight(SearchLight):
    ----------
    base_estimator : object
        The base estimator to iteratively fit on a subset of the dataset.
+    scoring : callable, string, defaults to None


scoring : callable | string | None

kingjr · 2016-09-25T18:09:50Z

Do you want to give it a look @agramfort ?

agramfort · 2016-09-25T19:25:52Z

LGTM

@kingjr merge if you're happy

fix the docstrings in master if needed or better in your next "privatization" PR

kingjr · 2016-09-25T21:54:57Z

I'll fix the docstring in the "privatizing" PR ;)

kingjr · 2016-09-25T22:34:15Z

Thanks @kaichogami !

kaichogami · 2016-09-26T01:59:35Z

@kingjr You are welcome! What is "privatizing" PR that is being talked about?

kingjr · 2016-09-26T02:05:44Z

We re making a release next week, but some of the recent decoding contribs
may be too unstable at the moment. I will temporarily make these changes
private until next week's release and put them back afterwards to give us
another 6 months to ensure the stability of the API.

Le 25 sept. 2016 9:59 PM, "Asish Panda" notifications@github.com a écrit :

@kingjr https://github.com/kingjr You are welcome! What is
"privatizing" PR that is being talked about?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3502 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEp7DIdhfjevhPzlmpYltEfx9qIgwZ1iks5qtycIgaJpZM4JfX2t
.

kaichogami reviewed Aug 8, 2016
View reviewed changes

kingjr mentioned this pull request Aug 9, 2016

ENH: decoding module 2017 #3442

Open

38 tasks

kaichogami reviewed Aug 10, 2016
View reviewed changes

kaichogami added 3 commits September 17, 2016 20:35

Added functionailty to change score method of searchlight

4de4e08

changed doc

b68973d

Used make scorer to change the type of prediction

179da28

kaichogami force-pushed the searchlight_score branch 2 times, most recently from 636d666 to e19ed6a Compare September 17, 2016 16:29

kaichogami force-pushed the searchlight_score branch from e19ed6a to 20a3267 Compare September 20, 2016 06:53

larsoner reviewed Sep 20, 2016

View reviewed changes

removed cross_val_score

029c716

kaichogami force-pushed the searchlight_score branch from b948831 to 029c716 Compare September 21, 2016 03:11

Resolved travis error

e419129

kingjr suggested changes Sep 21, 2016

View reviewed changes

Addressed review change

f84e9e8

kaichogami commented Sep 23, 2016

View reviewed changes

Pep8 issues

c74064d

kingjr suggested changes Sep 23, 2016

View reviewed changes

addressed comments

0bf0626

kaichogami force-pushed the searchlight_score branch from 24b48f0 to 0bf0626 Compare September 25, 2016 03:52

kingjr reviewed Sep 25, 2016

View reviewed changes

doc string changes

f164fa1

kingjr approved these changes Sep 25, 2016

View reviewed changes

kingjr changed the title ~~[WIP] Functionality to change scoring method of searchlight~~ MRG: Functionality to change scoring method of searchlight Sep 25, 2016

kingjr merged commit a294c33 into mne-tools:master Sep 25, 2016

kingjr mentioned this pull request Dec 11, 2016

MRG: scoring in generalization light #3833

Merged

kingjr mentioned this pull request Dec 22, 2016

ENH: specify scoring in search light #3475

Closed

kingjr mentioned this pull request Jan 20, 2017

MRG: search light n_jobs wrong split #3924

Merged

		@@ -4,6 +4,8 @@

		import numpy as np

		from sklearn.metrics import make_scorer


		if not (scoring is None or hasattr(scoring, '__call__')):
		raise ValueError("scoring must be None or a callable type")


		elif self.scoring is not None:
		self.scoring = get_scorer(self.scoring)

MRG: Functionality to change scoring method of searchlight #3502

MRG: Functionality to change scoring method of searchlight #3502

Uh oh!

Conversation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Current coverage is 87.35% (diff: 96.55%)

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment