8000 Name of test-fold results in GridSearchCV.results_ · Issue #7205 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

Name of test-fold results in GridSearchCV.results_ #7205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
amueller opened this issue Aug 18, 2016 · 14 comments
Closed

Name of test-fold results in GridSearchCV.results_ #7205

amueller opened this issue Aug 18, 2016 · 14 comments

Comments

@amueller
Copy link
Member

I forgot if we discussed this before, but I'm slightly uncertain about the naming in results_.
Before, we used mean_validation_score, now it's test_mean_score. I'm wondering if it should be test or validation. I would call this the mean cross-validation score (I guess I named the previous named tuple).

After all, it's called cross-validation, not cross-testing ;)

ping @jnothman @raghavrv @GaelVaroquaux

I don't feel very strongly but I think it deserves some consideration (I just saw me using it in the book and it felt strange).
@GaelVaroquaux
Copy link
Member
GaelVaroquaux commented Aug 18, 2016 via email

@raghavrv
Copy link
Member

It's inside the model_selection which is not released yet.

mean_validation_score / split0_validation_score Sounds okay to me... @jnothman?

@amueller
Copy link
Member Author

Yeah it's not released so we can still change it.

@jnothman
Copy link
Member

It's not only inside model_selection, it's part of a new GridSearch results format where the previous grid_scores_ was deprecated. It's certainly up for question and perhaps.

I think it'll make sense to have similar output from cross_val_score at some point soon (for multiple metric support and to allow it to return times and training scores). Whether or not this is "development/validation test score" or "final test score" is a matter of context. It's also a matter of difference across academic fields, I suspect. I think test_ is preferable for brevity, but I'm ambivalent, appreciating that we don't want people to misuse validation. We could make it cv_.

In terms of the ordering of components (scope/aggregate, train vs test, metric), I think these are the factors to take into consideration:

  • memorability for the user (although I'm finding my contribution of dict/dataframe key completion to IPython remarkably useful on this front)
  • avoiding ambiguity for readers of code (e.g. if we landed up with "precision" instead of "score" is "mean_precision" going to be confusing?)
  • usefulness of sort order, so that comparable things are likely to appear next to each other

On the basis of all of these things, I'd be okay seeing the scope/aggregate ("mean", "split0", ...) come first, as you suggest @amueller, and frankly can't recall (and the thread was too long!) why we did not land up choosing that (except perhaps impatience).

@amueller
Copy link
Member Author

@jnothman so you vote changing the word order, and +0 for keeping test?

Maybe @agramfort has a preference? (I don't want to bug @GaelVaroquaux too much with naming things ;) [TLDR this is a new name, we can pick whatever we want].

@amueller
Copy link
Member Author

I'm not sure I'm a fan of "cv". Maybe I have to sleep over it.

@jnothman
Copy link
Member

I suppose that's my vote. I'm only +1/2 on changing word order!

@agramfort
Copy link
Member

I would prefer scores_ to results_ (it's output of our scorers).

I am +0 on adding a word eg test_scores_

@raghavrv
Copy link
Member

I would prefer scores_ to results_ (it's output of our scorers).

If you are talking about the GridSearchCV(...).results_ attribute, I thought we agreed on renaming it cv_results_?

@jnothman
Copy link
Member

It's not just scores that go in there. There are also standard deviations
over scores and parameter settings and, soon, corresponding timings.

On 27 August 2016 at 19:55, Raghav RV notifications@github.com wrote:

I would prefer scores_ to results_ (it's output of our scorers).

If you are talking about the GridSearchCV(...).results_ attribute, I
thought we agreed on renaming it cv_results_?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7205 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz6yqpSZ_04hhpQB8reIaHLRl_0kVLks5qkAmugaJpZM4JnnzR
.

@raghavrv
Copy link
Member

@amueller @jnothman So the consensus is to rename it to split0_test_score and mean_test_score correct?

@raghavrv
Copy link
Member

I'll proceed? It would be painful to deprecate the dict keys... Let's decide and do it before 0.18...

@amueller
Copy link
Member Author

ok so we go ahead with cv_results_ and mean_test_score? (so more human-readable, but still called test?)

@jnothman
Copy link
Member

Sure, let's do it.

On 1 September 2016 at 04:02, Raghav RV notifications@github.com wrote:

I'll proceed? It would be painful to deprecate the dict keys... Let's
decide and do it before 0.18...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7205 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz607Tcb1X3y7rL-9lg3CNHpCU0RaFks5qlcG1gaJpZM4JnnzR
.

@raghavrv raghavrv mentioned this issue Sep 1, 2016
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jnothman @agramfort @GaelVaroquaux @amueller @raghavrv
0