how can cv folds can be more than number of groups in cross_validate? #13972

omarcr · 2019-05-29T00:47:23Z

In the following code:

from sklearn import datasets, linear_model
from sklearn.model_selection import cross_validate
from sklearn.metrics.scorer import make_scorer
from sklearn.metrics import confusion_matrix
from sklearn.svm import LinearSVC
import numpy as np

digits = datasets.load_digits()

X = digits.data[:500]
y = digits.target[:500]

counts, unique = np.unique(y, return_counts=True)

g1 = np.repeat(0, int(len(y)/2))
g2 = np.repeat(1, int(len(y)/2))

g = np.concatenate((g1, g2))

linear = LinearSVC()

cv_results = cross_validate(linear, X, y, cv=5)
sorted(cv_results.keys())
print(cv_results)


print('cv_results')
print(cv_results['test_score'])


print('cross-validate')
scores = cross_validate(linear, X, y, cv=5,
                        scoring=('f1_weighted'),
                        return_train_score=True, groups=g)
print(scores)

Shouldn't it be returned that:
groups should be equal to CV for the folds?

The text was updated successfully, but these errors were encountered:

jnothman · 2019-05-29T01:10:39Z

I don't think groups is being used in that case at all unless you specify a cv object that can utilise groups

omarcr · 2019-05-29T01:17:46Z

you mean something like cv=GroupKFold(n_splits=3) or what type of object?

jnothman · 2019-05-29T02:17:07Z

Yes. I think it would be good if the documentation of cross_validate was more explicit about this. A Pull Request clarifying the documentation would be welcome.

omarcr · 2019-05-29T15:56:52Z

I think it should be fixed in the library too. An exception should be shown if a group is declared but there is no object in the cv argument.

amueller · 2019-05-29T18:47:05Z

@omarcr I'm not sure that's easy to do in a very consistent way, and I'm also not sure it's desirable. We could raise an error any time groups is passed to a cv object that doesn't use it but that would be a pretty big change in behavior and potentially break many people's code.
If you change your CV object from a grouped one to a not-grouped one I feel like you shouldn't need to pass different data/groups to make the code work.
One option I could see is that if cv is a number we also validate groups. In a sense it's reasonable to expect the groups to be used if you pass cv=5. If you pass cv=KFold(n_splits=5) I think it's reasonable to ignore groups. But it's also a bit strange if these two cases have different behavior.

omarcr · 2019-05-29T19:40:04Z

@amueller The way I understand the method is the following:

if for example I have 3 groups of labels then there should be 3 folds. After cross validation, the 3 models built will have an independent group for testing.

If for example I don't declare the groups then the cross_validate function will just arrange the data by index to construct the 3 folds.

therefore if I pass cv=KFold(n_splits=5) I need to declare the groups labels of the data samples so cross_validate understands how to select the data for the folds.

Is this the way the method is implemented?

amueller · 2019-05-29T21:15:14Z

Groups are only used if you explicitly pass a cross-validation strategy that uses groups, so GroupKFold or LeaveOneGroupOut.
Whatever you pass to cv completely determines the behavior, and some values of cv require groups to be specified, all the others ignore it.
If you pass cv=5 for classification it will use stratified 5-fold cross-validation (so not actually sort by index, though we might be using a stable stratification strategy?).

jnothman · 2019-05-29T22:00:35Z

I agree that the logic of the code is not obvious from the API. I'd be happy to have cv=int fail if groups is passed for now. I would also consider warning if a non-groups splitter is used and groups is passed "Did you mean to use a group-based splitter?"

omarcr · 2019-05-30T15:16:34Z

Yes I agree. GroupKFold(n_splits=n) already has a routine checking for splits not to be greater than the number of folds: ValueError: Cannot have number of splits n_splits=4 greater than the number of groups: 3. maybe that can be reused here.

lucyleeow · 2024-05-08T11:49:53Z

I think this can be closed. With #28210 you now get a user warning when the splitter doesn't support group. And you will get an error if folds is greater than number of groups.

cmarmo added the module:model_selection label Feb 25, 2022

thomasjpfan moved this to Todo📬 in Quansight's scikit-learn Project Board Mar 28, 2022

thomasjpfan added this to Quansight's scikit-learn Project Board Mar 28, 2022

thomasjpfan moved this from Todo📬 to Delegate📪 in Quansight's scikit-learn Project Board Apr 10, 2022

thomasjpfan moved this from Delegate📪 to Todo📬 in Quansight's scikit-learn Project Board Apr 28, 2022

lucyleeow closed this as completed May 8, 2024

github-project-automation bot moved this from Todo📬 to Done🚀 in Quansight's scikit-learn Project Board May 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

how can cv folds can be more than number of groups in cross_validate? #13972

how can cv folds can be more than number of groups in cross_validate? #13972

how can cv folds can be more than number of groups in cross_validate? #13972

how can cv folds can be more than number of groups in cross_validate? #13972

Comments