8000 Difference in scores returned by cross_val_score when n_jobs is set and when not set? · Issue #13679 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

Difference in scores returned by cross_val_score when n_jobs is set and when not set? #13679

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
NagabhushanS opened this issue Apr 19, 2019 · 2 comments
Closed

Difference in scores returned by cross_val_score when n_jobs is set and when not set? #13679

NagabhushanS opened this issue Apr 19, 2019 · 2 comments

Comments

@NagabhushanS
Copy link
NagabhushanS commented Apr 19, 2019 •
edited
Loading

I came across a weird issue while cross validating my lightgbm model using a TimeSeriesSplit cv.
Following is the sample code:

model1 = LGBMClassifier(random_state=7)
scores1 = cross_val_score(model1, X, y, cv=TimeSeriesSplit(5), scoring='roc_auc')

model2 = LGBMClassifier(random_state=7)
scores2 = cross_val_score(model2, X, y, cv=TimeSeriesSplit(5), scoring='roc_auc', n_jobs=-1)

Here scores1.mean() is not equal to scores2.mean(). How is this possible? Is it that when n_jobs is None, it is uses warm_state to be True for subsequent TimeSeriesSplit folds and when n_jobs = -1, it has to train all models parallel and hence cannot reuse the parameters of previous split folds?
@NagabhushanS NagabhushanS changed the title Difference is scores returned by cross_val_score when n_jobs is set and when not set? Difference in scores returned by cross_val_score when n_jobs is set and when not set? Apr 19, 2019
@agramfort
Copy link
Member
agramfort commented Apr 19, 2019 via email

@jnothman
Copy link
Member
6DBD

That was my assumption too. I've not yet been able to confirm it with the lgbm code. If the random state is not seeded separately in each process, it won't give the same results.
Try asking at lightgbm. It's unlikely to be a big on our side as such.

@NicolasHug NicolasHug mentioned this issue Nov 17, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jnothman @agramfort @NagabhushanS
0