8000 GridSearchCV cannot be paralleled when custom scoring is used · Issue #10054 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

GridSearchCV cannot be paralleled when custom scoring is used #10054

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
shuuchen opened this issue Nov 1, 2017 · 25 comments
Closed

GridSearchCV cannot be paralleled when custom scoring is used #10054

shuuchen opened this issue Nov 1, 2017 · 25 comments

Comments

@shuuchen
Copy link
shuuchen commented Nov 1, 2017

Hi,

I met a problem with the code:

    from sklearn.model_selection import GridSearchCV
    model = ensemble.RandomForestRegressor()
    param = {'n_estimators': [500, 700, 1200],
             'max_depth': [3, 5, 7],
             'max_features': ['auto'],
             'n_jobs': [-1],
             'criterion': ['mae', 'mse'],
             'random_state': [300],
             }
    from sklearn.metrics import make_scorer
    def my_custom_loss_func(ground_truth, predictions):
        diff = np.abs(ground_truth - predictions) / ground_truth
        return np.mean(diff)
    loss = make_scorer(my_custom_loss_func, greater_is_better=False)
    model_cv = GridSearchCV(model, param, cv=5, n_jobs=2, scoring=loss, verbose=1)
    model_cv.fit(X, y.ravel())

in which I used custom scoring object in GridSearchCV(...) and set n_jobs = 2.

I got the following error message:

C:\Anaconda3\python.exe C:/Users/to/PycharmProjects/Toppan/[10-24]per_machine_vapor_pred_ver2.py
Fitting 5 folds for each of 18 candidates, totalling 90 fits
Traceback (most recent call last):
  File "C:/Users/to/PycharmProjects/Toppan/[10-24]per_machine_vapor_pred_ver2.py", line 172, in <module>
    models, scas = learn_all(X_train, y_train)
  File "C:/Users/to/PycharmProjects/Toppan/[10-24]per_machine_vapor_pred_ver2.py", line 108, in learn_all
    models[machine], scas[machine] = learn_cv(X, y)
  File "C:/Users/to/PycharmProjects/Toppan/[10-24]per_machine_vapor_pred_ver2.py", line 87, in learn_cv
    model_cv.fit(X, y.ravel())
  File "C:\Anaconda3\lib\site-packages\sklearn\model_selection\_search.py", line 638, in fit
    cv.split(X, y, groups)))
  File "C:\Anaconda3\lib\site-packages\sklearn\externals\joblib\parallel.py", line 789, in __call__
    self.retrieve()
  File "C:\Anaconda3\lib\site-packages\sklearn\externals\joblib\parallel.py", line 699, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "C:\Anaconda3\lib\multiprocessing\pool.py", line 608, in get
    raise self._value
  File "C:\Anaconda3\lib\multiprocessing\pool.py", line 385, in _handle_tasks
    put(task)
  File "C:\Anaconda3\lib\site-packages\sklearn\externals\joblib\pool.py", line 371, in send
    CustomizablePickler(buffer, self._reducers).dump(obj)
AttributeError: Can't pickle local object 'learn_cv.<locals>.my_custom_loss_func'

Process finished with exit code 1

It seems that if and only if n_jobs is set to 1 can the program be run.

Any ideas?
@jnothman
Copy link
Member
jnothman commented Nov 1, 2017

This is a limitation of pickling. Define my_custome_loss_func in a module you import, or at least not in a closure.

@jnothman jnothman closed this as completed Nov 1, 2017
@fx86
Copy link
Contributor
fx86 commented Oct 9, 2018

Wow. Was wondering what was wrong with my custom loss function. Turns out, it was n_jobs=-1 that was causing this issue! Removed that and the error vanishes.

def rmsle(y_true, y_pred, **kwargs):
    return np.sqrt(mean_squared_log_error(y_true, y_pred))


X, y = train.drop(target_col, 1), train[target_col]
cv = cross_validate(pipe, X, y, 
                     scoring=make_scorer(rmsle), 
                     n_jobs=-1, 
                     cv=5)

@jnothman
Copy link
Member
jnothman commented Oct 9, 2018 via email

@fx86
Copy link
Contributor
fx86 commented Oct 9, 2018 via email

@amueller
Copy link
Member
amueller commented Oct 9, 2018

@fx86 can you please provide code to reproduce or at least the error message?

@fx86
Copy link
Contributor
fx86 commented Oct 9, 2018

Wow. Was wondering what was wrong with my custom loss function. Turns out, it was n_jobs=-1 that was causing this issue! Removed that and the error vanishes.

def rmsle(y_true, y_pred, **kwargs):
    return np.sqrt(mean_squared_log_error(y_true, y_pred))


X, y = train.drop(target_col, 1), train[target_col]
cv = cross_validate(pipe, X, y, 
                     scoring=make_scorer(rmsle), 
                     n_jobs=-1, 
                     cv=5)

@amueller Code is above.
The error message is here:

---------------------------------------------------------------------------
_RemoteTraceback                          Traceback (most recent call last)
_RemoteTraceback: 
'''
Traceback (most recent call last):
  File "/Users/fibinse/anaconda3/lib/python3.6/site-packages/sklearn/externals/joblib/externals/loky/process_executor.py", line 393, in _process_worker
    call_item = call_queue.get(block=True, timeout=timeout)
  File "/Users/fibinse/anaconda3/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'rmsle' on <module 'sklearn.externals.joblib.externals.loky.backend.popen_loky_posix' from '/Users/fibinse/anaconda3/lib/python3.6/site-packages/sklearn/externals/joblib/externals/loky/backend/popen_loky_posix.py'>
'''

The above exception was the direct cause of the following exception:

BrokenProcessPool                         Traceback (most recent call last)
<ipython-input-63-ddb7fb8c151e> in <module>()
     28                      scoring=make_scorer(rmsle),
     29                      cv=5,
---> 30                     n_jobs=-1)
     31 print(cv)

~/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_validation.py in cross_val_score(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch, error_score)
    400                                 fit_params=fit_params,
    401                                 pre_dispatch=pre_dispatch,
--> 402                                 error_score=error_score)
    403     return cv_results['test_score']
    404 

~/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_validation.py in cross_validate(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch, return_train_score, return_estimator, error_score)
    238             return_times=True, return_estimator=return_estimator,
    239             error_score=error_score)
--> 240         for train, test in cv.split(X, y, groups))
    241 
    242     zipped_scores = list(zip(*scores))

~/anaconda3/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in __call__(self, iterable)
    994 
    995             with self._backend.retrieval_context():
--> 996                 self.retrieve()
    997             # Make sure that we get a last message telling us we are done
    998             elapsed_time = time.time() - self._start_time

~/anaconda3/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in retrieve(self)
    897             try:
    898                 if getattr(self._backend, 'supports_timeout', False):
--> 899                     self._output.extend(job.get(timeout=self.timeout))
    900                 else:
    901                     self._output.extend(job.get())

~/anaconda3/lib/python3.6/site-packages/sklearn/externals/joblib/_parallel_backends.py in wrap_future_result(future, timeout)
    515         AsyncResults.get from multiprocessing."""
    516         try:
--> 517             return future.result(timeout=timeout)
    518         except LokyTimeoutError:
    519             raise TimeoutError()

~/anaconda3/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    430                 raise CancelledError()
    431             elif self._state == FINISHED:
--> 432                 return self.__get_result()
    433             else:
    434                 raise TimeoutError()

~/anaconda3/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.```

@amueller
Copy link
Member

That's not the same issue. There might be an issue open for this one already, though. cc @ogrisel ?

@jnothman
Copy link
Member

The user is providing a function that isn't picklable. Please put rmsle into a module that can be imported.

@fx86
Copy link
Contributor
fx86 commented Oct 10, 2018 via email

@jnothman
Copy link
Member

Where in the docs would you see it? Somewhere specific to scoring, even though it is relevant to all parallel processing where a custom function can be provided? Under http://scikit-learn.org/0.20/glossary.html#term-n-jobs?? http://scikit-learn.org/0.20/faq.html??

@fx86
Copy link
Contributor
fx86 commented Oct 10, 2018 via email

@jnothman
Copy link
Member
jnothman commented Oct 10, 2018 via email

@fx86
Copy link
Contributor
fx86 commented Oct 10, 2018

Yes, I would. Is there a target date you guys are looking at for this ?

@amueller
Copy link
Member

@fx86 whenever you can ;) No rush

@mblouin02
Copy link
mblouin02 commented Oct 24, 2018 •
edited
Loading

Hi,

Note that this issue is also present when using k-fold validation with scikit-learn, using function cross_val_score().

This has been reported to the sclearn team, and the target to fix it is in the next version, 0.20.1.

See here for more info: #12250

@GinoWoz1
Copy link

Thanks. Newbie question here, how would one import a module with a custom scorer? I have the custom scorer below How would I import that in a module? Hope this helps others too as well.

def rmse_cv(y_true, y_pred) :
assert len(y_true) == len(y_pred)
y_pred = np.exp(y_pred)
y_true = np.exp(y_true)
return np.sqrt(mean_squared_error(y_true,y_pred))
"""
rmse_cv = make_scorer(rmse_cv,greater_is_better=False)

@jnothman
Copy link
Member

Put rmse_cv in a file called custom.py, for instance, then from custom import rmse_cv

@GinoWoz1
Copy link
GinoWoz1 commented Nov 19, 2018 •
edited
Loading

Thanks . Also with the make_scorer ? I imported the function and put make_scorer in my script (that I imported into) and it said ‘Predict score’ object has no attribute ‘__name’

@fx86
Copy link
Contributor
fx86 commented Nov 20, 2018

@GinoWoz1 I'm using it like this. How are you using it ?

from rmsle import rmsle
RandomizedSearchCV(pipe, 
                    param_distributions=random_grid_search_params, 
                    n_iter=30, 
                    scoring=make_scorer(rmsle), 
                    n_jobs=-1, 
                    cv=5, 
                    verbose=3, 
                    random_state=1001 )

@GinoWoz1
Copy link
GinoWoz1 commented Nov 20, 2018 •
edited
Loading

Thanks @fx86 , that helped to remove the name error. But still getting the below error with n_jobs=-1

image

Put this in a separate file and import it. I can't get the code block working for the life of me...I hit the "insert code" option but it just doesnt render...sorry

Function - need to indent after def...cant get it to work

def rmse_cv(y_true, y_pred) :
assert len(y_true) == len(y_pred)
y_pred = np.exp(y_pred)
y_true = np.exp(y_true)
return np.sqrt(mean_squared_error(y_true,y_pred))

Then run the below...
Code to run
from sklearn.model_selection import cross_val_score,RepeatedKFold
from sklearn.linear_model import ElasticNet
from sklearn.preprocessing import StandardScaler
import pandas as pd
from sklearn.pipeline import Pipeline
from YOURFILE import rmse_cv
from sklearn.metrics import make_scorer

rkfold = RepeatedKFold(n_splits=5,n_repeats=5)

url = 'https://github.com/GinoWoz1/Learnings/raw/master/'

X_train = pd.read_csv(url + 'X_trainGA.csv',index_col= 'Unnamed: 0')
y_train = pd.read_csv(url +'y_trainGA.csv',header=None,index_col=0)

X_train.rename(columns={'Constant Term':'tax'},inplace=True)

elnet_final = ElasticNet()

elnet_pipe = Pipeline([('std',StandardScaler()),
('elnet',elnet_final)])

cross_val = cross_val_score(elnet_pipe,X_train,y_train,scoring=make_scorer(rmse_cv,greater_is_better=False),cv=rkfold,n_jobs=-1)

@fx86
Copy link
Contributor
fx86 commented Nov 23, 2018

@GinoWoz1 This seems to be working okay for me. I'm on the following versions -

`Python 3.6.0 :: Continuum Analytics, Inc.

scikit-image==0.13.1

scikit-learn==0.20.0`

@GinoWoz1
Copy link
GinoWoz1 commented Dec 8, 2018

Thanks, it was a package conflict error. It is working now.

@fx86
Copy link
Contributor
fx86 commented Dec 15, 2018 •
edited
Loading

@fx86 whenever you can ;) No rush

@amueller should I open an issue for this before sending in the PR ?

@jnothman
Copy link
Member

No need to open an issue first, @fx86

@fx86 fx86 mentioned this issue Dec 18, 2018
Merged
@kirilldolmatov kirilldolmatov mentioned this issue Jun 17, 2019
Closed
@vbelz
Copy link
vbelz commented Oct 16, 2019

Same issue happened to me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@jnothman @amueller @fx86 @mblouin02 @shuuchen @GinoWoz1 @vbelz
0