cross_val_score issue with n_jobs = -1 on Windows #13228

rfernandezv · 2019-02-22T19:03:04Z

Description

The error is thrown when utilizing n_jobs = -1 with the function: cross_val_score. If I use n_jobs = 1, it works fine.

Steps/Code to Reproduce

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

dataset = pd.read_csv('example.csv')
X = dataset.iloc[:, 3:13].values    
y = dataset.iloc[:, 13].values      

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder_X_1 = LabelEncoder()
X[:, 1] = labelencoder_X_1.fit_transform(X[:, 1])  
labelencoder_X_2 = LabelEncoder()
X[:, 2] = labelencoder_X_2.fit_transform(X[:, 2])   

onehotencoder = OneHotEncoder(categorical_features = [1])
X = onehotencoder.fit_transform(X).toarray()
X = X[:, 1:]

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score
from keras.models import Sequential
from keras.layers import Dense


def build_classifier():
    classifier = Sequential()
    classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu', input_dim = 11))
    classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu'))
    classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))
    classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
    return classifier
classifier = KerasClassifier(build_fn = build_classifier, batch_size = 10, epochs = 100)
accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10, n_jobs = -1)

Expected Results

Expect my example to run multiple epochs at a time.

Actual Results

Error:

accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10, n_jobs = -1)
Traceback (most recent call last):

  File "<ipython-input-4-cc51c2d2980a>", line 1, in <module>
    accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10, n_jobs = -1)

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py", line 402, in cross_val_score
    error_score=error_score)

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py", line 240, in cross_validate
    for train, test in cv.split(X, y, groups))

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\externals\joblib\parallel.py", line 930, in __call__
    self.retrieve()

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\externals\joblib\parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\externals\joblib\_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)

  File "C:\Users\Richar\Anaconda3\lib\concurrent\futures\_base.py", line 432, in result
    return self.__get_result()

  File "C:\Users\Richar\Anaconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.

I think the problem is related to Windows, Keras and joblib

Versions

I'm using:

64-bit Windows 10
Python 3.7.1
scikit-learn 0.20.1
Spyder 3.3.3

Thanks for the help.

The text was updated successfully, but these errors were encountered:

jnothman · 2019-02-24T00:31:26Z

Put build_classifier in a separate module and import it

christian-steinmeyer · 2019-02-25T08:11:15Z

May I ask how this fixes the issue and is this intended behavior?

albertcthomas · 2019-02-25T09:30:07Z

@rfernandezv or anyone else having this error, could you please try @jnothman's solution and tell us if this solves the issue?

christian-steinmeyer · 2019-02-25T10:57:54Z

For me, this seems to do the trick. My code didn't run through yet (total runtime is many hours), but so far no errors (which was the case before).

jnothman · 2019-02-25T11:03:59Z

Something can only be unpickled if it can be imported by name.

albertcthomas · 2019-02-25T16:52:11Z

This should work with cloudpickle, maybe with the next release of joblib.

rfernandezv · 2019-02-26T00:57:35Z

@rfernandezv or anyone else having this error, could you please try @jnothman's solution and tell us if this solves the issue?

I tried but I still get the same error.

from newFile import build_classifier_function
# Evaluating the ANN

classifier = KerasClassifier(build_fn = build_classifier_function, batch_size = 10, epochs = 100)
accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10, n_jobs = -1)

Error:

Traceback (most recent call last):

  File "<ipython-input-50-cc51c2d2980a>", line 1, in <module>
    accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10, n_jobs = -1)

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py", line 402, in cross_val_score
    error_score=error_score)

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py", line 240, in cross_validate
    for train, test in cv.split(X, y, groups))

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\externals\joblib\parallel.py", line 930, in __call__
    self.retrieve()

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\externals\joblib\parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\externals\joblib\_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)

  File "C:\Users\Richar\Anaconda3\lib\concurrent\futures\_base.py", line 432, in result
    return self.__get_result()

  File "C:\Users\Richar\Anaconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.

Updated to version 0.13.2 (joblib) and still have the same error.
Only works for me with n_jobs = 1, no -1

albertcthomas · 2019-02-26T10:26:43Z

cross_val_score is using the version of joblib vendored with scikit-learn so updating joblib won't solve the error, you need to wait for the update of the vendored joblib to try it in your case.

However I don't know why you still have the error when importing from another module.

tomMoral · 2019-03-07T14:19:36Z

I could not reproduce your bug with the latest versiono of sklearn.
Could you update your version to 0.20.2 and report if there is any changes?

For the record, this should work even if you do not put the build_classifier function in a new module with newest versions of joblib as we are relying on cloudpickle.

rfernandezv · 2019-03-08T02:13:32Z

I could not reproduce your bug with the latest versiono of sklearn.
Could you update your version to 0.20.2 and report if there is any changes?

For the record, this should work even if you do not put the build_classifier function in a new module with newest versions of joblib as we are relying on cloudpickle.

I tried but the same error :(

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.

Updated to version 0.20.3

import sklearn print(sklearn.__version__) 0.20.3

tomMoral · 2019-03-08T06:39:02Z

This is strange. Can you verify that you paste the full error traceback. There should be an extra indication with `0.20.3` that give you the error that happened at unpickle. (Above the `BrokenProcessPool`) Le ven. 8 mars 2019 à 03:14, Richar Marvin Fernandez Vilchez < notifications@github.com> a écrit :

…

I could not reproduce your bug with the latest versiono of sklearn. Could you update your version to 0.20.2 and report if there is any changes? For the record, this should work even if you do not put the build_classifier function in a new module with newest versions of joblib as we are relying on cloudpickle. I tried but the same error :( BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable. Updated to version 0.20.3 import sklearn print(sklearn.__version__) 0.20.3 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#13228 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADKs-fW9_A7dw5F-K_fEFqrmIV2yQQIPks5vUcd-gaJpZM4bKNAb> .

rfernandezv · 2019-03-09T01:25:56Z

Look, this is the complete error:

accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10, n_jobs = -1)
Traceback (most recent call last):

  File "<ipython-input-6-cc51c2d2980a>", line 1, in <module>
    accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10, n_jobs = -1)

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py", line 402, in cross_val_score
    error_score=error_score)

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py", line 240, in cross_validate
    for train, test in cv.split(X, y, groups))

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\externals\joblib\parallel.py", line 930, in __call__
    self.retrieve()

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\externals\joblib\parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\externals\joblib\_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)

  File "C:\Users\Richar\Anaconda3\lib\concurrent\futures\_base.py", line 432, in result
    return self.__get_result()

  File "C:\Users\Richar\Anaconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.

albertcthomas · 2019-03-09T22:10:59Z

@rfernandezv I could not reproduce the error either with the following code on windows

from keras.layers import Dense
from keras.models import Sequential
from sklearn.model_selection import cross_val_score
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

from sklearn.datasets import make_classification

X, y = make_classification(n_features=11)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)


def build_classifier():
    classifier = Sequential()
    classifier.add(Dense(units=6, kernel_initializer='uniform',
                         activation='relu', input_dim=11))
    classifier.add(
        Dense(units=6, kernel_initializer='uniform', activation='relu'))
    classifier.add(
        Dense(units=1, kernel_initializer='uniform', activation='sigmoid'))
    classifier.compile(
        optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
    return classifier


classifier = KerasClassifier(
    build_fn=build_classifier, batch_size=10, epochs=100)
accuracies = cross_val_score(
    estimator=classifier, X=X_train, y=y_train, cv=10, n_jobs=-1)

The versions that I'm using:
Windows 10
Python 3.6.6
NumPy 1.15.4
SciPy 1.2.0
Scikit-Learn 0.20.2
Keras 2.2.4
Tensorflow 1.12.0

olgazju · 2019-03-11T12:48:28Z

I have the same error with cross_val_score and n_jobs=1

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.

Python 3.7.2
scikit-learn==0.20.3
joblib==0.13.2
windows 8.1

8000

christian-steinmeyer · 2019-03-19T12:55:19Z

I've ran into this problem again, in a slightly different form using scikit-learn 0.20.3:
I've set up a grid search in a for loop (in dependence of a balance strategy, as I have imbalanced data). Not, interestingly, this runs through smoothly when executed for the first time, however upon the second run, I get a similar error to above.

Here's a narrowed down version of my code:

from other.module import multi_metric_score, create_model, early_stopping

# ... define parameters

model = KerasClassifier(build_fn=create_model, epochs=100, batch_size=256, verbose=0)
multi_metric_scorer = make_scorer(multi_metric_score, greater_is_better=True)

for balance_strategy, _ in balance_strategies.items():
    # ... get X, y in dependence of balance_strategy

    skfs = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
    classifier = GridSearchCV(estimator=model, param_grid=parameters, cv=skfs, scoring=multi_metric_scorer, verbose=1, n_jobs=-1,
                              return_train_score=True)
    # ... define ratio
    results = classifier.fit(X, y, callbacks=early_stopping, class_weight=[ratio, 1.0-ratio])
    data = pd.DataFrame(results.cv_results_['params'])
    data['mean_test_score'] = results.cv_results_['mean_test_score']
    data['mean_fit_time'] = results.cv_results_['mean_fit_time']
    data['balance_strategy'] = balance_strategy

And here is the console output including the complete stack trace:

....    
Fitting 3 folds for each of 1 candidates, totalling 3 fits
[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
Using TensorFlow backend.
Using TensorFlow backend.
Using TensorFlow backend.
...
[Parallel(n_jobs=-1)]: Done   3 out of   3 | elapsed:   12.3s finished
2019-03-19 13:39:38.850213: I tensorflow/core/common_runtime/process_util.cc:71] Creating new thread pool with default inter op setting: 4. Tune using inter_op_parallelism_threads for best performance.
Fitting 3 folds for each of 1 candidates, totalling 3 fits
[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
sklearn.externals.joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\site-packages\sklearn\externals\joblib\externals\loky\backend\queues.py", line 150, in _feed
    obj_ = dumps(obj, reducers=reducers)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\site-packages\sklearn\externals\joblib\externals\loky\backend\reduction.py", line 243, in dumps
    dump(obj, buf, reducers=reducers, protocol=protocol)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\site-packages\sklearn\externals\joblib\externals\loky\backend\reduction.py", line 236, in dump
    _LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\site-packages\sklearn\externals\joblib\externals\cloudpickle\cloudpickle.py", line 284, in dump
    return Pickler.dump(self, obj)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 409, in dump
    self.save(obj)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 852, in _batch_setitems
    save(v)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 781, in save_list
    self._batch_appends(obj)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 808, in _batch_appends
    save(tmp[0])
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 736, in save_tuple
    save(element)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 781, in save_list
    self._batch_appends(obj)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 808, in _batch_appends
    save(tmp[0])
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 781, in save_list
    self._batch_appends(obj)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 808, in _batch_appends
    save(tmp[0])
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 781, in save_list
    self._batch_appends(obj)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 805, in _batch_appends
    save(x)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Pro
8000
gramData\Miniconda\envs\nnmp\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 634, in save_reduce
    save(state)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\pickle.py", line 496, in save
    rv = reduce(self.proto)
TypeError: can't pickle _thread.RLock objects
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:/Users/chris/Documents/Academics/Uni/Georg-August-University/Computer Science/Masterarbeit/code/src/model-grid-search.py", line 42, in <module>
    results = classifier.fit(X, y, callbacks=early_stopping, class_weight=[ratio, 1.0-ratio])
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\site-packages\sklearn\model_selection\_search.py", line 722, in fit
    self._run_search(evaluate_candidates)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\site-packages\sklearn\model_selection\_search.py", line 1191, in _run_search
    evaluate_candidates(ParameterGrid(self.param_grid))
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\site-packages\sklearn\model_selection\_search.py", line 711, in evaluate_candidates
    cv.split(X, y, groups)))
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\site-packages\sklearn\externals\joblib\parallel.py", line 930, in __call__
    self.retrieve()
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\site-packages\sklearn\externals\joblib\parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\site-packages\sklearn\externals\joblib\_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\concurrent\futures\_base.py", line 432, in result
    return self.__get_result()
  File "C:\ProgramData\Miniconda\envs\nnmp\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
_pickle.PicklingError: Could not pickle the task to send it to the workers.

rfernandezv · 2019-03-20T01:29:10Z

@albertcthomas, I ran your code but it does not work

Traceback (most recent call last):

  File "<ipython-input-5-8e611add1011>", line 2, in <module>
    estimator=classifier, X=X_train, y=y_train, cv=10, n_jobs=-1)

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py", line 402, in cross_val_score
    error_score=error_score)

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py", line 240, in cross_validate
    for train, test in cv.split(X, y, groups))

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\externals\joblib\parallel.py", line 930, in __call__
    self.retrieve()

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\externals\joblib\parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))

  File "C:\Users\Richar\Anaconda3\lib\site-packages\sklearn\externals\joblib\_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)

  File "C:\Users\Richar\Anaconda3\lib\concurrent\futures\_base.py", line 432, in result
    return self.__get_result()

  File "C:\Users\Richar\Anaconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.

shahabsh92 · 2019-04-26T11:03:46Z

Hi, I have exactly the same issue like by @rfernandezv
I get the same error if I use any other number than 1 for n_jobs in the cross_val_score function.
The Code is the same.

Python 3.7.1
Keras 2.2.4
TensorFlow 1.13.1
scipy 1.1.0
Numpy 1.15.4
Sklearn 0.20.2

rfernandezv · 2019-04-27T00:56:57Z

I have not found the solution yet. @shahabsh92 , did you try to update Sklearn version?
Updated to version 0.20.3 but it did not work

kush-daga · 2019-04-27T09:58:50Z

Has anyone been able to solve this issue, as im getting the same error in conda envt with python 3.7

shahabsh92 · 2019-04-27T16:27:12Z

@rfernandezv yes, I created another environment in the Anaconda with the latest version of sklearn 0.20.3 + keras-gpu and it didnt work again. It works only with n_jobs = 1 and nothing else.

albertcthomas · 2019-04-29T09:03:20Z

@shahabsh92 you mean that you have the same error as @rfernandezv when you run the code in this comment?

kush-daga · 2019-04-29T09:19:14Z

I feel it's an issue with the ipython console in spyder, i ran the same code in terminal of vscode using the python console and also in jupyter.. It worked.

nicholasg97 · 2019-05-01T07:47:59Z

I feel it's an issue with the ipython console in spyder, i ran the same code in terminal of vscode using the python console and also in jupyter.. It worked.

I concur, I'm having the same issue with spyder. Restarting the kernel doesn't work...

shahabsh92 · 2019-05-01T10:19:35Z

@albertcthomas yes exactly. I get both on spyder and Jupiter Errors. Both BrokenProcessPool like @rfernandezv.

That's the Error on IPython console of Spyder:

`exception calling callback for <Future at 0x2139edbbe10 state=finished raised BrokenProcessPool>
sklearn.externals.joblib.externals.loky.process_executor.RemoteTraceback:
'''
Traceback (most recent call last):
File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\externals\loky\process_executor.py", line 393, in process_worker
call_item = call_queue.get(block=True, timeout=timeout)
File "D:\Softwares\Anacoda\lib\multiprocessing\queues.py", line 113, in get
return ForkingPickler.loads(res)
File "D:\Softwares\Anacoda\lib\site-packages\keras_init.py", line 3, in
from . import utils
File "D:\Softwares\Anacoda\lib\site-packages\keras\utils_init.py", line 6, in
from . import conv_utils
File "D:\Softwares\Anacoda\lib\site-packages\keras\utils\conv_utils.py", line 9, in
from .. import backend as K
File "D:\Softwares\Anacoda\lib\site-packages\keras\backend_init.py", line 88, in
sys.stderr.write('Using TensorFlow backend.\n')
AttributeError: 'NoneType' object has no attribute 'write'
'''

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\externals\loky_base.py", line 625, in _invoke_callbacks
callback(self)
File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 375, in call
self.parallel.dispatch_next()
File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 797, in dispatch_next
if not self.dispatch_one_batch(self._original_iterator):
File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 825, in dispatch_one_batch
self._dispatch(tasks)
File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 782, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib_parallel_backends.py", line 506, in apply_async
future = self._workers.submit(SafeFunction(func))
File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\externals\loky\reusable_executor.py", line 151, in submit
fn, *args, **kwargs)
File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\externals\loky\process_executor.py", line 1016, in submit
raise self._flags.broken
sklearn.externals.joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.
Traceback (most recent call last):

File "", line 4, in
estimator=classifier, X=X_train, y=y_train, cv=10, n_jobs=-1)

File "D:\Softwares\Anacoda\lib\site-packages\sklearn\model_selection_validation.py", line 402, in cross_val_score
error_score=error_score)

File "D:\Softwares\Anacoda\lib\site-packages\sklearn\model_selection_validation.py", line 240, in cross_validate
for train, test in cv.split(X, y, groups))

File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 996, in call
self.retrieve()

File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 899, in retrieve
self._output.extend(job.get(timeout=self.timeout))

File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib_parallel_backends.py", line 517, in wrap_future_result
return future.result(timeout=timeout)

File "D:\Softwares\Anacoda\lib\concurrent\futures_base.py", line 405, in result
return self.__get_result()

File "D:\Softwares\Anacoda\lib\concurrent\futures_base.py", line 357, in __get_result
raise self._exception

File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\externals\loky_base.py", line 625, in _invoke_callbacks
callback(self)

File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 375, in call
self.parallel.dispatch_next()

File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 797, in dispatch_next
if not self.dispatch_one_batch(self._original_iterator):

File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 825, in dispatch_one_batch
self._dispatch(tasks)

File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 782, in _dispatch
job = self._backend.apply_async(batch, callback=cb)

File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib_parallel_backends.py", line 506, in apply_async
future = self._workers.submit(SafeFunction(func))

File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\externals\loky\reusable_executor.py", line 151, in submit
fn, *args, **kwargs)

File "D:\Softwares\Anacoda\lib\site-packages\sklearn\externals\joblib\externals\loky\process_executor.py", line 1016, in submit
raise self._flags.broken

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.

Traceback (most recent call last):