8000 ERROR: sklearn.tests.test_pipeline.test_feature_union_parallel · Issue #4016 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

ERROR: sklearn.tests.test_pipeline.test_feature_union_parallel #4016

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
chenyuqing opened this issue Dec 28, 2014 · 5 comments
Closed

ERROR: sklearn.tests.test_pipeline.test_feature_union_parallel #4016

chenyuqing opened this issue Dec 28, 2014 · 5 comments
Labels
Bug
Milestone
0.18

Comments

@chenyuqing
Copy link

When I test the scikit-learn installation , it occurs the error message like that:

======================================================================
ERROR: sklearn.externals.joblib.test.test_parallel.test_pre_dispatch_race_condit
ion
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\Python27\lib\site-packages\nose-1.3.4-py2.7.egg\nose\case.py", line 1
97, in runTest
    self.test(*self.arg)
  File "E:\Python27\lib\site-packages\sklearn\externals\joblib\test\test_paralle
l.py", line 335, in test_pre_dispatch_race_condition
    delayed(square)(i) for i in range(n_tasks))
  File "E:\Python27\lib\site-packages\sklearn\externals\joblib\parallel.py", lin
e 672, in __call__
    self._pool.terminate()  # terminate does a join()
  File "E:\Python27\lib\site-packages\sklearn\externals\joblib\pool.py", line 56
6, in terminate
    super(MemmapingPool, self).terminate()
  File "E:\Python27\lib\multiprocessing\pool.py", line 456, in terminate
    self._terminate()
  File "E:\Python27\lib\multiprocessing\util.py", line 207, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "E:\Python27\lib\multiprocessing\pool.py", line 504, in _terminate_pool
    p.terminate()
  File "E:\Python27\lib\multiprocessing\process.py", line 137, in terminate
    self._popen.terminate()
  File "E:\Python27\lib\multiprocessing\forking.py", line 312, in terminate
    _subprocess.TerminateProcess(int(self._handle), TERMINATE)
WindowsError: [Error 5] Access is denied

======================================================================
ERROR: Test that linear regression also works with sparse data
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\Python27\lib\site-packages\nose-1.3.4-py2.7.egg\nose\case.py", line 1
97, in runTest
    self.test(*self.arg)
  File "E:\Python27\lib\site-packages\sklearn\linear_model\tests\test_base.py",
line 78, in test_linear_regression_sparse
    ols.fit(X, y.ravel())
  File "E:\Python27\lib\site-packages\sklearn\linear_model\base.py", line 359, i
n fit
    out = lsqr(X, y)
  File "E:\Python27\lib\site-packages\scipy\sparse\linalg\isolve\lsqr.py", line
436, in lsqr
    test3 = 1 / acond
ZeroDivisionError: float division by zero

======================================================================
ERROR: sklearn.metrics.tests.test_pairwise.test_pairwise_parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\Python27\lib\site-packages\nose-1.3.4-py2.7.egg\nose\case.py", line 1
97, in runTest
    self.test(*self.arg)
  File "E:\Python27\lib\site-packages\sklearn\metrics\tests\test_pairwise.py", l
ine 128, in test_pairwise_parallel
    S2 = _parallel_pairwise(X, None, euclidean_distances, n_jobs=3)
  File "E:\Python27\lib\site-packages\sklearn\metrics\pairwise.py", line 972, in
 _parallel_pairwise
    for s in gen_even_slices(Y.shape[0], n_jobs))
  File "E:\Python27\lib\site-packages\sklearn\externals\joblib\parallel.py", lin
e 589, in __call__
    raise ImportError('[joblib] Attempting to do parallel computing '
ImportError: [joblib] Attempting to do parallel computing without protecting you
r import on a system that does not support forking. To use parallel-computing in
 a script, you must protect you main loop using "if __name__ == '__main__'". Ple
ase see the joblib documentation on Parallel for more information

======================================================================
ERROR: sklearn.tests.test_pipeline.test_feature_union_parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "E:\Python27\lib\site-packages\nose-1.3.4-py2.7.egg\nose\case.py", line 1
97, in runTest
    self.test(*self.arg)
  File "E:\Python27\lib\site-packages\sklearn\tests\test_pipeline.py", line 349,
 in test_feature_union_parallel
    fs_parallel.fit(X)
  File "E:\Python27\lib\site-packages\sklearn\pipeline.py", line 334, in fit
    for name, trans in self.transformer_list)
  File "E:\Python27\lib\site-packages\sklearn\externals\joblib\parallel.py", lin
e 589, in __call__
    raise ImportError('[joblib] Attempting to do parallel computing '
ImportError: [joblib] Attempting to do parallel computing without protecting you
r import on a system that does not support forking. To use parallel-computing in
 a script, you must protect you main loop using "if __name__ == '__main__'". Ple
ase see the joblib documentation on Parallel for more information

----------------------------------------------------------------------
Ran 3342 tests in 173.391s

FAILED (SKIP=25, errors=4)
@larsmans
Copy link
Member

More details needed. What platform? Which version? How did you install it?

@amueller
Copy link
Member

Windows, which seems to be the issue.... @ogrisel we do run continuous integration on appveyor, right? Btw, where is the output of that?

@amueller amueller added the Bug label Jan 22, 2015
@ogrisel
Copy link
Member
ogrisel commented May 25, 2016

AppVeyor tests pass:

https://ci.appveyor.com/project/sklearn-ci/scikit-learn/history

This seems to be an issue with multiprocessing (at least in Python 2.7):

http://stackoverflow.com/questions/17076679/windowserror-access-is-denied-on-calling-process-terminate

I never observed it myself when debugging stuff under windows.

@ogrisel
Copy link
Member
ogrisel commented May 25, 2016

In Python 3.5, the problem seems to be less likely has the timeout has been set to 1.0 second, here is the end of the multiprocessing/popen_spawn_win32.py file:

    def wait(self, timeout=None):
        if self.returncode is None:
            if timeout is None:
                msecs = _winapi.INFINITE
            else:
                msecs = max(0, int(timeout * 1000 + 0.5))

            res = _winapi.WaitForSingleObject(int(self._handle), msecs)
            if res == _winapi.WAIT_OBJECT_0:
                code = _winapi.GetExitCodeProcess(self._handle)
                if code == TERMINATE:
                    code = -signal.SIGTERM
                self.returncode = code

        return self.returncode

    def poll(self):
        return self.wait(timeout=0)

    def terminate(self):
        if self.returncode is None:
            try:
                _winapi.TerminateProcess(int(self._handle), TERMINATE)
            except OSError:
                if self.wait(timeout=1.0) is None:
                    raise

Under Python 2.7, the corresponding code (multiprocessing/forking.py at line 312) is not as robust as the time is 0.1s:

        def wait(self, timeout=None):
            if self.returncode is None:
                if timeout is None:
                    msecs = _subprocess.INFINITE
                else:
                    msecs = max(0, int(timeout * 1000 + 0.5))

                res = _subprocess.WaitForSingleObject(int(self._handle), msecs)
                if res == _subprocess.WAIT_OBJECT_0:
                    code = _subprocess.GetExitCodeProcess(self._handle)
                    if code == TERMINATE:
                        code = -signal.SIGTERM
                    self.returncode = code

            return self.returncode

        def poll(self):
            return self.wait(timeout=0)

        def terminate(self):
            if self.returncode is None:
                try:
                    _subprocess.TerminateProcess(int(self._handle), TERMINATE)
                except WindowsError:
                    if self.wait(timeout=0.1) is None:
                        raise

I will try to see if I can put a workaround in joblib.

@ogrisel ogrisel mentioned this issue May 25, 2016
Closed
@ogrisel ogrisel added this to the 0.18 milestone May 25, 2016
@krishnateja614 krishnateja614 mentioned this issue May 25, 2016
Closed
@nelson-liu nelson-liu mentioned this issue Jul 5, 2016
Merged
@lesteve
Copy link
Member
lesteve commented Sep 21, 2016

Closing. This was fixed in joblib/joblib#355 which is in joblib 0.10.0 and joblib 0.10.2 is bundled in the upcoming 0.18 release.

@lesteve lesteve closed this as completed Sep 21, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug
Projects
None yet
Milestone
0.18
Development

No branches or pull requests
5 participants
@ogrisel @larsmans @amueller @lesteve @chenyuqing
0