Description
It would be nice if loky worked with PyPy. In particular, scikit-learn 0.20 will include experimental support for PyPy. If loky doesn't support it, we would have to either switch default joblib backend on PyPy, or suggest users to install joblib 0.11 and use that (scikit-learn/scikit-learn#11971 (comment)). Neither of which is ideal.
Besides in my experience, PyPy is fairly stable and most things will just work. For things that don't there is a 50% chance of indeed some difference between PyPy / CPython, but the rest are due to some usage of private CPython API (as is for instance the case in https://github.com/tomMoral/loky/pull/159). So supporting PyPy can give you more confidence in the implementation even for running CPython.
Currently, there are the following issues,
privateimport _pickle(should be fixed in https://github.com/tomMoral/loky/pull/159)- error of the form,
usr/local/site-packages/sklearn/externals/joblib/externals/loky/backend/reduction.py:100: in
dispatcher
reduced = reduce_func(obj)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
m = <function _process_worker at 0x000000000320dec0>
def _reduce_method_descriptor(m):
> return getattr, (m.__objclass__, m.__name__)
E AttributeError: 'function' object has no attribute '__objclass__'- when running the test suite, I'm getting a deadlock in
TestsProcessPoolForkserverShutdown::test_interpreter_shutdown(which might be due to the above error though).
Full log of running the test suite with,
pip install pytest-instafail
pytest tests/ --instafail -v
can be found in loky-pypy-testing.log
To run the test suite, you can for instance download a binary of PyPy3, create a Python virtualenv with it then install dependencies as usual (see scikit-learn/scikit-learn#11010 (comment))