8000 Undeclared build dependency on cython · Issue #15298 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

Undeclared build dependency on cython #15298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jvesely opened this issue Oct 19, 2019 · 14 comments · Fixed by #15313 or #16244
Closed

Undeclared build dependency on cython #15298

jvesely opened this issue Oct 19, 2019 · 14 comments · Fixed by #15313 or #16244
Labels
Bug Build / CI
Milestone
0.23

Comments

@jvesely
Copy link
Contributor
jvesely commented Oct 19, 2019

Description

pip install scikit-learn --no-binary :all: fails to install scikit because of missing cython build dependency. Once cython is installed manually the above command works and installs scikit-learn.

Steps/Code to Reproduce

pip install scikit-learn --no-binary :all:
it's also the case for python implementations that don't have scikit-learn wheels available (PyPy3, Python3.8)

Expected Results

scikit-learn installs successfully

Actual Results

  Running setup.py install for scikit-learn ... error
    Complete output from command /home/user/pypy3-venv/bin/pypy3-c -u -c "import setuptools, tokenize;__file__='/var/tmp/pip-install-zf4xear5/scikit-learn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/tmp/pip-record-k9u2qwff/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/pypy3-venv/include/site/python3.6/scikit-learn:
    Partial import of sklearn during the build process.
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/var/tmp/pip-install-zf4xear5/scikit-learn/setup.py", line 290, in <module>
        setup_package()
      File "/var/tmp/pip-install-zf4xear5/scikit-learn/setup.py", line 286, in setup_package
        setup(**metadata)
      File "/home/user/pypy3-venv/site-packages/numpy/distutils/core.py", line 137, in setup
        config = configuration()
      File "/var/tmp/pip-install-zf4xear5/scikit-learn/setup.py", line 174, in configuration
        config.add_subpackage('sklearn')
      File "/home/user/pypy3-venv/site-packages/numpy/distutils/misc_util.py", line 1035, in add_subpackage
        caller_level = 2)
      File "/home/user/pypy3-venv/site-packages/numpy/distutils/misc_util.py", line 1004, in get_subpackage
        caller_level = caller_level + 1)
      File "/home/user/pypy3-venv/site-packages/numpy/distutils/misc_util.py", line 941, in _get_configuration_from_setup_py
        config = setup_module.configuration(*args)
      File "sklearn/setup.py", line 62, in configuration
        config.add_subpackage('utils')
      File "/home/user/pypy3-venv/site-packages/numpy/distutils/misc_util.py", line 1035, in add_subpackage
        caller_level = 2)
      File "/home/user/pypy3-venv/site-packages/numpy/distutils/misc_util.py", line 1004, in get_subpackage
        caller_level = caller_level + 1)
      File "/home/user/pypy3-venv/site-packages/numpy/distutils/misc_util.py", line 941, in _get_configuration_from_setup_py
        config = setup_module.configuration(*args)
      File "sklearn/utils/setup.py", line 8, in configuration
        from Cython import Tempita
    ModuleNotFoundError: No module named 'Cython'
    
    ----------------------------------------
Command "/home/user/pypy3-venv/bin/pypy3-c -u -c "import setuptools, tokenize;__file__='/var/tmp/pip-install-zf4xear5/scikit-learn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/tmp/pip-record-k9u2qwff/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/pypy3-venv/include/site/python3.6/scikit-learn" failed with error code 1 in /var/tmp/pip-install-zf4xear5/scikit-learn/

Versions
@lesteve
Copy link
Member
lesteve commented Oct 19, 2019

Indeed I can reproduce.

I seem to remember we included the .c files (generated from the Cython files) in the .tar.gz for source distribution (sdist). Maybe the problem is that we still go through the Cython route, even if we already have the .c files ... this needs to be double-checked.

@jeremiedbb jeremiedbb mentioned this issue Oct 21, 2019
Merged
@jeremiedbb jeremiedbb added this to the 0.22 milestone Oct 22, 2019
@jvesely
Copy link
Contributor Author
jvesely commented Dec 9, 2019

Hi @rth, @lesteve,

this issue still appears in 0.22. I think #15567 rendered changes in #15313 uneffective, and the cython dependency is still required.

user@machine ~ $ python3 -m venv test-env
user@machine ~ $ source test-env/bin/activate
(test-env) user@machine ~ $ TMPDIR=/var/tmp/ pip3 install --no-binary :all: scikit-learn
Collecting scikit-learn
  Using cached https://files.pythonhosted.org/packages/4f/2c/04e10167991ed6209fb251a212ca7c3148006f335f4aadf1808db2cbeda8/scikit-learn-0.22.tar.gz
Collecting numpy>=1.11.0 (from scikit-learn)
  Using cached https://files.pythonhosted.org/packages/ff/59/d3f6d46aa1fd220d020bdd61e76ca51f6548c6ad6d24ddb614f4037cf49d/numpy-1.17.4.zip
Collecting scipy>=0.17.0 (from scikit-learn)
  Using cached https://files.pythonhosted.org/packages/a7/5c/495190b8c7cc71977c3d3fafe788d99d43eeb4740ac56856095df6a23fbd/scipy-1.3.3.tar.gz
  Installing build dependencies ... done
Collecting joblib>=0.11 (from scikit-learn)
  Using cached https://files.pythonhosted.org/packages/0a/ae/89d32b92903241468d9e993d05c49e8e7d58bdc3db51ed276b01786760ac/joblib-0.14.0.tar.gz
Installing collected packages: numpy, scipy, joblib, scikit-learn
  Running setup.py install for numpy ... done
  Running setup.py install for scipy ... done
  Running setup.py install for joblib ... done
  Running setup.py install for scikit-learn ... error
    Complete output from command /home/user/test-env/bin/python3 -u -c "import setuptools, tokenize;__file__='/var/tmp/pip-install-r3bbyune/scikit-learn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/tmp/pip-record-wn_c4638/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/test-env/include/site/python3.6/scikit-learn:
    Partial import of sklearn during the build process.
    Traceback (most recent call last):
      File "/var/tmp/pip-install-r3bbyune/scikit-learn/sklearn/_build_utils/__init__.py", line 32, in _check_cython_version
        import Cython
    ModuleNotFoundError: No module named 'Cython'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/var/tmp/pip-install-r3bbyune/scikit-learn/setup.py", line 303, in <module>
        setup_package()
      File "/var/tmp/pip-install-r3bbyune/scikit-learn/setup.py", line 299, in setup_package
        setup(**metadata)
      File "/home/user/test-env/lib64/python3.6/site-packages/numpy/distutils/core.py", line 137, in setup
        config = configuration()
      File "/var/tmp/pip-install-r3bbyune/scikit-learn/setup.py", line 180, in configuration
        _check_cython_version()
      File "/var/tmp/pip-install-r3bbyune/scikit-learn/sklearn/_build_utils/__init__.py", line 35, in _check_cython_version
        raise ModuleNotFoundError(message)
    ModuleNotFoundError: Please install Cython with a version >= 0.28.5 in order to build a scikit-learn from source.
    
    ----------------------------------------
Command "/home/user/test-env/bin/python3 -u -c "import setuptools, tokenize;__file__='/var/tmp/pip-install-r3bbyune/scikit-learn/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/tmp/pip-record-wn_c4638/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/test-env/include/site/python3.6/scikit-learn" failed with error code 1 in /var/tmp/pip-install-r3bbyune/scikit-learn/

should I open a new issue, or can you reopen this one?

@rth
Copy link
Member
rth commented Dec 9, 2019

Thanks for reporting this!

I think #15567 rendered changes in #15313 uneffective,

Yes, makes sense.

@rth rth reopened this Dec 9, 2019
@jeremiedbb
Copy link
Member

I think #15567 rendered changes in #15313 uneffective, and the cython dependency is still required.

Indeed, but we made cython a build dependency for scikit-learn. So from 0.22, if you pip install with the no-binary flag, you need cython.

@jvesely
Copy link
Contributor Author
jvesely commented Dec 9, 2019 •
edited
Loading

I think #15567 rendered changes in #15313 uneffective, and the cython dependency is still required.

Indeed, but we made cython a build dependency for scikit-learn. So from 0.22, if you pip install with the no-binary flag, you need cython.

I understand that. It'd be nice if scikit-learn declared the build dependencies so that pip can install them automatically[0] rather than expecting the user to install it manually ahead of time.

[0] https://packaging.python.org/specifications/declaring-build-dependencies/

@jeremiedbb
Copy link
Member

We decided to not use pyproject.toml for this release because we didn't have time to test it properly before the release (you can see related discussion in #15567 and #15533). It's planed to add this for the next release

@jvesely
Copy link
Contributor Author
jvesely commented Dec 9, 2019

We decided to not use pyproject.toml for this release because we didn't have time to test it properly before the release (you can see related discussion in #15567 and #15533). It's planed to add this for the next release

thanks! Those are both closed/merged. Is there an open issue/pr that tracks this?

@jeremiedbb
Copy link
Member

Let's keep this one open to track this

@jnothman jnothman modified the milestones: 0.22, 0.22.1 Dec 12, 2019
@adrinjalali
Copy link
Member

why is this a blocker for 0.22.1? Are we going to have a pyproject.toml? I think this takes a while to fix and we shouldn't block the bug-fix release because of it.

@jnothman
Copy link
Member

Sorry, didn't see it tagged blocker. When it was reopened it was tagged 0.22

@jnothman jnothman removed the Blocker label Dec 12, 2019
@jnothman jnothman modified the milestones: 0.22.1, 0.23 Dec 12, 2019
@adrinjalali
Copy link
Member

This also seems to be moving to the scipy org, which will make writing a pyproject.toml easier: https://pypi.org/project/oldest-supported-numpy/

@jeremiedbb jeremiedbb mentioned this issue Jan 27, 2020
Merged
@sebnyberg sebnyberg mentioned this issue Nov 5, 2020
Closed
@jackjansen
Copy link

I have the feeling this bug has re-appeared with release scikit-learn 0.24.

As of a few days ago I see CI/CD builds failing for a project that has a deep indirect dependency on scikit-learn, with the ModuleNotFound error reported above. Sorry, I don't even know which package has the dependency on scikit-learn...

I can tell you the CI/CD machine runs Python 3.6 on Ubuntu 18.04.

@jackjansen jackjansen mentioned this issue Dec 24, 2020
Closed
@adrinjalali
Copy link
Member

@jackjansen we'll need more context to re-open this issue. We do have a pyproject.toml file in which Cython is a declared build dependency, so I'm not sure what the issue is on your side.

@jackjansen
Copy link

@adrinjalali no need to reopen. The issue has been solved, see #19068.

nils-drechsel added a commit to visokio/omniscope-custom-blocks that referenced this issue Jan 5, 2021
@nils-drechsel
workaround for sklearn bug scikit-learn/scikit-learn#15298
780f843
jackjansen added a commit to cwi-dis/cwipc_util that referenced this issue Jan 7, 2021
@jackjansen
Install scikit-learn version other than 0.24.0. Workaround for scikit…
5ef5b24 
…-learn/scikit-learn#15298
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Build / CI
Projects
None yet
Milestone
0.23
7 participants
@jnothman @rth @adrinjalali @lesteve @jackjansen @jvesely @jeremiedbb
0