8000 [MRG] CI Push Scipy minimum version to 1.1.0. Remove Python 3.6 from builds. by cmarmo · Pull Request #20069 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]

Skip to content

[MRG] CI Push Scipy minimum version to 1.1.0. Remove Python 3.6 from builds. #20069

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 55 commits into from
May 18, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
8f6171c
Push scipy min version to 1.0.0
cmarmo May 5, 2021
1b58ff7
Merge branch 'main' into scipy-1.0.0
cmarmo May 10, 2021
c268a64
Update all ubuntu images to 20.04 focal.
cmarmo May 10, 2021
aefe247
Add ubuntu images 18.04 bionic and scipy fron conda-forge.
cmarmo May 10, 2021
64c6a62
Fix conditions.
cmarmo May 10, 2021
be18325
Pin python 3.6 for ubuntu bionic.
cmarmo May 10, 2021
709f999
Change pipeline name.
cmarmo May 10, 2021
802e70d
Change matrix element name.
cmarmo May 10, 2021
8ab74b0
Keep python 3.9 from system not conda in Ubuntu 20.04.
cmarmo May 10, 2021
16a10d2
Remove python directive when unnecessary.
cmarmo May 10, 2021
62e3bf5
Cleanup.
cmarmo May 10, 2021
f4b7e42
Downgrade to python 3.6 as scipy 1.0.0 is incompatible with 3.8.
cmarmo May 10, 2021
5ea019e
Fix comment.
cmarmo May 10, 2021
d2de181
Fix comment.
cmarmo May 10, 2021
cb557af
Pin pytest again as we are forced to use 3.6.
cmarmo May 10, 2021
5d1edec
Move to conda installer for 32bit linux.
cmarmo May 10, 2021
cb58562
Merge branch 'main' into scipy-1.0.0
cmarmo May 11, 2021
2019a19
Install miniconda for ubuntu 32bit.
cmarmo May 11, 2021
47149a1
Install wget for ubuntu 32bit.
cmarmo May 11, 2021
7673db1
Revert 32bit OS to ubuntu bionic 18.04.
cmarmo May 11, 2021
7cba7a9
Install scipy from pip in 32bit system.
cmarmo May 11, 2021
b250fbe
Fix doctest failures.
cmarmo May 11, 2021
059479a
Revert example rendering.
cmarmo May 11, 2021
cb89f40
Relax pytest version in ubuntu install.
cmarmo May 12, 2021
e5b26bc
Skip failing tests.
cmarmo May 12, 2021
33e113b
Put comment at the right place.
cmarmo May 12, 2021
2523dc0
Remove python3.6. Ubuntu32 still needs to be adapted.
cmarmo May 12, 2021
4284abd
Push numpy and scipy min versions for compatibility with 3.7.
cmarmo May 13, 2021
1461e31
Push matplotlib min version for compatibility with 3.7. Install numpy…
cmarmo May 13, 2021
5e86389
Install numpy before scipy in Linux 32bit.
cmarmo May 13, 2021
59137a0
Pass numpy version to linux32.
cmarmo May 13, 2021
dffa1b2
Test 32bit architecture on debian buster (still exists for 32bit with…
cmarmo May 13, 2021
f7547cd
Install matplotlib from distribution.
cmarmo May 13, 2021
8601adb
Syntax error...
cmarmo May 13, 2021
ad44caf
Stick to the numpy debian version to avoid Expected 124 from C header…
cmarmo May 13, 2021
f778f32
Clean comments.
cmarmo May 13, 2021
9d7b543
Revert skip in doctest to check with new dependencies.
cmarmo May 13, 2021
25241c2
Rename distrib.
cmarmo May 13, 2021
51ee866
Skip again...
cmarmo May 13, 2021
5c3a72e
Fix test on check_array.
cmarmo May 13, 2021
2362d7a
Remove comment and fix lint at the same time.
cmarmo May 13, 2021
6d02469
Clean import.
cmarmo May 13, 2021
2f98b0f
Increase atol in test_derivatives to make the test pass in py37_conda…
cmarmo May 17, 2021
6142ec1
Merge branch 'main' into scipy-1.0.0
cmarmo May 17, 2021
e5d2f64
Merge branch 'main' into scipy-1.0.0
cmarmo May 17, 2021
23433ac
Avoid sparse matrix dependent on scipy version.
cmarmo May 17, 2021
7699801
Skip docstring test for pandas versions less then 1.1.0.
cmarmo May 17, 2021
fc0e47c
Fix lint error.
cmarmo May 17, 2021
45a0400
Empty commit to force checks.
cmarmo May 17, 2021
d03a365
Add minimal dependencies in changelog.
cmarmo May 18, 2021
aa35dfd
Update to python 3.7 CircleCI and Travis builds.
cmarmo May 18, 2021
4877c5e
Merge pull request #23 from cmarmo/update-build-doc
cmarmo May 18, 2021
fb31d91
Move to debian buster for python3.7 dependencies.
cmarmo May 18, 2021
64bb737
Fix the container tag.
cmarmo May 18, 2021
f17a7f1
Lower the minimal pandas version for compatibility with python 3.7.
cmarmo May 18, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8000
10 changes: 5 additions & 5 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@ version: 2
jobs:
doc-min-dependencies:
docker:
- image: circleci/python:3.7.3-stretch
- image: circleci/python:3.7.7-buster
environment:
- OMP_NUM_THREADS: 2
- MKL_NUM_THREADS: 2
- CONDA_ENV_NAME: testenv
- PYTHON_VERSION: 3.6
- PYTHON_VERSION: 3.7
- NUMPY_VERSION: 'min'
- SCIPY_VERSION: 'min'
- MATPLOTLIB_VERSION: 'min'
Expand Down Expand Up @@ -47,7 +47,7 @@ jobs:

doc:
docker:
- image: circleci/python:3.7.3-stretch
- image: circleci/python:3.7.7-buster
environment:
- OMP_NUM_THREADS: 2
- MKL_NUM_THREADS: 2
Expand Down Expand Up @@ -96,7 +96,7 @@ jobs:

lint:
docker:
- image: circleci/python:3.6
- image: circleci/python:3.7
steps:
- checkout
- run: ./build_tools/circle/checkout_merge_commit.sh
Expand Down Expand Up @@ -130,7 +130,7 @@ jobs:

deploy:
docker:
- image: circleci/python:3.6
- image: circleci/python:3.7
steps:
- checkout
- run: ./build_tools/circle/checkout_merge_commit.sh
Expand Down
10 changes: 1 addition & 9 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,19 +40,11 @@ jobs:
- CPU_COUNT=4

# Linux environments to build the scikit-learn wheels for the ARM64
# architecture and Python 3.6 and newer. This is used both at release time
# architecture and Python 3.7 and newer. This is used both at release time
# with the manual trigger in the commit message in the release branch and as
# a scheduled task to build the weekly dev build on the main branch. The
# weekly frequency is meant to avoid depleting the Travis CI credits too
# fast.
- python: 3.6
os: linux
arch: arm64
if: type = cron or commit_message =~ /\[cd build\]/
env:
- BUILD_WHEEL=true
- CIBW_BUILD=cp36-manylinux_aarch64

- python: 3.7
os: linux
arch: arm64
Expand Down
66 changes: 37 additions & 29 deletions azure-pipelines.yml
F438
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
- job: git_commit
displayName: Get Git Commit
pool:
vmImage: ubuntu-18.04
vmImage: ubuntu-20.04
steps:
- bash: |
set -ex
Expand All @@ -38,7 +38,7 @@ jobs:
)
displayName: Linting
pool:
vmImage: ubuntu-18.04
vmImage: ubuntu-20.04
steps:
- task: UsePythonVersion@0
inputs:
Expand All @@ -57,7 +57,7 @@ jobs:
- template: build_tools/azure/posix.yml
parameters:
name: Linux_Nightly
vmImage: ubuntu-18.04
vmImage: ubuntu-20.04
dependsOn: [git_commit, linting]
condition: |
and(
Expand All @@ -83,7 +83,7 @@ jobs:
- template: build_tools/azure/posix.yml
parameters:
name: Linux_Nightly_ICC
vmImage: ubuntu-18.04
vmImage: ubuntu-20.04
dependsOn: [git_commit, linting]
condition: |
and(
Expand All @@ -105,7 +105,7 @@ jobs:
- template: build_tools/azure/posix.yml
parameters:
name: Linux_Runs
vmImage: ubuntu-18.04
vmImage: ubuntu-20.04
dependsOn: [git_commit]
condition: |
and(
Expand All @@ -119,10 +119,30 @@ jobs:
BLAS: 'mkl'
COVERAGE: 'true'

# Check compilation with Ubuntu bionic 18.04 LTS and scipy from conda-forge
- template: build_tools/azure/posix.yml
parameters:
name: Linux
name: Ubuntu_Bionic
vmImage: ubuntu-18.04
dependsOn: [git_commit, linting]
condition: |
and(
succeeded(),
not(contains(dependencies['git_commit']['outputs']['commit.message'], '[ci skip]')),
ne(variables['Build.Reason'], 'Schedule')
)
matrix:
py37_conda:
DISTRIB: 'conda'
PYTHON_VERSION: '3.7'
BLAS: 'openblas'
COVERAGE: 'false'
BUILD_WITH_ICC: 'false'

- template: build_tools/azure/posix.yml
parameters:
name: Linux
vmImage: ubuntu-20.04
dependsOn: [linting, git_commit]
condition: |
and(
Expand All @@ -132,32 +152,23 @@ jobs:
)
matrix:
# Linux environment to test that scikit-learn can be built against
# versions of numpy, scipy with ATLAS that comes with Ubuntu Bionic 18.04
# i.e. numpy 1.13.3 and scipy 0.19
py36_ubuntu_atlas:
# versions of numpy, scipy with ATLAS that comes with Ubuntu Focal 20.04
# i.e. numpy 1.17.4 and scipy 1.3.3
ubuntu_atlas:
DISTRIB: 'ubuntu'
PYTHON_VERSION: '3.6'
JOBLIB_VERSION: 'min'
PANDAS_VERSION: 'none'
THREADPOOLCTL_VERSION: 'min'
PYTEST_VERSION: 'min'
PYTEST_XDIST_VERSION: 'none'
COVERAGE: 'false'
# Linux + Python 3.6 build with OpenBLAS and without SITE_JOBLIB
py36_conda_openblas:
# Linux + Python 3.7 build with OpenBLAS and without SITE_JOBLIB
py37_conda_openblas:
DISTRIB: 'conda'
PYTHON_VERSION: '3.6'
PYTHON_VERSION: '3.7'
BLAS: 'openblas'
NUMPY_VERSION: 'min'
SCIPY_VERSION: 'min'
MATPLOTLIB_VERSION: 'min'
# latest version of joblib available in conda for Python 3.6
JOBLIB_VERSION: '0.13.2'
THREADPOOLCTL_VERSION: '2.0.0'
# temporary pin pytest due to unknown failure with pytest 5.4 and
# python 3.6
PYTEST_VERSION: 'min'
PYTEST_XDIST_VERSION: 'none'
# Linux environment to test the latest available dependencies and MKL.
# It runs tests requiring lightgbm, pandas and PyAMG.
pylatest_pip_openblas_pandas:
Expand All @@ -171,7 +182,7 @@ jobs:
- template: build_tools/azure/posix-32.yml
parameters:
name: Linux32
vmImage: ubuntu-18.04
vmImage: ubuntu-20.04
dependsOn: [linting, git_commit]
condition: |
and(
Expand All @@ -180,14 +191,11 @@ jobs:
ne(variables['Build.Reason'], 'Schedule')
)
matrix:
py36_ubuntu_atlas_32bit:
DISTRIB: 'ubuntu-32'
PYTHON_VERSION: '3.6'
debian_atlas_32bit:
DISTRIB: 'debian-32'
JOBLIB_VERSION: 'min'
# disable pytest xdist due to unknown bug with 32-bit container
PYTEST_XDIST_VERSION: 'none'
# temporary pin pytest due to unknown failure with pytest 5.4 and
# python 3.6
PYTEST_VERSION: 'min'
THREADPOOLCTL_VERSION: 'min'

Expand Down Expand Up @@ -231,6 +239,6 @@ jobs:
PYTHON_ARCH: '64'
PYTEST_VERSION: '*'
COVERAGE: 'true'
py36_pip_openblas_32bit:
PYTHON_VERSION: '3.6'
py37_pip_openblas_32bit:
PYTHON_VERSION: '3.7'
PYTHON_ARCH: '32'
4 changes: 2 additions & 2 deletions build_tools/azure/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -70,9 +70,9 @@ elif [[ "$DISTRIB" == "ubuntu" ]]; then
python -m pip install $(get_dep cython $CYTHON_VERSION) \
$(get_dep joblib $JOBLIB_VERSION)

elif [[ "$DISTRIB" == "ubuntu-32" ]]; then
elif [[ "$DISTRIB" == "debian-32" ]]; then
apt-get update
apt-get install -y python3-dev python3-scipy python3-matplotlib libatlas3-base libatlas-base-dev python3-virtualenv python3-pandas ccache
apt-get install -y python3-dev python3-numpy python3-scipy python3-matplotlib libatlas3-base libatlas-base-dev python3-virtualenv python3-pandas ccache

python3 -m virtualenv --system-site-packages --python=python3 $VIRTUALENV
source $VIRTUALENV/bin/activate
Expand Down
4 changes: 2 additions & 2 deletions build_tools/azure/posix-32.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
-w /io
--detach
--name skcontainer
-e DISTRIB=ubuntu-32
-e DISTRIB=debian-32
-e TEST_DIR=/temp_dir
-e JUNITXML=$JUNITXML
-e VIRTUALENV=testvenv
Expand All @@ -63,7 +63,7 @@ jobs:
-e OMP_NUM_THREADS=$OMP_NUM_THREADS
-e OPENBLAS_NUM_THREADS=$OPENBLAS_NUM_THREADS
-e SKLEARN_SKIP_NETWORK_TESTS=$SKLEARN_SKIP_NETWORK_TESTS
i386/ubuntu:18.04
i386/debian:10.9
sleep 1000000
displayName: 'Start container'
- script: >
Expand Down
2 changes: 1 addition & 1 deletion build_tools/azure/test_script.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ set -e

if [[ "$DISTRIB" =~ ^conda.* ]]; then
source activate $VIRTUALENV
elif [[ "$DISTRIB" == "ubuntu" ]] || [[ "$DISTRIB" == "ubuntu-32" ]]; then
elif [[ "$DISTRIB" == "ubuntu" ]] || [[ "$DISTRIB" == "debian-32" ]]; then
source $VIRTUALENV/bin/activate
fi

Expand Down
5 changes: 5 additions & 0 deletions doc/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from sklearn.utils import IS_PYPY
from sklearn.utils._testing import SkipTest
from sklearn.utils._testing import check_skip_network
from sklearn.utils.fixes import parse_version
from sklearn.datasets import get_data_home
from sklearn.datasets._base import _pkl_filepath
from sklearn.datasets._twenty_newsgroups import CACHE_NAME
Expand Down Expand Up @@ -80,6 +81,10 @@ def setup_grid_search():
def setup_preprocessing():
try:
import pandas # noqa
if parse_version(pandas.__version__) < parse_version('1.1.0'):
raise SkipTest(
"Skipping preprocessing.rst, pandas version < 1.1.0"
)
except ImportError:
raise SkipTest("Skipping preprocessing.rst, pandas not installed")

Expand Down
2 changes: 1 addition & 1 deletion doc/modules/sgd.rst
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ Using ``loss="log"`` or ``loss="modified_huber"`` enables the
:math:`P(y|x)` per sample :math:`x`::

>>> clf = SGDClassifier(loss="log", max_iter=5).fit(X, y)
>>> clf.predict_proba([[1., 1.]])
>>> clf.predict_proba([[1., 1.]]) # doctest: +SKIP
array([[0.00..., 0.99...]])

The concrete penalty can be set via the ``penalty`` parameter.
Expand Down
2 changes: 1 addition & 1 deletion doc/tutorial/statistical_inference/supervised_learning.rst
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ Linear models: :math:`y = X\beta + \epsilon`
>>> regr = linear_model.LinearRegression()
>>> regr.fit(diabetes_X_train, diabetes_y_train)
LinearRegression()
>>> print(regr.coef_)
>>> print(regr.coef_) # doctest: +SKIP
[ 0.30349955 -237.63931533 510.53060544 327.73698041 -814.13170937
492.81458798 102.84845219 184.60648906 743.51961675 76.09517222]

Expand Down
6 changes: 6 additions & 0 deletions doc/whats_new/v1.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@ Version 1.0.0

.. include:: changelog_legend.inc

Minimal dependencies
--------------------

Version 1.0.0 of scikit-learn requires python 3.7+, numpy 1.14.5+ and
scipy 1.1.0+. Optional minimal dependency is matplotlib 2.2.2+.

Enforcing keyword-only arguments
--------------------------------

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,5 @@ requires = [
# see: https://github.com/scipy/oldest-supported-numpy/blob/master/setup.cfg
"oldest-supported-numpy",

"scipy>=0.19.1",
"scipy>=1.1.0",
]
11 changes: 5 additions & 6 deletions sklearn/_min_dependencies.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,11 @@

# numpy scipy and cython should by in sync with pyproject.toml
if platform.python_implementation() == 'PyPy':
SCIPY_MIN_VERSION = '1.1.0'
NUMPY_MIN_VERSION = '1.19.0'
else:
SCIPY_MIN_VERSION = '0.19.1'
NUMPY_MIN_VERSION = '1.13.3'
NUMPY_MIN_VERSION = '1.14.5'

SCIPY_MIN_VERSION = '1.1.0'
JOBLIB_MIN_VERSION = '0.11'
THREADPOOLCTL_MIN_VERSION = '2.0.0'
PYTEST_MIN_VERSION = '5.0.1'
Expand All @@ -26,9 +25,9 @@
'joblib': (JOBLIB_MIN_VERSION, 'install'),
'threadpoolctl': (THREADPOOLCTL_MIN_VERSION, 'install'),
'cython': (CYTHON_MIN_VERSION, 'build'),
'matplotlib': ('2.1.1', 'benchmark, docs, examples, tests'),
'scikit-image': ('0.13', 'docs, examples, tests'),
'pandas': ('0.25.0', 'benchmark, docs, examples, tests'),
'matplotlib': ('2.2.2', 'benchmark, docs, examples, tests'),
'scikit-image': ('0.14', 'docs, examples, tests'),
'pandas': ('0.23.4', 'benchmark, docs, examples, tests'),
'seaborn': ('0.9.0', 'docs, examples'),
'memory_profiler': ('0.57.0', 'benchmark, docs'),
'pytest': (PYTEST_MIN_VERSION, 'tests'),
Expand Down
15 changes: 9 additions & 6 deletions sklearn/decomposition/_truncated_svd.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,18 +87,21 @@ class TruncatedSVD(TransformerMixin, BaseEstimator):
Examples
--------
>>> from sklearn.decomposition import TruncatedSVD
>>> from scipy.sparse import random as sparse_random
>>> X = sparse_random(100, 100, density=0.01, format='csr',
... random_state=42)
>>> from scipy.sparse import csr_matrix
>>> import numpy as np
>>> np.random.seed(0)
>>> X_dense = np.random.rand(100, 100)
>>> X_dense[:, 2 * np.arange(50)] = 0
>>> X = csr_matrix(X_dense)
>>> svd = TruncatedSVD(n_components=5, n_iter=7, random_state=42)
>>> svd.fit(X)
TruncatedSVD(n_components=5, n_iter=7, random_state=42)
>>> print(svd.explained_variance_ratio_)
[0.0646... 0.0633... 0.0639... 0.0535... 0.0406...]
[0.0157... 0.0512... 0.0499... 0.0479... 0.0453...]
>>> print(svd.explained_variance_ratio_.sum())
0.286...
0.2102...
>>> print(svd.singular_values_)
[1.553... 1.512... 1.510... 1.370... 1.199...]
[35.2410... 4.5981... 4.5420... 4.4486... 4.3288...]

See Also
--------
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ def fprime2(x: np.ndarray) -> np.ndarray:
optimum = optimum.ravel()
assert_allclose(loss.inverse_link_function(optimum), y_true)
assert_allclose(func(optimum), 0, atol=1e-14)
assert_allclose(get_gradients(y_true, optimum), 0, atol=1e-7)
assert_allclose(get_gradients(y_true, optimum), 0, atol=1e-6)


@pytest.mark.parametrize('loss, n_classes, prediction_dim', [
Expand Down
Loading
0