From 62d9374b18a928b3da618234a13641a0bea42026 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lo=C3=AFc=20Est=C3=A8ve?= Date: Mon, 8 Jul 2024 17:00:27 +0200 Subject: [PATCH 1/6] DOC Add how to create an environment from lock-file to debug CI issues --- doc/developers/tips.rst | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/doc/developers/tips.rst b/doc/developers/tips.rst index 3dbc35cec68d0..c30093d59f623 100644 --- a/doc/developers/tips.rst +++ b/doc/developers/tips.rst @@ -242,6 +242,33 @@ PR: Don't change unrelated Please do not change unrelated lines. It makes your contribution harder to review and may introduce merge conflicts to other pull requests. +.. _debugging_ci_issues: + +Debugging CI issues +------------------- + +CI issues happen for a variety of reasons, so this is by no way a comprehensive +guide but more a list of useful tips and tricks. + +Using a lock-file to get an environment close to the CI ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +`conda-lock` can be used to create an conda environment with the exact same +conda and pip packages as on the CI. For example, the following command will +create a conda environment named `scikit-learn-doc` that is similar to the CI: + +.. prompt bash $ + + conda-lock install -n scikit-learn-doc build_tools/circle/doc_linux-64_conda.lock + +.. note:: + + It only works if you have the same OS as the CI build (check `platform:` in + the lock-file). For example, the previous command will only work if you are + on a Linux machine. Also this may not allow you to reproduce some of the + issues that are more tied to the particularities of the CI environment, for + example CPU architecture reported by OpenBLAS in `sklearn.show_versions()`. + .. highlight:: default Debugging memory errors in Cython with valgrind From d4a4873b2f796a178bcd96c764f43ef08f9553a6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lo=C3=AFc=20Est=C3=A8ve?= Date: Mon, 8 Jul 2024 18:15:41 +0200 Subject: [PATCH 2/6] rst what else --- doc/developers/tips.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/developers/tips.rst b/doc/developers/tips.rst index c30093d59f623..3a4aba8c088b7 100644 --- a/doc/developers/tips.rst +++ b/doc/developers/tips.rst @@ -259,7 +259,7 @@ create a conda environment named `scikit-learn-doc` that is similar to the CI: .. prompt bash $ - conda-lock install -n scikit-learn-doc build_tools/circle/doc_linux-64_conda.lock + conda-lock install -n scikit-learn-doc build_tools/circle/doc_linux-64_conda.lock .. note:: From b7455de1b92556b9e1469964d66582dbfe386c7f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lo=C3=AFc=20Est=C3=A8ve?= Date: Tue, 9 Jul 2024 10:47:57 +0200 Subject: [PATCH 3/6] Add backup plan using environment yaml file --- doc/developers/tips.rst | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/doc/developers/tips.rst b/doc/developers/tips.rst index 3a4aba8c088b7..3de4cb58bfa51 100644 --- a/doc/developers/tips.rst +++ b/doc/developers/tips.rst @@ -269,6 +269,30 @@ create a conda environment named `scikit-learn-doc` that is similar to the CI: issues that are more tied to the particularities of the CI environment, for example CPU architecture reported by OpenBLAS in `sklearn.show_versions()`. +If you don't have the same OS as the CI build you can still create a conda +environment from the right environment yaml file, although it won't be as close +as the CI environment as using a lock-file. For example for the doc build: + +.. prompt bash $ + + conda env create -n scikit-learn-doc -f build_tools/circle/doc_environment.yml -y + + +This may not give you exactly the same package versions as in the CI for a +variety of reasons, for example: + +- some packages may have been updated between the time the lock files were + last updated in the `main` branch and the time you run the `conda create` + command. You can always try to look at the version in the lock-file and + specify the versions by hand for some specific packages that you think would + help reproducing the issue. +- different package may be installed by default depending on the OS. For + example, the default BLAS library when installing numpy is OpenBLAS on Linux + and MKL on Windows. + +Also the problem may be OS specific so the only way to be able to reproduce +would be to have the same OS as the CI build. + .. highlight:: default Debugging memory errors in Cython with valgrind From 06a39e2a02a9ba82c4fc2c4f39e83882e27da53d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lo=C3=AFc=20Est=C3=A8ve?= Date: Tue, 9 Jul 2024 10:48:51 +0200 Subject: [PATCH 4/6] tweak --- doc/developers/tips.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/developers/tips.rst b/doc/developers/tips.rst index 3de4cb58bfa51..a477bc6161ad4 100644 --- a/doc/developers/tips.rst +++ b/doc/developers/tips.rst @@ -271,7 +271,8 @@ create a conda environment named `scikit-learn-doc` that is similar to the CI: If you don't have the same OS as the CI build you can still create a conda environment from the right environment yaml file, although it won't be as close -as the CI environment as using a lock-file. For example for the doc build: +as the CI environment as using the associated lock-file. For example for the +doc build: .. prompt bash $ From 35e6fbe742c2e6813a6219925cd14a647904a7ca Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lo=C3=AFc=20Est=C3=A8ve?= Date: Tue, 9 Jul 2024 11:09:33 +0200 Subject: [PATCH 5/6] Apply suggestions from code review Co-authored-by: Arturo Amor <86408019+ArturoAmorQ@users.noreply.github.com> --- doc/developers/tips.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/developers/tips.rst b/doc/developers/tips.rst index a477bc6161ad4..f63a23bf1c9f6 100644 --- a/doc/developers/tips.rst +++ b/doc/developers/tips.rst @@ -247,17 +247,17 @@ PR: Don't change unrelated Debugging CI issues ------------------- -CI issues happen for a variety of reasons, so this is by no way a comprehensive -guide but more a list of useful tips and tricks. +CI issues may arise for a variety of reasons, so this is by no means a +comprehensive guide, but rather a list of useful tips and tricks. Using a lock-file to get an environment close to the CI +++++++++++++++++++++++++++++++++++++++++++++++++++++++ -`conda-lock` can be used to create an conda environment with the exact same +`conda-lock` can be used to create a conda environment with the exact same conda and pip packages as on the CI. For example, the following command will create a conda environment named `scikit-learn-doc` that is similar to the CI: -.. prompt bash $ +.. prompt:: bash $ conda-lock install -n scikit-learn-doc build_tools/circle/doc_linux-64_conda.lock @@ -274,7 +274,7 @@ environment from the right environment yaml file, although it won't be as close as the CI environment as using the associated lock-file. For example for the doc build: -.. prompt bash $ +.. prompt:: bash $ conda env create -n scikit-learn-doc -f build_tools/circle/doc_environment.yml -y From f7b7c11d50b5248871015f6391928aac03299ddf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lo=C3=AFc=20Est=C3=A8ve?= Date: Tue, 9 Jul 2024 11:12:38 +0200 Subject: [PATCH 6/6] Tweaks --- doc/developers/tips.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/doc/developers/tips.rst b/doc/developers/tips.rst index f63a23bf1c9f6..1c6ea5ba6f6f4 100644 --- a/doc/developers/tips.rst +++ b/doc/developers/tips.rst @@ -278,16 +278,15 @@ doc build: conda env create -n scikit-learn-doc -f build_tools/circle/doc_environment.yml -y - This may not give you exactly the same package versions as in the CI for a variety of reasons, for example: -- some packages may have been updated between the time the lock files were +- some packages may have had new releases between the time the lock files were last updated in the `main` branch and the time you run the `conda create` command. You can always try to look at the version in the lock-file and specify the versions by hand for some specific packages that you think would help reproducing the issue. -- different package may be installed by default depending on the OS. For +- different packages may be installed by default depending on the OS. For example, the default BLAS library when installing numpy is OpenBLAS on Linux and MKL on Windows.