8000 ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 30, 2024) ⚠️ · Issue #29674 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 30, 2024) ⚠️ #29674

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
scikit-learn-bot opened this issue Aug 15, 2024 · 9 comments
Closed

⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 30, 2024) ⚠️ #29674

scikit-learn-bot opened this issue Aug 15, 2024 · 9 comments

Comments

@scikit-learn-bot
Copy link
Contributor
scikit-learn-bot commented Aug 15, 2024 •
edited
Loading

CI is still failing on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (Aug 30, 2024)

  • test_pairwise_distances_for_dense_data[float32]
  • test_pairwise_distances_for_sparse_data[float32-csr_matrix-bsr_matrix-csc_matrix-coo_matrix]
  • test_pairwise_distances_for_sparse_data[float32-csr_matrix-bsr_matrix-csc_matrix-coo_array]
  • test_pairwise_distances_for_sparse_data[float32-csr_matrix-bsr_matrix-csc_array-coo_matrix]
  • test_pairwise_distances_for_sparse_data[float32-csr_matrix-bsr_matrix-csc_array-coo_array]
  • test_pairwise_distances_for_sparse_data[float32-csr_matrix-bsr_array-csc_matrix-coo_matrix]
  • test_pairwise_distances_for_sparse_data[float32-csr_matrix-bsr_array-csc_matrix-coo_array]
  • test_pairwise_distances_for_sparse_data[float32-csr_matrix-bsr_array-csc_array-coo_matrix]
  • test_pairwise_distances_for_sparse_data[float32-csr_matrix-bsr_array-csc_array-coo_array]
  • test_pairwise_distances_for_sparse_data[float32-csr_array-bsr_matrix-csc_matrix-coo_matrix]
  • test_pairwise_distances_for_sparse_data[float32-csr_array-bsr_matrix-csc_matrix-coo_array]
  • test_pairwise_distances_for_sparse_data[float32-csr_array-bsr_matrix-csc_array-coo_matrix]
  • test_pairwise_distances_for_sparse_data[float32-csr_array-bsr_matrix-csc_array-coo_array]
  • test_pairwise_distances_for_sparse_data[float32-csr_array-bsr_array-csc_matrix-coo_matrix]
  • test_pairwise_distances_for_sparse_data[float32-csr_array-bsr_array-csc_matrix-coo_array]
  • test_pairwise_distances_for_sparse_data[float32-csr_array-bsr_array-csc_array-coo_matrix]
  • test_pairwise_distances_for_sparse_data[float32-csr_array-bsr_array-csc_array-coo_array]
  • test_numeric_pairwise_distances_datatypes[float32-Y is X-cosine]
  • test_numeric_pairwise_distances_datatypes[float32-Y is not X-cosine]
@github-actions github-actions bot added the Needs Triage Issue requires triage label Aug 15, 2024
@adrinjalali
Copy link
Member

The errors seem to be due to tolerance, For instance:

E           Not equal to tolerance rtol=1e-07, atol=0
E           
E           Mismatched elements: 15 / 25 (60%)
E           Max absolute difference among violations: 1.19654872e-07
E           Max relative difference among violations: 4.71704801e-06
E            ACTUAL: array([[0.039904, 0.199071, 0.090947, 0.133582, 0.025933],
E                  [0.065489, 0.059188, 0.024719, 0.06899 , 0.081717],
E                  [0.07393 , 0.402515, 0.137955, 0.257353, 0.054411],...
E            DESIRED: array([[0.039904, 0.19907 , 0.090947, 0.133582, 0.025933],
E                  [0.065489, 0.059188, 0.024719, 0.06899 , 0.081717],
E                  [0.07393 , 0.402515, 0.137955, 0.257353, 0.054411],...

Seems reasonable to me to have a lower tolerance for float32 maybe?

@adrinjalali adrinjalali removed the Needs Triage Issue requires triage label Aug 15, 2024
@adrinjalali adrinjalali mentioned this issue Aug 15, 2024
Closed
@adrinjalali
Copy link
Member

@Micky774 @jjerphan do you remember this being an issue before?

@jjerphan
Copy link
Member

There were similar issues for FastICA in the past, but I can not tell if the root cause is similar.

@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 15, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 16, 2024) ⚠️ Aug 16, 2024
@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 16, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 17, 2024) ⚠️ Aug 17, 2024
@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 17, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 18, 2024) ⚠️ Aug 18, 2024
@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last fa 8000 ilure: Aug 18, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 19, 2024) ⚠️ Aug 19, 2024
@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 19, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 20, 2024) ⚠️ Aug 20, 2024
@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 20, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 21, 2024) ⚠️ Aug 21, 2024
@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 21, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 22, 2024) ⚠️ Aug 22, 2024
@lesteve
Copy link
Member
lesteve commented Aug 22, 2024 •
edited
Loading

Honestly I am a bit of a loss about what is happening here, the tests started failing on
#29639 which seems unrelated. The previous scheduled build was on #29590 which is the previous commit and pass. Not sure what else could have changed ...

See build logs
image

The failure is deterministic and the results are the same so this is not due to global random seed.

The fact that at the same time two other builds fail with the same error (scipy-dev #29675 and free-threaded #29599) is also weird, since there is not much in common between them: Python 3.9 and latest versions from conda-forge for this build and Python 3.12 + dev versions from pip for the others.

@jjerphan
Copy link
Member

Have the machines changed on the CI?

@adrinjalali
Copy link
Member

I'm also at a loss, but the diff is in the order of 1e-5. So I wouldn't mind relaxing the tests, however, it's really odd.

@lesteve
Copy link
Member
lesteve commented Aug 22, 2024

Yeah I think relaxing the tests is the most pragmatic thing to do.

Have the machines changed on the CI?

I quickly tried to have a look, at least the Docker image has not been changed according to https://github.com/actions/runner-images?tab=readme-ov-file#faqs

last passing log and first failing log. Both says:

Image: ubuntu-22.04
Version: 20240811.1.0
Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240811.1/images/ubuntu/Ubuntu2204-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240811.1

@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 22, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 23, 2024) ⚠️ Aug 23, 2024
@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 23, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 24, 2024) ⚠️ Aug 24, 2024
@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 24, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 25, 2024) ⚠️ Aug 25, 2024
@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 25, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 26, 2024) ⚠️ Aug 26, 2024
@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 26, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 27, 2024) ⚠️ Aug 27, 2024
@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 27, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 28, 2024) ⚠️ Aug 28, 2024
@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 28, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 29, 2024) ⚠️ Aug 29, 2024
@scikit-learn-bot scikit-learn-bot changed the title ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 29, 2024) ⚠️ ⚠️ CI failed on Ubuntu_Jammy_Jellyfish.pymin_conda_forge_openblas_ubuntu_2204 (last failure: Aug 30, 2024) ⚠️ Aug 30, 2024
@lesteve lesteve mentioned this issue Aug 30, 2024
Merged
@scikit-learn-bot
Copy link
Contributor Author

CI is no longer failing! ✅

Successful run on Aug 31, 2024

@lesteve
Copy link
Member
lesteve commented Aug 31, 2024

This has been fixed by #29746.

@lesteve lesteve closed this as completed Aug 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@adrinjalali @lesteve @scikit-learn-bot @jjerphan
0