8000 CircleCI failure on "doc" build · Issue #21897 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

CircleCI failure on "doc" build #21897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
adrinjalali opened this issue Dec 6, 2021 · 10 comments · Fixed by #21922
Closed

CircleCI failure on "doc" build #21897

adrinjalali opened this issue Dec 6, 2021 · 10 comments · Fixed by #21922

Comments

@adrinjalali
Copy link
Member
adrinjalali commented Dec 6, 2021 •
edited by glemaitre
Loading

Since "CI Simplier wheel building config [cd build gh] (#21849)" circleci has been consistently failing around an example which does not take long at all when run locally. It also doesn't fail on doc-min-dependencies. Here's an example CI run with the following log.

Luckily this is not blocking our PR CIs since we don't build the full documentation there.

ping @glemaitre @lesteve and @thomasjpfan

generating gallery for auto_examples/neighbors... [  7%] plot_regression.py
generating gallery for auto_examples/neighbors... [ 15%] plot_lof_outlier_detection.py
generating gallery for auto_examples/neighbors... [ 23%] plot_nearest_centroid.py
generating gallery for auto_examples/neighbors... [ 30%] plot_digits_kde_sampling.py
generating gallery for auto_examples/neighbors... [ 38%] plot_classification.py
generating gallery for auto_examples/neighbors... [ 46%] plot_caching_nearest_neighbors.py
generating gallery for auto_examples/neighbors... [ 53%] plot_nca_illustration.py
generating gallery for auto_examples/neighbors... [ 61%] plot_nca_dim_reduction.py
generating gallery for auto_examples/neighbors... [ 69%] plot_lof_novelty_detection.py
generating gallery for auto_examples/neighbors... [ 76%] plot_nca_classification.py
generating gallery for auto_examples/neighbors... [ 84%] plot_species_kde.py
generating gallery for auto_examples/neighbors... [ 92%] plot_kde_1d.py
generating gallery for auto_examples/neighbors... [100%] approximate_nearest_neighbors.py

generating gallery for auto_examples/neural_networks... [ 25%] plot_mnist_filters.py
make: *** [Makefile:48: html] Killed

Too long with no output (exceeded 10m0s): context deadline exceeded
@glemaitre
Copy link
Member

I tried to reproduce locally with the same environment on a linux machine and building the full documentation and everything went fine. I will try to send a build and connect via ssh to the machine on CircleCI but this is a rather complicated issue.

@thomasjpfan
Copy link
Member

Copying my response from #21859:

From past experience with circleci + doc building, it usually comes down to memory issues. I recall doing experiments on fetch_openml + mnist that it takes quite a bit of memory to load: #19774 and the circleci instance only has 4gb.

@thomasjpfan thomasjpfan mentioned this issue Dec 6, 2021
Merged
@adrinjalali
Copy link
Member Author

Two things that come in mind:

We switched the mnist to use the one which has 70k samples instead of 62k. We could create a small mnist dataset on openml with 20k samples, which should be enough for our examples and use that one.

What happened to moving doc build to azure? I can't find the PR, was @jjerphan working on it? Not sure.

@jjerphan
Copy link
Member
jjerphan commented Dec 7, 2021

What happened to moving doc build to azure? I can't find the PR, was @jjerphan working on it? Not sure.

@adrinjalali: are you referring to #21721 (which was superseded by #21715)?

@adrinjalali
Copy link
Member Author
8000

No, the idea was to build the docs on azure, and then use circleci only to store the artifacts. Or maybe the idea was to build them on GitHub actions, not sure. But either way, it would make us not use circleci for such long builds anymore.

@jjerphan
Copy link
Member
jjerphan commented Dec 7, 2021

I see, apart from simple hotfixes I have not worked actively on the CI lately.

The two PR I worked on for the CI are #21744 and #21174.

@thomasjpfan
Copy link
Member
thomasjpfan commented Dec 7, 2021 •
edited
Loading

I will prioritize working on moving doc building away from circleci this week with my plan outlined in #21068.

@lesteve lesteve mentioned this issue Dec 8, 2021
Merged
@lesteve
Copy link
Member
lesteve commented Dec 8, 2021 •
edited
Loading

I opened #21922 as a potential quick fix (it considerably reduces memory usage for mnist dataset by using as_frame=False).

By the way the last build on master of the doc is green https://app.circleci.com/pipelines/github/scikit-learn/scikit-learn/20979/workflows/f4efceec-734e-4c10-88d9-9add8d632379/jobs/166261 whereas it has been consistently red for some time 🤷‍♂️ ...

@glemaitre
Copy link
Member

I opened #21922 as a potential quick fix (it considerably reduces memory usage for mnist dataset by using as_frame=False).

By how much? Is it that we have fewer copies going in the ARFF reader?

@lesteve
Copy link
Member
lesteve commented Dec 9, 2021

By how much? Is it that we have fewer copies going in the ARFF reader?

I sprinkled comments in different issues ... more details are here: #19774 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use as_frame=False for mnist_784 examples to reduce memory usage lesteve/scikit-learn
5 participants
@adrinjalali @lesteve @thomasjpfan @glemaitre @jjerphan
0