Restructuring GitHub Actions CI jobs #24410
Comments
|
Sounds good. One downside of breaking up the jobs is how to do a smoke test run before starting many other runs. I don't know how we could pause all other CI runs until it passes, extra points for pausing cirrus travis, and azure. It would be too disruptive to do that with a local precommit-hook before allowing the PR to be pushed, but is there such a thing for the repo? Another thing that would be nice is to be able to label a PR There is a persistent failure in travis, we should mark that as "not fail". |
That's not too difficult, I think smoke tests can be run for the 3 main files (simd, blas, linux). I had thought of that already, will keep it as a required feature for now.
Controls between different CI providers are a pain, and I think an orthogonal topic to GHA changes. I'd much prefer not to think about that here.
We should turn TravisCI off for anything but SIMD PRs. |
|
We discussed this in the community meeting today. Overall sentiment was that:
|
Also documents better what is being run. See numpygh-24410 for the overall restructuring plan for GitHub Actions CI.
Also documents better what is being run. See numpygh-24410 for the overall restructuring plan for GitHub Actions CI.
Also documents better what is being run. See numpygh-24410 for the overall restructuring plan for GitHub Actions CI.
Also documents better what is being run. See numpygh-24410 for the overall restructuring plan for GitHub Actions CI.
|
Here are the current workflows on github actions and the time they take to run (after merging #24493 and #24479)
In addition we have the
|
|
With GH Actions it should be possible to make a workflow that triggers other workflows. One could imagine a main Linux workflow (the wheel builder config?) that takes on the responsibility of doing the builds, that are then used by subsiduary workflows that do the different test runs. |
I think the vast majority of builds should be for different build configs. We're running the whole test suite, there's not much need for retesting the same wheel multiple times. The two most expensive files have a I have a decent idea of what's still left for |
|
Okay, here are all jobs that need work, based on the failing builds from gh-24519:
@ev-br will have a look at the Emscripten job. @mkoeppe would you be able to have a look at the Cygwin job? @seiko2plus could you look at the SIMD tests (Linux) jobs? @mattip there is still a problem with the PyPy 3.9 on Windows job in Azure, would you be able to take that one? I can have a look at the conda and macOS Azure jobs and the two to-be-removed jobs. Any takers for the CircleCI job or the GHA Linux tests jobs? After all that is done, we also still have jobs to add:
plus it'll be easier to clean up the docs if all the |
|
I keep on finding special-case test bits, like this one: We should keep all of those only on GitHub Actions, because they're much easier to debug there. And we should have them all together in a separate job, not mixed into some random job (macOS tests on Azure in this case). |
|
I believe we can get rid of Azure completely. It's mostly Windows jobs that are left after gh-24520 is merged. For Windows, we now have the following:
We only need 4 jobs, to cover:
This can be a sparse test matrix, there's no need for a dense one. |
|
I'll take the CircleCI one as well. |
|
Getting closer - I'll take GHA Linux tests, (3x, basic/debug/benchmark) as well. |
It's possible to set an Action to run only if certain files are modified; this example looks like a decent starting place for not running the main tests on a PR that only changes documentation, or configuration for other CI services. Unfortunately, it doesn't signal other CI services that they don't need to run either. |
This is a backport of numpy#24493 and numpy#24291. The purpose of this is to ease future backports that expect these files. - CI: move some jobs in `build_test.yml` to Meson - CI: split `build_test.yml` into three GHA jobs files Also documents better what is being run. See numpygh-24410 for the overall restructuring plan for GitHub Actions CI. - CI: merge `linux_meson.yml` into `linux_blas.yml` - TST: disable mypy tests in test suite unless an environment variable is set These tests are super slow, and they're effectively always passing in CI. Running them on all "full" test suite runs is too expensive. Note that SciPy has an XSLOW mark, NumPy does not. So use an env var for now. - CI: add new GHA CI jobs to run MyPy across OS/Python flavors
This is a backport of numpy#24493 and numpy#24291. The purpose of this is to ease future backports that expect these files. - CI: move some jobs in `build_test.yml` to Meson - CI: split `build_test.yml` into three GHA jobs files Also documents better what is being run. See numpygh-24410 for the overall restructuring plan for GitHub Actions CI. - CI: merge `linux_meson.yml` into `linux_blas.yml` - TST: disable mypy tests in test suite unless an environment variable is set These tests are super slow, and they're effectively always passing in CI. Running them on all "full" test suite runs is too expensive. Note that SciPy has an XSLOW mark, NumPy does not. So use an env var for now. - CI: add new GHA CI jobs to run MyPy across OS/Python flavors
|
@rgommers I think this can be closed? |
|
Yes, this is pretty much all done. We haven't killed the last Azure DevOps jobs yet, but they don't seem too troublesome right now. We can always circle back to this later if needed. |
These CI jobs are hard to navigate with several levels of indirection (a workflow file, then an action file, then two
tools/travis-*.shscripts), and now that we have to move a lot of jobs over to Meson it's a good time to clean up. Thebuild_test.ymlis particularly messy, and contains 22 jobs which is too much (quite a few are unnecessary too).A few thoughts:
windows_meson.ymlandwindows_clangcl.ymlcan be merged, and namedwindows.ymlemscripten.yml,cygwin.yml,linux_musl.ymlandlinux_compiler_sanitizers.yml.build_test.yml:linux_simd.yml: all the SIMD related jobs (~8 jobs) and theold_gcconelinux_blas.yml: all the BLAS/LAPACK related variations we're testing (should end up with ~6 jobs)linux.yml: testing across Python versions (PyPy, python-dbg, a pre-release Python in summer time), build-via-sdist, run benchmarks, measure code coverage, and other build options like relaxed-strides. This will also fold inlinux_meson.yml..shscript, for the checks that are now intravis-test.sh. For the rest, those shell script are just hard to read cruft.linux.ymlones, that would be really helpful.The text was updated successfully, but these errors were encountered: