Conversation
dba9b19 to
c55364a
Compare
|
@seiko2plus, cibuildwheel has an example CI setup that seems to cover what you're trying to do here,https://cibuildwheel.readthedocs.io/en/stable/faq/#emulation. It uses a GH Action to setup the emulation. It's probably worth investigating that path before iterating too much here. It may be easier to iterate CI configs on your personal fork, it saves a lot of CI triggering on the main repo. |
c55364a to
6144dff
Compare
I couldn't find a guide for native cross-compiling. What I'm trying to do is use binaries cross-compiled on the host system, which includes the "ninja" tool, while building. This mixes binaries from emulated architectures with the host's x86_64 architecture via binfmt.
My bad, I have moved the tests to my local machine |
6144dff to
30c232d
Compare
We've built macosx_arm64 wheels for scipy on an intel build machine using cibuildwheel+meson. So I'm not sure if this statement is true. Hmm. Do you mean that you have a cross compiler that builds on x86_64, but produces output for an armhf host? If so, then it's not clear why you have to use QEMU. If you're using QEMU aren't you using emulation to run a native armhf compiler to build for an armhf host? If this is the case then why not use https://github.com/docker/setup-qemu-action to start the docker container? |
30c232d to
66564cb
Compare
|
https://github.com/scipy/scipy/pull/17580/files shows the changeset for macosx cross compilation with scipy. |
|
macOS x86-64 to arm64 cross-compilation support is a special case, because Apple makes it extraordinarily easy. Beyond that, I think this |
|
Also: it'd be great to get this to work, so we can retire Travis CI usage. Thanks for working on it @seiko2plus. |
f2dc6f5 to
f192241
Compare
This patch implements cross-compile builds for armhf, ppc64le, and IBMZ architectures in the CI pipeline. While it might not be highly efficient due to qemu's quirks and slower performance, it still does extend testing to include umath, ufunc, and simd operations. In this setup, QEMU manages the Python interpreter, meson, and runtime tests, while ninja, the toolchain, and any binutils binaries are executed natively to speed up the build.
f192241 to
4be83a5
Compare
|
All green, the runtime tests only covers "test_kind, test_multiarray, test_simd, test_umath and test_ufunc" which is good enough I suppose. The full test takes up to 25m without container cache involved, not sure how to test the cache yet. |
|
I made a manual re-run for armhf with debug enabled but still missed the cache: ##[debug]Resolved Keys:
##[debug]["container-Linux-arm-linux-gnueabihf-arm32v7/ubuntu:22.04-8bc1d1beea3ee08d8a0df5594ff514fb36a313213aaada747cc83e9ba243a8e6"]
##[debug]Checking zstd --quiet --version
##[debug]1.5.5
##[debug]zstd version: 1.5.5
##[debug]Resource Url: https://acghubeus2.actions.githubusercontent.com/sH05vHzoTO8FskJ61ZFM2de2fOtiqhFE8SSrZeKfiH7OXqclo7/_apis/artifactcache/cache?keys=container-Linux-arm-linux-gnueabihf-arm32v7%2Fubuntu%3A22.04-8bc1d1beea3ee08d8a0df5594ff514fb36a313213aaada747cc83e9ba243a8e6&version=82cace457337bec689d5d2c52d6ee729c5a67a454f53c4372afbe7269f7d18f6
##[debug]Resource Url: https://acghubeus2.actions.githubusercontent.com/sH05vHzoTO8FskJ61ZFM2de2fOtiqhFE8SSrZeKfiH7OXqclo7/_apis/artifactcache/caches?key=container-Linux-arm-linux-gnueabihf-arm32v7%2Fubuntu%3A22.04-8bc1d1beea3ee08d8a0df5594ff514fb36a313213aaada747cc83e9ba243a8e6
##[debug]Failed to delete archive: Error: ENOENT: no such file or directory, unlink ''
Cache not found for input keys: container-Linux-arm-linux-gnueabihf-arm32v7/ubuntu:22.04-8bc1d1beea3ee08d8a0df5594ff514fb36a313213aaada747cc83e9ba243a8e6
##[debug]Node Action run completed with exit code 0
##[debug]Save intra-action state CACHE_KEY = container-Linux-arm-linux-gnueabihf-arm32v7/ubuntu:22.04-8bc1d1beea3ee08d8a0df5594ff514fb36a313213aaada747cc83e9ba243a8e6
##[debug]Finishing: Cache docker container |
The Post Cache docker container step contains: The path it's looking for is: and it's being saved to: with |
|
@rgommers, oh nice catch you made my day! |
|
Another thing we have a limit of 10GB cache currently about 8.5GB is been used, is there away to increase it? |
|
No, it's 10 GB per repository (https://github.blog/changelog/2021-11-23-github-actions-cache-size-is-now-increased-to-10gb-per-repository/) and you can't even pay to increase it. How much more do we need with these new jobs? Or is it 8.5 GB including these jobs, and you're worried about cache evictions on re-runs? |
c316a86 to
f012f1d
Compare
Currently three containers one for each architecture, all of them may exceed 1.5GB and we may need to adds support for risc-v, and ppc64(big-endian).
without including these jobs.
yes, saving about 8 minutes on each run may allow us to extend the current runtime tests to cover more unites. |
|
Can we cache in a third party location, akin to nightly wheels? E g. Dockerhub. An account for OSS projects can have up to 200 image pulls every six hours. |
Yes, we can since the cache hit can be detected to enable/disable steps.
LGTM, lets just see first how github manages the cache storage once its exceed 10GB. |
It's possible, but caching outside of GitHub is way slower. We used Docker Hub for the Gitpod images, which were also ~1.5GB and those took a couple of minutes to transfer and load. Plus we got rid of a lot of images after Docker Hub made changes to their free team plan. So I'm not very eager to look at it now.
+1, we can reassess after merging this. |
|
I made a re-run but it doesn't work, and it seems the cache will never hit, so after checking the debugging URLs, I just realized that its seem to be only allowed through the default branch maybe due to security concerns: {"$id":"1","innerException":null,"message":"The user 'System:PublicAccess;aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' is not authorized to access this resource.","typeName":"Microsoft.TeamFoundation.Framework.Server.UnauthorizedRequestException, Microsoft.TeamFoundation.Framework.Server","typeKey":"UnauthorizedRequestException","errorCode":0,"eventId":3000}
So it seem there's no way to handle it through github. |
|
According to the github doc https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache, there's still chance to works after the merge in order to gain the write access. |
|
Let's put this in and iterate on the caching setup as needed. |
|
Thanks @seiko2plus |
|
Awesome! @seiko2plus can we delete |
|
The cache page seems to show we have only one cached image: a 390MB one named |
|
That's from the Emscripten job, and it's because the We have nothing else that uses a cache right now, so we should be fine space-wise. |
I just left it for testing distutils, no other reason for keeping it. |
This patch implements cross-compile builds for armhf, ppc64le, and IBMZ architectures in the CI pipeline.
In this setup, QEMU manages the Python interpreter, meson, and runtime tests, while ninja,
the toolchain, and any binutils binaries are executed natively to speed up the build.
While it might not be highly efficient due to qemu's quirks and slower performance,
it still does extend testing to include multiarray, umath, ufunc, and simd operations.