ENH: Enable musllinux wheels #21200
ENH: Enable musllinux wheels #21200
Conversation
|
@mattip in this comment #20089 (comment) the GH action you linked shows the wheel build but doesn't seem to have the wheel tests. Am I missing something there? Also, it seems like the wheel builder was skipped here, is there a way to kick it off without waiting for the cron? |
I added the "build" label, which will trigger the cibuildwheel action whenever a commit is made. You can also trigger that by adding "[wheel build]" to the git commit message. See the documentation for more info
Open the "build wheel" task, you will see the "test wheel" subtask which runs |
|
Wheel building is very resource intensive: it runs ~14 jobs and each one requires between 16 and 75 minutes, so we try to keep the number of runs down to a minimum. |
|
I think you need to add musl to the buildplatform. I am not sure what github actions calls the platform. |
|
Of course, thank you for starting it. I'm surprised any pipelines run automatically on PRs at all. Good catch, I wasn't sure about that either. Seems like this pipeline is unfortunately a waste so you're welcome to cancel. I'll look into the platform. |
|
No dice on the github actions info https://docs.github.com/en/actions/using-jobs/using-a-build-matrix-for-your-jobs but I'm wondering if something like Probably best to find an example from here https://cibuildwheel.readthedocs.io/en/latest/working-examples/ like tornado https://github.com/tornadoweb/tornado/blob/master/.github/workflows/build.yml |
|
Looks like the musl failures are because of missing openblas64 libs, Also lots of warnings about distutils to take a look at. I need to figure out the differences between this environment and the default Alpine docker env. At least we have a starting point. |
|
I opened an issue MacPython/openblas-libs#72 for openblas artifacts, but it seems there is another link in the chain that has to happen first: multibuild support for musl multi-build/multibuild#430 |
|
Thanks Matt, Yuck, growing needed work, but that was expected. Seems like multibuild is close, hope that rolls out soon. |
|
Since the multibuild PR was merged, I started MacPython/openblas-libs#73 to create the OpenBLAS artifacts build for x86_64 and aarch64 (musllinux) |
|
It might be nice to have 1-2 jobs in the regular CI testing musllinux. The wheel builders are somewhat hard to debug if something goes wrong because they run infrequently(only once a week). |
That part should be solved in a few hours. |
|
Awesome, they're there! Will take another look at this this afternoon |
|
The musl OpenBLAS libraries seem to be present now. |
|
Sorry, haven't had time to work on this in a bit. Will try to get back on it this week |
|
Should have figured this out before, can specify the build identifier for cibw local |
|
Please revert the whitespace changes |
|
Our linting policy is to only reformat lines that the PR itself changes, not to reformat entire files. |
|
Yes that YAML file was an accident that I didn't notice until after my most recent commit, I just didn't want waste the CI time with a fix & push until I had something worth running it |
|
There is an error |
|
xref #21425 to skip the hanging PyPy test |
|
No clue yet, I'm still trying to get a good debug setup working that uses an Alpine docker container and the cibuildwheel-built wheel. How are you testing locally? It doesn't show up using the quick test I had here #20089 (comment) but I guess it must have something to do with openblas or other features enabled with cibuildwheel but not From here it seems like the problem is that MachAr seems deprecated - I don't really understand the module enough to know if the issue is
|
|
In an Alpine container (just using pip install rather than a built wheel) running
|
|
I added a known type for that key pointing to float80 (same as install via Also disabled other wheel builds to not waste your CPU time |
|
With adding that type, the following is the test report: This is at least closer to the result of running pytest on a |
|
The |
|
OK I am now able to run tests on wheels built locally with cibuildwheel and tested in docker, but I can't seem to exactly replicate test results in any way. Things I've tried:
Test summary
Test reportRunning test_format on its own with Dockerfile I've been using is in tools/Dockerfile.musl |
|
Alright, looks like Edit: looks like it's not just my machine, I see the same test mentioned in #21425 Test reportSince barrier to entry on debugging musl wheels was brought up as an annoyance in the mailing list, it's probably a good idea to clean up tools/Dockerfile.musl a bit and keep it around. My test setup kinda-sorta works, building the docker image then running {
"name": "Docker: Attach",
"type": "python",
"request": "attach",
"localRoot": "${workspaceFolder}",
"remoteRoot": "/numpy",
"port": 5678,
"host": "localhost",
"justMyCode": true
}@stefanv sorry to pull you out of the blue but I notice you did the Alpine docker work for multi-build from the thread here. Do you happen to have a good workflow for working with/debugging musl wheels? |
There should be O(50) tests, you can identify them by searching for |
|
A lot of the failures seem related to |
Looking at this again, I think it's actually fine to skip those tests for now. There's only a couple (e.g., the For keeping this in shape, we should add a regular musl CI job I think, rather than only a wheel build job. |
| if "musllinux" in auditwheel_plat: | ||
| suffix = f"musllinux_1_1_{arch}.tar.gz" | ||
| typ = "tar.gz" | ||
| elif osname == "linux": |
There was a problem hiding this comment.
Determining the glibc is a bit obscure. Perhaps this should undergo a deeper refactoring to better differentiate between musllinux and manylinux
|
After gh-22864, there is passing Getting this PR in would be quite nice I think. |
|
I don't think any other changes are needed. I experimented with this on my mac M1 and things mostly worked**. Note that I'm building the aarch64 architecture because that's the native CPU type for my machine, but it shouldn't be any different on x86_64. ** there are some test failures, next comment. |
|
@rgommers the test fails were observed by @tgross35 previously. It's not clear to me why they weren't picked up by (bear in mind that I'm testing musllinux_aarch64 not musllinux_x86_64, I don't know if the two will be different) |
|
Woah, it's been a while since I touched this. If the tests are now xfail, that's perfect. Silly long double... at least it's not i128 where llvm literally violates the Linux ABI with its alignment. Regarding the changes I had in this PR:
So building the wheels are probably pretty trivial, if that's what you'd like to do (based on your comment here #20089 (comment) I suppose you could build and publish to GH but not PyPi, or something like that). Sorry I never wrapped this up, I was waiting on openblas and some other things and forgot to ever come back to it. @andyfaff it looks like you're taking a look at this more than I currently have time to - care to open a new PR? If so, I'll close this one. |
It does. The default test command is Minor floating point printing differences like those are not all that surprising - and not very relevant imho. |
|
@mattip this can be closed now. |
This is intended to address #20089
Per my comment here there will likely be some failures, hopefully the actions here will show exactly what needs to be addressed.