Description
Hi setuptools devs, I'd like to use this issue to summarize issues in the interaction between setuptools and numpy.distutils, and then see if we can find a way to resolve those structurally.
The setuptools 50.0 release gave NumPy and SciPy (a heavy numpy.distutils users) a number of concrete problems:
- gh-2368,
CCompiler_spawn() got an unexpected keyword argument 'env', also reported to NumPy at distutils MSVC: CCompiler_spawn() got an unexpected keyword argument 'env' numpy/numpy#17216 - setuptools >= 48.0.0 broken in AIX for Python 3.7 #2367, AIX build of SciPy broken, also reported to SciPy at Scipy build failure in AIX with Python 3.7 scipy/scipy#12797
- WIP: CI: try building with setuptools 50.0 numpy/numpy#17209, CI break in NumPy,
from distutils import sysconfigbroken on TravisCI - WIP: BLD: try to fix build with setuptools 50.0 on Windows scipy/scipy#12798, SciPy CI breakage due to a valid LDFLAGS fix showing up in
setuptools._distutils, which gave the SciPy build a new-ffat-lto-objectsthat it couldn't handle. - There's likely more that's not yet reported; wheel builds of multiple packages using older NumPy versions that haven't pinned
setuptoolsare definitely broken. Same for CI/deployment pipelines on ARM64 and other platforms that don't have wheels.
All of the above are resolvable. The bigger issue here is the interaction between distutils, numpy.distutils and setuptools. Due to setuptools release cadence, such breaks will keep on happening due to incompatibilities between numpy.distutils and setuptools. Let me summarize the issue, and then some options for dealing with it.
Old situation, for setuptools <50.0:
numpy.distutilsextends and monkeypatchesdistutils.- the import order in NumPy and SciPy
setup.pyis:
import setuptools # unused, just to let setuptools do its monkeypatching dance first
from numpy.distutils.core import setup
This situation worked reasonably well, because:
distutilsmoved slowly- larger
distutilschanges typically showed up in pre-releases of new Python versions, or in bugfix releases (say 3.7.7) that vendors like Anaconda qualify before they ship it, so we could adjustnumpy.distutilsto those changes setuptoolsdidn't do all that much that was relevant build-wise
New situation, for setuptools 50.0:
(default situation, no env var set)
setuptoolsreplacesdistutilswith its vendoredsetuptools._distutils, even if plaindistuilsis imported firstnumpy.distutilsis unchanged, so it still extends and monkeypatchesdistutils- which is nowsetuptools._distutils
So numpy.distutils finds itself monkeypatching setuptools code all of sudden, and that
setuptools code includes patches from Python 3.9-dev that are therefore now released into the wild with new setuptools releases without any alpha/beta/QA trajectory like we had before.
The releasing new patches quickly without testing will still be an issue for numpy.distutils probably, even if the SETUPTOOLS_USE_DISTUTILS="local" behaviour gets reverted for the time being (which seems likely, since there's a ton of other issues right now, like the Debian breakage).
What now?
Longer-term I don't think it's feasible for numpy.distutils to work as it does today and extend setuptools; the release cycle mismatch will be too much of a problem. I'm not quite sure what the best solution for that is though. Options could include:
- for every NumPy release, add a hard
setuptools <= current_setuptools_versionconstraint (NumPy releases get supported for 2.5 years by most of the scientific ecosystem though (i.e. ~50% of Python's user base), so that'll mean pretty oldsetuptoolsversions for building, e.g., wheels of packages that rely on NumPy - example: scikit-learn needs to build with NumPy 1.13.3 right now, so it would get setuptools 36.5) - make
numpy.distutilsa standalone package on top ofsetuptools(then at least release cycle can be matched, but situation remains unstable) - integrate
numpy.distutilsintosetuptools(more testing will then be done before setuptools releases and no extending/monkeypatching is needed, but is it a good idea knowledge and work load sharing wise?)
Let me emphasize that I do see the upsides in merging disutils into setuptools, both design and maintenance wise. And we (NumPy & scientific packages) have been burned in the past by distutils patches going unmerged for ages, so setuptools being better maintained is great.
Also, for context, it may be useful to list what numpy.distutils adds besides carrying around needed distutils patches:
- Fortran build support
- SIMD intrinsics support
- BLAS/LAPACK library support (OpenBLAS, MKL, ATLAS, Netlib LAPACK/BLAS, BLIS, 64-bit ILP interface, etc.)
- Support for a few other scientific libraries, like FFTW and UMFPACK (less often used)
- Better MinGW support
- Per-compiler build flag customization (e.g.
-O3andSSE2flags are default) - EDIT: a simple user build config system, see site.cfg.example
Looking forward to hearing your thoughts on this topic.