8000 PEP 632, deprecation of distutils and numpy.distutils · Issue #18588 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

PEP 632, deprecation of distutils and numpy.distutils #18588

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
rth opened this issue Mar 10, 2021 · 21 comments
Closed

PEP 632, deprecation of distutils and numpy.distutils #18588

rth opened this issue Mar 10, 2021 · 21 comments
Assignees
@rgommers
Labels
component: numpy.distutils
Milestone
1.26.0 release

Comments

@rth
Copy link
Contributor
rth commented Mar 10, 2021

It looks like PEP 632 with the deprecation of distutils in stdlib and removal in Python 3.12 was accepted.

How would it impact numpy.distutils ? Does it mean that distutils would be added as a stand-alone build dependency (even if it's deprecated) or are there some other long term plans for numpy.distutils?
@rth rth mentioned this issue Mar 10, 2021
Closed
@kmaehashi kmaehashi mentioned this issue Mar 27, 2021
Open
@rgommers
Copy link
Member

If we're just going to deal with the setuptools vendoring distutils and then eventually exposing it with a new API, here is the plan:

If we're going to do a ton of work, the time is now though to move to a build system that doesn't hate compilers and has a sane design. Here is the plan for SciPy and I'm slowly moving that forward: scipy/scipy#13615 (likely will start moving faster soon).

If that move is successful, we should do the same for NumPy.

@saimn saimn mentioned this issue Jul 20, 2021
Merged
9 tasks
@pllim pllim mentioned this issue Sep 9, 2021
Closed
@rgommers
Copy link
Member
rgommers commented Dec 7, 2021 •
edited
Loading

My thinking on this topic has evolved a bit. setuptools enabled its vendored copy of distutils for the first time in Sep'21, and after that broke a lot of things reverted that. Re-enabling was planned a few months later, and I said back in Jan'22 that setuptools can go ahead without worrying about numpy.distutils (in pypa/setuptools#2372 (comment)). I expected that to happen soon, and then setuptools was going to establish a new API (original estimate in Sep'21 was 6-9 months for that). So we'd wait, and then fix up numpy.distutils. We're now 15 months further, and setuptools is still very far from exposing that new API (EDIT: setuptools is re-enabling its vendored copy in 2 weeks, see here). Based on the current trajectory, I don't think my original plan is working out. So my new thinking is:

  • Go with the pinning strategy that we've been using in NumPy and SciPy so far (in pyproject.toml, pin setuptools to the last version that is known to work for each release).
  • Fix things up in numpy.distutils when they break on a best-effort basis.
  • Projects that need significant features from numpy.distutils (Fortran support or BLAS/LAPACK detection for example) should migrate to a better build system next year. The two reasonable choices (Meson and CMake/scikit-build) are also still works-in-progress, but they're much more maintainable and performant.

@rgommers
Copy link
Member
rgommers commented Dec 8, 2021

The question then becomes how we communicate changes. We could:

  • deprecate numpy.distutils now, and say it will be removed from NumPy once Python 3.12 is our lowest supported version (should happen around 2025-2026), and will not be installed in any Python >=3.12 wheels and sdists.
  • have a separate doc somewhere that gives more detailed migration advice, and for complicated cases offer help
  • for projects/people that do want to stay with setuptools and need numpy.distutils features, say they are invited to help port those features to setuptools.

@rgommers rgommers added this to the 1.23.0 release milestone Dec 8, 2021
@rgommers rgommers self-assigned this Dec 8, 2021
@thomasjpfan thomasjpfan mentioned this issue Dec 21, 2021
Merged
This was referenced Jan 16, 2022
Closed
Merged
@rgommers
Copy link
Member

By far the most widely used numpy.distutils feature is nested setup.py files. It would make sense to port that to setuptools I think. It does need a volunteer to work on this (or, if there's a strong need but no one volunteers, we may consider funding this work through NumPy project funds at some point). In the migration guide from numpy.distutils to other build systems (see gh-20875) I will link to this issue as the tracking issue.

@MarkWieczorek MarkWieczorek mentioned this issue Apr 1, 2022
Closed
@seberg seberg modified the milestones: 1.23.0 release, 1.24.0 release May 4, 2022
@seberg
Copy link
Member
seberg commented May 4, 2022

Bumping the milestone, but if there is anyting important for the 1.23.0 release with respect to this issue, please make a note!

@rgommers
Copy link
Member
rgommers commented Nov 6, 2022

Given that distutils was removed from the stdlib for the Python 3.12 development cycle 3 days ago (python/cpython#99061), this issue is likely to get some traffic. Here is the up-to-date info: https://numpy.org/devdocs/reference/distutils_status_migration.html.

NumPy does not build with the distutils that setuptools vendored and then keeps modifying, so I expect that if someone is trying to build NumPy with a CPython 3.12 development version, things will not work.

We are moving away from disutils, so are not going to spend the time keeping up with setuptools as a moving target. If someone wants to take this on, I'm happy to review and merge patches (however, keep in mind that it's nontrivial, patches should work also with older setuptools versions that we actually need on Python 3.8-3.11, e.g. we're pinning setuptools 59.2.0 in our pyproject.toml, and users of numpy.distutils may be using other versions). For the most recent relevant discussion on this, see #21114 (comment).

I'll link to a new tracking issue for the build system migration to Meson soon, need to summarize the current plan and status.

@tacaswell
Copy link
Contributor

NumPy does not build with the distutils that setuptools vendored and then keeps modifying, so I expect that if someone is trying to build NumPy with a CPython 3.12 development version, things will not work.

I can confirm numpy does not install with either setuptool 59 or the latest (did not check any in between) with py312.

@CAM-Gerlach
Copy link
Contributor

As a stopgap, someone could potentially put a forward-port of distutils on PyPI, and NumPy could depend on that at build and/or runtime. You could also potentially use the distutils vendored into Setuptools 59, which presumably has substantially fewer (if any) than in later versions, and would not be a moving target.

@mattip
Copy link
Member
mattip commented Nov 7, 2022

You could also potentially use the distutils vendored into Setuptools 59

Does this work to build NumPy? If I understand correctly, @tacaswell commented above that it does not.

@CAM-Gerlach
Copy link
Contributor
CAM-Gerlach commented Nov 7, 2022 •
edited
Loading

On Setuptools <=59, Setuptools' copy of distutils doesn't replace the (missing in 3.12) stdlib version. However, NumPy could modify its imports to use that (assuming the changes/adjustments to work with that target would not be so much).

However, the forward-port on PyPI would be a much better solution overall, since it wouldn't require any changes to NumPy other than adding it to NumPy's build time and runtime requirements, and/or users manually installing it themselves. This will unblock building NumPy from source/sdist for 3.12, to not block early porting and testing to that version while the Meson migration is underway, or as a fallback plan in case it isn't fully ready. If it was not needed at runtime to support numpy.distutils, then it could be specified as a build-time requirement only, and NumPy binary wheels would not require installing it,

Of course, up to you all how you feel its best to approach that...though, IMO, we should still go ahead with the distutils forward-port regardless, for other packages that might need it.

@rgommers
Copy link
Member
rgommers commented Nov 7, 2022

My $2c:

  • Easiest is to do nothing, it's too early in the Python 3.12 release cycle to care about it, so it's not urgent for us. We typically start testing with Python development versions around March, somewhere around the alpha/beta stage.
  • Second easiest is for CPython to revert the deletion and wait till right before the alpha stage to remove distutils again.
  • Third option is for someone to work on one of the suggested workarounds and take responsibility for it. I recommend that that's not a NumPy maintainer, because we have more urgent/important things to do - like moving to Meson so we're actually ready for Python 3.12

@CAM-Gerlach
Copy link
Contributor

Just FYI, the first alpha is typically released right around the same time as the previous version .0 release, and the nominal goal is to have as many of the planned removals in by the first alpha as practical, so that downstream packages can start testing with it early and maintainers, contributors, distributors, etc. can identify issues as soon as possible to make progress on fixing them, and provide feedback upstream.

In this case, Alpha 1 was released a bit later than usual, around a week or two ago, and while it was specifically planned to be sure that the distutils removal was included for those reasons, due to an oversight it was still left in and only removed now, for Alpha 2. It is possible that the removal is reverted and deferred to the next release, if it looks like it would still be too disruptive to the ecosystem, but that would be probably at the end of the alpha stage or during the beta, if it happens.

Its certainly a possibility its reverted, but for better or for worse, the Setuptools changes with overriding the stdlib distutils mean that any packages, whether with legacy or PEP 517 builds, that are building in an environment with Setuptools installed (which is the great majority of the time) and don't explicitly require <=59 or happen to have it (probably not so likely with CPython 3.12), will already behave the same whether or not distutils is removed from the stdlib on the CPython side. because they will always be getting the Setuptools' bundled distutils.

@tacaswell
Copy link
Contributor

I also have little interest in doing the work to put a forward port on pypi. I would much rather put the effort into helping packages affected by this move to meson.

Does this work to build NumPy? If I understand correctly, @tacaswell commented above that it does not.

It does not work out of the box, but I have not tried setting the relevant ENV yet.

@rgommers rgommers mentioned this issue Nov 7, 2022
Closed
8000
@rgommers
Copy link
Member
rgommers commented Nov 14, 2022 •
edited
Loading

Okay, if someone really needs NumPy for Python 3.12 right now, the meson branch should work on Linux and macOS (see #22546 (comment)). For early adopters only, and there are no docs yet - but there's a GitHub Actions CI file in that branch, so I'm sure it can be figured out without too much trouble.

EDIT: scratch that, NumPy is updated but Meson itself still relies on distutils: mesonbuild/meson#7702. Guess we'll have to wait a little while.

@eli-schwartz
Copy link

Sysconfig is not (yet) a full replacement for distutils, because Meson sneakily uses a distutils Extension() object and checks whether it depends on libpython to figure out whether extensions need to be linked to the DLL on the current platform.

I intend to discuss this with CPython soon, but in the meantime, setuptools on python 3.12 will inject distutils and Meson's use is so simple that it currently still works with setuptools' patches (obviously due to setuptools' lack of API stability, no promises...), so...

... Installing setuptools as a build dependency should work as a quick hack to run Meson on the alphas. Obviously, this is a quick hack and Meson is not going to rely on this for the final release of python 3.12. ;)

@tacaswell
Copy link
Contributor
tacaswell commented Nov 15, 2022 •
edited
Loading

With the hint to install setuptools > 60 I was able to get numpy to compile and install

pip install -v . --no-build-isolation
meson setup build --prefix=/home/tcaswell/.virtualenvs/bleeding/
ninja -C build
meson install -C build

from the meson branch.

@tacaswell
Copy link
Contributor

The above commands does not install the dist-info, copying that directory (and accepting the lies) seems to get the rest of the stack building again.

@tacaswell
Copy link
Contributor

I'm also getting numpy reporting a version of '0+unknown' which is breaking scipy's build, will have to look into why that is ...

@rgommers
Copy link
Member

Should be fixed in the meson branch now, I hooked up versioneer.

@rgommers rgommers mentioned this issue Jan 4, 2023
Closed
@amotl amotl mentioned this issue Feb 9, 2023
Merged
@nilskattenbeck-bosch
Copy link

What is the intended plan for the usage of distutils inside numpy.ctypeslib.load_library?

@seberg seberg modified the milestones: 1.25.0 release, 2.0.0 release May 17, 2023
@mattip mattip mentioned this issue May 18, 2023
Merged
@matusvalo matusvalo mentioned this issue Aug 9, 2023
Closed
@dshick-nydig dshick-nydig mentioned this issue Oct 25, 2023
Closed
@rgommers
Copy link
Member

I'll close this issue, since we are basically done with this transition. NumPy 1.26.x defaults to Meson as the new build system, and numpy.distutils is deprecated and remains available for Python version up to 3.11.x (it is no longer installed for >=3.12).

No distutils code is used anymore on 3.12; there may be a handful of usages left (in f2py) and in the docs. We'll clean those up as we go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rgommers rgommers

Labels
component: numpy.distutils
Projects
None yet
Milestone
1.26.0 release
Development

No branches or pull requests
9 participants
@seberg @charris @rgommers @tacaswell @rth @mattip @eli-schwartz @CAM-Gerlach @nilskattenbeck-bosch
0