8000 MAINT: update supported versions of Python and NumPy to follow NEP29 by tupui · Pull Request #17959 · scipy/scipy · GitHub
[go: up one dir, main page]
Skip to content

MAINT: update supported versions of Python and NumPy to follow NEP29 #17959

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 15 commits into from
Feb 21, 2023
Merged

MAINT: update supported versions of Python and NumPy to follow NEP29 #17959

merged 15 commits into from
Feb 21, 2023

Conversation

tupui
Copy link
Member
@tupui tupui commented Feb 10, 2023 •
edited
Loading

This PR does the work to follow NEP29 for the next release (1.11) which probably is going to be scheduled around June.

  • Drop support for Python 3.8, hence NumPy >=1.22.4.
  • Change the dev job to Python 12-dev instead of 11-dev. And bump all jobs 1 version up.
  • Update ubuntu image from 20 to 22 (Focal to Jammy).

Not sure about a few things and could have forgot to update some lines, please check carefully.

Closes #17959

@tupui tupui added the maintenance Items related to regular maintenance tasks label Feb 10, 2023
@tupui tupui added this to the 1.11.0 milestone Feb 10, 2023
@tupui tupui force-pushed the ci_3_12 branch 2 times, most recently from f3e3c62 to b9e7b1f Compare February 10, 2023 17:59
@andyfaff
Copy link
Contributor
andyfaff commented Feb 11, 2023 •
edited
Loading

I'd probably want to merge this after #17950. Oh, and don't you mean drop 3.8, not 3.9?

@tupui
Copy link
Member Author
tupui commented Feb 11, 2023

Correct!

What is the link with your PR?

@andyfaff
Copy link
Contributor

#17950 . There's a lot of overlap with the ci configs.

rgommers
rgommers requested changes Feb 11, 2023
View reviewed changes
Copy link
Member
@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tupui. A few initial comments.

The bump in NumPy versions from 1.19.5 to 1.22.4 (which is less than 9 months old) is quite aggressive. Unless there's a good reason for that, we should be more conservative and do a regular bump to 1.20.3 I'd say.

tupui
tupui commented Feb 13, 2023
View reviewed changes
Copy link
Member Author
@tupui tupui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rgommers

The bump in NumPy versions from 1.19.5 to 1.22.4 (which is less than 9 months old) is quite aggressive. Unless there's a good reason for that, we should be more conservative and do a regular bump to 1.20.3 I'd say.

I just think we should follow the NEP. 1.21 is supposed to be dropped around our release time in June.

On Apr 14, 2023 drop support for Python 3.8 (initially released on Oct 14, 2019)
On Jun 23, 2023 drop support for NumPy 1.21 (initially released on Jun 22, 2021)

Also, from what I have seen, our deps tend to update NumPy way faster than SciPy. By the time they update to 1.11, I don't think it would be an issue to require NumPy 1.21.

@tupui
8000
Copy link
Member Author
tupui commented Feb 13, 2023 •
edited
Loading

Not sure how to fix some of the failures otherwise (the linter could be ignored since it's only complaining about the setup.py).

tupui
tupui commented Feb 13, 2023
View reviewed changes
azure-pipelines.yml Outdated
@@ -177,29 +177,30 @@ stages:
set -euo pipefail
docker pull i386/ubuntu:bionic
docker run -v $(pwd):/scipy i386/ubuntu:bionic /usr/bin/linux32 /bin/bash -c "cd scipy && \
Copy link
Member Author
@tupui tupui Feb 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could use this instead? https://hub.docker.com/r/i386/python

Would mean going from an Ubuntu to a Debian. Which might be a good idea to test.

i386/python:3.9-bullseye for instance

Also Python3.9 is available on Bullseye as opposed to later Ubuntu.

tupui
tupui commented Feb 13, 2023
View reviewed changes
tupui
tupui commented Feb 13, 2023
View reviewed changes
@tupui
Copy link
Member Author
tupui commented Feb 13, 2023 •
edited
Loading

The Mypy error is a known bug and we would need to update mypy to work on Python 3.10. At least 0.981. python/mypy#13627

Not sure why we are pinning to 0.931. Also mypy 1.0 is out with great speed bump.

@tupui
Copy link
Member Author
tupui commented Feb 13, 2023

Locally I could bump mypy, there was just a very minor things to adjust. We could remove the mypy requirements file too. Not done yet in case we want to keep it for some reason.

@tupui
Copy link
Member Author
tupui commented Feb 13, 2023 •
edited
Loading

@rgommers CI is mostly happy now, only 3 things remain.

  • Linux_Python_39_32bit_full fails with a timeout. Seems like the setup is waiting for user input.

  • scipy.scipy (Main Windows Python311-64bit-full-ilp64) fails with:

scipy\_lib\_ccallback_c.c(198): fatal error C1083: Cannot open include file: 'longintrepr.h': No such file or directory

Not sure, some suggest to update Cython? Might be due to these lines PyLong_AsVoidPtr(long(context_obj))

  • macOS tests (setup.py) / macOS Test Matrix (3.9, --upgrade numpy) (pull_request) fails with:

I don't know. @andyfaff I guess your PR could fix this as you also update the Python version.

Any help appreciated 🙇 Feel free to push on the branch directly.

@mdhaber
Copy link
Contributor
mdhaber commented Feb 14, 2023 •
edited
Loading

I just think we should follow the NEP. 1.21 is supposed to be dropped around our release time in June.

Perhaps we can compromise. To get back on track with NEP 29 without a huge jump in one release:

  • Release SciPy 1.11.0 on or after June 23, 2023, and make NumPy 1.21 the minimum supported version for that release.
  • Release SciPy 1.12.0 on or after January 1, 2024, and make NumPy 1.23 the minimum supported version for that release.

This would be consistent with the support table, and it wouldn't require a jump of more than two minor NumPy versions per SciPy release.

@rgommers
Copy link
Member

The Mypy error is a known bug and we would need to update mypy to work on Python 3.10. At least 0.981. python/mypy#13627

Not sure why we are pinning to 0.931. Also mypy 1.0 is out with great speed bump.

The exact version doesn't matter, a bump is usually fine. We do need to pin to an exact version because new Mypy versions very often have both regressions and changes in behavior. And even plain improvements result in errors, because #type: ignore commands for things that are supported in a newer version result in a complaint about an unnecessary skip IIRC.

@rgommers
Copy link
Member

scipy.scipy (Main Windows Python311-64bit-full-ilp64) fails with:

I suggest separating the problems here a bit. Just do not bump the version from 3.10 to 3.11 and open an issue for this one to fix in a separate PR.

@rgommers
Copy link
Member

just think we should follow the NEP. 1.21 is supposed to be dropped around our release time in June.

Perhaps we can compromise. To get back on track with NEP 29 without a huge jump in one release:

* Release SciPy 1.11.0 on or after June 23, 2023, and make NumPy 1.21 the minimum supported version for that release.

* Release SciPy 1.12.0 on or after January 1, 2024, and make NumPy 1.23 the minimum supported version for that release.

This would be consistent with the support table, and it wouldn't require a jump of more than two minor NumPy versions per SciPy release.

There's two things here to keep in mind:

  1. NEP 29 doesn't explicitly account for bug-fix versions, and NumPy has started to do those for older minor releases more frequently,
  2. SciPy and NumPy release schedules more or less coincide, with releases around Dec/Jan and Jun/Jul.

For (1), we should be careful not to pick very recent bug-fix releases. NEP 29 aims for NumPy versions up to 24 months old; picking a version that's only 9 months old is going to result in problems for users too often.

For (2), we should aim slightly on the conservative side here, rather than effectively making our support window 18 instead of 24 months.

Now for the actual versions: we're aiming to release SciPy 1.11.0 around the middle of the year. NumPy 1.21.0 was released on June 22, 2021. So that's roughly 24 months. So I'd say that should be included, and 1.22.x is definitely too aggressive. The latest 1.21.x release is 1.21.6, released on April 12, 2022. So that will be 14 months old at that point. That's probably okay. So let's go with that, and we're adhering to NEP 29 as is.

@tupui tupui mentioned this pull request Feb 14, 2023
Closed
@tupui tupui requested a review from rgommers February 14, 2023 11:50
rgommers
rgommers reviewed Feb 15, 2023
View reviewed changes
Copy link
Member
@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two more comments on the pyproject.toml changes.

@tupui tupui force-pushed the ci_3_12 branch 2 times, most recently from 1a0f029 to b572734 Compare February 15, 2023 13:46
@tupui tupui requested a review from rgommers February 15, 2023 14:51
rgommers
rgommers reviewed Feb 15, 2023
View reviewed changes
@rgommers
Copy link
Member

What should we do? xfail on 32bits?

It looks like the test needs tweak. The _nulp assert seems to be failing on comparing 0 and 3.0...e-34. So changing it to use a different test function with atol=1e-33 is probably fine. Not 100% sure though without spending more time - for signal processing functions such small absolute numbers are sometimes relevant.

So how about skipping (not xfailing) the test here, and then opening a follow-up PR changing to assert_allclose? That way we can get this merged, and get some help from the signal experts on a much smaller PR.

@tupui
Copy link
Member Author
tupui commented Feb 17, 2023

@rgommers CI is green.

@tupui
Copy link
Member Author
tupui commented Feb 17, 2023

Mistake rebased push the wrong thing, hopefully I did not break anything 😒

@tupui
Copy link
Member Author
tupui commented Feb 17, 2023 •
edited
Loading

Azure itself did not like the rebase, but seems all good otherwise.

EDIT: looks all good now.

@tupui tupui requested a review from rgommers February 18, 2023 12:17
@tupui
Copy link
Member Author
tupui commented Feb 20, 2023

Anything else to do here?

rgommers
rgommers approved these changes Feb 21, 2023
View reviewed changes
Copy link
Member
@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM now, let's give it a go. Thanks @tupui! And thanks @andyfaff for the reviews

@rgommers rgommers merged commit 890ffe9 into scipy:main Feb 21, 2023
@tupui
Copy link
Member Author
tupui commented Feb 21, 2023

Thanks all 😃

@tupui tupui deleted the ci_3_12 branch February 21, 2023 23:41
rgommers added a commit to rgommers/scipy that referenced this pull request Feb 23, 2023
@rgommers
BLD: build Windows wheel for py39 against numpy 1.22.3
437d88c 
This is a follow-up to scipygh-17959, which bumped numpy from
1.19.5 to 1.21.6, which is wrong because that version was
built with vc142 (see comment above the line added in this PR).

[wheel build]
[skip circle] [skip cirrus] [skip azp]
@rgommers rgommers mentioned this pull request Feb 23, 2023
Merged
rgommers added a commit to rgommers/scipy that referenced this pull request Feb 23, 2023
@rgommers
BLD: build Windows wheel for py39 against numpy 1.22.3
f1369e2 
This is a follow-up to scipygh-17959, which bumped numpy from
1.19.5 to 1.21.6, which is wrong because that version was
built with vc142 (see comment above the line added in this PR).

[wheel build]
[skip circle] [skip cirrus] [skip azp]
@lesteve lesteve mentioned this pull request May 26, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andyfaff andyfaff andyfaff left review comments

@rgommers rgommers rgommers approved these changes

@larsoner larsoner Awaiting requested review from larsoner larsoner is a code owner

Assignees
No one assigned
Labels
maintenance Items related to regular maintenance tasks
Projects
None yet
Milestone
1.11.0
Development

Successfully merging this pull request may close these issues.
4 participants
@tupui @andyfaff @mdhaber @rgommers
0