8000 BUG: Factor out slow `getenv` call used for memory policy warning by seberg · Pull Request #24248 · numpy/numpy · GitHub
[go: up one dir, main page]

Skip to content

BUG: Factor out slow getenv call used for memory policy warning #24248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 24, 2023

Conversation

seberg
Copy link
Member
@seberg seberg commented Jul 24, 2023

Using getenv regularly is probably not great anyway, but it seems very slow on windows which leads to a large overhead for every array deallocation here.

Refactor it out to only check on first import and add helper because the tests are set up slightly differently.
(Manually checked that the startup works, tests run with policy set to 1, not modifying it and passing.)

Closes gh-24232.

Using `getenv` regularly is probably not great anyway, but it seems
very slow on windows which leads to a large overhead for every array
deallocation here.

Refactor it out to only check on first import and add helper because
the tests are set up slightly differently.
(Manually checked that the startup works, tests run with policy
set to 1, not modifying it and passing.)
@seberg seberg requested a review from mattip July 24, 2023 08:13
@seberg seberg added the 09 - Backport-Candidate PRs tagged should be backported label Jul 24, 2023
@seberg seberg added this to the 1.25.2 release milestone Jul 24, 2023
@mattip
Copy link
Member
mattip commented Jul 24, 2023

LGTM. I am trying to run the benchmark comparison on this, but something seems off

python runtests.py --bench-compare HEAD main
...
·· Building 1d0487b5 <getenv-slow-warn-mem-policy> for virtualenv-py3.11-Cython-packaging-setuptools59.2.0..........................................................
·· Installing 1d0487b5 <getenv-slow-warn-mem-policy> into virtualenv-py3.11-Cython-packaging-setuptools59.2.0
·· Failed to build the project and import the benchmark suite.

It would be nice to see that this has some echo in the benchmarks.

@seberg
Copy link
Member Author
seberg commented Jul 24, 2023

Hmmm, good question, I do think a few of our benchmarks should notice this, although I don't think we have dealloc benchmarks specifically.

spin bench --compare -t bench_scalar

works for me on main on MacOS, but I don't have access to my windows test machine on which I dug into this right now.

(scalar, because I suspect a few of them create temporary 0-D arrays making up a big chunk of their overhead. Another benchmark to try could be ArrayCoercionSmall)


I cannot see a performance difference on MacOS. I suspect the only reason why this getenv call was there in the first place is because we couldn't measure its performance impact on linux/mac.

@mattip
Copy link
Member
mattip commented Jul 24, 2023

Once I worked out that I had pkg-config set up to use OpenBLAS on windows, which does not work (see #24251), I could disable it and run benchmarks. It seems benchmarks for bench_scalar on windows are quite random. Running the reproducer at the top of the issue printed, for 1.25.1

Function completed in: 0.64 seconds
Function returned after 2.4 seconds
Function completed in: 6.30 seconds
Function returned after 24.0 seconds

For this PR

Function completed in: 0.53 seconds
Function returned after 0.6 seconds
Function completed in: 5.66 seconds
Function returned after 6.7 seconds

Looks good!

@mattip mattip merged commit f1eb09d into numpy:main Jul 24, 2023
@mattip
Copy link
Member
mattip commented Jul 24, 2023

Thanks @seberg

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG: Garbage collection order of magnitude slower in numpy>=1.22
3 participants
0