TST: Remove memory leak test #27732
TST: Remove memory leak test #27732
Conversation
It is flaky, and our memory "limit" is pretty much arbitrary. Testing with `tools/memleak.py` shows that memory usage generally rises and only stabilizes close to 200 figures, while this test only does a warmup of 2 figures. Waiting for equilibrium would take too long for a general test, so we might move this to a weekly job later.
|
I'd be on board with a weekly run that's not tied to PRs. After all we're not really worried about any specific threshold, just if things are growing excessively, and a weekly trend would be easy to bisect, at least compared to bug reports. Maybe it would be worth putting together a whole pythonspeed thing with benchmarks and memory usage? |
|
I presume the x-axis in those plots is elapsed time in seconds? I'm 👍 to removing these to stop the CI failing for now given it doesn't seem like there's an easy or quick fix to the tests, but would be worth opening an issue to discuss ways to fix these tests or add replacement memory leak tests. |
The x-axis is cycle count (so how many Figures we have created and destroyed). |
PR summary
It is flaky, and our memory "limit" is pretty much arbitrary. Testing with
tools/memleak.pyshows that memory usage generally rises and only stabilizes close to 200 figures, while this test only does a warmup of 2 figures. Waiting for equilibrium would take too long for a general test, so we might move this to a weekly job later.For example, with just Agg:
and with the flaky wxAgg:
It's true that the latter uses more memory, but it still takes about 200 or so figure to reach equilibrium.
Closes #27635
PR checklist