"import numpy" leaks memory #10157
Comments
|
The build with Python 3.4 also leaks. |
|
Yes, it does, and I doubt it will be fixed soon. I am not sure how much is leaked, so it might be possible to make sure that it leaks, but does not leak more every time. (I.e. numpy explicitly caches some stuff, and not all of that is cleaned up, it might be that there are worse leaking bugs around though, especially interned strings for example should not matter much overall). |
|
In other words, do you know how bad it leaks? It is a bug, but until now it never seemed like a big priority I guess. Of course any work improving is always welcome. |
|
The leak isn't big, just a couple of KB. I think ideally cached values should not show up under "definitely lost", i.e. it's fine if they aren't cleaned up as long as they are still reachable at the end of the program. "definitely lost" is a minor inconvenience if numpy is imported in another test suite (example: test_buffer in the Python stdlib) and you run Valgrind on that. I understand if it isn't a big priority, perhaps I'll dig around a bit myself some day. |
|
A couple of KB does sound like it might be more then a few cached strings and dtype objects though, hmmm. |
|
This is a minimal My guess is that something added by |
|
That sounds like nice narrowing down! More out of curiosity (but also to help someone else if they want to take this up): what do you run to produce the above output? And does this mean memory is not even recovered when you exit python? |
|
Its based on valgrind, in principle you should compile a debug python with valgrind support, other then that there are some suppression files you can use (I can search for mine/what Julian once also uploaded mostly). Then can run things like: (valgrind-py is just my silly alias to load the python suppression file as well, always run it first without valgrind, otherwise compliation will never finish ;)) @skrah yeah, I guess ADDDOC is the biggest part, and my guess is we can keep the string objects around and make sure to dealloc them on module destruction. Doesn't sound all that bad. |
|
@mhvk of course it is recovered on python exit, the OS will make sure of that, its just the point in time where you notice this kind of leak. |
|
With Python >=3.6, it may be enough to set the PYTHONMALLOC=malloc (sp?)
env variable to use an allocator more compatible with valgrind, so
Python debug build may not be needed.
|
|
In principle it would be possible to change add_newdocs to e.g. generate
a header file at compile time with the docs (as is done in
scipy.special), rather than patching them at runtime. Requires some
hunting for the correct places to specify the docstrings but should be
doable in practice.
|
|
@pv thanks, really good to know about that PYTHONMALLOC env variable! |
|
Adding the docstrings differently is on my TODO list, The way |
|
@mattip - if you have a minute, and won't fix it in the next day, maybe raise a separate issue just for this concrete item? (We even have the |
|
#10286 seems to fix the "definitely lost" issue here. |
|
Is this resolved? Looks like PR ( #10286 ) went in. Or are there other leaks from importing? |
|
I think it was intended to close this. |
Just executing
import numpy as npleaks memory:Python version: 3.6.3
Numpy version: 0d749ad
This could also be a CPython issue, but other extensions I have tested don't leak
on 3.6.3. I have seen similar leak patterns with Valgrind if an extension did not use
the GC alloc functions or if there was a cyclic GC problem.
Example (there are more):
The text was updated successfully, but these errors were encountered: