np.ctypeslib.as_array leaks memory when used on a pointer #6511

msebregts · 2015-10-19T09:13:44Z

I have a C++ function returning an array, which I convert to a numpy array using np.ctypeslib.as_array(pointer_from_C++_function, (size_of_array,)).

This works as expected, but when I repeatedly call this function (about a million times) I found a substantial increase of memory usage for my python process.

Running a sample script (attached, output attached) through valgrind's memcheck (output attached), it appears that the problem is in ctors.c, which calls the python C API function PyErr_WarnEx. The strange thing is that I never see a warning appear in my python output, so this could also be a bug in python.

For now I will try to work around this problem, but it would be great if this could be fixed.

Details of installation:
Ubuntu 15.04 x86-64
Python: 3.4.3 (installed from ubuntu repo), but problem arises with python 2.7.9 too (also from ubuntu repo)
Numpy 1.10.1 (from pip), but problem also present in numpy 1.8.2 (from ubuntu repo)

Working example:

#!/usr/bin/env python

from __future__ import print_function

import ctypes
import sys
import numpy as np

print("python version:", sys.version)
print("numpy version:", np.version.full_version)

def printvminfo():
  with open('/proc/self/status', 'r') as f:
    print(''.join(line for line in f.readlines() if line.startswith('Vm')))

# create array to work with
N = 100
a = np.arange(N)

# get pointer to array
pnt = np.ctypeslib.as_ctypes(a)

printvminfo()

Nrun = 1000
if len(sys.argv)>=2:
  Nrun = int(sys.argv[1])
print("Running", Nrun, "times")

for i in range(Nrun):
  # create a raw pointer (this is how my real c function works)
  newpnt = ctypes.cast(pnt, ctypes.POINTER(ctypes.c_long))
  # and construct an array using this data
  b = np.ctypeslib.as_array(newpnt, (N,))
  # now delete both, which should cleanup both objects
  del newpnt, b

# except it doesn't, RSS memory increases as a function of Nrun!
printvminfo()

Output of script

python version: 3.4.3 (default, Mar 26 2015, 22:03:40) 
[GCC 4.9.2]
numpy version: 1.10.1
VmPeak:    75088 kB
VmSize:    75088 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:     20820 kB
VmRSS:     20820 kB
VmData:    13448 kB
VmStk:       136 kB
VmExe:      3408 kB
VmLib:     10920 kB
VmPTE:       168 kB
VmSwap:        0 kB

Running 10000 times
VmPeak:    86960 kB
VmSize:    86960 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:     32636 kB
VmRSS:     32636 kB
VmData:    25320 kB
VmStk:       136 kB
VmExe:      3408 kB
VmLib:     10920 kB
VmPTE:       188 kB
VmSwap:        0 kB

Relevant valgrind output

<lots of output before>
==2956== 11,086,528 bytes in 9,952 blocks are definitely lost in loss record 720 of 720
==2956==    at 0x4C2BBA0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2956==    by 0x4C2DF4F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2956==    by 0x4D5123: _PyMem_RawRealloc (obmalloc.c:68)
==2956==    by 0x4D5123: PyMem_Realloc (obmalloc.c:308)
==2956==    by 0x4D5123: data_stack_grow (_sre.c:237)
==2956==    by 0x4D5123: sre_ucs1_match (sre_lib.h:506)
==2956==    by 0x4D8442: sre_match (_sre.c:511)
==2956==    by 0x4D8442: pattern_match (_sre.c:579)
==2956==    by 0x594194: PyObject_Call (abstract.c:2040)
==2956==    by 0x594194: call_function_tail.lto_priv.2483 (abstract.c:2078)
==2956==    by 0x5946ED: callmethod (abstract.c:2147)
==2956==    by 0x5946ED: _PyObject_CallMethodId (abstract.c:2192)
==2956==    by 0x5074D6: check_matched.part.1.lto_priv.2234 (_warnings.c:29)
==2956==    by 0x53369B: check_matched (_warnings.c:154)
==2956==    by 0x53369B: get_filter (_warnings.c:149)
==2956==    by 0x53369B: warn_explicit (_warnings.c:414)
==2956==    by 0x533884: do_warn (_warnings.c:665)
==2956==    by 0x53398C: warn_unicode (_warnings.c:790)
==2956==    by 0x5E503E: PyErr_WarnEx (_warnings.c:829)
==2956==    by 0x72D4599: _array_from_buffer_3118 (ctors.c:1277)
==2956== 
==2956== LEAK SUMMARY:
==2956==    definitely lost: 11,246,547 bytes in 10,035 blocks
==2956==    indirectly lost: 0 bytes in 0 blocks
==2956==      possibly lost: 208,484 bytes in 130 blocks
==2956==    still reachable: 1,390,813 bytes in 2,937 blocks
==2956==         suppressed: 0 bytes in 0 blocks
<some output after>

The text was updated successfully, but these errors were encountered:

pliskowski · 2016-06-14T20:24:14Z

I've observed similar behavior in my application. Without a doubt, there is a memory leak on np.ctypeslib.as_array. Is there any known workaround?

msebregts · 2016-06-19T16:07:48Z

In my application both the memory address and size remained constant most of the time. This allowed me to work around the bug by saving the previously created array:

_cache = None
def array_from_pointer(pnt, shape):
  if (_cache is None or
      _cache.shape != shape or
      _cache.__array_interface__['data'][0] != ctypes.addressof(pnt.contents) ):
    _cache = np.ctypeslib.as_array(pnt, shape)
  return _cache

Note: instead of a single item cache, it is of course possible to do memoization, there's good implementations of memoize decorators floating around on the web.

charris · 2016-06-19T17:07:41Z

I'm not familiar enough with ctypes to make a stab at this myself. A PR would be welcome.

andersjel · 2016-06-21T13:44:53Z

As a workaround, instead of

array = numpy.ctypeslib.as_array(c_array, (size,))

how about

ptr = ctypes.cast(c_array, ctypes.POINTER(ctypes.c_double * size))
array = numpy.frombuffer(ptr.contents)

pliskowski · 2016-06-21T13:47:58Z

Yeah, that's exactly what I've been doing. Prevents the leak in my case.

eric-wieser · 2018-04-26T06:20:46Z

Using np.frombuffer here is dangerous - it assumes a np.float64 dtype.

Simply using np.asarray(ptr.contents) will do the trick

eric-wieser · 2018-04-28T21:20:36Z

using the np.testing.assert_no_gc_cycles function on b5c1bcf reveals that the local class Stream in np.core._internals._dtype_from_pep3118 is causing the leak - in cpython, classes create reference cycles, which won't be cleaned up till the garbage collector runs.

import ctypes
import sys
import numpy as np

# create array to work with
N = 100
a = np.arange(N)

# get pointer to array
pnt = np.ctypeslib.as_ctypes(a)

with np.testing.assert_no_gc_cycles():
    # create a raw pointer (this is how my real c function works)
    newpnt = ctypes.cast(pnt, ctypes.POINTER(ctypes.c_long))
    # and construct an array using this data
    b = np.ctypeslib.as_array(newpnt, (N,))
    # now delete both, which should cleanup both objects
    del newpnt, b

(currently this test doesn't work at all on master, untion #10970 is merged)

Simply changing from

def _dtype_from_pep3118(spec):
    class Stream: ...

to

class _Stream: ...

def _dtype_from_pep3118(spec):

makes that leak go away.

Previously a local `Stream` class would be defined every time a format needed parsing. Classes in cpython create reference cycles, which create load on the GC. This may or may not resolve numpygh-6511

charris added 00 - Bug component: Other labels Jun 19, 2016

eric-wieser mentioned this issue Apr 26, 2018

WIP: Remove fragile use of __array_interface__ in ctypeslib.as_array #10970

Merged

eric-wieser mentioned this issue Jul 9, 2018

MAINT: Improve memory usage in PEP3118 format parsing #11535

Merged

charris closed this as completed in #11535 Aug 4, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

np.ctypeslib.as_array leaks memory when used on a pointer #6511

np.ctypeslib.as_array leaks memory when used on a pointer #6511

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

np.ctypeslib.as_array leaks memory when used on a pointer #6511

np.ctypeslib.as_array leaks memory when used on a pointer #6511

Comments

Working example:

Output of script

Relevant valgrind output

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!