返回ndarray的字典会导致使用Boost Python的内存泄漏 [英] Returning a dictionary of ndarray causes memory leaks using boost python

问题描述

我正在为python编写一个c ++模块.它获取图像，进行一些处理，然后返回图像字典.我遇到内存泄漏，无法弄清原因.

I am writing a c++ module for python. It takes a image, does some processing and returns a dictionary of images. I am having memory leaks which I can't figure out why..

我使用 opencv-ndarray-conversion 在cv::Mat和numpy.ndarray

我使用 Boost.Python 来转换c ++代码到python模块.

I use Boost.Python to convert c++ code to python module.

在运行htop来检查内存使用情况时，我使用以下python代码测试c ++模块.

I use the following python code to test the c++ module, while running htop to check the memory usage.

import cv2
import this_cpp_module

for i in xrange(100000):
    img = cv2.imread('a_640x480x3_image.png')
    ret = this_cpp_module.func(img)
    #this 'func' is mapping to one of the following c++ functions, using Boost.Python:
    #    func1, func2 or func3.

1，转换图像不会导致内存泄漏

1, Converting the image does not cause memory leaks

using namespace boost::python;
PyObject * func1(PyObject *image)
{
    NDArrayConverter cvt;
    cv::Mat mat;
    mat = cvt.toMat(image);
    PyObject* ret = cvt.toNDArray(mat);
    return ret;
}

2，构造字典并将图像放入其中不会导致内存泄漏

2, Constructing a dictionary and putting the image into it do not cause memory leaks

using namespace boost::python;
dict func2(PyObject *image)
{
    dict pyDict;    
    object objImage(handle<>(borrowed(image)));
    pyDict[std::string("key")] = objImage;    
    return pyDict;
}

3，但是将它们组合会导致内存泄漏(每个循环大约1MB)

3, But combining them causes the memory leaks (around 1MB per loop)

dict func3(PyObject *image)
{
    return func2(func1(image));
}

我无法弄清楚.一切对我来说似乎都是正确的，但将它们组合在一起只会导致此问题.

I cannot figure it out. Everything seems to be correct to me but combining them together just causes this problem.

推荐答案

泄漏是func3()从未正确处置func1()返回的临时拥有引用的结果.要解决此问题，func3()需要执行以下操作之一:

The leak is a result of func3() never properly disposing the temporary owned reference returned by func1(). To resolve this, func3() needs to do one of the following:

• 在返回的自有引用上明确调用 Py_DECREF() 从func3()返回之前.
• 使用func1()返回的值. rel ="nofollow"> boost::python::handle ，因为它会在handle被销毁时减少对象的引用计数.

• Explicitly invoke Py_DECREF() on the owned reference returned from func1() before returning from func3().
• Manage the value returned by func1() with a boost::python::handle, as it will decrement the object's reference count when the handle is destroyed.

例如，func3()可以写为:

boost::python::dict func3(PyObject* image)
{
  // func1() returns an owned reference, so create a handle to keep the
  // object alive for at least as long as the handle remains alive.  The
  // handle will properly dispose of the reference.
  boost::python::handle<> handle(func1(image));
  return func2(handle.get());
}

有关原始问题的详细信息，当func1()返回时，返回的对象具有

For details on the original problem, when func1() returns, the returned object has a reference count of 1. Upon returning from func2() and func3(), the object has a reference count of 2. When the dict returned from func3() is destroyed, the object initially returned from func1() will have its reference count decremented by 1, resulting in the leaked object having a reference count of 1.

这是一个基于原始代码的完整的最小示例:

Here is a complete minimal example based on the original code:

#include <boost/python.hpp>

PyObject* func1(PyObject*)
{
  return PyList_New(0);
}

boost::python::dict func2(PyObject* obj)
{
  namespace python = boost::python;
  python::dict dict;
  python::handle<> handle(python::borrowed(obj));
  dict[std::string("key")] = python::object(handle);
  return dict;
}

boost::python::dict func3(PyObject* obj)
{
  // Fails to properly dispose of the owned reference returned by func1(),
  // resulting in a leak.
  return func2(func1(obj));
}

boost::python::dict func4(PyObject* obj)
{
  // func1() returns an owned reference, so create a handle to keep the
  // object alive for at least as long as the handle remains alive.  The
  // handle will properly dispose of the reference.
  boost::python::handle<> handle(func1(obj));
  return func2(handle.get());
}

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;
  python::def("func1", &func1);
  python::def("func2", &func2);
  python::def("func3", &func3);
  python::def("func4", &func4);
}

互动用法:

>>> from sys import getrefcount
>>> import example
>>> x = example.func1(None)
>>> assert(2 == getrefcount(x)) # refs: x and getrefcount
>>> d = example.func2(x)
>>> assert(3 == getrefcount(x)) # refs: x, d["key"], and getrefcount
>>> d = None
>>> assert(2 == getrefcount(x)) # refs: x and getrefcount
>>> d = example.func3(None)
>>> x = d["key"]
>>> assert(4 == getrefcount(x)) # refs: x, d["key"], getrefcount, and one leak
>>> d = None
>>> assert(3 == getrefcount(x)) # refs: x, getrefcount, and one leak
>>> d = example.func4(None)
>>> x = d["key"]
>>> assert(3 == getrefcount(x)) # refs: x, d["key"], and getrefcount
>>> d = None
>>> assert(2 == getrefcount(x)) # refs: x and getrefcount

这篇关于返回ndarray的字典会导致使用Boost Python的内存泄漏的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！