混淆的引用所有权:如何正确地(通过Py_DECREF)释放对象的对象? [英] Confusing reference ownership: how to properly deallocate (via Py_DECREF) objects of an object?

问题描述

我正在分析以下代码，这些代码可以正确编译和运行，但是会产生内存泄漏.

I was analysing the following code, which compiles and runs correctly, but generates a memory leak.

cfiboheap是Fibonacci堆的C实现，下面的代码是cfiboheap的Cython包装器(其一部分).

The cfiboheap is a C implementation of a Fibonacci Heap and the following code is a Cython wrapper (a part of it) for cfiboheap.

我对插入函数的怀疑开始了.对象data已在某处创建，并传递给函数insert().由于该函数希望将此对象添加到fiboheap中，因此它增加了其引用计数.但是之后呢?所有权归谁所有?以我的理解，C函数fh_insertkey()只是借用了所有权.然后，它返回需要封装的专有指针，然后由insert()返回.凉爽的.但是我的对象data及其引用计数?通过创建胶囊，我创建了一个新对象，但是我没有减少data的引用计数.这会导致内存泄漏.

My doubts starts on the insert function. The object data has been created somewhere and passed to the function insert(). Since the function wants to add this object to the fiboheap it increases its reference count. But afterwards? To whom the ownership goes? In my understanding, the C function fh_insertkey() just borrows the ownership. Then it returns a proprietary pointer that needs to be incapsulated, and then returned by insert(). Cool. But my object data and its ref count? By creating the capsule I'm creating a new object, but I'm not decreasing the ref count of data. This produces the memory leak.

(请注意，在insert()返回之前注释掉Py_INCREF或添加Py_DECREF会导致分段错误.)

(Note that commenting out Py_INCREF or adding Py_DECREF before the return of insert() results in a segmentation fault.)

我的问题是:

1)为什么在insert()期间必须增加data的引用计数?

1) Why is it necessary to increment the ref count of data during the insert()?

2)为什么在extract()期间不必使用Py_DECREF?

2) Why is it not necessary to use a Py_DECREF during the extract()?

3)更笼统地说，当在C和Python之间跳转时，如何准确跟踪引用所有权?

3) More generally, how can I exactly keep track of the reference ownership when jumping between C and Python?

4)如何正确地释放诸如FiboHeap这样的对象?我应该在__dealloc__()中预防性地使用Py_XDECREF，如果可以的话，如何使用?

4) How to properly deallocate an object like this FiboHeap? Should I use preventively a Py_XDECREF in __dealloc__() and, if yes, how?

谢谢！

cimport cfiboheap
from cpython.pycapsule cimport PyCapsule_New, PyCapsule_GetPointer
from python_ref cimport Py_INCREF, Py_DECREF 

cdef inline object convert_fibheap_el_to_pycapsule(cfiboheap.fibheap_el* element):
    return PyCapsule_New(element, NULL, NULL)

cdef class FiboHeap:

    def __cinit__(FiboHeap self):
        self.treeptr = cfiboheap.fh_makekeyheap()
        if self.treeptr is NULL:
            raise MemoryError()

    def __dealloc__(FiboHeap self):
        if self.treeptr is not NULL:
            cfiboheap.fh_deleteheap(self.treeptr)

    cpdef object insert(FiboHeap self, double key, object data=None):
        Py_INCREF(data)
        cdef cfiboheap.fibheap_el* retValue = cfiboheap.fh_insertkey(self.treeptr, key, <void*>data)
        if retValue is NULL:
            raise MemoryError()

        return convert_fibheap_el_to_pycapsule(retValue)

    cpdef object extract(FiboHeap self):
        cdef void* ret = cfiboheap.fh_extractmin(self.treeptr)
        if ret is NULL:
            raise IndexError("FiboHeap is empty")

        return <object> ret

    cpdef object decrease_key(FiboHeap self,  object element, double newKey):
        cdef void* ret = cfiboheap.fh_replacekey(self.treeptr, convert_pycapsule_to_fibheap_el(element), newKey)
        if ret is NULL:
            raise IndexError("New Key is Bigger")

        return <object> ret 

请注意，这不是我写的，但是我正在使用此示例来更好地理解obj引用并阻止泄漏(因为我实际上是在使用代码).

Note that this hasn't been written by me, but I'm using this example to better understand obj referencing and to stop the leak (since I am actually using the code).

使用FiboHeap(以及发生泄漏的位置)的主要代码如下:

The main code that makes use of FiboHeap (and where the leak happens) looks like this:

cdef dijkstra(Graph G, int start_idx, int end_idx):

    cdef np.ndarray[object, ndim=1] fiboheap_nodes = np.empty([G.num_nodes], dtype=object) # holds all of our FiboHeap Nodes Pointers
    Q = FiboHeap()
    fiboheap_nodes[start_idx] = Q.insert(0, start_idx)
    # Then occasionally:
    Q.insert(...)
    Q.decrease_key(...)
    Q.extract()

    return

extract不是偷看，而是正确的弹出窗口，因此它正在删除C fiboheap中的C元素.

extract is not a peek, but a proper pop, so it is deleting the C element in the C fiboheap.

结论:显然data的引用计数会导致内存泄漏，但是为什么呢?以及如何阻止它?

In conclusion: it seems clear that the ref count of data causes a memory leak, but why? And how to stop it?

推荐答案

1)必须增加insert中的引用计数，因为它的引用计数将在插入结束时自动减少. Cython不知道您要存储该对象以备后用. (您可以检查生成的C代码，以在函数末尾查看DECREF).如果使用引用计数为1的对象(即.insert(SomeObject()))调用insert，则在没有INCREF

1) It is necessary to increase the reference count in insert because its reference count will be automatically decreased at the end of insert. Cython does not know you are storing the object for later. (You can inspect the generated C code to see the DECREF at the end of the function). If insert is called with an object of reference count 1 (i.e. .insert(SomeObject()), then the object would be destroyed at the end of insert without the INCREF

2)如果在extract期间从cfiboheap中删除了对象，则应执行DECREF确认不再持有该对象.首先将其投射到对象上(因此您仍然拥有对其的引用)

2) If the object is removed from the cfiboheap during extract then you should do a DECREF to acknowledge the fact that you no longer hold it. Cast it to object first (so you still hold a reference to it)

   cdef void* ret = cfiboheap.fh_extractmin(self.treeptr) # refcount is 1 here (from the INCREF when it was stored)
   if ret==NULL:
        # ...

   ret_obj = <object>ret
   # reference count should be 2 here - one for being on the heap and one for ret_obj. Casting to object increases the refcount in Cython
   Py_DECREF(ret_obj) # 1 here
   return ret_obj

3)老实说，如果可以避免的话，请尽量不要使用PyObject*！最好让Cython来完成这项工作.如果无法避免，则只需确保存储对象时调用INCREF一次，而停止存储对象时调用一次DECREF.

3) Honestly you try not to use PyObject* if you can avoid it! It's much better to let Cython do the work. If you can't avoid it then you just need to ensure INCREF is called once when you store the object, and DECREF is called once when you stop storing it.

4)您确实需要对__dealloc__中堆上的其余对象进行解引用.一个简单的方法可能是对所有extract进行操作，直到cfiboheap为空:

4) You do need to decref the remaining objects on the heap in __dealloc__. A very easy way to do that might be to all extract until the cfiboheap is empty:

try:
    while True:
        self.extract()
except IndexError:
    pass # ignore the error - we're done emptying the heap

---

关于胶囊使用的评论:谁拥有它们指向的fibheap_el(何时销毁)?如果cfiboheap被破坏时它被破坏，则可能是一个胶囊的问题，其中无效的指针仍然有效.在某个地方使用该胶囊可能会导致问题.如果它没有被cfiboheap破坏，则可能存在另一个内存泄漏.

---

A comment about the use of capsules: who owns the fibheap_el that they point to (and when does this get destructed)? If it gets destructed when the cfiboheap gets destructed then you have the issue of a capsule with an invalid pointer still being alive. Using this capsule somewhere might end up causing problems. If it doesn't get destructed by the cfiboheap then you potentially have another memory leak.

这篇关于混淆的引用所有权:如何正确地(通过Py_DECREF)释放对象的对象?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！