指向相同地址的空指针 [英] Void pointer pointing to the same address
问题描述
问题
指向cdef类的无效指针指向相同的内存地址，而没有强制使用python引用计数器.
说明
我有一个简单的类，希望通过将其转换为void指针存储在cpp向量中.但是，在打印了指针指向的内存地址之后，它会在第二次迭代之后重复执行，除非，我通过将新对象添加到列表中来强制增加引用计数器.有人可以在没有引用计数器强制的情况下为何返回内存吗?
# distutils: language = c++
# distutils: extra_compile_args = -std=c++11
from libcpp.vector cimport vector
from libc.stdio cimport printf
cdef class Temp:
cdef int a
def __init__(self, a):
self.a = a
def f():
cdef vector[void *] vec
cdef int i, n = 3
cdef Temp tmp
cdef list ids = []
# cdef list classes = [] # force reference counter?
for i in range(n):
tmp = Temp(1)
# classes.append(tmp)
vec.push_back(<void *> tmp)
printf('%p ', <void *> tmp)
ids.append(id(tmp))
print(ids)
f()
哪个输出:
[140137023037824, 140137023037848, 140137023037824]
但是，如果我通过将引用计数器添加到类列表中来强制引用计数器:
[140663518040448, 140663518040472, 140663518040496]
答案很长，因此可以快速浏览内容:
- 观察到的行为的解释
- 避免问题的天真方法
- 更系统，更典型的c ++解决方案
- "nogil"模式下的多线程代码说明问题
- 扩展针对nogil模式的c ++典型解决方案
观察到的行为的解释
与Cython的交易:只要您的变量的类型为object或从中继承(在您的情况下为cdef Temp)，cython就会为您管理引用计数.一旦将其强制转换为PyObject *或任何其他指针，引用计数就是您的责任.
很明显，对已创建对象的唯一引用是变量tmp，将其重新绑定到新创建的Temp -object时，旧对象的引用计数器将变为0，并且破坏-向量中的指针悬空.但是，相同的内存可以重用(很可能是这样)，因此您总是看到相同的重用地址.
天真解决方案
您如何进行引用计数?例如(我使用的是PyObject *而不是void *):
...
from cpython cimport PyObject,Py_XINCREF, Py_XDECREF
...
def f():
cdef vector[PyObject *] vec
cdef int i, n = 3
cdef Temp tmp
cdef PyObject *tmp_ptr
cdef list ids = []
for i in range(n):
tmp = Temp(1)
tmp_ptr = <PyObject *> tmp
Py_XINCREF(tmp_ptr) # ensure it is not destroyed
vec.push_back(tmp_ptr)
printf('%p ', tmp_ptr)
ids.append(id(tmp))
#free memory:
for i in range(n):
Py_XDECREF(vec.at(i))
print(ids)
现在，所有对象均保持活动状态，并且仅在显式调用Py_XDECREF之后才消亡".
C ++典型解决方案
以上不是一种非常典型的c ++处理方式，我宁愿介绍一种自动管理引用计数的包装器(与std::shared_ptr相同):
...
cdef extern from *:
"""
#include <Python.h>
class PyObjectHolder{
public:
PyObject *ptr;
PyObjectHolder():ptr(nullptr){}
PyObjectHolder(PyObject *o):ptr(o){
Py_XINCREF(ptr);
}
//rule of 3
~PyObjectHolder(){
Py_XDECREF(ptr);
}
PyObjectHolder(const PyObjectHolder &h):
PyObjectHolder(h.ptr){}
PyObjectHolder& operator=(const PyObjectHolder &other){
Py_XDECREF(ptr);
ptr=other.ptr;
Py_XINCREF(ptr);
return *this;
}
};
"""
cdef cppclass PyObjectHolder:
PyObjectHolder(PyObject *o)
...
def f():
cdef vector[PyObjectHolder] vec
cdef int i, n = 3
cdef Temp tmp
cdef PyObject *tmp_ptr
cdef list ids = []
for i in range(n):
tmp = Temp(1)
vec.push_back(PyObjectHolder(<PyObject *> tmp)) # vector::emplace_back is missing in Cython-wrappers
printf('%p ', <PyObject *> tmp)
ids.append(id(tmp))
print(ids)
# PyObjectHolder automatically decreases ref-counter as soon
# vec is out of scope, no need to take additional care
值得注意的事情:
-
PyObjectHolder一旦拥有PyObject指针，便增加ref-counter，并在释放指针时立即将其降低. - 三个规则意味着我们还必须注意复制构造函数和赋值运算符
- 我已经省略了c ++ 11的move-stuff，但是您也需要注意这一点.
使用nogil模式的问题
但是，有一件非常重要的事情:您不应该通过上述实现发布GIL (即，将其导入为PyObjectHolder(PyObject *o) nogil，但是当C ++复制向量和类似物时也会出现问题)-因为否则Py_XINCREF和Py_XDECREF可能无法正常工作.
为说明这一点，让我们看下面的代码，该代码释放gil并并行执行一些愚蠢的计算(整个魔术单元格位于答案的结尾处):
%%cython --cplus -c=/openmp
...
# importing as nogil - A BAD THING
cdef cppclass PyObjectHolder:
PyObjectHolder(PyObject *o) nogil
# some functionality using a lot of incref/decref
cdef int create_vectors(PyObject *o) nogil:
cdef vector[PyObjectHolder] vec
cdef int i
for i in range(100):
vec.push_back(PyObjectHolder(o))
return vec.size()
# using PyObjectHolder without gil - A BAD THING
def run(object o):
cdef PyObject *ptr=<PyObject*>o;
cdef int i
for i in prange(10, nogil=True):
create_vectors(ptr)
现在:
import sys
a=[1000]*1000
print("Starts with", sys.getrefcount(a[0]))
# prints: Starts with 1002
run(a[0])
print("Ends with", sys.getrefcount(a[0]))
#prints: Ends with 1177
我们很幸运，该程序没有崩溃(但是可以！).但是由于竞争条件，我们最终发生了内存泄漏-a[0]的引用计数为1177，但是仅存在1000个引用(sys.getrefcount内部+2)活着的引用，因此该对象永远不会销毁. /p>
使PyObjectHolder线程安全
那该怎么办?最简单的解决方案是使用互斥锁来保护对ref-counter的访问(即，每次调用Py_XINCREF或Py_XDECREF时).这种方法的缺点是可能会降低单个核心代码的速度(例如，请参见这篇较旧的文章是关于一种较旧的尝试通过互斥锁相似的方法替换GIL的文章.
这是一个原型:
%%cython --cplus -c=/openmp
...
cdef extern from *:
"""
#include <Python.h>
#include <mutex>
std::mutex ref_mutex;
class PyObjectHolder{
public:
PyObject *ptr;
PyObjectHolder():ptr(nullptr){}
PyObjectHolder(PyObject *o):ptr(o){
std::lock_guard<std::mutex> guard(ref_mutex);
Py_XINCREF(ptr);
}
//rule of 3
~PyObjectHolder(){
std::lock_guard<std::mutex> guard(ref_mutex);
Py_XDECREF(ptr);
}
PyObjectHolder(const PyObjectHolder &h):
PyObjectHolder(h.ptr){}
PyObjectHolder& operator=(const PyObjectHolder &other){
{
std::lock_guard<std::mutex> guard(ref_mutex);
Py_XDECREF(ptr);
ptr=other.ptr;
Py_XINCREF(ptr);
}
return *this;
}
};
"""
cdef cppclass PyObjectHolder:
PyObjectHolder(PyObject *o) nogil
...
现在，运行从上面截断的代码会产生预期的/正确的行为:
import sys
a=[1000]*1000
print("Starts with", sys.getrefcount(a[0]))
# prints: Starts with 1002
run(a[0])
print("Ends with", sys.getrefcount(a[0]))
#prints: Ends with 1002
但是，正如@DavidW指出的那样，使用std::mutex仅适用于openmp-threads，而不适用于Python解释器创建的线程.
以下是互斥体解决方案将失败的示例.
首先，将nogil函数包装为def -function:
%%cython --cplus -c=/openmp
...
def single_create_vectors(object o):
cdef PyObject *ptr=<PyObject *>o
with nogil:
create_vectors(ptr)
现在使用threading -module创建
import sys
a=[1000]*10000 # some safety, so chances are high python will not crash
print(sys.getrefcount(a[0]))
#output: 10002
from threading import Thread
threads = []
for i in range(100):
t = Thread(target=single_create_vectors, args=(a[0],))
threads.append(t)
t.start()
for t in threads:
t.join()
print(sys.getrefcount(a[0]))
#output: 10015 but should be 10002!
使用 Void pointer to a cdef class is pointing to the same memory address without forcing the the python reference counter. I have a simple class that I want to store in a cpp vector by casting it to a void pointer. However, after printing the memory addresses the pointer is pointing to, it repeats after the second iteration, unless I force the reference counter to be increased by adding the new object to a list. Can somebody why the memory loops back without the reference counter enforcement? Which outputs: However if I force the reference counter by adding it to the classes list:
This answer became quite long, so there is a quick overview of the content: Explanation of the observed behavior The deal with Cython: as long as your variables are of type Obviously, the only reference to the created object is the variable Naive solution How could you do the reference counting? For example (I use rather Now all objects stay alive and "die" only after C++-typical solution The above is not a very typical c++-way of doing things, I would rather introduce a wrapper which manages the reference counting automatically (not unlike Noteworthy things: Problems with nogil-mode There is however one very important thing: You shouldn't release GIL with the above implementation (i.e. import it as To illustrate that let's take a look at the following code, which releases gil and does some stupid calculations in parallel (the whole magic cell is in listings at the end of the answer): And now: We got lucky, the program didn't crash (but could!). However due to race conditions, we ended up with memory leak - Making So what to do? The simplest solution is to use a mutex to protect the accesses to ref-counter(i.e. every time Here is a prototype: And now, running the code snipped from above yields the expected/right behavior: However, as @DavidW has pointed out, using Here is an example for which the mutex-solution will fail. First, wrapping nogil-function as And now using An alternative to using This would also work for the Listing complete thread-unsafe version:
这篇关于指向相同地址的空指针的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！std::mutex的替代方法是使用Python机器，即Problem
Description
# distutils: language = c++
# distutils: extra_compile_args = -std=c++11
from libcpp.vector cimport vector
from libc.stdio cimport printf
cdef class Temp:
cdef int a
def __init__(self, a):
self.a = a
def f():
cdef vector[void *] vec
cdef int i, n = 3
cdef Temp tmp
cdef list ids = []
# cdef list classes = [] # force reference counter?
for i in range(n):
tmp = Temp(1)
# classes.append(tmp)
vec.push_back(<void *> tmp)
printf('%p ', <void *> tmp)
ids.append(id(tmp))
print(ids)
f()
[140137023037824, 140137023037848, 140137023037824]
[140663518040448, 140663518040472, 140663518040496]
object or inherit from it (in your case cdef Temp) cython manages the reference counting for you. As soon as you cast it to PyObject * or any other pointer - the reference counting is your responsibility.tmp, as soon as you rebind it to the newly created Temp-object, the reference-counter of the old object becomes 0 and it is destroyed - the pointers in the vector becomes dangling. However, the same memory can be reused (it is quite probably) and thus you see always the same reused address.PyObject * than void *):...
from cpython cimport PyObject,Py_XINCREF, Py_XDECREF
...
def f():
cdef vector[PyObject *] vec
cdef int i, n = 3
cdef Temp tmp
cdef PyObject *tmp_ptr
cdef list ids = []
for i in range(n):
tmp = Temp(1)
tmp_ptr = <PyObject *> tmp
Py_XINCREF(tmp_ptr) # ensure it is not destroyed
vec.push_back(tmp_ptr)
printf('%p ', tmp_ptr)
ids.append(id(tmp))
#free memory:
for i in range(n):
Py_XDECREF(vec.at(i))
print(ids)
Py_XDECREF is called explicitly.std::shared_ptr):...
cdef extern from *:
"""
#include <Python.h>
class PyObjectHolder{
public:
PyObject *ptr;
PyObjectHolder():ptr(nullptr){}
PyObjectHolder(PyObject *o):ptr(o){
Py_XINCREF(ptr);
}
//rule of 3
~PyObjectHolder(){
Py_XDECREF(ptr);
}
PyObjectHolder(const PyObjectHolder &h):
PyObjectHolder(h.ptr){}
PyObjectHolder& operator=(const PyObjectHolder &other){
Py_XDECREF(ptr);
ptr=other.ptr;
Py_XINCREF(ptr);
return *this;
}
};
"""
cdef cppclass PyObjectHolder:
PyObjectHolder(PyObject *o)
...
def f():
cdef vector[PyObjectHolder] vec
cdef int i, n = 3
cdef Temp tmp
cdef PyObject *tmp_ptr
cdef list ids = []
for i in range(n):
tmp = Temp(1)
vec.push_back(PyObjectHolder(<PyObject *> tmp)) # vector::emplace_back is missing in Cython-wrappers
printf('%p ', <PyObject *> tmp)
ids.append(id(tmp))
print(ids)
# PyObjectHolder automatically decreases ref-counter as soon
# vec is out of scope, no need to take additional care
PyObjectHolder increases ref-counter as soon as it take possession of a PyObject-pointer and decreases it as soon as it releases the pointer.PyObjectHolder(PyObject *o) nogil but there are also problems when C++ copies the vectors and similar) - because otherwise Py_XINCREF and Py_XDECREF might not work correctly.%%cython --cplus -c=/openmp
...
# importing as nogil - A BAD THING
cdef cppclass PyObjectHolder:
PyObjectHolder(PyObject *o) nogil
# some functionality using a lot of incref/decref
cdef int create_vectors(PyObject *o) nogil:
cdef vector[PyObjectHolder] vec
cdef int i
for i in range(100):
vec.push_back(PyObjectHolder(o))
return vec.size()
# using PyObjectHolder without gil - A BAD THING
def run(object o):
cdef PyObject *ptr=<PyObject*>o;
cdef int i
for i in prange(10, nogil=True):
create_vectors(ptr)
import sys
a=[1000]*1000
print("Starts with", sys.getrefcount(a[0]))
# prints: Starts with 1002
run(a[0])
print("Ends with", sys.getrefcount(a[0]))
#prints: Ends with 1177
a[0] has reference count of 1177 but there are only 1000 references(+2 inside of sys.getrefcount) references alive, so this object will never be destroyed.PyObjectHolder thread-safePy_XINCREF or Py_XDECREF is called). The downside of this approach is that it might slowdown the single core code considerable (see for example this old article about an older try to replace GIL by mutex-similar approach).%%cython --cplus -c=/openmp
...
cdef extern from *:
"""
#include <Python.h>
#include <mutex>
std::mutex ref_mutex;
class PyObjectHolder{
public:
PyObject *ptr;
PyObjectHolder():ptr(nullptr){}
PyObjectHolder(PyObject *o):ptr(o){
std::lock_guard<std::mutex> guard(ref_mutex);
Py_XINCREF(ptr);
}
//rule of 3
~PyObjectHolder(){
std::lock_guard<std::mutex> guard(ref_mutex);
Py_XDECREF(ptr);
}
PyObjectHolder(const PyObjectHolder &h):
PyObjectHolder(h.ptr){}
PyObjectHolder& operator=(const PyObjectHolder &other){
{
std::lock_guard<std::mutex> guard(ref_mutex);
Py_XDECREF(ptr);
ptr=other.ptr;
Py_XINCREF(ptr);
}
return *this;
}
};
"""
cdef cppclass PyObjectHolder:
PyObjectHolder(PyObject *o) nogil
...
import sys
a=[1000]*1000
print("Starts with", sys.getrefcount(a[0]))
# prints: Starts with 1002
run(a[0])
print("Ends with", sys.getrefcount(a[0]))
#prints: Ends with 1002
std::mutex works only for openmp-threads, but not threads created by the Python-interpreter. def-function:%%cython --cplus -c=/openmp
...
def single_create_vectors(object o):
cdef PyObject *ptr=<PyObject *>o
with nogil:
create_vectors(ptr)
threading-module to create import sys
a=[1000]*10000 # some safety, so chances are high python will not crash
print(sys.getrefcount(a[0]))
#output: 10002
from threading import Thread
threads = []
for i in range(100):
t = Thread(target=single_create_vectors, args=(a[0],))
threads.append(t)
t.start()
for t in threads:
t.join()
print(sys.getrefcount(a[0]))
#output: 10015 but should be 10002!
std::mutex would be to use the Python-machinery, i.e. PyGILState_STATE, which would lead to code similar to...
PyObjectHolderPy(PyObject *o):ptr(o){
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();
Py_XINCREF(ptr);
PyGILState_Release(gstate);
}
...
threading-example above. However, PyGILState_Ensure has just too much overhead - for the example above it would be about 100 times slower than the mutex-solution. One more lightweight solution with Python-machinery would mean also much more hassle.
%%cython --cplus -c=/openmp
from libcpp.vector cimport vector
from libc.stdio cimport printf
from cpython cimport PyObject
from cython.parallel import prange
import sys
cdef extern from *:
"""
#include <Python.h>
class PyObjectHolder{
public:
PyObject *ptr;
PyObjectHolder():ptr(nullptr){}
PyObjectHolder(PyObject *o):ptr(o){
Py_XINCREF(ptr);
}
//rule of 3
~PyObjectHolder(){
Py_XDECREF(ptr);
}
PyObjectHolder(const PyObjectHolder &h):
PyObjectHolder(h.ptr){}
PyObjectHolder& operator=(const PyObjectHolder &other){
{
Py_XDECREF(ptr);
ptr=other.ptr;
Py_XINCREF(ptr);
}
return *this;
}
};
"""
cdef cppclass PyObjectHolder:
PyObjectHolder(PyObject *o) nogil
cdef int create_vectors(PyObject *o) nogil:
cdef vector[PyObjectHolder] vec
cdef int i
for i in range(100):
vec.push_back(PyObjectHolder(o))
return vec.size()
def run(object o):
cdef PyObject *ptr=<PyObject*>o;
cdef int i
for i in prange(10, nogil=True):
create_vectors(ptr)
