Python C API:创建一定数量的PyObject后出现WindowsError [英] Python C API: WindowsError after creating some number of PyObjects

问题描述

在获取Python C API时不会出现错误，我一直遇到问题.

I've been having an issue getting the Python C API to not give me errors.

背景:一段时间以来，我一直在使用ctypes运行本机代码(C ++)，但是直到现在我还从未真正使用Python C API做过任何特定的事情.我基本上只是从Python传递结构，然后从C ++填充它们.我使用结构的方法变得很麻烦，因此我决定尝试使用C ++直接创建Python对象，然后将它们传递回我的Python脚本.

Background: I've been using ctypes to run native code (C++) for a while, but until now I had never actually done anything specific with the Python C API. I had mostly just been passing in structs from Python and filling them from C++. The way I was using structs was becoming cumbersome, so I decided I would try creating Python objects directly in C++ and just pass them back to my Python script.

代码: 我有一个仅具有一个功能的DLL(Foo.dll):

Code: I have a DLL (Foo.dll) with only one function:

#define N 223
__declspec(dllexport) void Bar(void)
{
    std::cout << "Bar" << std::endl;

    for (int i = 0; i < N; ++i)
    {
        auto list = PyList_New(0);
        std::cout << "Created: " << i << std::endl;
        //Py_DECREF(list);
    }
}

然后我有正在运行的Python脚本:

And then I have the Python script I'm running:

import ctypes as C
dll = r"C:\path\to\dll\Foo.dll"
Foo = C.CDLL(dll)
# lists = [[] for _ in range(0)]
Foo.Bar()
print "Done."

会发生什么:如果我在上述DLL中将N定义为222或以下，则代码工作正常(内存泄漏除外，但这不是问题) .

What happens: If I define N in the above DLL to be 222 or below, the code works fine (except for the memory leak, but that isn't the problem).

如果我取消注释//Py_DECREF(list)行，则代码可以正常工作.

If I uncomment the //Py_DECREF(list) line, the code works fine.

但是，通过上面的代码，我得到了:

However, with the above code, I get this:

Bar
Created: 0
Created: 1
Created: 2
...(omitted for your sake)
Created: 219
Created: 220
Created: 221
Traceback (most recent call last):
  File "C:\path_to_script\script.py", line 9, in <module>
    Foo.Bar()
WindowsError: exception: access violation reading 0x00000028

事实上，我用字典，列表，元组等也得到了相同的结果.如果创建一个列表，然后将空的子列表附加到该列表，则会得到相同的结果.

In fact, I get this same result with dictionaries, lists, tuples and so on. I get the same result if I create a list and then append empty sublists to that list.

奇怪的是，我在实际的Python脚本中创建的每个列表都会减少DLL在出现此Windows错误之前可以创建的列表数量.

What's weirder, every list that I make from within the actual Python script will decrease the number of lists the DLL can make before getting this windows error.

更奇怪的是，如果我在python脚本中创建了超过222个列表，则DLL在创建720个更多列表之类的内容之前，不会遇到此错误.

Weirder still, if I make more than 222 lists in my python script, then the DLL won't encounter this error until it's created something like 720 more lists.

**其他详细信息:**

**Other details: **

• Windows 10
• 使用Anaconda2 32位Python 2.7发行版
• (使用该发行版中的Python.h和python27.lib
• python.exe --version:2.7.13 :: Anaconda custom (32-bit)
• 使用Visual Studio 2017创建DLL

• Windows 10
• Using the Anaconda2 32-bit Python 2.7 distribution
• (Using Python.h and python27.lib from that distribution
• python.exe --version: 2.7.13 :: Anaconda custom (32-bit)
• Creating DLL with Visual Studio 2017

只要我不从我的C ++代码创建许多PyObject，一切似乎都可以正常工作.我可以在Python代码之间传递PyObject，并且它可以正常工作..直到在我的C ++代码中创建了太多"的对象.

As long as I don't create many PyObjects from my C++ code, everything seems to work fine. I can pass PyObjects to and from the Python code and it works fine.. until I've created "too many" of the objects from within my C++ code.

发生了什么事?

推荐答案

摘自CDLL文档:

Python全局解释器锁在调用这些库导出的任何函数之前已释放，并在以后重新获取.

The Python global interpreter lock is released before calling any function exported by these libraries, and reacquired afterwards.

这使得使用Python C API代码不安全.正如您所发现的，它到底如何失败是无法预测的.我猜想这与分配是否触发垃圾收集器运行有关，但我认为花太多时间试图找出确切原因是不值得的.

This makes it unsafe to use Python C API code. Exactly how it fails is unpredictable, as you are finding. I'd guess it has to do with if the allocation triggers a run of the garbage collector, but I don't think it's worth spending too much time trying to work out the exact cause.

(至少)有两种解决方案可供选择:

There's (at least) two solutions to chose from:

1. 使用ctypes.PyDLL(文档注释与CDLL相似，只是它不会释放GIL)

2. 在C ++代码中重新获得GIL-一种简单的方法是:

1. Use ctypes.PyDLL (which the documentation notes is like CDLL except that it does not release the GIL)

2. Reacquire the GIL within your C++ code - an easy way to do this is:

auto state = PyGILState_Ensure();
// C++ code requiring GIL - probably your entire loop
PyGILState_Release(state);

这篇关于Python C API:创建一定数量的PyObject后出现WindowsError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！