PyGILState_Ensure()导致死锁 [英] PyGILState_Ensure() Causing Deadlock

问题描述

我正在用C ++编写Python扩展，包装了我无法控制的第三方库.该库创建了一个Python无关的线程，并且从该线程中调用了我提供给该库的C ++回调.我希望该回调函数调用Python函数，但是使用从文档中读取的方法却遇到了死锁.这是我对这些的解释.

I'm writing a Python extension in C++, wrapping a third-party library I do not control. That library creates a thread Python knows nothing about, and from that thread, calls a C++ callback I provide to the library. I want that callback to call a Python function, but I get a deadlock using the approach I read from the docs. Here's my interpretation of those.

void Wrapper::myCallback()
{
   PyGILState_STATE gstate=PyGILState_Ensure();
   PyObject *result=PyObject_CallMethod(_pyObj,"callback",nullptr);
   if (result) Py_DECREF(result);
   PyGILState_Release(gstate);
}

我的代码没有执行任何其他与线程相关的操作，尽管我尝试了许多其他操作.例如，基于此，我尝试调用PyEval_InitThreads()，但是在哪里调用并不明显应该进行扩展.我把它放在PyMODINIT_FUNC中.这些尝试均会导致Python死锁，崩溃或神秘的致命错误，例如 PyEval_ReleaseThread:错误的线程状态.

My code does nothing else related to threads, though I've tried a number of other things that have. Based on this, for example, I tried calling PyEval_InitThreads(), but it's not obvious where that call should be made for an extension. I put it in the PyMODINIT_FUNC. These attempts have all lead to deadlock, crashes, or mysterious fatal errors from Python, e.g., PyEval_ReleaseThread: wrong thread state.

这是在具有Python 3.6.1的Linux上.有什么想法可以使此简单"回调起作用吗?

This is on Linux with Python 3.6.1. Any ideas how I can get this "simple" callback to work?

我没有意识到在另一个线程中，该库处于繁忙/等待循环中，正在等待回调的线程.在gdb中，info threads使这一点显而易见.我唯一能看到的解决方案是跳过那些对回调的特定调用.考虑到繁忙/等待循环，我看不到任何使它们安全的方法.在这种情况下，这是可以接受的，并且这样做可以消除死锁.

I didn't realize that in another thread, the library was in a busy/wait loop waiting on the callback's thread. In gdb, info threads made this apparent. The only solution I can see is to skip those particular calls to the callback; I don't see a way to make them safe, given the busy/wait loop. In this case, that's acceptable, and doing so eliminates the deadlock.

此外，看来我确实还需要在任何此方法之前调用PyEval_InitThreads().在C ++扩展中，不清楚该去哪里.答复之一建议在Python中通过创建和删除一次性threading.Thread间接执行此操作.这似乎并没有解决，而是触发了致命的Python错误:take_gil:NULL tstate ，我认为这意味着仍然没有GIL.根据 this 及其所指问题，我的猜测是PyEval_InitThreads()使当前线程成为GIL的主线程.如果该调用是在短暂的一次性线程中进行的，则可能是一个问题.是的，我只是在猜测，并且会感谢不需要这样做的人的解释.

Also, it appears that I do need to also call PyEval_InitThreads() before any of this. In a C++ extension, it's not clear where that should go though. One of the replies suggested doing it indirectly in Python by creating and deleting a throwaway threading.Thread. That didn't seem to fix it, triggering instead a Fatal Python error: take_gil: NULL tstate, which I think means there's still no GIL. My guess, based on this and the issue it refers to, is that PyEval_InitThreads() causes the current thread to become the main thread for the GIL. If that call is made in the short-lived throwaway thread, maybe that's a problem. Yeah, I'm only guessing and would appreciate an explanation from someone who doesn't have to.

推荐答案

我是StackOverflow的新手，但最近几天我一直在将python嵌入多线程C ++系统中，并且遇到了很多代码自身陷入僵局的情况.这是我用来确保线程安全的解决方案:

I'm new to StackOverflow, but I've been working on embedding python in a multithreaded C++ system for the last few days and run into a fair number of situations where the code has deadlocked itself. Here's the solution that I've been using to ensure thread safety:

class PyContextManager {
   private:
      static volatile bool python_threads_initialized;
   public:
      static std::mutex pyContextLock;
      PyContextManager(/* if python_threads_initialized is false, call PyEval_InitThreads and set the variable to true */);
      ~PyContextManager();
};

#define PY_SAFE_CONTEXT(expr)                   \
{                                               \
   std::unique_lock<std::mutex>(pyContextLock); \
   PyGILState_STATE gstate;                     \
   gstate = PyGILState_Ensure();                \
      expr;                                     \
   PyGILState_Release(gstate);                  \
}

初始化.cpp文件中的布尔值和互斥量.

Initializing the boolean and the mutex in the .cpp file.

我注意到没有互斥锁，PyGILState_Ensure()命令会导致线程死锁.同样，在另一个PySafeContext的expr中调用PySafeContext会导致线程在等待其互斥锁时发生阻塞.

I've noticed that without the mutex, the PyGILState_Ensure() command can cause a thread to deadlock. Likewise, calling PySafeContext within the expr of another PySafeContext will cause the thread to brick while it waits on its mutex.

使用这些函数，我相信您的回调函数将如下所示:

Using these functions, I believe your callback function would look like this:

void Wrapper::myCallback()
{
   PyContextManager cm();
   PY_SAFE_CONTEXT(
       PyObject *result=PyObject_CallMethod(_pyObj,"callback",nullptr);
       if (result) Py_DECREF(result);
   );
}

如果您不认为自己的代码可能需要对Python的多个多线程调用，则可以轻松地扩展宏并将静态变量从类结构中删除.这就是我处理未知线程启动并确定是否需要启动系统并避免重复写出GIL函数的乏味的方式.

If you don't believe that your code is likely to ever need more than one multithreaded call to Python, you can easily expand the macro and take the static variables out of a class structure. This is just how I've handled an unknown thread starting and determining whether it needs to start up the system, and dodging the tedium of writing out the GIL functions repeatedly.

希望这会有所帮助！

这篇关于PyGILState_Ensure()导致死锁的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！