Python C-Api 线程问题 [英] Python C-Api Threading issues

问题描述

我正在编写一个 C 程序，它使用一个用 Python 编写的网络库.我用 python C api 嵌入了 python 库.该库异步发送所有请求，并在请求完成时通过信号通知我.

I am writing a C program which uses a networking lib written in python. I embed the python lib with the python C api. The library sends all requests async and informs me through signals when the request is done.

这意味着理论上.

实际上我有两个与线程相关的问题:

In reality I have two threading related problems problems:

1. 从 c 对 python 库的所有调用都是块状的(它们应该立即返回)
2. python 库异步调用注册的回调(thread.start_new_thread(callback, args)).这不起作用(没有任何反应).如果我将 python 代码更改为 callback(args) 那么它就可以工作了.

我做错了什么?我需要做些什么才能使多线程工作?

What I am doing wrong? Is there something I have to do to make multithreading work?

推荐答案

我有类似的情况.

初始工作流程

1. 应用从 C++ 层开始
2. C++层在主线程中调用Python层的函数
3. 主线程中的Python层函数创建事件线程
4. 在Python层启动事件线程并返回到C++层
5. 主循环从 C++ 层开始
6. 事件线程根据需要调用C++层的回调函数

从一开始，事件线程的工作就出乎意料.我想这是由于我遇到的情况的 GIL，所以我试图从 GIL 解决这个问题.这是我的解决方案.

From the beginning, the event thread works unexpected. I guess this is due to GIL from the situation I encountered so I tried to solve this from GIL. Here is my solution.

分析

首先，来自 PyEval_InitThreads 中的注释，

First, from note in PyEval_InitThreads,

只有主线程存在时，不需要GIL操作....因此，最初不会创建锁....

When only the main thread exists, no GIL operations are needed. ... Therefore, the lock is not created initially. ...

所以如果需要多线程，必须在主线程中调用PyEval_InitThreads().我在 Py_Initialize() 之前调用了 PyEval_InitThreads().现在 GIL 已初始化，主线程获取 GIL.

So if multi-thread is needed, PyEval_InitThreads() must be called in main thread. And I call PyEval_InitThreads() before Py_Initialize(). Now GIL is initialized and main thread acquires GIL.

其次，每次从C++层调用Python函数之前，都会调用PyGILState_Ensure()获取GIL.另外，在调用Python函数后，会调用PyGILState_Release(state)回到之前的GIL状态.结果，在第2步之前，调用了PyGILState_Ensure()，在第4步之后，调用了PyGILState_Release(state).

Second, each time before Python function is invoked from C++ layer, PyGILState_Ensure() is called to get GIL. In addition, after Python function is invoked, PyGILState_Release(state) is called to go back to previous GIL state. As a result, before step 2, PyGILState_Ensure() is called, and after step 4, PyGILState_Release(state) is called.

但是有一个问题.来自 PyGILState_Ensure 和 PyGILState_Release，这两个函数是保存当前GIL状态获取GIL和恢复之前的GIL状态释放 GIL.但是，在主线程调用PyEval_InitThreads()后，主线程肯定拥有GIL.并且主线程中的 GIL 状态如下:

But there is a problem. From PyGILState_Ensure and PyGILState_Release, these two functions are to save current GIL state to get GIL and restore previous GIL state to release GIL. However, after calling PyEval_InitThreads() in main thread, main thread owns GIL definitely. And the GIL state in main thread is as follows:

/* main thread owns GIL by PyEval_InitThreads */

state = PyGILState_Ensure();
/* main thread owns GIL by PyGILState_Ensure */

...
/* invoke Python function */
...

PyGILState_Release(state);
/* main thread owns GIL due to go back to previous state */

从上面的代码示例中，主线程始终拥有 GIL，因此事件线程永远不会运行.为了克服这种情况，让主线程在调用 PyGILState_Ensure() 之前不要获取 GIL.因此，调用PyGILState_Release(state)后，主线程可以释放GIL让事件线程运行.所以 GIL 应该在 GIL 初始化时立即在主线程中释放.

From above code sample, main thread always owns GIL so the event thread never runs. To overcome this situation, let main thread not acquire GIL before calling PyGILState_Ensure(). Therefore, after calling PyGILState_Release(state), main thread could release GIL to let event thread run. So GIL should be released in main thread immediately when GIL is initialized.

这里使用了 PyEval_SaveThread().来自 PyEval_SaveThread，

Here PyEval_SaveThread() is used. From PyEval_SaveThread,

释放全局解释器锁(如果已创建并启用线程支持)并将线程状态重置为NULL，...

Release the global interpreter lock (if it has been created and thread support is enabled) and reset the thread state to NULL, ...

通过这样做，嵌入 Python 的多线程工作.

By doing so, embedding Python with multi-thread works.

修改后的工作流程

1. 应用从 C++ 层开始
2. PyEval_InitThreads(); 开启多线程
3. save = PyEval_SaveThread(); 在主线程中释放 GIL
4. state = PyGILState_Ensure(); 在主线程中获取 GIL
5. C++层在主线程中调用Python层的函数
6. 主线程中的Python层函数创建事件线程
7. 在Python层启动事件线程并返回到C++层
8. PyGILState_Release(state); 在主线程中释放 GIL
9. 主循环从 C++ 层开始
10. 事件线程根据需要调用C++层的回调函数

1. Application starts from C++ layer
2. PyEval_InitThreads(); to enable multi-thread
3. save = PyEval_SaveThread(); to release GIL in main thread
4. state = PyGILState_Ensure(); to acquire GIL in main thread
5. C++ layer invokes function in Python layer in main thread
6. The Python layer function in main thread creates an event thread
7. Starts the event thread in Python layer and go back to C++ layer
8. PyGILState_Release(state); to release GIL in main thread
9. Main loop starts in C++ layer
10. The event thread invokes callback function in C++ layer if needed

这篇关于Python C-Api 线程问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！