为什么GIL不能使在DLL中运行本机C ++代码的Python线程同步化? [英] Why GIL is not synchrionizing Python threads that are running native C++ code inside a DLL?

问题描述

我已经开发了一个简单的多线程Python应用程序(Python 3.7)，可以在8个线程中调用8个不同的计算密集型任务.这些任务可以是嵌入在DLL中并且可以通过ctypes包进行访问的Python代码或C ++代码.我正在Windows的8核计算机上运行实验.

I have developed a simple multi-threaded Python application (Python 3.7) to call 8 different compute-intensive tasks within 8 threads . The tasks can be either a Python code or a C++ code that is embedded inside a DLL and is accessible through ctypes package. I am running the experiments on a 8-core machine on Windows.

奇怪的是，当所有线程都调用Python代码时，似乎在特定时间只有一个线程处于活动状态，CPU利用率约为12.5％.但是，当在DLL中调用C ++代码时，将使用整个内核，并且CPU利用率为100％.

The strange point is that when all the threads call the Python code, it seems that only one thread is active at an specific time and the CPU utilization is around 12.5%. But when calling the C++ code inside the DLL, the whole cores are used and the CPU utilization is 100%.

现在，问题是为什么GIL(全局解释器锁)不同步调用本地C ++代码的Python线程?调用本地C ++代码时，ctypes实现是否释放GIL?

Now, the question is that why GIL (Global Interpreter Lock) is not synchronizing Python threads that call native C++ codes? Is the ctypes implementation releases the GIL when calling the native C++ code?

没有像

Edit 1: No macro like Py_BEGIN_ALLOW_THREADS is used inside the Native C++ DLL.

推荐答案

来自

From [Python 3]: ctypes - Loading shared libraries (emphasis is mine; thanks @user2357112 for pointing out this very explicit quote (waay better than what I've originally posted)):

Python 全局解释器锁定会在调用这些库导出的任何函数之前被释放，然后重新获取.

The Python global interpreter lock is released before calling any function exported by these libraries, and reacquired afterwards.

您还可以在同一页面上以其他形式找到此语句(请检查 PyDLL ， CFUNCTYPE ).

You can also find this statement in other forms on the same page (check PyDLL, CFUNCTYPE).

有一些方法可以绕过 GIL 限制:

There are ways of going around the GIL limitation:

• 通过 multiprocessing (

• Replacing the threading module usage by multiprocessing ([Python 3]: multiprocessing - Process-based parallelism). This is the most common one

包含可在 Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS 中并行执行的代码块.缺点是 .dll (s)现在将依赖于 Python

Enclosing code blocks that can be executed in parallel in Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS. The drawback would be that the .dll(s) will now depend on Python

这篇关于为什么GIL不能使在DLL中运行本机C ++代码的Python线程同步化?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！