使用子流程模块会释放python GIL吗? [英] Does using the subprocess module release the python GIL?

问题描述

通过Python的subprocess模块调用花费相对较长时间的linux二进制文件时，这会释放GIL吗?

When calling a linux binary which takes a relatively long time through Python's subprocess module, does this release the GIL?

我想并行化一些从命令行调用二进制程序的代码.使用线程(通过threading和multiprocessing.pool.ThreadPool)还是multiprocessing更好?我的假设是，如果subprocess释放GIL，则选择threading选项会更好.

I want to parallelise some code which calls a binary program from the command line. Is it better to use threads (through threading and a multiprocessing.pool.ThreadPool) or multiprocessing? My assumption is that if subprocess releases the GIL then choosing the threading option is better.

推荐答案

通过Python的subprocess模块调用花费相对较长时间的linux二进制文件时，这会释放GIL吗?

When calling a linux binary which takes a relatively long time through Python's subprocess module, does this release the GIL?

是的，它会在调用过程中释放全局解释器锁定(GIL).

Yes, it releases the Global Interpreter Lock (GIL) in the calling process.

您可能已经知道，在POSIX平台上，subprocess在fork，execve和waitpid的原始"组件上提供了方便的界面.

As you are likely aware, on POSIX platforms subprocess offers convenience interfaces atop the "raw" components from fork, execve, and waitpid.

通过检查CPython 2.7.9源，fork和execve不会 释放GIL.但是，这些调用不会阻塞，因此我们不希望GIL被释放.

By inspection of the CPython 2.7.9 sources, fork and execve do not release the GIL. However, those calls do not block, so we'd not expect the GIL to be released.

waitpid当然是做块，但是我们看到它的实现确实使用ALLOW_THREADS宏放弃了GIL:

waitpid of course does block, but we see it's implementation does give up the GIL using the ALLOW_THREADS macros:

static PyObject *
posix_waitpid(PyObject *self, PyObject *args)
{
....
Py_BEGIN_ALLOW_THREADS
pid = waitpid(pid, &status, options);
Py_END_ALLOW_THREADS
....

也可以通过调用一些长期运行的程序(例如 sleep 来自演示的多线程python脚本.

This could also be tested by calling out to some long running program like sleep from a demonstration multithreaded python script.

这篇关于使用子流程模块会释放python GIL吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！