每个进程有多个解释器？ [英] More than one interpreter per process?
问题描述
Python有一个GIL会损害一台处理器超过
的计算机的可扩展性。问题似乎是每个
流程只有一个GIL。删除GIL的解决方案总是搁置在引用计数上的''细粒度锁定'的需要
。我认为第二种方式是
，这被忽略了：每个翻译一个GIL
而不是每个进程一个GIL。
目前，Python C API - 据我所知 - 每个进程只允许一个
单个解释器。下面是如何在多线程C程序中嵌入Python
，其中GIL在C
线程中共享：
#include< windows.h>
#include< Python.h>
#include< process.h>
void threadproc（void * data）
{
/ *为此线程创建一个线程状态* /
PyThreadState * mainstate = NULL;
mainstate = PyThreadState_Get（）;
PyThreadState * threadstate = PyThreadState_New（mainstate）;
PyEval_ReleaseLock（）;
/ *交换此线程，做我们需要的任何事情* /
PyEval_AcquireLock（）;
PyThreadState_Swap（threadstate）;
PyRun_SimpleString（" print''Hello World1'''\\ n）;
PyThreadState_Swap（NULL）;
PyEval_ReleaseLock（）;
/ *此线程的明确线程状态* /
PyEval_AcquireLock（）;
PyThreadState_Swap（NULL）;
PyTh readState_Clear（threadstate）;
PyThreadState_Delete（threadstate）;
PyEval_ReleaseLock（）;
/ *告诉Windows此线程已完成* /
_endthread（）;
}
int main（int argc，char * argv []）
{
HANDLE t1，t2，t3;
Py_Initialize（）;
PyEval_InitThreads（）;
t1 = _beginthread（threadproc，0，NULL）;
t2 = _beginthread（threadproc，0，NULL）;
t3 = _beginthread（threadproc，0，NULL ）;
WaitForMultipleObjects（3，{t1，t2，t3}，TRUE，INFINITE）;
Py_Finalize（）;
返回0;
}
在Java本机接口（JNI）中，所有函数都为VM提供了一个en b / b
环境变量。对于
Python也可以做同样的事情，将包含GIL的VM封装在一个对象中：
#include< windows.h>
#include< Python.h>
#include< process.h>
void threadproc（void * data）
{
PyVM * vm = Py_Initialize（）; / *创建一个新的解释器* /
PyRun_SimpleString（vm，" print''Hello World1'''\\ n）;
Py_Finalize（vm）;
_endthread（）;
}
int main（int argc，char * argv []）
{
HANDLE t1 = _beginthread（threadproc，0，NULL）;
HANDLE t2 = _beginthread（threadproc，0，NULL）;
HANDLE t3 = _beginthread（threadproc，0，NULL）;
WaitForMultipleObjects（3，{t1，t2，t3}，TRUE，INFINITE）;
返回0;
}
看起来好不好？
如果一个人可以有多个翻译一个进程，它可以创建一个池并实现并发编程
范例，例如''forkjoin''（出现在Java 7中，已经在C＃3.0）。
可以在不支持
本机fork（）的平台上模拟fork，例如Windows。 Perl在''perlfork'中做到这一点。这个
将处理具有多个CPU的计算机上的GIL问题。
实际上可以使用ctypes嵌入Python解释器池
已经在运行Python的过程中。
当前Python C API的大部分转换都可以自动化。
Python也需要与多线程版本链接
的C库。
Python has a GIL that impairs scalability on computers with more than
one processor. The problem seems to be that there is only one GIL per
process. Solutions to removing the GIL has always stranded on the need
for ''fine grained locking'' on reference counts. I believe there is a
second way, which has been overlooked: Having one GIL per interpreter
instead of one GIL per process.
Currently, the Python C API - as I understand it - only allows for a
single interpreter per process. Here is how Python would be embedded
in a multi-threaded C program today, with the GIL shared among the C
threads:
#include <windows.h>
#include <Python.h>
#include <process.h>
void threadproc(void *data)
{
/* create a thread state for this thread */
PyThreadState *mainstate = NULL;
mainstate = PyThreadState_Get();
PyThreadState *threadstate = PyThreadState_New(mainstate);
PyEval_ReleaseLock();
/* swap this thread in, do whatever we need */
PyEval_AcquireLock();
PyThreadState_Swap(threadstate);
PyRun_SimpleString("print ''Hello World1''\n");
PyThreadState_Swap(NULL);
PyEval_ReleaseLock();
/* clear thread state for this thread */
PyEval_AcquireLock();
PyThreadState_Swap(NULL);
PyThreadState_Clear(threadstate);
PyThreadState_Delete(threadstate);
PyEval_ReleaseLock();
/* tell Windows this thread is done */
_endthread();
}
int main(int argc, char *argv[])
{
HANDLE t1, t2, t3;
Py_Initialize();
PyEval_InitThreads();
t1 = _beginthread(threadproc, 0, NULL);
t2 = _beginthread(threadproc, 0, NULL);
t3 = _beginthread(threadproc, 0, NULL);
WaitForMultipleObjects(3, {t1, t2, t3}, TRUE, INFINITE);
Py_Finalize();
return 0;
}
In the Java native interface (JNI) all functions take an en
environment variable for the VM. The same thing could be done for
Python, with the VM including GIL encapsulated in a single object:
#include <windows.h>
#include <Python.h>
#include <process.h>
void threadproc(void *data)
{
PyVM *vm = Py_Initialize(); /* create a new interpreter */
PyRun_SimpleString(vm, "print ''Hello World1''\n");
Py_Finalize(vm);
_endthread();
}
int main(int argc, char *argv[])
{
HANDLE t1 = _beginthread(threadproc, 0, NULL);
HANDLE t2 = _beginthread(threadproc, 0, NULL);
HANDLE t3 = _beginthread(threadproc, 0, NULL);
WaitForMultipleObjects(3, {t1, t2, t3}, TRUE, INFINITE);
return 0;
}
Doesn''t that look a lot nicer?
If one can have more than one interpreter in a single process, it is
possible to create a pool of them and implement concurrent programming
paradigms such as ''forkjoin'' (to appear in Java 7, already in C# 3.0).
It would be possible to emulate a fork on platforms not supporting a
native fork(), such as Windows. Perl does this in ''perlfork''. This
would deal with the GIL issue on computers with more than one CPU.
One could actually use ctypes to embed a pool of Python interpreters
in a process already running Python.
Most of the conversion of the current Python C API could be automated.
Python would also need to be linked against a multi-threaded version
of the C library.
推荐答案
sturlamolden写道：
sturlamolden wrote:
Python有一个GIL会损害一台处理器超过
的计算机的可扩展性。问题似乎是每个
流程只有一个GIL。删除GIL的解决方案总是搁置在引用计数上的''细粒度锁定'的需要
。我认为第二种方式是
，这被忽略了：每个翻译一个GIL
而不是每个进程一个GIL。
Python has a GIL that impairs scalability on computers with more than
one processor. The problem seems to be that there is only one GIL per
process. Solutions to removing the GIL has always stranded on the need
for ''fine grained locking'' on reference counts. I believe there is a
second way, which has been overlooked: Having one GIL per interpreter
instead of one GIL per process.
这将如何处理程序员想要在线程中分享
的python资源？将使用
口译员之间的IPC设施？
How would this handle python resources that a programmer would want to
share among the threads? What facilities for IPC between the
interpreters would be used?
sturlamolden写道：
sturlamolden wrote:
如果一个进程中可以有多个解释器，
If one can have more than one interpreter in a single process,
你可以。看一下mod_python和mod_wsgi，它确实可以确定
。但是使用简化GIL api的扩展模块不能和它们一起工作（好吧，如果有的话）。
You can. Have a look at mod_python and mod_wsgi which does exactly
this. But extension modules that use the simplified GIL api don''t work
with them (well, if at all).
大多数当前Python C API的转换可以自动化。
Most of the conversion of the current Python C API could be automated.
最大的绊脚石是当外部环境
创建新线程然后最终回调到Python时该怎么办。很难知道回调应该转向哪个解释器。
您还要求每个扩展模块都必须更改。
绝大多数都不是Python源代码树的一部分，并且在这样的更改之前还需要支持版本。
你可以更好地将这种变化转化为Python 3，因为
需要对大多数扩展模块进行一些修改（例如用字符串和unicode问题处理
） 。
但在此之前，为什么不表明你的计划会有多好？
制作东西。这样做的成本是可以理解的，但是在cpu消耗，内存消耗，操作系统响应，缓存利用率，多核利用率等方面，有什么好处？b $ b bb b。如果
的答案是1％，那就是噪音。
Roger
The biggest stumbling block is what to do when the external environment
makes a new thread and then eventually calls back into Python. It is
hard to know which interpretter that callback should go to.
You are also asking for every extension module to have to be changed.
The vast majority are not part of the Python source tree and would also
have to support the versions before a change like this.
You would have more luck getting this sort of change into Python 3 since
that requires most extension modules to be modified a bit (eg to deal
with string and unicode issues).
But before doing that, why not show how much better your scheme would
make things. The costs of doing it are understood, but what are the
benefits in terms of cpu consumption, memory consumption, OS
responsiveness, cache utilisation, multi-core utilisation etc. If the
answer is 1% then that is noise.
Roger
On 18 Des ，05：46，Michael L Torrie< torr ... @ chem.byu.eduwrote：
On 18 Des, 05:46, Michael L Torrie <torr...@chem.byu.eduwrote:
这将如何处理程序员想要的python资源
在线程中分享？将使用
口译员之间的IPC设施？
How would this handle python resources that a programmer would want to
share among the threads? What facilities for IPC between the
interpreters would be used?
没有IPC，因为他们会住在同一个过程中。一个线程
安全队列就足够了。在放入队列之前，必须将Python对象序列化为
。也可以允许两个单独的解释器中的类似NumPy的数组共享相同的内存缓冲区。
使用模拟fork（），整个解释器将是克隆，
可能在''写入副本'方案中推迟。
多个流程和IPC就是我们今天所拥有的，例如mpi4py。
There would be no IPC as they would live in the same process. A thread-
safe queue would suffice. Python objects would have to be serialized
before placed in the queue. One could also allow NumPy-like arrays in
two separate interpreters to share the same memory buffer.
With an emulated fork() the whole interpreter would be cloned,
possibly deferred in a ''copy on write'' scheme.
Multiple processes and IPC is what we have today with e.g. mpi4py.
这篇关于每个进程有多个解释器？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
