2.6,3.0，真正独立的解释器 [英] 2.6, 3.0, and truly independent intepreters
问题描述
亲爱的Python开发社区，
我是一家制作音乐可视化的小型软件公司的CTO
软件（你可以在 www.soundspectrum.com）。大约两年前，我们决定在几个新产品中使用嵌入式python，考虑到python的所有优点。我们接近使用lua
但由于各种原因我们决定使用
python。然而，在过去的两年中，有一个地方感到悲伤，有时让我三思而后行决定用python ...
... br />
首先是一些背景...我们的软件用于娱乐和
围绕实时，高性能图形，所以python'的
性能，嵌入式灵活性和稳定性是我们最重要的问题。我们的软件针对大型横截面
的硬件，我们目前为Win32，OS X和
iPhone发货，因为我们的客户是最终用户，我们的产品有要坚固，安装足够整洁，并且万无一失。基本上，
我们使用嵌入式python并用它来包装我们的高性能C ++
类集，它包含了OpenGL，DirectX和我们自己的软件渲染器。
除了包装我们的C ++框架之外，我们还使用python来执行各种工作和/或工作。工作线程上的任务（例如图像加载和
处理）。但是，我们要求* true * thread / interpreter
独立，所以python 2一直令人沮丧，说
最少。请不要以但实际上，python支持多个
口译员开头。因为我和很多人一起去过那里。
而且，是的，我知道在2.6中添加了多处理模块，但
那些东西不是重量轻，并不适合许多
环境（包括我们的）。最重要的是，如果你想在不同的线程上执行独立的处理（在python中），使用
机器的多个内核，那么你运气不好
在python 2下。
可悲的是，我们能够获得真正独立翻译的唯一方法是
将python放在一个动态库中，让我们的安装程序在安装过程中制作一个*复制*
的副本（例如python.dll / .bundle - >
python2.dll / .bundle）并在我们的应用程序中明确加载每一个，所以我们
可以获得真正独立的解释器。换句话说，我们为每个与线程无关的解释器加载了一个新的动态库（你不能再次使用同一个动态库，因为操作系统只会引用它
已经加载了一个。）
从我从python社区收集的内容来看，不是提供给b
的基础。真实"多线程的支持是它会增加很多内部开销 - 而且我不能同意。作为一个高性能的C $ / b $ b和C ++家伙，我完全同意线程安全应该处于高水平，而不是低水平。但是，缺乏真正独立的
解释器是最终阻止在酷炫，强大的方式中使用python的原因。仅仅这个缺点已经导致游戏开发者 - 无论大小 - b / b
选择其他嵌入式解释器而不是
python（例如暴雪选择了lua而不是python）。例如，Apple的
QuickTime API非常强大，因为高级实例对象可以利用
来提高与多线程处理相关的性能提升。
与此同时，QuickTime API只列出了
调用者关于线程安全的责任以及所有需要做的事情。用
换句话说，CPython不需要介入提供线程安全的
环境;它只需要建立规则并确保
它自己的实现支持这些规则。
不止一次，我实际上考虑过花费公司资源
开发一个高性能，真正独立的解释器
实现python核心语言和模块但最终
估计那个大小项目只会太多，因为我们公司目前的资源是b
。如果要开发这样的实施，那么对于公司来说，支持，b $ b b和b或许可将是非常有吸引力的。事实上，我们只是喜欢python作为一种
语言，但它缺乏真正的解释器独立性（在一个
解释器以及线程意义上）仍然是一个*巨大的*责任。
所以，我的问题变成：python 3是否准备好真正的多线程
支持？我们能否最终放弃我们的弗兰肯斯坦方法来加载
多个相同的动态库来实现真正独立的
解释器？我已经回顾了所有新的python 3 C API模块的东西，
，所有我要说的是：哇 - 好迟到然后永远！因此，虽然
解决了提供真正独立解释器支持的模块，但是仍然存在以下问题：
- 在python 3中， C模块API现在支持真正的解释器
独立性，但是python代码库中的所有模块都被转换过来了吗？它们现在都真正合规吗？它只需要一个模块中的单个静态/全局状态变量，可能会导致多个解释器环境中的痛苦结束！哎呀！
- python 3与真正的多线程使用有多接近？这里的
假设是调用者确保安全性（例如确保在将数据从一个序列化为另一个b / b时，确保
两个解释器都没有使用）。
我相信真正的python独立线程/解释器支持是
paramount并且应该成为头等大事，因为这是关键
开发人员在决定将哪个
口译员嵌入他们的应用程序时使用的考虑因素。直到有一个hello world
演示在多个应用程序上运行独立的python解释器
线程，lua仍然是python的明确选择。 Python 3需要
真正的解释器独立性和多线程支持！
谢谢，
Andy O''Meara
Dear Python dev community,
I''m CTO at a small software company that makes music visualization
software (you can check us out at www.soundspectrum.com). About two
years ago we went with decision to use embedded python in a couple of
our new products, given all the great things about python. We were
close to using lua but for various reasons we decided to go with
python. However, over the last two years, there''s been one area of
grief that sometimes makes me think twice about our decision to go
with python...
Some background first... Our software is used for entertainment and
centers around real time, high-performance graphics, so python''s
performance, embedded flexibility, and stability are the most
important issues for us. Our software targets a large cross section
of hardware and we currently ship products for Win32, OS X, and the
iPhone and since our customers are end users, our products have to be
robust, have a tidy install footprint, and be foolproof. Basically,
we use embedded python and use it to wrap our high performance C++
class set which wraps OpenGL, DirectX and our own software renderer.
In addition to wrapping our C++ frameworks, we use python to perform
various "worker" tasks on worker thread (e.g. image loading and
processing). However, we require *true* thread/interpreter
independence so python 2 has been frustrating at time, to say the
least. Please don''t start with "but really, python supports multiple
interpreters" because I''ve been there many many times with people.
And, yes, I''m aware of the multiprocessing module added in 2.6, but
that stuff isn''t lightweight and isn''t suitable at all for many
environments (including ours). The bottom line is that if you want to
perform independent processing (in python) on different threads, using
the machine''s multiple cores to the fullest, then you''re out of luck
under python 2.
Sadly, the only way we could get truly independent interpreters was to
put python in a dynamic library, have our installer make a *duplicate*
copy of it during the installation process (e.g. python.dll/.bundle ->
python2.dll/.bundle) and load each one explicitly in our app, so we
can get truly independent interpreters. In other words, we load a
fresh dynamic lib for each thread-independent interpreter (you can''t
reuse the same dynamic library because the OS will just reference the
already-loaded one).
From what I gather from the python community, the basis for not
offering "real" muti-threaded support is that it''d add to much
internal overhead--and I couldn''t agree more. As a high performance C
and C++ guy, I fully agree that thread safety should be at the high
level, not at the low level. BUT, the lack of truly independent
interpreters is what ultimately prevents using python in cool,
powerful ways. This shortcoming alone has caused game developers--
both large and small--to choose other embedded interpreters over
python (e.g. Blizzard chose lua over python). For example, Apple''s
QuickTime API is powerful in that high-level instance objects can
leverage performance gains associated with multi-threaded processing.
Meanwhile, the QuickTime API simply lists the responsibilities of the
caller regarding thread safety and that''s all its needs to do. In
other words, CPython doesn''t need to step in an provide a threadsafe
environment; it just needs to establish the rules and make sure that
its own implementation supports those rules.
More than once, I had actually considered expending company resources
to develop a high performance, truly independent interpreter
implementation of the python core language and modules but in the end
estimated that the size of that project would just be too much, given
our company''s current resources. Should such an implementation ever
be developed, it would be very attractive for companies to support,
fund, and/or license. The truth is, we just love python as a
language, but it''s lack of true interpreter independence (in a
interpreter as well as in a thread sense) remains a *huge* liability.
So, my question becomes: is python 3 ready for true multithreaded
support?? Can we finally abandon our Frankenstein approach of loading
multiple identical dynamic libs to achieve truly independent
interpreters?? I''ve reviewed all the new python 3 C API module stuff,
and all I have to say is: whew--better late then never!! So, although
that solves modules offering truly independent interpreter support,
the following questions remain:
- In python 3, the C module API now supports true interpreter
independence, but have all the modules in the python codebase been
converted over? Are they all now truly compliant? It will only take
a single static/global state variable in a module to potentially cause
no end of pain in a multiple interpreter environment! Yikes!
- How close is python 3 really to true multithreaded use? The
assumption here is that caller ensures safety (e.g. ensuring that
neither interpreter is in use when serializing data from one to
another).
I believe that true python independent thread/interpreter support is
paramount and should become the top priority because this is the key
consideration used by developers when they''re deciding which
interpreter to embed in their app. Until there''s a hello world that
demonstrates running independent python interpreters on multiple app
threads, lua will remain the clear choice over python. Python 3 needs
true interpreter independence and multi-threaded support!
Thanks,
Andy O''Meara
推荐答案
Andy schrieb：
Andy schrieb:
亲爱的Python开发社区，
[...]基本上，
我们使用嵌入式python并使用它来包装我们的高性能C ++
类集，它包含了OpenGL，DirectX和我们自己的软件渲染器。 br />
除了包装我们的C ++框架之外，我们还使用python执行
各种worker工作。工作线程上的任务（例如图像加载和
处理）。但是，我们要求* true * thread / interpreter
独立，所以python 2一直令人沮丧，说
最少。
Dear Python dev community,
[...] Basically,
we use embedded python and use it to wrap our high performance C++
class set which wraps OpenGL, DirectX and our own software renderer.
In addition to wrapping our C++ frameworks, we use python to perform
various "worker" tasks on worker thread (e.g. image loading and
processing). However, we require *true* thread/interpreter
independence so python 2 has been frustrating at time, to say the
least.
[...]
[...]
>
可悲的是，只有这样我们才能获得真正独立的口译员才能将bthon放入动态库中，让我们的安装人员在安装过程中制作一份*重复*
的副本（例如python.dll / .bundle - >
python2.dll / .bundle）并在我们的应用程序中明确加载每一个，所以我们
可以获得真正独立的解释器。换句话说，我们为每个与线程无关的解释器加载了一个新的动态库（你不能再次使用同一个动态库，因为操作系统只会引用它
已装载一个）。
>
Sadly, the only way we could get truly independent interpreters was to
put python in a dynamic library, have our installer make a *duplicate*
copy of it during the installation process (e.g. python.dll/.bundle ->
python2.dll/.bundle) and load each one explicitly in our app, so we
can get truly independent interpreters. In other words, we load a
fresh dynamic lib for each thread-independent interpreter (you can''t
reuse the same dynamic library because the OS will just reference the
already-loaded one).
你问的有趣问题。
随机说明：py2exe也为可执行文件构建类似的内容
，带有''bundle = 1''选项。在这种情况下，python.dll和.pyd扩展模块
不会以正常方式加载到进程中（使用
某种窗口LoadLibrary（ ）调用，而不是它们由代码加载
在py2exe中/ emulate / LoadLibrary - 代码段加载到
内存中，修复程序用于导入的函数，并标记可执行文件。
结果是，作为Python模块实现的单独COM对象和由py2exe转换为单独的dll的
不会共享他们的解释器
如果它们在同一个进程中运行。当然这只适用于windows。
实际上这类似于使用/静态/链接的python解释器
在不同的dll。你不能做那样的事吗？
Interesting questions you ask.
A random note: py2exe also does something similar for executables build
with the ''bundle = 1'' option. The python.dll and .pyd extension modules
in this case are not loaded into the process in the ''normal'' way (with
some kind of windows LoadLibrary() call, instead they are loaded by code
in py2exe that /emulates/ LoadLibrary - the code segments are loaded into
memory, fixups are made for imported functions, and marked executable.
The result is that separate COM objects implemented as Python modules and
converted into separate dlls by py2exe do not share their interpreters even
if they are running in the same process. Of course this only works on windows.
In effect this is similar to using /statically/ linked python interpreters
in separate dlls. Can''t you do something like that?
所以，我的问题变成：python 3准备好真正的多线程
支持??我们终于可以放弃我的弗兰肯斯坦方法吗？ oading
多个相同的动态库实现真正独立的
解释器？我已经回顾了所有新的python 3 C API模块的东西，
，所有我要说的是：哇 - 好迟到然后永远！因此，虽然
解决了提供真正独立解释器支持的模块，但是仍然存在以下问题：
- 在python 3中， C模块API现在支持真正的解释器
独立性，但是python代码库中的所有模块都被转换过来了吗？它们现在都真正合规吗？它只需要一个模块中的单个静态/全局状态变量，可能会导致多个解释器环境中的痛苦结束！哎呀！
So, my question becomes: is python 3 ready for true multithreaded
support?? Can we finally abandon our Frankenstein approach of loading
multiple identical dynamic libs to achieve truly independent
interpreters?? I''ve reviewed all the new python 3 C API module stuff,
and all I have to say is: whew--better late then never!! So, although
that solves modules offering truly independent interpreter support,
the following questions remain:
- In python 3, the C module API now supports true interpreter
independence, but have all the modules in the python codebase been
converted over? Are they all now truly compliant? It will only take
a single static/global state variable in a module to potentially cause
no end of pain in a multiple interpreter environment! Yikes!
我不认为是这种情况（目前）。但是你可以向Python提交补丁
，这样至少''官方''模块（内置和扩展）
在多个解释器的情况下会表现得很好。至少
这比编写你自己的无GIL翻译要轻得多。
我的2美分，
Thomas
I don''t think this is the case (currently). But you could submit patches
to Python so that at least the ''official'' modules (builtin and extensions)
would behave corectly in the case of multiple interpreters. At least
this is a much lighter task than writing your own GIL-less interpreter.
My 2 cents,
Thomas
- 在python 3中，C模块API现在支持真正的解释器
- In python 3, the C module API now supports true interpreter
independent，但是python代码库中的所有模块都被转换过来了吗？
independence, but have all the modules in the python codebase been
converted over?
不，没有。
No, none of them.
它们现在都真正合规吗？它只需要一个模块中的单个静态/全局状态变量，可能会导致多个解释器环境中的痛苦结束！哎呀！
Are they all now truly compliant? It will only take
a single static/global state variable in a module to potentially cause
no end of pain in a multiple interpreter environment! Yikes!
所以你将不得不忍受痛苦。
So you will have to suffer pain.
- python 3与真正的多线程有多接近使用？
- How close is python 3 really to true multithreaded use?
Python一如既往的线程安全（即完全线程安全）。
Python is as thread-safe as ever (i.e. completely thread-safe).
我相信真正的python独立线程/解释器支持是
paramount并且应该成为首要任务因为这是开发人员在决定哪个时候使用的关键因素/>
解释器嵌入他们的应用程序。直到有一个hello world
演示在多个应用程序上运行独立的python解释器
线程，lua仍然是python的明确选择。 Python 3需要
真正的解释器独立性和多线程支持！
I believe that true python independent thread/interpreter support is
paramount and should become the top priority because this is the key
consideration used by developers when they''re deciding which
interpreter to embed in their app. Until there''s a hello world that
demonstrates running independent python interpreters on multiple app
threads, lua will remain the clear choice over python. Python 3 needs
true interpreter independence and multi-threaded support!
那么到目前为止你为实现这个目标做了哪些补丁？
在开源中，请求几乎为零效果;代码贡献是
有什么影响。
我认为目前的任何提交者都没有重大的兴趣
支持多个口译员（我说那个人写了
并实施了PEP 3121）。为了做出重大改变，你需要从一个PEP开始，b
$，一旦接受就提供实施它，并提供
维持该功能五年。
问候，
马丁
So what patches to achieve that goal have you contributed so far?
In open source, pleas have nearly zero effect; code contributions is
what has effect.
I don''t think any of the current committers has a significant interest
in supporting multiple interpreters (and I say that as the one who wrote
and implemented PEP 3121). To make a significant change, you need to
start with a PEP, offer to implement it once accepted, and offer to
maintain the feature for five years.
Regards,
Martin
嗨托马斯 -
感谢您对此主题的想法和时间。
Hi Thomas -
I appreciate your thoughts and time on this subject.
>
结果是单独的COM实现为Python模块的对象和由py2exe转换为单独的dll的
如果它们在同一进程中运行，则不会共享他们的解释器
。 *当然这仅适用于Windows。
实际上这类似于在单独的dll中使用/静态/链接的python解释器
。你不能做那样的事吗？
>
The result is that separate COM objects implemented as Python modules and
converted into separate dlls by py2exe do not share their interpreters even
if they are running in the same process. *Of course this only works on windows.
In effect this is similar to using /statically/ linked python interpreters
in separate dlls. *Can''t you do something like that?
你肯定是正确的，自制加载和链接可以做到这一点
。但是，因为我们的python东西会在我们的b / b $ C / C ++中进行回调，这使得链接过程变得复杂（如果我理解你正确的话）。此外，还有OS X的问题。
You''re definitely correct that homebrew loading and linking would do
the trick. However, because our python stuff makes callbacks into our
C/C++, that complicates the linking process (if I understand you
correctly). Also, then there''s the problem of OS X.
- 在python 3中，C模块API现在支持真正的解释器
独立，但是python代码库中的所有模块都被转换过来了吗？ *他们现在都真正合规吗？ *在一个模块中只需要一个静态/全局状态变量，就可能导致多个解释器环境中的痛苦无法结束！ *哎呀！
- In python 3, the C module API now supports true interpreter
independence, but have all the modules in the python codebase been
converted over? *Are they all now truly compliant? *It will only take
a single static/global state variable in a module to potentially cause
no end of pain in a multiple interpreter environment! *Yikes!
我不认为是这种情况（目前）。 *但是你可以向Python提交补丁
，这样至少官方模块（内置和扩展）
在多个解释器的情况下会表现得很好。 *至少
这比编写自己的无GIL解释器要轻得多。
I don''t think this is the case (currently). *But you could submit patches
to Python so that at least the ''official'' modules (builtin and extensions)
would behave corectly in the case of multiple interpreters. *At least
this is a much lighter task than writing your own GIL-less interpreter.
我同意 - 而且我''我一直在考虑（或者更确切地说，让我们的b
公司雇佣/支付部分python开发社区来完成这项工作）。要考虑到这个问题，问题就变成了，我们说了多少个模块
你觉得怎么样？ 10？ 100？我承认我不够熟悉
使用完整的C python套件来了解我们在这里讨论的工作量是多少。
问候，
Andy
I agree -- and I''ve been considering that (or rather, having our
company hire/pay part of the python dev community to do the work). To
consider that, the question becomes, how many modules are we talking
about do you think? 10? 100? I confess that I''m no familiar enough
with the full C python suite to have a good idea of how much work
we''re talking about here.
Regards,
Andy
这篇关于2.6,3.0，真正独立的解释器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
