如何从内存中加载已编译的python模块? [英] How to load compiled python modules from memory?

问题描述

我需要从一个zip文件(由py2exe压缩构建)中读取所有模块(预编译)到内​​存中，然后将它们全部加载. 我知道这可以通过直接从zip文件加载来完成，但是我需要从内存中加载它们. 有任何想法吗? (我在Windows上使用python 2.5.2) TIA史蒂夫

解决方案

这取决于您所拥有的模块(预编译)"的确切含义.假设它恰好是.pyc文件的内容，例如ciao.pyc的构建者:

$ cat>'ciao.py'
def ciao(): return 'Ciao!' 
$ python -c'import ciao; print ciao.ciao()'
Ciao!

如此构建ciao.pyc的

IOW表示您现在要这样做:

$ python
Python 2.5.1 (r251:54863, Feb  6 2009, 19:02:12) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> b = open('ciao.pyc', 'rb').read()
>>> len(b)
200

，您的目标是从该字节字符串b转到可导入的模块ciao.方法如下:

>>> import marshal
>>> c = marshal.loads(b[8:])
>>> c
<code object <module> at 0x65188, file "ciao.py", line 1>

这是从.pyc二进制内容中获取代码对象的方式. 编辑:如果您好奇的话，前8个字节是一个魔术数字"和一个时间戳记-此处不需要(除非您想进行健全性检查并在有必要的情况下引发异常，但是似乎不在问题的范围内；如果marshal.loads检测到损坏的字符串，反正会引发该问题.

然后:

>>> import types
>>> m = types.ModuleType('ciao')
>>> import sys
>>> sys.modules['ciao'] = m
>>> exec c in m.__dict__

即:创建一个新的模块对象，将其安装在sys.modules中，通过执行其__dict__中的代码对象来填充它. 编辑:sys.modules插入和exec的插入顺序在且仅当您可能具有循环导入时才重要-但这是Python自己的import通常使用的顺序，因此最好模仿它(没有具体缺点).

您可以通过几种方式(例如，从标准库模块中的函数，例如new和imp中)创建新模块对象"，但是调用类型以获取实例"是Python的常规方法如今，从中获取类型的正常位置(除非它具有内置名称，否则您已经很方便了)是从标准库模块types中获得的，因此，我建议这样做.

现在，终于:

>>> import ciao
>>> ciao.ciao()
'Ciao!'
>>> 

...您可以导入模块并使用其功能，类等.然后，其他import(和from)语句将找到该模块为sys.modules['ciao']，因此您无需重复此操作序列(实际上，您不需要 这最后一个import在此声明，如果您只想确保该模块可用于从其他位置导入，那么我将其添加只是为了显示其功能；-).

编辑:如果您绝对必须以这种方式从其中导入软件包和模块，而不是像我刚才所示的纯模块"那样导入，那也是可行的，但要复杂一些.由于这个答案已经很长了，希望我可以坚持使用简单的模块来简化您的工作，我将回避答案的那一部分；-).

还请注意，在多次从内存中加载同一模块"的情况下，此操作可能会或可能不会执行您想要的操作(这将每次重新生成模块；您可能需要检查sys.modules，并且如果模块已经存在)，特别是当从多个线程(需要锁)中发生这种重复的从内存中加载"时，但是更好的体系结构是只有一个专用线程来执行任务，而其他模块则通过队列).

最后，没有讨论如何将此功能安装为透明的导入钩子"，该钩子自动地介入了import语句内部机制本身—这也是可行的，但并不完全是您所要的.问一个问题，所以在这里，我也希望您可以按照此答案的概述，通过简单的方式来做事来简化您的生活.

I need to read all modules (pre-compiled) from a zipfile (built by py2exe compressed) into memory and then load them all. I know this can be done by loading direct from the zipfile but I need to load them from memory. Any ideas? (I'm using python 2.5.2 on windows) TIA Steve

解决方案

It depends on what exactly you have as "the module (pre-compiled)". Let's assume it's exactly the contents of a .pyc file, e.g., ciao.pyc as built by:

$ cat>'ciao.py'
def ciao(): return 'Ciao!' 
$ python -c'import ciao; print ciao.ciao()'
Ciao!

IOW, having thus built ciao.pyc, say that you now do:

$ python
Python 2.5.1 (r251:54863, Feb  6 2009, 19:02:12) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> b = open('ciao.pyc', 'rb').read()
>>> len(b)
200

and your goal is to go from that byte string b to an importable module ciao. Here's how:

>>> import marshal
>>> c = marshal.loads(b[8:])
>>> c
<code object <module> at 0x65188, file "ciao.py", line 1>

this is how you get the code object from the .pyc binary contents. Edit: if you're curious, the first 8 bytes are a "magic number" and a timestamp -- not needed here (unless you want to sanity-check them and raise exceptions if warranted, but that seems outside the scope of the question; marshal.loads will raise anyway if it detects a corrupt string).

Then:

>>> import types
>>> m = types.ModuleType('ciao')
>>> import sys
>>> sys.modules['ciao'] = m
>>> exec c in m.__dict__

i.e: make a new module object, install it in sys.modules, populate it by executing the code object in its __dict__. Edit: the order in which you do the sys.modules insertion and exec matters if and only if you may have circular imports -- but, this is the order Python's own import normally uses, so it's better to mimic it (which has no specific downsides).

You can "make a new module object" in several ways (e.g., from functions in standard library modules such as new and imp), but "call the type to get an instance" is the normal Python way these days, and the normal place to obtain the type from (unless it has a built-in name or you otherwise have it already handy) is from the standard library module types, so that's what I recommend.

Now, finally:

>>> import ciao
>>> ciao.ciao()
'Ciao!'
>>> 

...you can import the module and use its functions, classes, and so on. Other import (and from) statements will then find the module as sys.modules['ciao'], so you won't need to repeat this sequence of operations (indeed you don't need this last import statement here if all you want is to ensure the module is available for import from elsewhere -- I'm adding it only to show it works;-).

Edit: If you absolutely must import in this way packages and modules therefrom, rather than "plain modules" as I just showed, that's doable, too, but a bit more complicated. As this answer is already pretty long, and I hope you can simplify your life by sticking to plain modules for this purpose, I'm going to shirk that part of the answer;-).

Also note that this may or may not do what you want in cases of "loading the same module from memory multiple times" (this rebuilds the module each time; you might want to check sys.modules and just skip everything if the module's already there) and in particular when such repeated "load from memory" occurs from multiple threads (needing locks -- but, a better architecture is to have a single dedicated thread devoted to performing the task, with other modules communicating with it via a Queue).

Finally, there's no discussion of how to install this functionality as a transparent "import hook" which automagically gets involved in the mechanisms of the import statement internals themselves -- that's feasible, too, but not exactly what you're asking about, so here, too, I hope you can simplify your life by doing things the simple way instead, as this answer outlines.

这篇关于如何从内存中加载已编译的python模块?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！