如何从内存加载已编译的python模块? [英] How to load compiled python modules from memory?

问题描述

我需要将 zipfile(由 py2exe 压缩构建)中的所有模块(预编译)读取到内存中，然后将它们全部加载.我知道这可以通过直接从 zipfile 加载来完成，但我需要从内存中加载它们.有任何想法吗?(我在 Windows 上使用 python 2.5.2)蒂亚史蒂夫

解决方案

这取决于您拥有的模块(预编译)"究竟是什么.让我们假设它正是 .pyc 文件的内容，例如，ciao.pyc 由:

$ cat>'ciao.py'def ciao(): 返回 'Ciao!'$ python -c'import ciao;打印 ciao.ciao()'再见！

IOW，这样构建了 ciao.pyc，说你现在做:

$ pythonPython 2.5.1(r251:54863，2009 年 2 月 6 日，19:02:12)[GCC 4.0.1 (Apple Inc. build 5465)] 在达尔文上输入帮助"、版权"、信用"或许可证"以获取更多信息.>>>b = open('ciao.pyc', 'rb').read()>>>连(b)200

你的目标是从那个字节串 b 到一个可导入的模块 ciao.方法如下:

<预><代码>>>>进口元帅>>>c = marshal.loads(b[8:])>>>C<代码对象<模块>在 0x65188，文件ciao.py"，第 1 行>

这是从 .pyc 二进制内容中获取代码对象的方式.编辑:如果你很好奇，前 8 个字节是一个幻数"和一个时间戳——这里不需要(除非你想对它们进行完整性检查并在有必要的情况下引发异常，但是似乎超出了问题的范围；如果检测到损坏的字符串，marshal.loads 无论如何都会引发).

那么:

<预><代码>>>>进口类型>>>m = types.ModuleType('ciao')>>>导入系统>>>sys.modules['ciao'] = m>>>在 m.__dict__ 中执行 c

即:创建一个新的模块对象，将其安装在 sys.modules 中，通过在其 __dict__ 中执行代码对象来填充它.编辑:您执行 sys.modules 插入和 exec 的顺序当且仅当您可能有循环导入时才重要——但是，这是 Python 自己的 import 通常使用的顺序，所以最好模仿它(没有特定的缺点).

您可以通过多种方式创建一个新的模块对象"(例如，从标准库模块中的函数，例如 new 和 imp)，但是调用类型获取实例"是现在 Python 的正常方式，并且从标准库模块 types，所以这就是我推荐的.

现在，终于:

<预><代码>>>>进口巧>>>ciao.ciao()'再见！'>>>

...您可以导入模块并使用其函数、类等.其他import(和from)语句会找到模块为sys.modules['ciao']，所以你不需要重复此操作序列(实际上，如果您只想确保该模块可从其他地方导入，则无需 需要 最后一个 import 语句 - 我'我添加它只是为了表明它有效;-)

编辑:如果您绝对必须以这种方式从中导入包和模块，而不是我刚刚展示的普通模块"，那也是可行的，但有点复杂.由于这个答案已经很长了，我希望您可以通过为此目的坚持使用普通模块来简化您的生活，我将回避答案的那部分;-)

另请注意，在多次从内存中加载同一个模块"的情况下，这可能会或可能不会执行您想要的操作(这每次都会重建模块；您可能想要检查 sys.modules 并跳过所有内容，如果模块已经存在)，特别是当多个线程(需要锁)发生这种重复的从内存加载"时(需要锁——但是，更好的架构是有一个专门用于执行任务的线程，其他模块通过排队).

最后，没有讨论如何将这个功能安装为一个透明的导入钩子"，它会自动参与到 import 语句内部的机制中——这也是可行的，但不是正是您要问的问题，所以在这里，我也希望您可以通过以简单的方式做事来简化您的生活，正如本答案所概述的那样.

I need to read all modules (pre-compiled) from a zipfile (built by py2exe compressed) into memory and then load them all. I know this can be done by loading direct from the zipfile but I need to load them from memory. Any ideas? (I'm using python 2.5.2 on windows) TIA Steve

解决方案

It depends on what exactly you have as "the module (pre-compiled)". Let's assume it's exactly the contents of a .pyc file, e.g., ciao.pyc as built by:

$ cat>'ciao.py'
def ciao(): return 'Ciao!' 
$ python -c'import ciao; print ciao.ciao()'
Ciao!

IOW, having thus built ciao.pyc, say that you now do:

$ python
Python 2.5.1 (r251:54863, Feb  6 2009, 19:02:12) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> b = open('ciao.pyc', 'rb').read()
>>> len(b)
200

and your goal is to go from that byte string b to an importable module ciao. Here's how:

>>> import marshal
>>> c = marshal.loads(b[8:])
>>> c
<code object <module> at 0x65188, file "ciao.py", line 1>

this is how you get the code object from the .pyc binary contents. Edit: if you're curious, the first 8 bytes are a "magic number" and a timestamp -- not needed here (unless you want to sanity-check them and raise exceptions if warranted, but that seems outside the scope of the question; marshal.loads will raise anyway if it detects a corrupt string).

Then:

>>> import types
>>> m = types.ModuleType('ciao')
>>> import sys
>>> sys.modules['ciao'] = m
>>> exec c in m.__dict__

i.e: make a new module object, install it in sys.modules, populate it by executing the code object in its __dict__. Edit: the order in which you do the sys.modules insertion and exec matters if and only if you may have circular imports -- but, this is the order Python's own import normally uses, so it's better to mimic it (which has no specific downsides).

You can "make a new module object" in several ways (e.g., from functions in standard library modules such as new and imp), but "call the type to get an instance" is the normal Python way these days, and the normal place to obtain the type from (unless it has a built-in name or you otherwise have it already handy) is from the standard library module types, so that's what I recommend.

Now, finally:

>>> import ciao
>>> ciao.ciao()
'Ciao!'
>>> 

...you can import the module and use its functions, classes, and so on. Other import (and from) statements will then find the module as sys.modules['ciao'], so you won't need to repeat this sequence of operations (indeed you don't need this last import statement here if all you want is to ensure the module is available for import from elsewhere -- I'm adding it only to show it works;-).

Edit: If you absolutely must import in this way packages and modules therefrom, rather than "plain modules" as I just showed, that's doable, too, but a bit more complicated. As this answer is already pretty long, and I hope you can simplify your life by sticking to plain modules for this purpose, I'm going to shirk that part of the answer;-).

Also note that this may or may not do what you want in cases of "loading the same module from memory multiple times" (this rebuilds the module each time; you might want to check sys.modules and just skip everything if the module's already there) and in particular when such repeated "load from memory" occurs from multiple threads (needing locks -- but, a better architecture is to have a single dedicated thread devoted to performing the task, with other modules communicating with it via a Queue).

Finally, there's no discussion of how to install this functionality as a transparent "import hook" which automagically gets involved in the mechanisms of the import statement internals themselves -- that's feasible, too, but not exactly what you're asking about, so here, too, I hope you can simplify your life by doing things the simple way instead, as this answer outlines.

这篇关于如何从内存加载已编译的python模块?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！