如何腌制__main__(python)中定义的函数/类 [英] How to pickle functions/classes defined in __main__ (python)

问题描述

我希望能够从__main__内部对函数或类进行酸洗，而存在一个明显的问题(在其他文章中提到)，即被酸洗的函数/类位于__main__命名空间中，并且无法在其他脚本/模块中进行酸洗.

I would like to be able to pickle a function or class from within __main__, with the obvious problem (mentioned in other posts) that the pickled function/class is in the __main__ namespace and unpickling in another script/module will fail.

我有以下可行的解决方案，是否有理由不应该这样做?

I have the following solution which works, is there a reason this should not be done?

以下内容在myscript.py中:

The following is in myscript.py:

import myscript
import pickle

if __name__ == "__main__":               

    print pickle.dumps(myscript.myclass())

else:

    class myclass:
        pass

编辑:将在可以访问 myscript.py并可以执行import myscript的脚本/模块中完成拆开操作.目的是使用 parallel python 之类的解决方案来远程调用函数，并能够编写一个简短的独立脚本，其中包含可以远程访问的功能/类.

edit: The unpickling would be done in a script/module that has access to myscript.py and can do an import myscript. The aim is to use a solution like parallel python to call functions remotely, and be able to write a short, standalone script that contains the functions/classes that can be accessed remotely.

推荐答案

通过导入__main__并使用该模块中可用的方法，可以更好地处理全局对象.这就是莳萝的作用，目的是在python中序列化几乎所有内容.基本上，当莳萝序列化一个交互式定义的函数时，它会在序列化和反序列化方面在__main__上使用一些名称修饰，从而使__main__成为有效模块.

You can get a better handle on global objects by importing __main__, and using the methods available in that module. This is what dill does in order to serialize almost anything in python. Basically, when dill serializes an interactively defined function, it uses some name mangling on __main__ on both the serialization and deserialization side that makes __main__ a valid module.

>>> import dill
>>> 
>>> def bar(x):
...   return foo(x) + x
... 
>>> def foo(x):
...   return x**2
... 
>>> bar(3)
12
>>> 
>>> _bar = dill.loads(dill.dumps(bar))
>>> _bar(3)
12

实际上，莳萝将其类型注册到pickle注册表中，因此，如果您有一些使用pickle的黑匣子代码，并且您无法真正对其进行编辑，那么仅导入莳萝就可以神奇地使其工作而无需猴子补丁第三方代码.

Actually, dill registers it's types into the pickle registry, so if you have some black box code that uses pickle and you can't really edit it, then just importing dill can magically make it work without monkeypatching the 3rd party code.

或者，如果您希望将整个解释器会话作为"python映像"发送出去，则莳萝也可以做到这一点.

Or, if you want the whole interpreter session sent over as an "python image", dill can do that too.

>>> # continuing from above
>>> dill.dump_session('foobar.pkl')
>>>
>>> ^D
dude@sakurai>$ python
Python 2.7.5 (default, Sep 30 2013, 20:15:49) 
[GCC 4.2.1 (Apple Inc. build 5566)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> dill.load_session('foobar.pkl')
>>> _bar(3)
12

您可以轻松地将图像通过ssh发送到另一台计算机，然后从您离开的地方开始，只要有pickle的版本兼容性以及有关python更改和安装的常见警告.

You can easily send the image across ssh to another computer, and start where you left off there as long as there's version compatibility of pickle and the usual caveats about python changing and things being installed.

我实际上使用莳萝序列化对象，并通过并行python ，多处理和 mpi4py .我将它们方便地汇总到 pathos 包(和 pyina (用于MPI)，它为不同的并行批处理后端提供统一的map接口.

I actually use dill to serialize objects and send them across parallel resources with parallel python, multiprocessing, and mpi4py. I roll these up conveniently into the pathos package (and pyina for MPI), which provides a uniform map interface for different parallel batch processing backends.

>>> # continued from above
>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> Pool(4).map(foo, range(10))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>>
>>> from pyina.launchers import MpiPool
>>> MpiPool(4).map(foo, range(10))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

也有非阻塞和迭代映射以及非平行管道连接.我也有一个pp的pathos模块，但是，对于__main__中定义的函数来说，它有些不稳定.我正在努力改善这一点.如果愿意，可以在__main__中定义 github上的代码，并使pp更好地适用于. pp表现不佳的原因是pp通过使用临时文件对象并读取解释器会话的历史记录来完成序列化技巧...因此它不会像多处理或mpi4py那样对对象进行序列化.我有一个莳萝模块dill.source，它无缝地执行与pp所用的相同类型的酸洗，但是它是相当新的.

There are also non-blocking and iterative maps as well as non-parallel pipe connections. I also have a pathos module for pp, however, it is somewhat unstable for functions defined in __main__. I'm working on improving that. If you like, fork the code on github and help make the pp better for functions defined in __main__. The reason pp doesn't pickle well is that pp does it's serialization tricks through using temporary file objects and reading the interpreter session's history... so it doesn't serialize objects in the same way that multiprocessing or mpi4py do. I have a dill module dill.source that seamlessly does the same type of pickling that pp uses, but it's rather new.

这篇关于如何腌制__main__(python)中定义的函数/类的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！