用莳萝序列化SWIG扩展 [英] Serialize SWIG extension with dill

问题描述

最近，有人要求我使我们的C ++库在云中工作". 基本上，lib是计算机密集型的(计算价格)，因此这是有道理的. 我已经构建了一个SWIG接口，以制作一个python版本，并考虑将MapReduce与MRJob结合使用. 我想序列化文件中的对象，然后使用映射器反序列化并计算价格.

Recently, I have been asked to make "our C++ lib work in the cloud". Basically, the lib is computer intensive (calculating prices), so it would make sense. I have constructed a SWIG interface to make a python version with in the mind to use MapReduce with MRJob. I wanted to serialize the objects in a file, and using a mapper, deserialize and calculate the price.

例如:

class MRTest(MRJob):
    def mapper(self,key,value):
        obj = dill.loads(value)
        yield (key, obj.price())

但是现在我走到了尽头，因为似乎莳萝无法处理SWIG扩展名:

But now I reach a dead end since it seems that dill cannot handle SWIG extension:

PicklingError: Can't pickle <class 'SwigPyObject'>: it's not found as builtins.SwigPyObject

有没有办法使它正常工作?

Is there a way to make this work properly?

推荐答案

我是dill的作者.没错，dill不能腌制C ++对象.当您看到it's not found as builtin. some_object …时，几乎总是意味着您正在尝试腌制不是用python编写的 some object ，而是使用python绑定到C/C ++ (即扩展名类型).您不希望使用python序列化器直接对此类对象进行腌制.

I'm the dill author. That's correct, dill can't pickle C++ objects. When you see it's not found as builtin.some_object… that almost invariably means that you are trying to pickle some object that is not written in python, but uses python to bind to C/C++ (i.e. an extension type). You have no hope of directly pickling such objects with a python serializer.

但是，由于您有兴趣对扩展类型的子类进行酸洗，因此实际上可以这样做.您需要做的就是为您的对象提供要保存为一个或多个实例属性的适当状态，并提供一个__reduce__方法来告诉dill(或pickle)如何保存您的状态.目的.此方法是python处理序列化扩展类型的方式.看: https://docs.python.org/2/library/pickle.html#pickling-and-unpickling-extension-types

However, since you are interested in pickling a subclass of an extension type, you can actually do it. All you will need to do is to give your object the appropriate state you want to save as an instance attribute or attributes, and provide a __reduce__ method to tell dill (or pickle) how to save the state of your object. This method is how python deals with serializing extension types. See: https://docs.python.org/2/library/pickle.html#pickling-and-unpickling-extension-types

也许有更好的例子，但是至少有一个例子: https://stackoverflow.com/a/19874769/4646678

There are probably better examples, but here's at least one example: https://stackoverflow.com/a/19874769/4646678

这篇关于用莳萝序列化SWIG扩展的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！