UnpicklingError: NEWOBJ 类参数不是类型对象 [英] UnpicklingError: NEWOBJ class argument isn't a type object

问题描述

我正在使用自定义pickler，根据Shane Hathaway的代码，用它们的字符串表示替换任何不可pickle的对象(例如套接字或文件):Python:使用一些不可选择的项目腌制字典

它大部分时间都可以工作，但是当我尝试解开 Django HttpResponse 时，出现以下错误:UnpicklingError: NEWOBJ 类参数不是类型对象

我不知道错误的实际含义.如果可以腌制，为什么不能脱酸呢?我在 Google 上发现了三个对此错误的引用，但没有真正解释它发生的原因或如何修复它.

这是我的代码:

from cPickle import Pickler, Unpickler, UnpicklingError类过滤对象:def __init__(self, about):self.about = 关于def __repr__(self):返回 '​​FilteredObject(%s)' % repr(self.about)类 MyPickler(对象):def __init__(self, file, protocol=2):Pickler = Pickler(文件，协议)pickler.persistent_id = self.persistent_idself.dump = pickler.dumpself.clear_memo = pickler.clear_memodefpersistent_id(self, obj):如果不是 hasattr(obj, '__getstate__') 并且不是 isinstance(obj,(basestring, bool, int, long, float, complex, tuple, list, set, dict)):返回 ["过滤:%s" % str(obj)]别的:返回无类 MyUnpickler(对象):def __init__(self, 文件):unpickler = Unpickler(文件)unpickler.persistent_load = self.persistent_loadself.load = unpickler.loadself.noload = unpickler.noloaddefpersistent_load(self, obj_id):如果 obj_id[0].startswith('filtered:'):返回过滤对象(obj_id[0][9:])别的:引发 UnpicklingError('无效的持久 ID')######序列化到文件f = open('test.txt','wb')p = MyPickler(f)p.dump(数据)f.close()###### 从文件中反序列化f = open('test.txt','rb')pickled_data = f.read()f.seek(0)u = MyUnpickler(f)数据 = u.load()

解决方案

成功的pickle 分两步进行，pickler 的pickle.dump 和Unpickler 的pickle.load.Pickler 将对象转换为序列化格式(例如字符串)，而 Unpickler 消化pickle 对象并生成一个应该与原始对象等效的新对象.Pickle 有几个函数可以用来转储 pickle...所以第 1 部分是让对象转换为序列化格式.使用自定义pickler，您可以绕过python 的一些保护措施来pickle pickle 本身无法pickle 的对象.按照您的示例，我可以创建一个简单的 Pickler，它可以通过将每个对象转换为 __repr__ 来将 lambdas 和诸如此类的东西转换为字符串.

<预><代码>>>>x = λ x:x>>>代表(x)'<函数<lambda>在 0x4d39cf0>'>>>>>>进口泡菜>>>l = repr(x)>>>pickle.dumps(l)S'<function <lambda> at 0x4d39cf0>'\np0\n."

这肯定是可腌制的，因为它是一个字符串.但是，问题是如何从保存的字符串构建对象.对于 lambda，如果您有一个可以查找内存的函数字符串中注明的引用，您可以取回对象......但前提是您仍然拥有原来的东西活在你的记忆中......所以那不好.因此，转换为字符串的技巧仅在 __repr__ 字符串中包含足够的信息以根据存储的字符串信息构建新对象时才有效.您可能会更喜欢存储的内容，但最终很可能会遇到将对象转换为字符串的问题.所以在这种情况下，您的 Pickler 可以工作，但您的 Unpickler 会失败.

字典很有趣，因为它们可以包含任何内容，而且通常很快就会完成.最讨厌的字典之一是 globals() 字典.要序列化它，我会使用 dill，它可以序列化 Python 中的几乎所有内容.

<预><代码>>>>进口莳萝>>>>>>定义 foo(a):...定义栏(x):...返回 a*x...返回栏...>>>类巴兹(对象):... def __call__(self, a,x):... 返回 foo(a)(x)...>>>b = baz()>>>b(3,2)6>>>c = baz.__call__>>>c(b,3,2)6>>>g = dill.loads(dill.dumps(globals()))>>>G{'dill':<模块'dill'来自'/Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/site-packages/dill-0.2a.dev-py2.7.egg/dill/__init__.pyc'>, 'c': <unbound method baz.__call__>, 'b': <__main__.baz object at 0x4d61970>, 'g': {...}, '__builtins__': <;模块'__builtin__'(内置)>, 'baz': <class '__main__.baz'>, '_version': '2', '__package__': None, '__name__': '__main__','foo':<函数 foo 在 0x4d39d30>，'__doc__':无}

实际上，dill 将其类型注册到 pickle 注册表中，因此如果您有一些使用 pickle 的黑盒代码并且您无法真正编辑它，那么只需导入 dill 可以神奇地使其工作，而无需对 3rd 方代码进行猴子修补.

或者，如果您希望将整个解释器会话作为python 图像"发送，dill 也可以这样做.

<预><代码>>>># 从上面继续>>>dill.dump_session('foobar.pkl')>>>>>>^D老兄@sakurai>$蟒蛇Python 2.7.5(默认，2013 年 9 月 30 日，20:15:49)[GCC 4.2.1 (Apple Inc. build 5566)] 在达尔文上输入帮助"、版权"、信用"或许可"以获取更多信息.>>>进口莳萝>>>dill.load_session('foobar.pkl')>>>c(b,3,2)6

Dill 还有一些很好的工具来帮助你理解当您的代码失败时，是什么导致酸洗失败.

I'm using a custom pickler that replaces any un-pickleable objects (such as sockets or files) with a string representation of them, based on the code from Shane Hathaway here: Python: Pickling a dict with some unpicklable items

It works most of the time, but when I try to unpickle a Django HttpResponse, I get the following error: UnpicklingError: NEWOBJ class argument isn't a type object

I have no clue what the error actually means. If it pickles okay, why should it not be able to unpickle? I've found three references to this error on Google, but no real explanation of why it occurs or how to fix it.

Here is my code:

from cPickle import Pickler, Unpickler, UnpicklingError

class FilteredObject:
    def __init__(self, about):
        self.about = about
    def __repr__(self):
        return 'FilteredObject(%s)' % repr(self.about)

class MyPickler(object):
    def __init__(self, file, protocol=2):
        pickler = Pickler(file, protocol)
        pickler.persistent_id = self.persistent_id
        self.dump = pickler.dump
        self.clear_memo = pickler.clear_memo

    def persistent_id(self, obj):
           if not hasattr(obj, '__getstate__') and not isinstance(obj,
        (basestring, bool, int, long, float, complex, tuple, list, set, dict)):
            return ["filtered:%s" % str(obj)]
        else:
            return None

class MyUnpickler(object):
    def __init__(self, file):
        unpickler = Unpickler(file)
        unpickler.persistent_load = self.persistent_load
        self.load = unpickler.load
        self.noload = unpickler.noload

    def persistent_load(self, obj_id):
        if obj_id[0].startswith('filtered:'):
            return FilteredObject(obj_id[0][9:])
        else:
            raise UnpicklingError('Invalid persistent id')

###### serialize to file

f = open('test.txt','wb')
p = MyPickler(f)
p.dump(data)
f.close()

###### unserialize from file

f = open('test.txt','rb')
pickled_data = f.read()
f.seek(0)
u = MyUnpickler(f)
data = u.load()    

解决方案

Successful pickling happens in two steps, the pickle.dump by the Pickler, and the pickle.load by the Unpickler. The Pickler converts an object to a serialized format (e.g. strings), and the Unpickler digests the pickled object and generates a new object that should be equivalent to the original. Pickle has several functions that can be used to dump pickles... so part 1 is getting the objects to convert to the serialized format. With a custom pickler, you can bypass some of python's safeguards to pickle objects that pickle itself can't pickle. Following your example, I could create a simple Pickler, that would convert lambdas and whatnot to strings by converting each object to it's __repr__.

>>> x = lambda x:x
>>> repr(x)
'<function <lambda> at 0x4d39cf0>'
>>> 
>>> import pickle
>>> l = repr(x)
>>> pickle.dumps(l)
"S'<function <lambda> at 0x4d39cf0>'\np0\n."

This would definitely be pickleable, as it's a string. However, the problem is how to build an object from the saved string. For a lambda, if you had a function that could look-up the memory reference noted in the string, you could get the object back... but only if you still had the original object alive in your memory... so that's no good. So the trick of converting to strings only works when there's enough information contained in the __repr__ string to build a new object from the stored string's information. You could get fancier about what you store, but you will eventually run into problems by converting the objects to strings, most likely. So this is a case where your Pickler would work, but your Unpickler would fail.

Dictionaries are interesting to pickle, because they can have anything in them, and usually do pretty quickly. One of the nastiest dictionaries is the globals() dictionary. To serialize it, I'd use dill, which can serialize almost anything in python.

>>> import dill
>>> 
>>> def foo(a):
...   def bar(x):
...     return a*x
...   return bar
... 
>>> class baz(object):
...   def __call__(self, a,x):
...     return foo(a)(x)
... 
>>> b = baz()
>>> b(3,2)
6
>>> c = baz.__call__
>>> c(b,3,2)
6
>>> g = dill.loads(dill.dumps(globals()))
>>> g
{'dill': <module 'dill' from '/Library/Frameworks/Python.framework/Versions/7.2/lib/python2.7/site-packages/dill-0.2a.dev-py2.7.egg/dill/__init__.pyc'>, 'c': <unbound method baz.__call__>, 'b': <__main__.baz object at 0x4d61970>, 'g': {...}, '__builtins__': <module '__builtin__' (built-in)>, 'baz': <class '__main__.baz'>, '_version': '2', '__package__': None, '__name__': '__main__', 'foo': <function foo at 0x4d39d30>, '__doc__': None}

Actually, dill registers it's types into the pickle registry, so if you have some black box code that uses pickle and you can't really edit it, then just importing dill can magically make it work without monkeypatching the 3rd party code.

Or, if you want the whole interpreter session sent over as an "python image", dill can do that too.

>>> # continuing from above
>>> dill.dump_session('foobar.pkl')
>>>
>>> ^D
dude@sakurai>$ python
Python 2.7.5 (default, Sep 30 2013, 20:15:49) 
[GCC 4.2.1 (Apple Inc. build 5566)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> dill.load_session('foobar.pkl')
>>> c(b,3,2)
6

Dill also has some good tools for helping you understand what is causing your pickling to fail when your code fails.

这篇关于UnpicklingError: NEWOBJ 类参数不是类型对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！