RecursionError:使用pickle.load()时调用Python对象时超出了最大递归深度 [英] RecursionError: maximum recursion depth exceeded while calling a Python object when using pickle.load()
问题描述
首先，我知道有关此特定错误已经有多个问题，但我找不到能解决此错误的确切上下文.我还尝试了为其他类似错误提供的解决方案，但没有任何区别.
我正在使用python模块pickle将对象保存到文件中，并使用以下代码重新加载它:
I'm using the python module pickle to save an object to file and the reload it using the following code:
with open('test_file.pkl', 'wb') as a:
pickle.dump(object1, a, pickle.HIGHEST_PROTOCOL)
这不会引发任何错误，但是当我尝试使用以下代码打开文件时:
This doesn't throw any error but then when I try and open the file using the following code:
with open('test_file.pkl', 'rb') as a:
object2 = pickle.load(a)
我收到此错误:
---------------------------------------------------------------------------
RecursionError Traceback (most recent call last)
<ipython-input-3-8c5a70d147f7> in <module>()
1 with open('2test_bolfi_results.pkl', 'rb') as a:
----> 2 results = pickle.load(a)
3
~/.local/lib/python3.5/site-packages/elfi/methods/results.py in __getattr__(self, item)
95 def __getattr__(self, item):
96 """Allow more convenient access to items under self.meta."""
---> 97 if item in self.meta.keys():
98 return self.meta[item]
99 else:
... last 1 frames repeated, from the frame below ...
~/.local/lib/python3.5/site-packages/elfi/methods/results.py in __getattr__(self, item)
95 def __getattr__(self, item):
96 """Allow more convenient access to items under self.meta."""
---> 97 if item in self.meta.keys():
98 return self.meta[item]
99 else:
RecursionError: maximum recursion depth exceeded while calling a Python object
我知道其他人也看到了相同的错误(达到最大递归在执行pickle.dump时使用Pickle/cPickle 进行深度操作，而我尝试通过执行sys.setrecursionlimit()来增加最大递归深度，但这是行不通的，我要么得到了与上述相同的错误，要么我进一步将其增加了并且python崩溃并显示消息:Segmentation fault (core dumped).
I'm aware other people have seen this same error (Hitting Maximum Recursion Depth Using Pickle / cPickle) when doing pickle.dump and I've tried increasing the maximum recursion depth by doing sys.setrecursionlimit() but this doesn't work, I either get the same error as above or I increase it further and python crashes with the message: Segmentation fault (core dumped).
我怀疑问题的根源实际上是当我用pickle.load()保存对象时，但我真的不知道如何诊断它.
I suspect that the root of the problem is actually when I save the object with pickle.load() but I don't really know how to diagnose it.
有什么建议吗?
(我正在Windows 10计算机上运行python3)
(I'm running python3 on a windows 10 machine)
推荐答案
Here's a fairly minimal class derived from collections.UserDict which performs the same trick that your problem object does. It's a dictionary which allows you to access its items either via normal dict syntax, or as attributes. I've thrown in a few print calls so we can see when the main methods get called.
import collections
class AttrDict(collections.UserDict):
''' A dictionary that can be accessed via attributes '''
def __setattr__(self, key, value):
print('SA', key, value)
if key == 'data':
super().__setattr__('data', value)
else:
self.data[key] = value
def __getattr__(self, key):
print('GA', key)
if key in self.data:
return self.data[key]
else:
print('NOKEY')
raise AttributeError
def __delattr__(self, key):
del self.data[key]
# test
keys = 'zero', 'one', 'two', 'three'
data = {k: i for i, k in enumerate(keys)}
d = AttrDict(data)
print(d)
print(d.zero, d.one, d.two, d['three'])
输出
SA data {}
{'zero': 0, 'one': 1, 'two': 2, 'three': 3}
GA zero
GA one
GA two
0 1 2 3
到目前为止，太好了.但是，如果我们尝试腌制d实例，则会得到RecursionError，因为该__getattr__可以将属性访问神奇地转换为键查找.我们可以通过为类提供 __getstate__ 来克服这一问题和__setstate__方法.
So far, so good. But if we try to pickle our d instance, we get RecursionError because of that __getattr__ which does the magic conversion of attribute access to key lookup. We can overcome that by providing the class with __getstate__ and __setstate__ methods.
import pickle
import collections
class AttrDict(collections.UserDict):
''' A dictionary that can be accessed via attributes '''
def __setattr__(self, key, value):
print('SA', key, value)
if key == 'data':
super().__setattr__('data', value)
else:
self.data[key] = value
def __getattr__(self, key):
print('GA', key)
if key in self.data:
return self.data[key]
else:
print('NOKEY')
raise AttributeError
def __delattr__(self, key):
del self.data[key]
def __getstate__(self):
print('GS')
return self.data
def __setstate__(self, state):
print('SS')
self.data = state
# tests
keys = 'zero', 'one', 'two', 'three'
data = {k: i for i, k in enumerate(keys)}
d = AttrDict(data)
print(d)
print(d.zero, d.one, d.two, d['three'])
print('Pickling')
s = pickle.dumps(d, pickle.HIGHEST_PROTOCOL)
print(s)
print('Unpickling')
obj = pickle.loads(s)
print(obj)
输出
SA data {}
{'zero': 0, 'one': 1, 'two': 2, 'three': 3}
GA zero
GA one
GA two
0 1 2 3
Pickling
GS
b'\x80\x04\x95D\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x08AttrDict\x94\x93\x94)\x81\x94}\x94(\x8c\x04zero\x94K\x00\x8c\x03one\x94K\x01\x8c\x03two\x94K\x02\x8c\x05three\x94K\x03ub.'
Unpickling
SS
SA data {'zero': 0, 'one': 1, 'two': 2, 'three': 3}
{'zero': 0, 'one': 1, 'two': 2, 'three': 3}
但是，我们应该怎么做才能用这种行为修复现有的班级呢?幸运的是，Python允许我们轻松地向现有类添加新方法，甚至可以通过导入获得新方法.
But what can we do to repair an existing class with this behaviour? Fortunately, Python allows us to easily add new methods to an existing class, even one that we obtain via importing.
import pickle
import collections
class AttrDict(collections.UserDict):
''' A dictionary that can be accessed via attributes '''
def __setattr__(self, key, value):
print('SA', key, value)
if key == 'data':
super().__setattr__('data', value)
else:
self.data[key] = value
def __getattr__(self, key):
print('GA', key)
if key in self.data:
return self.data[key]
else:
print('NOKEY')
raise AttributeError
def __delattr__(self, key):
del self.data[key]
# Patch the existing AttrDict class with __getstate__ & __setstate__ methods
def getstate(self):
print('GS')
return self.data
def setstate(self, state):
print('SS')
self.data = state
AttrDict.__getstate__ = getstate
AttrDict.__setstate__ = setstate
# tests
keys = 'zero', 'one', 'two', 'three'
data = {k: i for i, k in enumerate(keys)}
d = AttrDict(data)
print(d)
print(d.zero, d.one, d.two, d['three'])
print('Pickling')
s = pickle.dumps(d, pickle.HIGHEST_PROTOCOL)
print(s)
print('Unpickling')
obj = pickle.loads(s)
print(obj)
此代码产生的输出与以前的版本相同，因此在此不再赘述.
This code produces the same output as the previous version, so I won't repeat it here.
希望这为您提供了足够的信息来修复有故障的对象.我的__getstate__& __setstate__方法仅 保存并恢复.data词典中的内容.要正确腌制您的物品，我们可能需要更严格一些.例如，我们可能需要保存并恢复实例的.__dict__属性，而不仅仅是保存与问题对象中.meta属性相对应的.data属性.
Hopefully, this gives you enough info to repair your faulty object. My __getstate__ & __setstate__ methods only save and restore the stuff in the .data dictionary. To properly pickle your object, we may need to be a bit more drastic. For example, we may need to save and restore the instance's .__dict__ attribute, rather than just the .data attribute, which corresponds to the .meta attribute in your problem object.
这篇关于RecursionError:使用pickle.load()时调用Python对象时超出了最大递归深度的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
