不同服务器上的 Python 3.5 莳萝酸洗/解压:“KeyError: 'ClassType'" [英] Python 3.5 dill pickling/unpickling on different servers: "KeyError: 'ClassType'"
问题描述
在底部查看更新
--
这里有人问过类似的问题，但从未解决:pickling 和 unpickling 用户定义的类
我正在开发一个项目，该项目需要对用户定义的类进行酸洗，并将它们发送到远程服务器，在那里对它们进行解压和调用.我们使用 Dill 库来实现这一点，并取得了很多成功.
不幸的是，我遇到了一个问题，我很难调试.我创建并pickle一个类如下:
导入莳萝，base64导入时间，随机类周期源(对象):def __call__(self):为真:时间.睡眠(0.1)产量 random.uniform(20,100)period_src =periodicSource()a = base64.b64encode(dill.dumps(periodic_src)).decode("ascii")打印(一)它创建了 dilled 类的 ascii 表示.
<代码> gANjZGlsbC5kaWxsCl9jcmVhdGVfdHlwZQpxAChjZGlsbC5kaWxsCl9sb2FkX3R5cGUKcQFYCQAAAENsYXNzVHlwZXEChXEDUnEEWA4AAABwZXJpb2RpY1NvdXJjZXEFaAFYBgAAAG9iamVjdHEGhXEHUnEIhXEJfXEKKFgIAAAAX19jYWxsX19xC2NkaWxsLmRpbGwKX2NyZWF0ZV9mdW5jdGlvbgpxDChoAVgIAAAAQ29kZVR5cGVxDYVxDlJxDyhLAUsASwFLA0tjQyl4IgB0AABqAQBkAQCDAQABdAIAagMAZAIAZAMAgwIAVgFxAwBXZAAAU3EQKE5HP7mZmZmZmZpLFEtkdHERKFgEAAAAdGltZXESWAUAAABzbGVlcHETWAYAAAByYW5kb21xFFgHAAAAdW5pZm9ybXEVdHEWWAQAAABzZWxmcReFcRhYHwAAADxpcHl0aG9uLWlucHV0LTIwLTdhNGU5MDIwYWM2Yz5xGWgLSwdDBgABAwENAXEaKSl0cRtScRx9cR0oWAYAAAByYW5kb21xHmNkaWxsLmRpbGwKX2ltcG9ydF9tb2R1bGUKcR9oFIVxIFJxIVgEAAAAdGltZXEiaB9YBAAAAHRpbWVxI4VxJFJxJXVoC05OfXEmdHEnUnEoWAoAAABfX21vZHVsZV9fcSlYCAAAAF9fbWFpbl9fcSpYBwAAAF9fZG9jX19xK05YDQAAAF9fc2xvdG5hbWVzX19xLF1xLXV0cS5ScS8pgXEwLg ==
当我在另一台服务器上反序列化它时:
<预> <代码>一个= 'gANjZGlsbC5kaWxsCl9jcmVhdGVfdHlwZQpxAChjZGlsbC5kaWxsCl9sb2FkX3R5cGUKcQFYCQAAAENsYXNzVHlwZXEChXEDUnEEWA4AAABwZXJpb2RpY1NvdXJjZXEFaAFYBgAAAG9iamVjdHEGhXEHUnEIhXEJfXEKKFgIAAAAX19jYWxsX19xC2NkaWxsLmRpbGwKX2NyZWF0ZV9mdW5jdGlvbgpxDChoAVgIAAAAQ29kZVR5cGVxDYVxDlJxDyhLAUsASwFLA0tjQyl4IgB0AABqAQBkAQCDAQABdAIAagMAZAIAZAMAgwIAVgFxAwBXZAAAU3EQKE5HP7mZmZmZmZpLFEtkdHERKFgEAAAAdGltZXESWAUAAABzbGVlcHETWAYAAAByYW5kb21xFFgHAAAAdW5pZm9ybXEVdHEWWAQAAABzZWxmcReFcRhYHwAAADxpcHl0aG9uLWlucHV0LTIwLTdhNGU5MDIwYWM2Yz5xGWgLSwdDBgABAwENAXEaKSl0cRtScRx9cR0oWAYAAAByYW5kb21xHmNkaWxsLmRpbGwKX2ltcG9ydF9tb2R1bGUKcR9oFIVxIFJxIVgEAAAAdGltZXEiaB9YBAAAAHRpbWVxI4VxJFJxJXVoC05OfXEmdHEnUnEoWAoAAABfX21vZHVsZV9fcSlYCAAAAF9fbWFpbl9fcSpYBwAAAF9fZG9jX19xK05YDQAAAF9fc2xvdG5hbWVzX19xLF1xLXV0cS5ScS8pgXEwLg =='a = dill.loads(base64.b64decode(a.encode()))打印(一)我收到以下错误:
/home/streamsadmin/anaconda3/bin/python/home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py回溯(最近一次调用最后一次):文件/home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py"，第 40 行，在 <module> 中.a = dill.loads(base64.b64decode(a.encode()))文件/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py"，第277行，加载返回负载(文件)加载中的文件/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py"，第 266 行obj = pik.load()文件/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py"，第524行，_load_type返回_reverse_typemap[名称]KeyError: 'ClassType'如果我在远程系统上使用不同版本的 Python，我会期待这一点，但它们是相同的:
服务器 1:
<预><代码>>>>导入系统>>>系统版本'3.5.2 |蟒蛇 4.1.1(64 位)|(默认，2016 年 7 月 2 日，17:53:06)\n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]'服务器 2:
<预><代码>>>>导入系统>>>系统版本'3.5.2 |Anaconda 4.2.0(64 位)|(默认，2016 年 7 月 2 日，17:53:06)\n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]'另外，Dill 的两个版本都是 0.2.6.有什么想法可以调试吗?
我认为这可能与我的环境有关.我使用的是 Python 3.5，但列出了内置类型:
<预><代码>>>>进口类型>>>目录(类型)['内置功能类型'，'内置方法类型','类类型','代码类型',...]似乎 ClassType 在输出中应该 NOT 是这种情况，因为 ClassType 在 Python 3.5 中被删除了.这是非常奇怪的.
我在安装了 Python 2.7 和 Python 3.5 的系统上运行.2.7 安装会以某种方式污染 3.5 安装吗?
罪魁祸首是 cloudpickle.默认情况下，在 Python 3.5 中， types.ClassType 未设置.
当导入 cloudpickle 时，突然，types.ClassType 变成了定义.
服务器 A 使用 dill 来序列化对象，并导入 cloudpickle.因此，它在序列化期间包含对 ClassType 的引用.
服务器 B 不导入 cloudpickle，然后在反序列化过程中尝试查找对 ClassType 的引用并失败.引发错误:
回溯(最近一次调用最后一次):文件/home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py"，第 40 行，在 <module> 中.a = dill.loads(base64.b64decode(a.encode()))文件/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py"，第277行，加载返回负载(文件)加载中的文件/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py"，第 266 行obj = pik.load()文件/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py"，第524行，_load_type返回_reverse_typemap[名称]KeyError: 'ClassType'在我们的系统上，我们无法从环境中删除 cloudpickle，因此我们必须执行以下解决方法.
在服务器 B 上，在我们导入 dill 之后和第一次调用 dill.loads 之前的某个时间，我们调用以下行代码:
dill._dill._reverse_typemap['ClassType'] = 类型这适当地定义了 ClassType.并导致 dill 反序列化按预期工作.
See updates at the bottom
--
A similar question was asked here, but never resolved: pickling and unpickling user-defined class
I'm working on a project which necessitates pickling user defined classes, and sending them to a remote server where they are unpickled and called. We use the Dill library to accomplish this, and have had a lot of success.
Unfortunately, I've run into an issue I'm having a hard time debugging. I create and pickle a class as follows:
import dill, base64
import time, random
class periodicSource(object):
def __call__(self):
while True:
time.sleep(0.1)
yield random.uniform(20,100)
periodic_src = periodicSource()
a = base64.b64encode(dill.dumps(periodic_src)).decode("ascii")
print(a)
It creates an ascii representation of the dilled class.
gANjZGlsbC5kaWxsCl9jcmVhdGVfdHlwZQpxAChjZGlsbC5kaWxsCl9sb2FkX3R5cGUKcQFYCQAAAENsYXNzVHlwZXEChXEDUnEEWA4AAABwZXJpb2RpY1NvdXJjZXEFaAFYBgAAAG9iamVjdHEGhXEHUnEIhXEJfXEKKFgIAAAAX19jYWxsX19xC2NkaWxsLmRpbGwKX2NyZWF0ZV9mdW5jdGlvbgpxDChoAVgIAAAAQ29kZVR5cGVxDYVxDlJxDyhLAUsASwFLA0tjQyl4IgB0AABqAQBkAQCDAQABdAIAagMAZAIAZAMAgwIAVgFxAwBXZAAAU3EQKE5HP7mZmZmZmZpLFEtkdHERKFgEAAAAdGltZXESWAUAAABzbGVlcHETWAYAAAByYW5kb21xFFgHAAAAdW5pZm9ybXEVdHEWWAQAAABzZWxmcReFcRhYHwAAADxpcHl0aG9uLWlucHV0LTIwLTdhNGU5MDIwYWM2Yz5xGWgLSwdDBgABAwENAXEaKSl0cRtScRx9cR0oWAYAAAByYW5kb21xHmNkaWxsLmRpbGwKX2ltcG9ydF9tb2R1bGUKcR9oFIVxIFJxIVgEAAAAdGltZXEiaB9YBAAAAHRpbWVxI4VxJFJxJXVoC05OfXEmdHEnUnEoWAoAAABfX21vZHVsZV9fcSlYCAAAAF9fbWFpbl9fcSpYBwAAAF9fZG9jX19xK05YDQAAAF9fc2xvdG5hbWVzX19xLF1xLXV0cS5ScS8pgXEwLg==
When I go to deserialize it on the other server:
a = 'gANjZGlsbC5kaWxsCl9jcmVhdGVfdHlwZQpxAChjZGlsbC5kaWxsCl9sb2FkX3R5cGUKcQFYCQAAAENsYXNzVHlwZXEChXEDUnEEWA4AAABwZXJpb2RpY1NvdXJjZXEFaAFYBgAAAG9iamVjdHEGhXEHUnEIhXEJfXEKKFgIAAAAX19jYWxsX19xC2NkaWxsLmRpbGwKX2NyZWF0ZV9mdW5jdGlvbgpxDChoAVgIAAAAQ29kZVR5cGVxDYVxDlJxDyhLAUsASwFLA0tjQyl4IgB0AABqAQBkAQCDAQABdAIAagMAZAIAZAMAgwIAVgFxAwBXZAAAU3EQKE5HP7mZmZmZmZpLFEtkdHERKFgEAAAAdGltZXESWAUAAABzbGVlcHETWAYAAAByYW5kb21xFFgHAAAAdW5pZm9ybXEVdHEWWAQAAABzZWxmcReFcRhYHwAAADxpcHl0aG9uLWlucHV0LTIwLTdhNGU5MDIwYWM2Yz5xGWgLSwdDBgABAwENAXEaKSl0cRtScRx9cR0oWAYAAAByYW5kb21xHmNkaWxsLmRpbGwKX2ltcG9ydF9tb2R1bGUKcR9oFIVxIFJxIVgEAAAAdGltZXEiaB9YBAAAAHRpbWVxI4VxJFJxJXVoC05OfXEmdHEnUnEoWAoAAABfX21vZHVsZV9fcSlYCAAAAF9fbWFpbl9fcSpYBwAAAF9fZG9jX19xK05YDQAAAF9fc2xvdG5hbWVzX19xLF1xLXV0cS5ScS8pgXEwLg=='
a = dill.loads(base64.b64decode(a.encode()))
print(a)
I get the following error:
/home/streamsadmin/anaconda3/bin/python /home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py
Traceback (most recent call last):
File "/home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py", line 40, in <module>
a = dill.loads(base64.b64decode(a.encode()))
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 277, in loads
return load(file)
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 266, in load
obj = pik.load()
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 524, in _load_type
return _reverse_typemap[name]
KeyError: 'ClassType'
I would expect this if I were using different version of Python on the remote system, but they're the same:
Server 1:
>>> import sys
>>> sys.version
'3.5.2 |Anaconda 4.1.1 (64-bit)| (default, Jul 2 2016, 17:53:06) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]'
Server 2:
>>> import sys
>>> sys.version
'3.5.2 |Anaconda 4.2.0 (64-bit)| (default, Jul 2 2016, 17:53:06) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]'
Additionally, both versions of Dill are 0.2.6. Any ideas how I could debug this?
EDIT: I've think it might be something with my environment. I'm using Python 3.5, but list the builtin types:
>>> import types
>>> dir(types)
['BuiltinFunctionType',
'BuiltinMethodType',
'ClassType',
'CodeType',
...
]
It seems ClassType is in the output which should NOT be the case since ClassType was removed in Python 3.5. This is exceedingly strange.
I'm running on a system that has both Python 2.7 and Python 3.5 installed. Could the 2.7 installation somehow be polluting the 3.5 installation?
The culprit is cloudpickle. By default in Python 3.5, types.ClassType is left unset.
>>> import types
>>> dir(types)
['BuiltinFunctionType', 'BuiltinMethodType', 'CodeType', ...]
When cloudpickle is imported, suddenly, types.ClassType becomes defined.
>>> import cloudpickle
>>> dir(types)
['BuiltinFunctionType', 'BuiltinMethodType', 'ClassType', 'CodeType', ...]
Server A uses dill to serialize objects, and also imports cloudpickle. Therefore it includes a reference to ClassType during serialization.
Server B does NOT import cloudpickle, and then tries to find a reference to ClassType during deserialization and fails. Raising the error:
Traceback (most recent call last):
File "/home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py", line 40, in <module>
a = dill.loads(base64.b64decode(a.encode()))
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 277, in loads
return load(file)
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 266, in load
obj = pik.load()
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 524, in _load_type
return _reverse_typemap[name]
KeyError: 'ClassType'
On our system, we can't remove cloudpickle from our environment, so we had to do the following workaround.
On server B, right after we import dill and sometime before the first call to dill.loads, we invoke the following line of code:
dill._dill._reverse_typemap['ClassType'] = type
This defines ClassType appropriately. And causes dill deserialization to work as expected.
这篇关于不同服务器上的 Python 3.5 莳萝酸洗/解压:“KeyError: 'ClassType'"的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
