多重处理:池和泡菜错误-泡菜错误:无法泡菜< type'instancemethod'&gt ;:属性查找__builtin __.instancemethod失败 [英] Multiprocessing: Pool and pickle Error -- Pickling Error: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

问题描述

我有两个文件:

x.py

class BF(object)
   def __init__():
   .
   .
   def add(self,z):
   .
   .

y.py

from y import BF
def FUNC((a,b,bf))
   .
   .
   bf.add(x)
   .
   .
   return bf

.
.
if __name__ == '__main__':
    pool = multiprocessing.Pool(3)
    for i in range(len(sl)):
         bf_set.append(BF())
    results = pool.map(FUNC,zip(sl, itertools.repeat(aa), bf_set))

我也尝试在FUNC内定义BF，但是我得到了:

I also tried to define BF inside FUNC, but sill I got:

PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

我已经阅读了一些有关相关问题的文章，但是它们在类中都有其pool.map()，因此解决方案无法应用于该问题(我想).

I've read some posts for related issues, but they have their pool.map() inside of the class, so the solutions cannot be applied to this problem (i guess).

有什么主意吗?

推荐答案

我将基本上使用上面的内容，但是将其变成有效的代码.如果使用dill，则序列化没有问题.我正在使用名为pathos.multiprocessing的multiprocessing叉，它使用dill而不是pickle.

I'm going to basically use what you have above, but turn it into working code. There is no problem serializing, if you use dill. I'm using a fork of multiprocessing called pathos.multiprocessing, which uses dill instead of pickle.

>>> def FUNC((a,b,bf)):
...   z = a+b
...   bf.add(z)
...   return bf
... 
>>> class BF(object):
...   def add(self, z):
...     self.z += z
...   def __init__(self):
...     self.z = 0
... 
>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> pool = Pool()
>>> 
>>> f = BF()
>>> f.add(1)
>>> f.z
1
>>> 
>>> FUNC((0,1,f))
<__main__.BF object at 0x10d387f50>
>>> 
>>> FUNC((0,1,f)).z
2
>>> 
>>> sl = [BF() for i in range(10)]
>>> results = pool.map(FUNC, zip(range(len(sl)), range(len(sl)), sl))
>>> [bf.z for bf in results]
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

这行得通，因为pathos使用dill，它可以在python中序列化几乎所有内容.

This works, because pathos uses dill, which can serialize almost anything in python.

>>> import dill as pickle
>>> pickle.loads(pickle.dumps(bf.add))
<bound method BF.add of <__main__.BF object at 0x10d383950>>
>>> pickle.loads(pickle.dumps(BF.add))
<unbound method BF.add>

在以下位置获取pathos和dill: https://github.com/uqfoundation

这篇关于多重处理:池和泡菜错误-泡菜错误:无法泡菜< type'instancemethod'&gt ;:属性查找__builtin __.instancemethod失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！