在绑定方法上使用multiprocessing.Pool [英] using multiprocessing.Pool on bound methods

问题描述

我正在尝试在代码中使用multiprocessing.Pool，但出现了以下异常:

I'm trying to use multiprocessing.Pool in my code but I got this exception:

PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

我发现了这，它是首选解决方案配方

我的问题是我不知道如何在我的代码中实现此解决方案.

my problem is that I don't know how to implement this solution in my code.

我的代码是这样的:

class G(class):
    def submit(self,data):
        cmd = self.createCommand(data)
        subprocess.call(cmd, shell=True)
        # call for a short command

    def main(self):
        self.pool = multiprocessing.Pool()
        while(True):
            data = self.GenerateData()
            self.pool.apply_async(self.Submit, args=(data,))

一些注意事项:

• 主while应该可以工作很长时间(几天)
• 我将pool用于性能目的，如果您有更好的解决方案，我会很高兴在这里

• the main while should work for a long time (few days)
• I'm using pool for performance purposes, if you have a better solution I will be glad to here it

使用@unutbu解决方案后，我得到了下一个异常: PicklingError: Can't pickle <type 'thread.lock'>: attribute lookup thread.lock failed

after using @unutbu solution I got the next exception: PicklingError: Can't pickle <type 'thread.lock'>: attribute lookup thread.lock failed

现在，我发现的所有解决方案都在谈论Queue.Queue和mp.Pool.map，但是我没有使用这些属性，所以我无法弄清楚.

now , all the solutions I found were talking about Queue.Queue and mp.Pool.map but I'm not using those attributes so I can't figure it out.

推荐答案

这是Steven Bethard解决方案针对您的情况的应用:

This is an application of Steven Bethard's solution to your situation:

import multiprocessing as mp
import time
import copy_reg
import types

def _pickle_method(method):
    """
    Author: Steven Bethard 
    http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-instancemethods
    """
    func_name = method.im_func.__name__
    obj = method.im_self
    cls = method.im_class
    cls_name = ''
    if func_name.startswith('__') and not func_name.endswith('__'):
        cls_name = cls.__name__.lstrip('_')
    if cls_name:
        func_name = '_' + cls_name + func_name
    return _unpickle_method, (func_name, obj, cls)


def _unpickle_method(func_name, obj, cls):
    """
    Author: Steven Bethard
    http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-instancemethods
    """
    for cls in cls.mro():
        try:
            func = cls.__dict__[func_name]
        except KeyError:
            pass
        else:
            break
    return func.__get__(obj, cls)

# This call to copy_reg.pickle allows you to pass methods as the first arg to
# mp.Pool methods. If you comment out this line, `pool.map(self.foo, ...)` results in
# PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
# __builtin__.instancemethod failed

copy_reg.pickle(types.MethodType, _pickle_method, _unpickle_method)

class G(object):
    def submit(self, data):
        print('processing {}'.format(data))
        # cmd = self.createCommand(data)
        # subprocess.call(cmd, shell=True)
        # call for a short command
        time.sleep(2)

    def main(self):
        pool = mp.Pool()
        while True:
            data = (1, 2, 3)
            pool.apply_async(self.submit, args=(data,))

if __name__ == '__main__':
    g = G()
    g.main()

这篇关于在绑定方法上使用multiprocessing.Pool的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！