Python NotImplementedError:无法在进程之间传递池对象 [英] Python NotImplementedError: pool objects cannot be passed between processes

问题描述

当页面附加到页面列表时，我正在尝试交付工作，但是我的代码输出返回NotImplementedError.这是我要执行的代码:

I'm trying to deliver work when a page is appended to the pages list, but my code output returns a NotImplementedError. Here is the code with what I'm trying to do:

代码:

from multiprocessing import Pool, current_process
import time
import random
import copy_reg
import types
import threading


class PageControler(object):
    def __init__(self):
        self.nProcess = 3
        self.pages = [1,2,3,4,5,6,7,8,9,10]
        self.manageWork()


    def manageWork(self):

        self.pool = Pool(processes=self.nProcess)

        time.sleep(2)
        work_queue = threading.Thread(target=self.modifyQueue)
        work_queue.start()

        #pool.close()
        #pool.join()

    def deliverWork(self):    
        if self.pages != []:
            pag = self.pages.pop()
            self.pool.apply_async(self.myFun)


    def modifyQueue(self):
        t = time.time()
        while (time.time()-t) < 10:
            time.sleep(1)
            self.pages.append(99)
            print self.pages
            self.deliverWork()

    def myFun(self):
        time.sleep(2)


if __name__ == '__main__':
    def _pickle_method(m):
        if m.im_self is None:
            return getattr, (m.im_class, m.im_func.func_name)
        else:
            return getattr, (m.im_self, m.im_func.func_name)

    copy_reg.pickle(types.MethodType, _pickle_method)

    PageControler()

输出:

NotImplementedError: pool objects cannot be passed between processes or pickled

是否可以在进程之间传递池对象?

It's any way to pass the pool object between the processes ?

我正在使用Python 2.6

I'm using Python 2.6

推荐答案

为了使您尝试传递给Pool的实例方法腌制，Python需要腌制整个PageControler对象，包括其实例.变量.这些实例变量之一是Pool对象本身，而Pool对象不能被腌制，因此会产生错误.您可以通过在对象上实现 __getstate__ 来解决此问题，并在酸洗之前使用该方法从实例中删除pool对象:

In order to pickle the instance method you're trying to pass to the Pool, Python needs to pickle the entire PageControler object, including its instance variables. One of those instance variables is the Pool object itself, and Pool objects can't be pickled, hence the error. You can work around this by implementing __getstate__ on the object, and using that to remove the pool object from the instance prior to pickling:

class PageControler(object):
    def __init__(self):
        self.nProcess = 3
        self.pages = [1,2,3,4,5,6,7,8,9,10]
        self.manageWork()


    def manageWork(self):

        self.pool = Pool(processes=self.nProcess)

        time.sleep(2)
        work_queue = threading.Thread(target=self.modifyQueue)
        work_queue.start()

        #pool.close()
        #pool.join()

    def deliverWork(self):    
        if self.pages != []:
            pag = self.pages.pop()
            self.pool.apply_async(self.myFun)


    def modifyQueue(self):
        t = time.time()
        while (time.time()-t) < 10:
            time.sleep(1)
            self.pages.append(99)
            print self.pages
            self.deliverWork()

    def myFun(self):
        time.sleep(2)

    def __getstate__(self):
        self_dict = self.__dict__.copy()
        del self_dict['pool']
        return self_dict

    def __setstate__(self, state):
        self.__dict__.update(state)

始终在腌制对象之前调用

__getstate__，并允许您确切指定实际应腌制对象状态的哪些部分.然后在解开时，如果 __setstate__(state) 被调用， (在我们的示例中)，否则，由__getstate__返回的dict将用作未腌制实例的__dict__.在上面的示例中，我们将__dict__显式设置为在__getstate__中返回的dict，但是我们可能没有实现__setstate__并获得了相同的效果.

__getstate__ is always called prior to pickling an object, and allow you to specify exactly which pieces of the object's state should actually be pickled. Then upon unpickling, __setstate__(state) will be called if its implemented (it is in our case), or if it's not, the dict returned by __getstate__ will be used as the __dict__ for the unpickled instance. In the above example, we're explicitly setting __dict__ to the dict we returned in __getstate__, but we could have just not implemented __setstate__ and gotten the same effect.

这篇关于Python NotImplementedError:无法在进程之间传递池对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！