函数初始化和对象初始化(多处理) [英] function initialize and object initialization (multiprocessing)

问题描述

我最近看到了有关函数是python中的对象的答案/评论.因此，我想知道为什么当我举这个例子，并在初始化变量时围绕它创建一个类时，它的工作方式不同. (该类示例收到一个酸洗错误): PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

I recently saw a answer/comment about how functions are objects in python. So, I wondering why when I take that example, and create a class around it while initializing a variable, it doesn't work the same way. (The class example receives a pickling error): PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

有人知道这是为什么吗?

Does anyone know why this is?

链接中的示例代码:

import multiprocessing as mp

def f(x):
    f.q.put('Doing: ' + str(x))
    return x*x

def f_init(q):
    f.q = q

def main():
    jobs = range(1,6)

    q = mp.Queue()
    p = mp.Pool(None, f_init, [q])
    results = p.imap(f, jobs)
    p.close()

    for i in range(len(jobs)):
        print q.get()
        print results.next()

if __name__ == '__main__':
    main()

将f放入类中的相同示例:

Same example while puttin f into a class:

import multiprocessing as mp

class F:
    def __init__(self, q):
        self.q = q
    def f(x):
        self.q.put('Doing: ' + str(x))
        return x*x

def main():
    jobs = range(1,6)

    q = mp.Queue()
    p = mp.Pool(None)
    f = F(q)
    results = p.imap(f.f, jobs)
    p.close()

    for i in range(len(jobs)):
        print q.get()
        print results.next()

if __name__ == '__main__':
    main()

推荐答案

实例方法不会自动执行.所以

Instance methods are not automatically picklable. So

p.imap(f.f, jobs)

失败，因为p.imap试图使参数变酸. 有一种方法可以教导"腌菜如何腌制实例方法(请参阅史蒂文·贝萨德的回答)， 但是您的代码还有另一个问题:将队列传递给实例会导致RuntimeError:

fails because p.imap tries to pickle the arguments. There is a way to "teach" pickle how to pickle instance methods (see Steven Bethard's answer), but your code has another problem: Passing the queue to the instance leads to a RuntimeError:

RuntimeError: Queue objects should only be shared between processes through inheritance

错误消息有些混乱(至少对我来说是这样)，因为您可以将队列作为参数传递给p.imap，但是您不能先将其传递给类F ，然后通过f.f将其传输到辅助进程.

The error message is a little confusing (at least to me) since you can pass the queue as an argument to p.imap, but you can not pass it to the class F first, and then transfer it to the worker processes through f.f.

无论如何，由于这些问题，我建议坚持使用原始代码，而不是尝试将代码包装在一个类中.

Anyway, because of these problems, I'd suggest sticking with the original code instead of trying to wrap the code in a class.

以下是如何腌制实例方法的示例:

Here is an example of how to pickle instance methods:

import multiprocessing as mp
import copy_reg
import types

def _pickle_method(method):
    # Author: Steven Bethard (author of argparse)
    # http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-
    # instancemethods
    func_name = method.im_func.__name__
    obj = method.im_self
    cls = method.im_class
    cls_name = ''
    if func_name.startswith('__') and not func_name.endswith('__'):
        cls_name = cls.__name__.lstrip('_')
    if cls_name:
        func_name = '_' + cls_name + func_name
    return _unpickle_method, (func_name, obj, cls)


def _unpickle_method(func_name, obj, cls):
    # Author: Steven Bethard
    # http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-
    # instancemethods
    for cls in cls.mro():
        try:
            func = cls.__dict__[func_name]
        except KeyError:
            pass
        else:
            break
    return func.__get__(obj, cls)

# This call to copy_reg.pickle allows you to pass methods as the first arg to
# mp.Pool methods. If you comment out this line, `pool.map(self.foo, ...)` results in
# PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
# __builtin__.instancemethod failed

copy_reg.pickle(types.MethodType, _pickle_method, _unpickle_method)

class F(object):
    def f(self, x):
        fq.put('Doing: ' + str(x))        
        return x*x

def f_init(q):
    # http://stackoverflow.com/a/3843313/190597 (Olson)
    global fq
    fq = q

def main():
    jobs = range(1,6)
    q = mp.Queue()
    p = mp.Pool(None, f_init, [q]) 
    f = F()
    results = p.imap(f.f, jobs)
    p.close()

    for r in results:
        print(r, q.get())

if __name__ == '__main__':
    main()

收益

(1, 'Doing: 2')
(4, 'Doing: 3')
(9, 'Doing: 4')
(16, 'Doing: 1')
(25, 'Doing: 5')

这篇关于函数初始化和对象初始化(多处理)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！