带有多处理的Python装饰器失败 [英] Python decorator with multiprocessing fails

问题描述

我想在一个函数上使用装饰器，然后将其传递给多处理池.但是，该代码失败，并显示"PicklingError:无法进行腌制:属性查找__builtin__ .function失败".我不太明白为什么它在这里失败.我确信这很简单，但是我找不到.下面是一个最小的工作"示例.我认为使用functools函数足以完成这项工作.

I would like to use a decorator on a function that I will subsequently pass to a multiprocessing pool. However, the code fails with "PicklingError: Can't pickle : attribute lookup __builtin__.function failed". I don't quite see why it fails here. I feel certain that it's something simple, but I can't find it. Below is a minimal "working" example. I thought that using the functools function would be enough to let this work.

如果我注释掉功能修饰，它可以正常工作.我在这里误解了multiprocessing是什么?有什么办法可以做到这一点?

If I comment out the function decoration, it works without an issue. What is it about multiprocessing that I'm misunderstanding here? Is there any way to make this work?

编辑:添加了可调用类装饰器和函数装饰器后，事实证明函数装饰器可以按预期工作.可调用类装饰器继续失败.可调用类的版本是什么使它不会被腌制?

Edit: After adding both a callable class decorator and a function decorator, it turns out that the function decorator works as expected. The callable class decorator continues to fail. What is it about the callable class version that keeps it from being pickled?

import random
import multiprocessing
import functools

class my_decorator_class(object):
    def __init__(self, target):
        self.target = target
        try:
            functools.update_wrapper(self, target)
        except:
            pass

    def __call__(self, elements):
        f = []
        for element in elements:
            f.append(self.target([element])[0])
        return f

def my_decorator_function(target):
    @functools.wraps(target)
    def inner(elements):
        f = []
        for element in elements:
            f.append(target([element])[0])
        return f
    return inner

@my_decorator_function
def my_func(elements):
    f = []
    for element in elements:
        f.append(sum(element))
    return f

if __name__ == '__main__':
    elements = [[random.randint(0, 9) for _ in range(5)] for _ in range(10)]
    pool = multiprocessing.Pool(processes=4)
    results = [pool.apply_async(my_func, ([e],)) for e in elements]
    pool.close()
    f = [r.get()[0] for r in results]
    print(f)

推荐答案

问题是，泡菜需要某种方式来重新组装您泡菜的所有东西.请参阅此处以获取可腌制食品的列表:

The problem is that pickle needs to have some way to reassemble everything that you pickle. See here for a list of what can be pickled:

http://docs.python .org/library/pickle.html#what-c​​an-pickled和unpickled

酸洗my_func时，需要酸洗以下成分:

When pickling my_func, the following components need to be pickled:

• my_decorator_class的一个实例，称为my_func.

这很好. Pickle将存储该类的名称，并对其__dict__内容进行pickle.取消拾取时，它将使用名称查找类，然后创建一个实例并填写__dict__内容.但是，__dict__内容存在问题...

This is fine. Pickle will store the name of the class and pickle its __dict__ contents. When unpickling, it uses the name to find the class, then creates an instance and fills in the __dict__ contents. However, the __dict__ contents present a problem...

存储在my_func.target中的原始my_func的实例.

The instance of the original my_func that's stored in my_func.target.

这不是很好.这是顶层的功能，通常可以将其腌制. Pickle将存储函数的名称.但是，问题在于名称"my_func"不再绑定到未修饰的函数，而是绑定到修饰的函数.这意味着pickle将无法查找未修饰的函数来重新创建对象.可悲的是，泡菜没有任何办法知道它尝试泡菜的对象总是可以在名称__main__.my_func下找到.

This isn't so good. It's a function at the top-level, and normally these can be pickled. Pickle will store the name of the function. The problem, however, is that the name "my_func" is no longer bound to the undecorated function, it's bound to the decorated function. This means that pickle won't be able to look up the undecorated function to recreate the object. Sadly, pickle doesn't have any way to know that object it's trying to pickle can always be found under the name __main__.my_func.

您可以像这样更改它，它将起作用:

You can change it like this and it will work:

import random
import multiprocessing
import functools

class my_decorator(object):
    def __init__(self, target):
        self.target = target
        try:
            functools.update_wrapper(self, target)
        except:
            pass

    def __call__(self, candidates, args):
        f = []
        for candidate in candidates:
            f.append(self.target([candidate], args)[0])
        return f

def old_my_func(candidates, args):
    f = []
    for c in candidates:
        f.append(sum(c))
    return f

my_func = my_decorator(old_my_func)

if __name__ == '__main__':
    candidates = [[random.randint(0, 9) for _ in range(5)] for _ in range(10)]
    pool = multiprocessing.Pool(processes=4)
    results = [pool.apply_async(my_func, ([c], {})) for c in candidates]
    pool.close()
    f = [r.get()[0] for r in results]
    print(f)

---

您已经观察到装饰器函数在类不起作用时起作用.我相信这是因为functools.wraps修改了修饰的函数，以便它具有其包装的函数的名称和其他属性.就pickle模块所能分辨的而言，它与正常的顶级函数没有区别，因此它通过存储其名称来对其进行腌制.取消拾取后，该名称便绑定到装饰的函数上，因此一切正常.

---

You have observed that the decorator function works when the class does not. I believe this is because functools.wraps modifies the decorated function so that it has the name and other properties of the function it wraps. As far as the pickle module can tell, it is indistinguishable from a normal top-level function, so it pickles it by storing its name. Upon unpickling, the name is bound to the decorated function so everything works out.

这篇关于带有多处理的Python装饰器失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！