如何修复或重组此多处理模式以避免出现酸洗错误? [英] How to fix or reorganize this multiprocessing pattern to avoid pickling errors?

问题描述

另一个酸洗问题...以下导致酸洗错误.我认为这与范围界定或其他有关.还不确定.

Another pickling question ... The following leads to pickling errors. I think it is to do with the scoping or something. Am not sure yet.

目标是要有一个接受参数并使用方法丰富函数的装饰器.如果最好的方法是简单地显式构造类，那很好，但这意味着对编写内容"的用户隐藏了一些东西.

The goal is to have a decorator that takes arguments and enriches a function with methods. If the best way is to simply construct classes explicitly then that is fine but this is meant to hide things from users writing "content".

import concurrent.futures
import functools

class A():
    def __init__(self, fun, **kwargs):
        self.fun = fun
        self.stuff = kwargs
        functools.update_wrapper(self, fun)
    def __call__(self, *args, **kwargs):
        print(self.stuff, args, kwargs)
        return self.fun(*args, **kwargs)

def decorator(**kwargs):
    def inner(fun):
        return A(fun, **kwargs)
    return inner

@decorator(a=1, b=2)
def f():
    print('f called')

executor = concurrent.futures.ProcessPoolExecutor(max_workers=10)

tasks = [f for x in range(10)]
fut = list()
for task in tasks:
    fut.append(executor.submit(task))
res = [x.result() for x in fut]
print(res)

错误是:

_pickle.PicklingError: Can't pickle <function f at 0x7fe37da121e0>: it's not the same object as __main__.f

推荐答案

我最终做了这样的事情:

I ended up doing something like this:

def dill_wrapped(dilled, *args, **kwargs):
    fun = dill.loads(dilled)
    return wrapped(fun, *args, **kwargs)

您可能应该努力避免这种情况的发生，但是有时在装饰函数时确实确实需要它.

You should probably try hard to avoid the need for this but you really do need it sometimes when decorating functions.

这篇关于如何修复或重组此多处理模式以避免出现酸洗错误?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！