在装饰器中使用多处理会产生错误:无法腌制功能...找不到 [英] Using multiprocessing inside decorator generates error: can't pickle function...it's not found as

问题描述

我遇到了一个无法解决的问题，它与多重处理相关联，并在装饰器中使用.

I came across a problem that I can't solve and it's associated with multiprocessing and use it inside the decorator.

当我使用多重处理调用方法run_in_parallels时，我得到了错误:

When I'm calling the method run_in_parallels using multiprocessing I 'm getting the error:

Can't pickle <function run_testcase at 0x00000000027789C8>: it's not found as __main__.run_testcase

该调用在装饰器内部进行，然后出现上述问题.在没有装饰器的情况下调用同一方法run_in_parallels时，所有方法都正常工作.

The call takes place inside the decorator, then followed the above-mentioned problem. At the time of calling the same method run_in_parallels without a decorator all working properly.

此问题的原因是什么?

文件:w_PythonHelper.py

file: w_PythonHelper.py

描述:函数"run_in_parallel"用于同时运行多个进程.第一种方法将终止操作，从而停止其他操作.

desc: Function 'run_in_parallel' is used to run multiple processes simultaneously. The first method, which will end operation stops the others.

from multiprocessing import Process,Event

class ExtProcess(Process):
    def __init__(self, event,*args,**kwargs):
        self.event=event
        Process.__init__(self,*args,**kwargs)

    def run(self):
        Process.run(self)
        self.event.set()

class PythonHelper(object):
    @staticmethod
    def run_in_parallel(*functions):
        event=Event()
        processes=dict()
        for function in functions:
            fname=function[0]
            try:fargs=function[1]
            except:fargs=list()
            try:fproc=function[2]
            except:fproc=1
            for i in range(fproc):
                process=ExtProcess(event,target=fname,args=fargs)
                process.start()
                processes[process.pid]=process
        event.wait()
        for process in processes.values():
            process.terminate()
        for process in processes.values():
            process.join()

文件:w_Recorder.py

file: w_Recorder.py

描述:功能捕获"用于抓取屏幕截图

desc: function 'capture' is used to grab a screenshot

from PIL import ImageGrab
import time

class Recorder(object):
    def capture(self):
        ImageGrab.grab().save("{f}.{e}".format(f=time.time(),e="png"))

文件:w_Decorators.py

file: w_Decorators.py

描述:并行运行给定功能以及类'Recorder'的方法'capture'

desc: Running parallel a given function along with a method 'capture' of class 'Recorder'

from w_Recorder import Recorder
from w_PythonHelper import PythonHelper

def check(function):
    def wrapper(*args):
        try:
            recorder=Recorder()
            PythonHelper.run_in_parallel([function,args],[recorder.capture])
            print("success")
        except Exception as e:
            print("failure: {}".format(e))
        return function
    return wrapper

文件: w_Logger.py

file: w_Logger.py

描述:主程序(生成错误)

from w_Decorators import check
import time

class Logger(object):

    @check
    def run_testcase(self):
        # example function (runtime: 20s)
        for i in range(20):
            print("number: {}".format(i))
            time.sleep(1)

    def run_logger(self):
        self.run_testcase()


if __name__=="__main__":
    logger=Logger()
    logger.run_logger()

文件: w_Logger.py

file: w_Logger.py

描述:主程序(完全起作用)

desc: Main program (works corectly)

from w_PythonHelper import PythonHelper
from w_Recorder import Recorder
import time

class Logger(object):

    def run_testcase(self):
        # example function (runtime: 20s)
        for i in range(20):
            print("number: {}".format(i))
            time.sleep(1)

    def run_logger(self):
        recorder=Recorder()
        PythonHelper.run_in_parallel([self.run_testcase],[recorder.capture])

if __name__=="__main__":
    logger=Logger()
    logger.run_logger()

在两种情况下介绍的这些相同方法的不同之处是什么?

What is the difference that these same methods presented in the two cases work differently?

有没有人有解决这个问题的想法(这是Python错误)吗?如果不是，也许有人知道在应用程序运行时捕获屏幕截图的好方法吗?

Does anyone has an idea to solve this problem (is this Python bug)? If not, maybe someone knows a good way to capture screen shots when the application is running?

实际上，我发现了一个类似的问题: multiprocessing.Process子类起作用在Linux而非Windows上

Actually I found similiar question: multiprocessing.Process subclass works on Linux but not Windows

答案是:To fix this, you can remove the process member.，但是对于我的示例我该怎么做.

The answer is: To fix this, you can remove the process member., but how can I do this for my example.

在调试时，在run_in_parallel(*functions)

就像 ivan_pozdeev 写道:我可以将包装器用作函数，但不能将其用作装饰器.我有许多由该装饰器装饰的功能，最简单的方法是在装饰器内部使用多处理.但是不幸的是我无法解决这个问题.也许有人已经解决了类似的问题.我将不胜感激.

like ivan_pozdeev wrote: i can use wrapper as a function, but can't use it as decorator. I have many functions decorated by this decorator and the simpliest way is to use multiprocessing inside decorator. But unfortunatly I can't solve this problem. Maybe someone has already solved a similar problem. I would be grateful for any hint.

'run_in_parallel'函数的工作方式与我想要的一样.两个或多个功能并行运行，完成的第一个功能将强制终止第二个功能.当我调用 wrapper(function，* args)时，函数工作正常，当我将此机制放入装饰器中时，我得到无法腌制函数...找不到" 错误.详细信息可以在上面找到

'run_in_parallel' function works like I want. Two or more functions run in parallel and the first function, which is completed forces the termination of second function. When I call wrapper(function,*args) then functions works ok, when I put this mechanism inside decorator I get 'can't pickle function...it's not found as' error. Details can be found above

我的回溯:

Traceback (most recent call last):
  File "C:\Interpreters\Python32\lib\pickle.py", line 679, in save_global
    klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'run_testcase'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\EskyTests\w_Logger.py", line 19, in <module>
    logger.run_logger()
  File "C:\EskyTests\w_Logger.py", line 14, in run_logger
    self.run_testcase()
  File "C:\EskyTests\w_Decorators.py", line 14, in wrapper
    PythonHelper.run_in_parallel([function,args],[recorder.capture])
  File "C:\EskyTests\w_PythonHelper.py", line 25, in run_in_parallel
    process.start()
  File "C:\Interpreters\Python32\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Interpreters\Python32\lib\multiprocessing\forking.py", line 267, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Interpreters\Python32\lib\multiprocessing\forking.py", line 190, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Interpreters\Python32\lib\pickle.py", line 237, in dump
    self.save(obj)
  File "C:\Interpreters\Python32\lib\pickle.py", line 344, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Interpreters\Python32\lib\pickle.py", line 432, in save_reduce
    save(state)
  File "C:\Interpreters\Python32\lib\pickle.py", line 299, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Interpreters\Python32\lib\pickle.py", line 623, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Interpreters\Python32\lib\pickle.py", line 656, in _batch_setitems
    save(v)
  File "C:\Interpreters\Python32\lib\pickle.py", line 299, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Interpreters\Python32\lib\pickle.py", line 683, in save_global
    (obj, module, name))
_pickle.PicklingError: Can't pickle <function run_testcase at 0x00000000027725C8>: it's not found as __main__.run_testcase

推荐答案

您要传递给Process.__init__()的函数在Windows中不可用.阅读 16.6多重处理-编程指南-Windows .

The function you're passing to Process.__init__() is not picklable in Windows. Read 16.6 multiprocessing - Programming guidelines - Windows .

关于您使用顶级函数的错误-我怀疑您定义错误的方式，它每次生成的方式都不相同，因此在子级中实际上是不是同一对象". 我建议传递一个简单的顶级函数，如果您确实需要这种复杂程度，可以使用反射调用run_testcase.更新:这没有帮助

Regarding your error with a top-level function - I suspect the way you defined it, it is generated differently each time and thus is really "not the same object" in child. I suggest passing a simple top-level function that would call run_testcase using reflection if you really need this level of sophistication. Update: this didn't help

更新:

我通过取消装饰run_testcase，run_in_parallel和capture来完成这项工作. @check装饰器替换为具有相同功能的def wrapper(function,*args):

I made this work by undecorating run_testcase,run_in_parallel and capture. @check decorator was replaced with def wrapper(function,*args) with the same functionality:

import traceback
def wrapper(function,*args):
    try:
        recorder=Recorder()
        PythonHelper().run_in_parallel([function,args],[recorder.capture])
        print("success")
    except Exception,e:
        print("failure: "+traceback.format_exc(10))

主要:

from w_Decorators import wrapper

if __name__=="__main__":
    logger=Logger()
    wrapper(logger.run_testcase)

就像我想的那样-装饰对象是不可腌制的.

Just as I thought - decorated objects are not picklable.

这篇关于在装饰器中使用多处理会产生错误:无法腌制功能...找不到的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！