Python 多处理:如何在异常时关闭多处理池 [英] Python multiprocessing: How to close the multiprocessing pool on exception

问题描述

我正在使用 python 多处理来拆分较长的进程之一并并行运行.它工作正常，除非其中一个子进程出现异常，在这种情况下，进程池没有关闭，我仍然可以在服务器上看到这些进程.

I am using python multiprocessing to split one of the longer processes and run parallelly. It is working fine except when there is an exception in one of the child processes, in which case, process pool is not closed and I can still see those processes on the server.

代码如下:

from multiprocessing import Pool
pool = Pool(processes=4)
from functools import partial
param_data = "Test Value"
func = partial(test_function, param_data)
r = pool.map(func, range(3))
pool.close()

def test_function(param_data,index):
   try:
      # The process here;
   except Exception as e:
      # Close the process pool;

在给出 pool.close 里面的 except 块中，它说

On giving pool.close inside the except block it says

NameError: 全局名称 'pool' 未定义

NameError: global name 'pool' is not defined

我尝试使用以下代码终止异常进程.

I tried to kill the process on Exception with the following code.

    except Exception as e:
       import os
       import signal
       pid = os.getpid()
       os.kill(pid, signal.SIGTERM) 

但是我仍然可以在服务器上看到进程.这仍然不是最好的解决方案，因为这只会终止遇到异常的子进程，其他进程仍将继续.

But I can still see the process on the server. This still not the best solution as this will only terminate the child process which encountered an exception other processes will still go on.

我希望所有进程在完成时终止，无论它们是否遇到异常.

I want all processes to terminate on completion, irrespective of if they encounter with an exception or not.

我正在运行 Python2.7

I am running Python2.7

Ps:我无法在服务器上安装像 psutil 这样的新库，我正在尝试使用标准 python 库的解决方案.

Ps: I cannot install a new library like psutil on the server, I am trying for a solution using standard python library.

我检查了几个类似的问题，例如，自动杀死进程和子进程 在此论坛中，但它们并不是真正的问题.

I checked few similar questions such as, auto kill process and child in this forum but they were not really this issue.

推荐答案

我得到了解决方案 - 在父进程捕获异常.

I got the solution for this - Catch exception at the parent process.

try:
   pool = Pool(processes=4)
   from functools import partial
   param_data = "Test Value"
   func = partial(test_function, param_data)
   r = pool.map(func, range(3))
except Exception as e:
   pool.close()
pool.close()

并且还在子进程函数中添加一个try/catch:

And also add a try/catch in the child process function:

def test_function(param_data,index):
    try:
       # The process here;
    except Exception as e:
       raise Exception(e.message)          

这里的 except 块看起来是多余的，但事实并非如此.原因是，子进程引发的一些异常没有被抛出到父进程.

The except block here looks redundant but it is not. The reason is, some of the exceptions raised by child process were not thrown to the parent process.

例如，我在函数中引发了一个自定义异常并且它没有抛出到父进程，因此建议捕获子进程中的所有异常并从那里引发标准异常，然后在父进程中处理它进程，您可以在其中关闭进程池或进行其他清理.

For example, I was raising a custom exception in the function and it was not thrown to the parent process, So it is advised to catch all exceptions within the child process and raise the standard Exception from there, Then handle it at parent process, where you can close the process pool or do other cleanups.

这篇关于Python 多处理:如何在异常时关闭多处理池的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！