Python 多处理模块:加入超时的进程 [英] Python multiprocessing module: join processes with timeout

问题描述

我正在优化复杂模拟的参数.我使用多处理模块来提高优化算法的性能.我在 http://pymotw.com/2/multiprocessing/basics.html<学到的多处理基础知识/a>.根据优化算法的给定参数，复杂模拟持续不同的时间，大约 1 到 5 分钟.如果参数选择得非常糟糕，模拟可能会持续 30 分钟或更长时间，并且结果没有用.所以我在考虑建立多处理超时，终止所有持续超过定义时间的模拟.这是问题的抽象版本:

im doing an optimization of parameters of a complex simulation. Im using the multiprocessing module for enhancing the performance of the optimization algorithm. The basics of multiprocessing I learned at http://pymotw.com/2/multiprocessing/basics.html. The complex simulation lasts different times depending on the given parameters from the optimization algorithm, around 1 to 5 minutes. If the parameters are chosen very badly, the simulation can last 30 minutes or more and the results are not useful. So I was thinking about build in a timeout to the multiprocessing, that terminates all simulations that last more than a defined time. Heres an abstracted version of the problem:

import numpy as np
import time
import multiprocessing

def worker(num):

    time.sleep(np.random.random()*20)

def main():

    pnum = 10    

    procs = []
    for i in range(pnum):
        p = multiprocessing.Process(target=worker, args=(i,), name = ('process_' + str(i+1)))
        procs.append(p)
        p.start()
        print 'starting', p.name

    for p in procs:
        p.join(5)
        print 'stopping', p.name

if __name__ == "__main__":
    main()

p.join(5) 行定义了 5 秒的超时时间.由于 for 循环 for p in procs: 程序等待 5 秒直到第一个进程完成，然后再次等待 5 秒直到第二个进程完成等等，但我希望程序终止所有持续时间超过 5 秒的进程.此外，如果没有一个进程的持续时间超过 5 秒，则程序不得等待这 5 秒.

The line p.join(5) defines the timeout of 5 seconds. Because of the for-loop for p in procs: the program waits 5 seconds until the first process is finished and then again 5 seconds until the second process is finished and so on, but i want the program to terminate all processes that last more than 5 seconds. Additionally, if none of the processes last longer than 5 seconds the program must not wait this 5 seconds.

推荐答案

您可以通过创建一个循环来实现此目的，该循环将等待一些超时秒数，经常检查是否所有进程都已完成.如果它们没有在分配的时间内全部完成，则终止所有进程:

You can do this by creating a loop that will wait for some timeout amount of seconds, frequently checking to see if all processes are finished. If they don't all finish in the allotted amount of time, then terminate all of the processes:

TIMEOUT = 5 
start = time.time()
while time.time() - start <= TIMEOUT:
    if not any(p.is_alive() for p in procs):
        # All the processes are done, break now.
        break

    time.sleep(.1)  # Just to avoid hogging the CPU
else:
    # We only enter this if we didn't 'break' above.
    print("timed out, killing all processes")
    for p in procs:
        p.terminate()
        p.join()

这篇关于Python 多处理模块:加入超时的进程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！