在“多处理"模块中模拟"as_completed" [英] Analog `as_completed` in the `multiprocessing` module

问题描述

我正在Python 2.7模块multiprocessing中寻找as_completed函数的类似物(来自Python 3 concurrent.futures).我当前的解决方案:

I'm looking for an analog of the as_completed function (from Python 3 concurrent.futures) in the Python 2.7 module multiprocessing. My current solution:

import time
from multiprocessing import Pool
def f(x):
    time.sleep(x)
    return x
if __name__ == '__main__':
    pool = Pool()
    a = pool.apply_async(f, [4])
    b = pool.apply_async(f, [2])
    while any([a,b]):
        if a and a.ready(): print a.get(); a=False 
        if b and b.ready(): print b.get(); b=False

推荐答案

一种快速而肮脏的方法是将异步结果对象存储在可迭代的状态中，并定期轮询其状态.

A quick and dirty way is to store the async result objects in an iterable and periodically poll for their status.

from multiprocessing import Pool
from random import random
from time import sleep


def wrapped_sleep(n, i):
    sleep(n)
    return n, i

if __name__ == '__main__':
    pool = Pool()
    random_sleep_durations = [random() * 10 for _ in xrange(100)]
    results = [
        pool.apply_async(wrapped_sleep, (n, i, ))
        for i, n in enumerate(random_sleep_durations)
    ]

    while results:
        sleep(0.1)
        mature_indices = []
        mature_results = []

        for i, candidate in enumerate(results):
            if candidate.ready():
                mature_indices.append(i)
                break

        for i in mature_indices:
            mature_results.append(results.pop(i).get())

        for result in mature_results:
            print result

这篇关于在“多处理"模块中模拟"as_completed"的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！