Python:如何检查multiprocessing.Pool中待处理任务的数量? [英] Python: How can I check the number of pending tasks in a multiprocessing.Pool?

问题描述

我只有一小群工人(4)和大量任务(5000〜).我正在使用一个池，并使用map_async()发送任务.因为我正在运行的任务相当长，所以我将块大小强制为1，这样一个长进程就无法容纳一些较短的进程.

I have a small pool of workers (4) and a very large list of tasks (5000~). I'm using a pool and sending the tasks with map_async(). Because the task I'm running is fairly long, I'm forcing a chunksize of 1 so that one long process can't hold up some shorter ones.

我想做的是定期检查还有多少任务要提交.我知道最多4个会处于活动状态，我担心还有多少要处理.

What I'd like to do is periodically check how many tasks are left to be submitted. I know at most 4 will be active, I'm concerned with how many are left to process.

我在Google周围搜索，找不到任何人这样做.

I've googled around and I can't find anybody doing this.

一些简单的代码可以帮助您

Some simple code to help:

import multiprocessing
import time

def mytask(num):
    print('Started task, sleeping %s' % num)
    time.sleep(num)

pool = multiprocessing.Pool(4)
jobs = pool.map_async(mytask, [1,2,3,4,5,3,2,3,4,5,2,3,2,3,4,5,6,4], chunksize=1)
pool.close()

while True:
    if not jobs.ready():
        print("We're not done yet, %s tasks to go!" % <somethingtogettasks>)
        jobs.wait(2)
    else:
        break

推荐答案

像Jobs._number_left是您想要的. _表示它是一个内部值，可能会随开发人员的想法而改变，但这似乎是获取该信息的唯一方法.

Looks like jobs._number_left is what you want. _ indicates that it is an internal value which may change at the whim of the developers, but it seems to be the only way to get that info.

这篇关于Python:如何检查multiprocessing.Pool中待处理任务的数量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！