显示Python多处理池映射调用的进度? [英] Show the progress of a Python multiprocessing pool map call?

问题描述

我有一个脚本可以通过imap_unordered()调用成功完成多处理池任务集:

I have a script that's successfully doing a multiprocessing Pool set of tasks with a imap_unordered() call:

p = multiprocessing.Pool()
rs = p.imap_unordered(do_work, xrange(num_tasks))
p.close() # No more work
p.join() # Wait for completion

但是，我的num_tasks大约为250,000，因此join()锁定主线程10秒钟左右，我希望能够逐步回显命令行以显示主进程没有锁定.像这样:

However, my num_tasks is around 250,000, and so the join() locks the main thread for 10 seconds or so, and I'd like to be able to echo out to the command line incrementally to show the main process isn't locked. Something like:

p = multiprocessing.Pool()
rs = p.imap_unordered(do_work, xrange(num_tasks))
p.close() # No more work
while (True):
  remaining = rs.tasks_remaining() # How many of the map call haven't been done yet?
  if (remaining == 0): break # Jump out of while loop
  print "Waiting for", remaining, "tasks to complete..."
  time.sleep(2)

是否有用于结果对象或池本身的方法来指示剩余任务数?我尝试使用multiprocessing.Value对象作为计数器(do_work在完成其任务后调用counter.value += 1动作)，但是在停止递增之前，计数器只能达到总值的〜85％.

Is there a method for the result object or the pool itself that indicates the number of tasks remaining? I tried using a multiprocessing.Value object as a counter (do_work calls a counter.value += 1 action after doing its task), but the counter only gets to ~85% of the total value before stopping incrementing.

推荐答案

无需访问结果集的私有属性:

There is no need to access private attributes of the result set:

from __future__ import division
import sys

for i, _ in enumerate(p.imap_unordered(do_work, xrange(num_tasks)), 1):
    sys.stderr.write('\rdone {0:%}'.format(i/num_tasks))

这篇关于显示Python多处理池映射调用的进度?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！