显示 Python 多处理池 imap_unordered 调用的进度? [英] Show the progress of a Python multiprocessing pool imap_unordered call?

问题描述

我有一个脚本，它通过 imap_unordered() 调用成功执行了多处理池任务集:

I have a script that's successfully doing a multiprocessing Pool set of tasks with a imap_unordered() call:

p = multiprocessing.Pool()
rs = p.imap_unordered(do_work, xrange(num_tasks))
p.close() # No more work
p.join() # Wait for completion

但是，我的 num_tasks 大约为 250,000，因此 join() 将主线程锁定了 10 秒左右，我希望能够以增量方式回显到命令行以显示主进程未锁定.类似的东西:

However, my num_tasks is around 250,000, and so the join() locks the main thread for 10 seconds or so, and I'd like to be able to echo out to the command line incrementally to show the main process isn't locked. Something like:

p = multiprocessing.Pool()
rs = p.imap_unordered(do_work, xrange(num_tasks))
p.close() # No more work
while (True):
  remaining = rs.tasks_remaining() # How many of the map call haven't been done yet?
  if (remaining == 0): break # Jump out of while loop
  print("Waiting for", remaining, "tasks to complete...")
  time.sleep(2)

结果对象或池本身是否有指示剩余任务数的方法?我尝试使用 multiprocessing.Value 对象作为计数器(do_work 在完成任务后调用 counter.value += 1 操作)，但是在停止递增之前，计数器仅达到总值的 85% 左右.

Is there a method for the result object or the pool itself that indicates the number of tasks remaining? I tried using a multiprocessing.Value object as a counter (do_work calls a counter.value += 1 action after doing its task), but the counter only gets to ~85% of the total value before stopping incrementing.

推荐答案

不需要访问结果集的私有属性:

There is no need to access private attributes of the result set:

from __future__ import division
import sys

for i, _ in enumerate(p.imap_unordered(do_work, xrange(num_tasks)), 1):
    sys.stderr.write('\rdone {0:%}'.format(i/num_tasks))

这篇关于显示 Python 多处理池 imap_unordered 调用的进度?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！