来自IPython并行引擎的实时输出？ [英] Real-time output from engines in IPython parallel?

问题描述

我正在使用IPython的并行化功能运行一系列长时间运行的任务。

I am running a bunch of long-running tasks with IPython's great parallelization functionality.

如何在我的IPython客户端中从ipengines的标准输出获得实时输出？

How can I get real-time output from the ipengines' stdout in my IPython client?

例如，我正在运行 dview.map_async（fun，lots_of_args）和 fun 打印到stdout。我希望看到输出正在发生。

E.g., I'm running dview.map_async(fun, lots_of_args) and fun prints to stdout. I would like to see the outputs as they are happening.

我知道 AsyncResult.display_output（），但是只有在所有任务完成后才可用。

I know about AsyncResult.display_output(), but it's only available after all tasks have finished.

推荐答案

您可以通过访问同时查看stdout AsyncResult.stdout ，它将返回一个字符串列表，这是每个引擎的标准输出。

You can see stdout in the meantime by accessing AsyncResult.stdout, which will return a list of strings, which are the stdout from each engine.

最简单案例是：

print ar.stdout

您可以将此包装在一个简单的函数中，该函数在您等待AsyncResult完成时打印stdout：

You can wrap this in a simple function that prints stdout while you wait for the AsyncResult to complete:

import sys
import time
from IPython.display import clear_output

def wait_watching_stdout(ar, dt=1, truncate=1000):
    while not ar.ready():
        stdouts = ar.stdout
        if not any(stdouts):
            continue
        # clear_output doesn't do much in terminal environments
        clear_output()
        print '-' * 30
        print "%.3fs elapsed" % ar.elapsed
        print ""
        for eid, stdout in zip(ar._targets, ar.stdout):
            if stdout:
                print "[ stdout %2i ]\n%s" % (eid, stdout[-truncate:])
        sys.stdout.flush()
        time.sleep(dt)

示例笔记本说明了这个函数。

现在，如果你使用的是较旧的IPython，你可以看到对stdout属性访问的人为限制（'结果未就绪'错误）。
信息在元数据中可用，因此您可以在任务未完成时获取信息：

Now, if you are using older IPython, you may see an artificial restriction on access of the stdout attribute ('Result not ready' errors). The information is available in the metadata, so you can still get at it while the task is not done:

rc.spin()
stdout = [ rc.metadata[msg_id]['stdout'] for msg_id in ar.msg_ids ]

这与 ar.stdout 属性访问基本相同。

Which is essentially the same thing that the ar.stdout attribute access does.

这篇关于来自IPython并行引擎的实时输出？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！