在IPython并行进程中打印到stdout [英] Printing to stdout in IPython parallel processes

问题描述

我是IPython的新手，想在运行IPython并行集群功能时将中间结果打印到stdout。 （我知道有多个进程，这可能会破坏输出，但这很好 - 它只是用于测试/调试，而我正在运行的进程足够长，以至于不太可能发生此类冲突。）我检查了IPython的文档，但找不到并行化函数打印的示例。
基本上，我正在寻找一种方法将子流程的打印输出重定向到主stdout，IPython相当于

I'm new to IPython and would like to print intermediate results to stdout while running IPython parallel cluster functions. (I'm aware that with multiple processes, this might mangle the output, but that's fine--it's just for testing/debugging, and the processes I'd be running are long enough that such a collision is unlikely.) I checked the documentation for IPython but can't find an example where the parallelized function prints. Basically, I'm looking for a way to redirect the print output of the subprocesses to the main stdout, the IPython equivalent of

subprocess.Popen( ... , stdout=...)

在过程不起作用：

rc = Client()
dview = rc()
def ff(x):
    print(x)
    return x**2
sync = dview.map_sync(ff,[1,2,3,4])
print('sync res=%s'%repr(sync))
async = dview.map_async(ff,[1,2,3,4])
print('async res=%s'%repr(async))
print(async.display_outputs())

返回

sync res=[1, 4, 9, 16]
async res=[1, 4, 9, 16]

因此计算执行正确，但函数ff中的print语句从不打印，即使在所有进程中也是如此已经回来了。
我做错了什么？如何使打印工作？

So the computation executes correctly, but the print statement in the function ff is never printed, not even when all the processes have returned. What am I doing wrong? How do I get "print" to work?

推荐答案

它实际上更类似于 subprocess.Popen（ ......，stdout = PIPE）比你想象的还要多。
就像 Popen 对象有一个 stdout 属性，你可以阅读该属性，看看它的标准输出subprocess，
AsyncResult有一个 stdout 属性，其中包含从引擎捕获的stdout。
确实不同之处在于 AsyncResult.stdout 是字符串列表，其中列表中的每个项目都是单个的标准输出引擎作为一个字符串。

It's actually more similar to subprocess.Popen( ... , stdout=PIPE) than you seem to be expecting. Just like the Popen object has a stdout attribute, which you can read to see the stdout of the subprocess, An AsyncResult has a stdout attribute that contains the stdout captured from the engines. It does differ in that AsyncResult.stdout is a list of strings, where each item in the list is the stdout of a single engine as a string.

所以，开始：

rc = parallel.Client()
dview = rc[:]
def ff(x):
    print(x)
    return x**2
sync = dview.map_sync(ff,[1,2,3,4])
print('sync res=%r' % sync)
async = dview.map_async(ff,[1,2,3,4])
print('async res=%r' % async)
async.get()

给出

sync res=[1, 4, 9, 16]
async res=<AsyncMapResult: ff>

我们可以看到 AsyncResult.stdout 列表字符串：

We can see the AsyncResult.stdout list of strings:

print(async.stdout)
['1\n2\n', '3\n4\n']

我们可以看到异步结果的标准输出：

We can see the stdout of the async result:

print('async output:')
async.display_outputs()

打印：

async output:
[stdout:0] 
1
2
[stdout:1] 
3
4

这是一个笔记本，所有这些都证明了这一点。

And here is a notebook with all of this demonstrated.

根据您的问题需要注意的一些事项：

Some things to note, based on your question:

1. 你必须等待在输出就绪之前完成AsyncResult（ async.get（））


2. display_outputs（）不返回任何内容 - 它实际上是打印/显示本身，所以 print（async.display_outputs（）） 没有意义。

1. you have to wait for the AsyncResult to finish, before outputs are ready (async.get())
2. display_outputs() does not return anything - it actually does the printing/displaying itself, so print(async.display_outputs()) doesn't make sense.

这篇关于在IPython并行进程中打印到stdout的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！