Python“打印"嵌入MPI程序时不起作用 [英] Python "print" not working when embedded into MPI program

问题描述

我在C ++ MPI应用程序中嵌入了Python 3解释器.此应用程序加载脚本并将其传递给解释器.

I have an Python 3 interpreter embedded into an C++ MPI application. This application loads a script and passes it to the interpreter.

当我在没有MPI启动器的情况下(仅调用./myprogram)在1个进程上执行程序时，脚本将正确执行，并且其"print"语句将输出到终端.当脚本有错误时，我会使用PyErr_Print()在C ++端进行打印.

When I execute the program on 1 process without the MPI launcher (simply calling ./myprogram), the script is executed properly and its "print" statements output to the terminal. When the script has an error, I print it on the C++ side using PyErr_Print().

但是，当我通过mpirun(甚至在单个进程中)对程序进行放血时，我没有从python代码中的打印"中获得任何输出.当我的脚本有错误时，我也无法从PyErr_Print()得到任何信息.

However when I lauch the program through mpirun (even on a single process), I don't get any output from the "print" in the python code. I also don't get anything from PyErr_Print() when my script has errors.

我猜想Python处理标准输出的方式与MPI(此处为actuall Mpich)处理将进程的输出重定向到启动器并最终重定向到终端的方式不匹配.

I guess there is something in the way Python deals with standard output that do not match the way MPI (actuall Mpich here) deals with redirecting the processes' output to the launcher and finally to the terminal.

关于如何解决此问题的任何想法?

Any idea on how to solve this?

推荐答案

我最终遇到了同样的问题( PyErr_Print 在mpirun上不起作用).追溯(涉及到python3的某些gdb)并比较工作的东西(./myprogram)和不工作的东西(mpirun -np 1 ./myprogram)，我最终在的 _io_TextIOWrapper_write_impl ./Modules/_io/textio.c:1277 (顺便说一下，python-3.6.0).

I ended up with the same issue (PyErr_Print not working from an mpirun). Tracing back (some gdb of python3 involved) and comparing the working thing (./myprogram) and non-working thing (mpirun -np 1 ./myprogram), I ended up in _io_TextIOWrapper_write_impl at ./Modules/_io/textio.c:1277 (python-3.6.0 by the way).

这两次运行之间的唯一区别是 self-> line_buffering 是1比0(此时 self 代表 sys.stderr ).然后，在 pylifecycle.c:1128 中，我们可以看到谁决定了该值:

The only difference between the 2 runs is that self->line_buffering is 1 vs. 0 (at this point self represents sys.stderr). Then, in pylifecycle.c:1128, we can see who decided this value:

if (isatty || Py_UnbufferedStdioFlag)
    line_buffering = Py_True;

因此，似乎MPI在启动程序之前会对stderr做些操作，这使其不是tty.我尚未调查mpirun中是否存在将tty标志保留在stderr上的选项...如果有人知道，这会很有趣(尽管第二个想法是mpi可能有充分的理由将他的文件描述符替换为stdout&.stderr，例如其--output-filename).

So it seems that MPI does something to stderr before launching the program, which makes it not a tty. I haven't investigated if there's an option in mpirun to keep the tty flag on stderr ... if someone knows, it'd be interesting (though on second thought mpi probably has good reasons to put his file descriptors in place of stdout&stderr, for its --output-filename for example).

有了此信息，我可以提出3个解决方案(前2个是快速修复，第3个是更好的解决方案):

With this info, I can come out with 3 solutions (the first 2 are quick-fixes, the 3rd is better):

1/在启动python解释器的C代码中，在创建sys.stderr之前设置缓冲标志.代码变为:

1/ in the C code that starts the python interpreter, set the buffering flag before creating sys.stderr. The code becomes :

Py_UnbufferedStdioFlag = 1;   // force line_buffering for _all_ I/O
Py_Initialize(); 

在所有情况下，这都会使Python的追溯回到屏幕上；但可能会带来灾难性的I/O ...，因此只有在调试模式下可以接受的解决方案.

This brings Python's traceback back to screen in all situations; but will probably give catastrophic I/O ... so only an acceptable solution in debug mode.

2/在python(嵌入式)脚本中，一开始就添加以下内容:

2/ in the python (embedded) script, at the very beginning add this :

import sys
#sys.stderr.line_buffering = True  # would be nice, but readonly attribute !
sys.stderr = open("error.log", 'w', buffering=1 )

然后，脚本将回溯转储到此文件error.log.

The script then dumps the traceback to this file error.log.

我还尝试在PyErr_Print()之后立即添加对fflush(stderr)或fflush(NULL)的调用...但这没有用(因为sys.stderr具有自己的内部缓冲).不过那将是一个不错的解决方案.

I also tried adding a call to fflush(stderr) or fflush(NULL) right after the PyErr_Print() ... but this didn't work (because sys.stderr has its own internal buffering). That'd be a nice solution though.

3/经过进一步的挖掘，我在

3/ After a little more digging, I found the perfect function in

Python/pythonrun.c:57:static void flush_io(void);

实际上在此文件中的每个PyErr_Print之后都调用它.不幸的是，它是静态的(仅存在于该文件中，至少在3.6.0中没有在Python.h中对其进行引用).我将该函数从该文件复制到了myprogram中，结果证明它确实可以完成这项工作.

It is in fact called after each PyErr_Print in this file. Unfortunately it's static (only exists in that file, no reference to it in Python.h, at least in 3.6.0). I copied the function from this file to myprogram and it turns out to do exactly the job.

这篇关于Python“打印"嵌入MPI程序时不起作用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！