更好地多线程使用Python子进程.交流()? [英] Better multithreaded use of Python subprocess.Popen & communicate()?

问题描述

我正在运行Python 2.6的Linux机器上并行运行多个命令，这可能需要一些时间.

I'm running multiple commands which may take some time, in parallel, on a Linux machine running Python 2.6.

因此，我使用subprocess.Popen类和process.communicate()方法并行化多个命令组的执行，并在执行后立即捕获输出.

So, I used subprocess.Popen class and process.communicate() method to parallelize execution of mulitple command groups and capture the output at once after execution.

def run_commands(commands, print_lock):
    # this part runs in parallel.
    outputs = []
    for command in commands:
        proc = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True)
        output, unused_err = proc.communicate()  # buffers the output
        retcode = proc.poll()                    # ensures subprocess termination
        outputs.append(output)
    with print_lock: # print them at once (synchronized)
        for output in outputs:
            for line in output.splitlines():
                print(line)

在其他地方这样称呼:

processes = []
print_lock = Lock()
for ...:
    commands = ...  # a group of commands is generated, which takes some time.
    processes.append(Thread(target=run_commands, args=(commands, print_lock)))
    processes[-1].start()
for p in processes: p.join()
print('done.')

预期结果是在并行执行命令时，一次显示一组命令的每个输出.

The expected result is that each output of a group of commands is displayed at once while execution of them is done in parallel.

但是从第二个输出组(当然，由于调度不确定性而改变了第二个线程)，它开始打印时没有换行，并增加了与前一行和输入中打印的字符数一样多的空格回声已关闭-终端状态为乱码"或崩溃". (如果发出reset shell命令，它将恢复正常.)

But from the second output group (of course, the thread that become the second is changed due to scheduling indeterminism), it begins to print without newlines and adding spaces as many as the number of characters printed in each previous line and input echo is turned off -- the terminal state is "garbled" or "crashed". (If I issue reset shell command, it restores normal.)

起初，我试图通过处理'\r'找出原因，但这不是原因.如您在我的代码中看到的，我使用splitlines()正确处理了该代码，并确认已将repr()函数应用于输出.

At first, I tried to find the reason from handling of '\r', but it was not the reason. As you see in my code, I handled it properly using splitlines(), and I confirmed that with repr() function applied to the output.

我认为原因是同时在Popen和communicate()中将管道用于stdout/stderr.我在Python 2.7中尝试了check_output快捷方式，但没有成功.当然，如果我序列化所有命令执行并打印，则不会发生上述问题.

I think the reason is concurrent use of pipes in Popen and communicate() for stdout/stderr. I tried check_output shortcut method in Python 2.7, but no success. Of course, the problem described above does not occur if I serialize all command executions and prints.

还有什么更好的方法可以并行处理Popen和communicate()?

Is there any better way to handle Popen and communicate() in parallel?

推荐答案

最终结果受塞巴斯蒂安(J.F. Sebastian)的评论启发.

A final result inspired by the comment from J.F.Sebastian.

http://bitbucket.org/daybreaker/kaist- cs443/src/247f9ecf3cee/tools/manage.py

这似乎是Python错误.

It seems to be a Python bug.

这篇关于更好地多线程使用Python子进程.交流()?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！