为什么与多个 Popen 子进程一起使用时会发生通信死锁? [英] Why does communicate deadlock when used with multiple Popen subprocesses?

问题描述

以下问题在 Python 2.7.3 中不会发生.但是，它发生在我的机器(64 位 Mac OSX 10.7.3)上的 Python 2.7.1 和 Python 2.6 上.这是我最终会分发的代码，所以我想知道是否有任何方法可以完成这项任务，而不会如此显着地依赖于 Python 版本.

The following issue does not occur in Python 2.7.3. However, it occurs with both Python 2.7.1 and Python 2.6 on my machine (64-bit Mac OSX 10.7.3). This is code I will eventually distribute, so I would like to know if there is any way to complete this task that does not depend so dramatically on the Python version.

我需要并行打开多个子进程并将 STDIN 数据写入每个子进程.通常我会使用 Popen.communicate 方法来做到这一点.但是，每当我同时打开多个进程时，communicate 就会死锁.

I need to open multiple subprocesses in parallel and write STDIN data to each of them. Normally I would do this using the Popen.communicate method. However, communicate is deadlocking whenever I have multiple processes open at the same time.

import subprocess

cmd = ["grep", "hello"]
processes = [subprocess.Popen(cmd, stdin=subprocess.PIPE,
                              stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                                for _ in range(2)]

for p in processes:
    print p.communicate("hello world\ngoodbye world\n")

如果我将进程数更改为 for _ in range(1)，则输出与预期的一样:

If I change the number of processes to for _ in range(1), the output is just as expected:

('hello world\n', '')

但是，当有两个进程(for _ in range(2))时，进程会无限阻塞.我尝试了手动写入标准输入的替代方法:

However, when there are two processes (for _ in range(2)), the process blocks indefinitely. I've tried the alternative of writing to stdin manually:

for p in processes:
    p.stdin.write("hello world\ngoodbye world\n")

但是任何从进程中读取数据的尝试(例如，p.stdout.read())仍然会死锁.

But then any attempt to read from the processes, (p.stdout.read(), for example) still deadlocks.

起初这个似乎是相关的，但它指定它在使用多个线程时发生，并且死锁很少发生(而在这里它总是发生).有没有办法让它在 2.7.3 之前的 Python 版本上工作?

At first this appears to be related, but it specifies that it occurs when multiple threads are being used, and that the deadlocking occurs only very infrequently (while here it always occurs). Is there any way to get this to work on versions of Python before 2.7.3?

推荐答案

我不得不为这个方案挖掘一下.(我曾经遇到过类似的问题，所以以为我知道答案，但错了.)

I had to dig a bit for this one. (I ran into a similar problem once, so thought I knew the answer, but was wrong.)

此处描述了该问题(以及 2.7.3 的补丁):

The issue (and patch for 2.7.3) is described here:

http://bugs.python.org/issue12786

问题是子进程继承了 PIPE.答案是在 Popen 调用中使用 'close_fds=True'.

The issue is that the PIPEs get inherited by subprocesses. The answer is to use 'close_fds=True' in your Popen call.

processes = [subprocess.Popen(cmd, stdin=subprocess.PIPE,
               stdout=subprocess.PIPE, stderr=subprocess.PIPE,close_fds=True)
                            for _ in range(2)]

如果这导致您想要重用的其他文件描述符出现问题(如果这是一个简化的示例)，那么您可以使用子进程wait()/communicate()它们的创建顺序相反，它似乎有效.

If that causes issues with other file descriptors you want to re-use (if this was a simplified example), it turns out that you can wait()/communicate() with the subprocesses in the reverse order they were created, and it seems to work.

即，而不是:

for p in processes:
    print p.communicate("hello world\ngoodbye world\n")

使用:

while processes:
    print processes.pop().communicate("hello world\ngoodbye world\n")

(或者，我想，只需在现有循环之前执行 'processes.reverse()'.)

(Or, I guess, just do 'processes.reverse()' before your existing loop.)

这篇关于为什么与多个 Popen 子进程一起使用时会发生通信死锁?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！