带有Python子进程的IPC [英] IPC with a Python subprocess

问题描述

我正在尝试在Python中进行一些简单的IPC，如下所示:一个Python进程使用subprocess启动另一个.子进程将一些数据发送到管道中，而父进程则将其接收.

I'm trying to do some simple IPC in Python as follows: One Python process launches another with subprocess. The child process sends some data into a pipe and the parent process receives it.

这是我当前的实现方式:

Here's my current implementation:

# parent.py
import pickle
import os
import subprocess
import sys
read_fd, write_fd = os.pipe()
if hasattr(os, 'set_inheritable'):
    os.set_inheritable(write_fd, True)
child = subprocess.Popen((sys.executable, 'child.py', str(write_fd)), close_fds=False)
try:
    with os.fdopen(read_fd, 'rb') as reader:
        data = pickle.load(reader)
finally:
    child.wait()
assert data == 'This is the data.'

# child.py
import pickle
import os
import sys
with os.fdopen(int(sys.argv[1]), 'wb') as writer:
    pickle.dump('This is the data.', writer)

在Unix上这可以按预期工作，但是如果我在Windows上运行此代码，则会收到以下错误，此后程序将挂起直到被中断:

On Unix this works as expected, but if I run this code on Windows, I get the following error, after which the program hangs until interrupted:

Traceback (most recent call last):
  File "child.py", line 4, in <module>
    with os.fdopen(int(sys.argv[1]), 'wb') as writer:
  File "C:\Python34\lib\os.py", line 978, in fdopen
    return io.open(fd, *args, **kwargs)
OSError: [Errno 9] Bad file descriptor

我怀疑问题在于子进程没有继承write_fd文件描述符.我该如何解决?

I suspect the problem is that the child process isn't inheriting the write_fd file descriptor. How can I fix this?

代码必须与Python 2.7、3.2和所有后续版本兼容.这意味着该解决方案不能依赖于

The code needs to be compatible with Python 2.7, 3.2, and all subsequent versions. This means that the solution can't depend on either the presence or the absence of the changes to file descriptor inheritance specified in PEP 446. As implied above, it also needs to run on both Unix and Windows.

(回答两个明显的问题:我不使用multiprocessing的原因是，在我的现实生活中非简化的代码中，这两个Python程序是Django项目的一部分，具有不同的设置模块.这表示它们不能共享任何全局状态.此外，子进程的标准流正在用于其他目的，因此无法使用.)

(To answer a couple of obvious questions: The reason I'm not using multiprocessing is because, in my real-life non-simplified code, the two Python programs are part of Django projects with different settings modules. This means they can't share any global state. Also, the child process's standard streams are being used for other purposes and are not available for this.)

更新:设置close_fds参数后，该代码现在可以在Unix上的所有Python版本中使用.但是，它在Windows上仍然失败.

UPDATE: After setting the close_fds parameter, the code now works in all versions of Python on Unix. However, it still fails on Windows.

推荐答案

subprocess.PIPE已在所有平台上实现.您为什么不只使用它呢?

subprocess.PIPE is implemented for all platforms. Why don't you just use this?

如果要手动创建和使用os.pipe()，则需要注意Windows不支持fork()的事实.而是使用CreateProcess()，默认情况下，它不使子级继承打开的文件.但是有一种方法:每个文件描述符都可以显式地继承.这需要调用Win API.我已经在gipc中实现了此功能，请参见_pre/post_createprocess_windows()方法这里.

If you want to manually create and use an os.pipe(), you need to take care of the fact that Windows does not support fork(). It rather uses CreateProcess() which by default not make the child inherit open files. But there is a way: each single file descriptor can be made explicitly inheritable. This requires calling Win API. I have implemented this in gipc, see the _pre/post_createprocess_windows() methods here.

这篇关于带有Python子进程的IPC的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！