Python中主进程和子进程之间的动态通信 [英] Dynamic communication between main and subprocess in Python

问题描述

我在 Python 中工作，我想找到一个工作流，使两个进程(main-process 和 sub-process)能够相互通信.我的意思是主进程能够将一些数据发送到子进程(也许，通过写入子进程的标准输入) 以及子进程将一些数据发送回主进程的能力.这也意味着两者都可以读取发送给他们的数据(我正在考虑从 stdin 读取).

I work in Python, and I want to find a workflow for enabling two processes (main-process and sub-process) to communicate with each other. By that, I mean the ability of main-process to send some data to sub-process (perhaps, by writing to sub-process's stdin) and the ability of sub-process to send some data back to the main one. This also implies that both can read the data sent to them (I was thinking of reading from stdin).

我试图使用 subprocess 库，但它似乎旨在与旨在仅提供一次输出然后终止的进程一起工作，而我想交换数据动态并仅在收到此类命令时关闭子进程.

I was trying to use subprocess library, but it seems that it's intended to work with processes that are designed to give an output only once and then terminate, whereas I want to exchange data dynamically and shut the sub-process down only when such a command is received.

我在 StackOverflow 上阅读了许多解决与我的问题密切相关的问题的答案，但没有一个让我感到满意，因为这些答案所针对的问题在一个重要细节上与我的不同:我需要我的 主进程能够交换数据与其子进程根据需要动态多次，而不仅仅是一次，这反过来意味着子进程应该运行，直到它从主进程接收到某个命令来终止.

I've read lots of answers here on StackOverflow tackling problems closely related to mine, but none of them did I find satisfying, as the questions those answers were meant to were different from mine in one important detail: I need my main-process to be able to exchange data with its sub-process dynamically as many times as needed, not just once, which in turn implies that the sub-process should run until it receives a certain command from main-process to terminate.

我愿意使用第三方库，但如果您提出一个仅基于 Python 标准库的解决方案会更好.

I'm open to using third-party libraries, but it would be much better if you proposed a solution based solely on the Python Standard Library.

推荐答案

你想用 subprocess.PIPE 制作一个 Popen 对象，用于标准输入和输出，并使用它的文件对象来进行通信——而不是使用像 run 这样的 cantrips 之一(以及像 check_output 这样的旧的、更具体的).挑战在于避免死锁:很容易陷入这样一种情况:每个进程都在尝试写入，管道缓冲区已满(因为没有人从它们读取)，然后一切都挂了.您还必须记住在两个进程中flush，以避免请求或响应卡在 file 对象的缓冲区中.

You want to make a Popen object with subprocess.PIPE for standard input and output and use its file objects to communicate—rather than using one of the cantrips like run (and the older, more specific ones like check_output). The challenge is avoiding deadlock: it’s easy to land in a situation where each process is trying to write, the pipe buffers fill (because no one is reading from them), and everything hangs. You also have to remember to flush in both processes, to avoid having a request or response stuck in a file object’s buffer.

Popen.communicate 旨在避免这些问题，但它仅支持单个字符串(而不是正在进行的对话).传统的解决方案是select，但它也可以使用单独的线程来发送请求和读取结果.(这是尽管有 GIL 仍然使用 CPython 线程的原因之一:每个线程都可以运行而另一个被阻塞，因此很少有争用.)当然，同步是一个问题，并且您可能需要做一些工作才能使多线程客户端在外部表现得像一个简单的同步函数调用.

Popen.communicate is provided to avoid these issues, but it supports only a single string (rather than an ongoing conversation). The traditional solution is select, but it also works to use separate threads to send requests and read results. (This is one of the reasons to use CPython threads in spite of the GIL: each exists to run while the other is blocked, so there’s very little contention.) Of course, synchronization is then an issue, and you may need to do some work to make the multithreaded client act like a simple, synchronous function call on the outside.

注意两个进程都需要flush，但是如果其中一个实现这种非阻塞I/O就足够了；一个通常在启动另一个的进程中完成这项工作，因为这是已知的必要的地方(此类程序是例外).

Note that both processes need to flush, but it’s enough if either implements such non-blocking I/O; one normally does that job in the process that starts the other because that’s where it’s known to be necessary (and such programs are the exception).

这篇关于Python中主进程和子进程之间的动态通信的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！