Python asyncio 子进程连续写入标准输入并读取标准输出/标准错误 [英] Python asyncio subprocess write stdin and read stdout/stderr continuously

问题描述

我目前正在使用 python3 asyncio 中的子进程执行任务.我的代码只是写入标准输入并同时读取标准输出/标准错误:

I'm currently on a task with subprocess in python3 asyncio. My code is simply write to stdin and read stdout/stderr simultaneously:

import asyncio


async def read_stdout(stdout):
    print('read_stdout')
    while True:
        buf = await stdout.read(10)
        if not buf:
            break

        print(f'stdout: { buf }')


async def read_stderr(stderr):
    print('read_stderr')
    while True:
        buf = await stderr.read()
        if not buf:
            break

        print(f'stderr: { buf }')


async def write_stdin(stdin):
    print('write_stdin')
    for i in range(100):
        buf = f'line: { i }\n'.encode()
        print(f'stdin: { buf }')

        stdin.write(buf)
        await stdin.drain()
        await asyncio.sleep(0.5)


async def run():
    proc = await asyncio.create_subprocess_exec(
        '/usr/bin/tee',
        stdin=asyncio.subprocess.PIPE,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE)

    await asyncio.gather(
        read_stderr(proc.stderr),
        read_stdout(proc.stdout),
        write_stdin(proc.stdin))


asyncio.run(run())

效果很好，但我在 Python3 文档页面:

Warning 使用 communicate() 方法而不是 process.stdin.write(), await process.stdout.read() 或 await process.stderr.read.这避免了由于流暂停读取或写入并阻塞子进程而导致的死锁.

Warning Use the communicate() method rather than process.stdin.write(), await process.stdout.read() or await process.stderr.read. This avoids deadlocks due to streams pausing reading or writing and blocking the child process.

这是否意味着上面的代码在某些情况下会陷入死锁?如果是这样，如何在python3 asyncio中连续编写stdin和读取stdout/stderr而不会死锁?

Does that mean the above code will fall into deadlock in some scenarios? If so how to write stdin and read stdout/stderr continuously in python3 asyncio without deadlock?

非常感谢.

推荐答案

警告是从 regular 子进程模块，并对试图实现看起来完全正确的简单通信的幼稚代码发出警告，例如:

The warning was carried over from the regular subprocess module, and warns against naive code that tries to implement simple communication that appears perfectly correct, such as:

# write the request to the subprocess
await proc.stdin.write(request)
# read the response
response = await proc.stdout.readline()

如果子进程在读取整个请求之前开始写入响应，这可能会导致死锁.如果响应足够大，子进程将阻塞，等待父进程读取其中的一部分并在管道缓冲区中腾出空间.但是，父级不能这样做，因为它仍在写入响应并在开始读取之前等待写入完成.因此，子进程等待父进程读取(部分)响应，父进程等待子进程完成接受请求.由于双方都在等待对方的当前操作完成，所以这是一个死锁.

This can cause a deadlock if the subprocess starts writing the response before it has read the whole request. If the response is large enough, the subprocess will block, waiting for the parent to read some of it and make room in the pipe buffer. However, the parent cannot do so because it is still writing the response and waiting for the write to complete before starting reading. So, the child waits for the parent to read (some of) its response, and the parent waits for the child to finish accepting the request. As both are waiting for the other's current operation to complete, it's a deadlock.

您的代码没有这个问题，因为您的读取和写入是并行执行的.由于读者从不等待作者，反之亦然，因此没有机会出现(那种)死锁.如果你看看 communicate 是如何已实现，您会发现，除非进行一些调试日志记录，否则它的工作方式与您的代码非常相似.

Your code doesn't have that issue simply because your reads and writes are executed in parallel. Since the reader never waits for the writer and vice versa, there is no opportunity for (that kind of) deadlock. If you take a look at how communicate is implemented, you will find that, barring some debug logging, it works pretty much like your code.

这篇关于Python asyncio 子进程连续写入标准输入并读取标准输出/标准错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！