使用Popen.wait()时重现死锁 [英] Reproducing deadlock while using Popen.wait()

问题描述

从文档中使用 Popen.wait()可以:

在使用stdout = PIPE和/或stderr = PIPE并使用子级时出现死锁进程生成足够的输出到管道，从而阻止等待让OS管道缓冲区接受更多数据.使用communication()避免那个.

deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

在交流中

读取的数据缓存在内存中，因此如果数据量大或无限制

The data read is buffered in memory, so do not use this method if the data size is large or unlimited

如何重现这种有问题的行为，并使用 Popen.communicate()修复了吗?

How to reproduce such a problematic behavior and see that using Popen.communicate() fixes it?

死锁意味着持有资源的进程之间会发生一些循环等待，并永远被阻塞.这里的循环依赖是什么?等待子进程终止的Python进程是一个等待.还有什么在以下情况下，谁在等待什么?

Deadlock means some circular wait between processes holding resources happens and is stuck forever. What is the circular dependency here? Python process which waits for the child process to terminate is one wait. What's the other? Who waits for what in the below scenario?

它阻止等待OS管道缓冲区接受更多数据

it blocks waiting for the OS pipe buffer to accept more data

推荐答案

简单.

创建一个输出大量文本的进程，并且不要读取输出:

Create a process which outputs a lot of text and don't read the output:

p = subprocess.Popen(["ls","-R"],stdout=subprocess.PIPE)
p.wait()

一段时间后，标准输出管道已满，并且过程被阻塞.

after a while the standard output pipe is full and process is blocked.

这是一个僵局，因为子进程在消耗完之前无法再将输出写入输出(即:永不)，而python进程等待子进程完成.

为避免死锁，可以使用读取行循环:

To avoid the deadlock, you could use a read line loop:

p = subprocess.Popen(["ls","-R"],stdout=subprocess.PIPE)
for line in p.stdout:
    # do something with the line
p.wait()

communication 不仅可以解决此问题，而且还可以解决更为棘手的情况，在这种情况下，输出和错误流都将重定向到单独的流(在这种情况下，上面的幼稚循环仍可能死锁).

communicate also fixes that but also fixes the much trickier case where both output and error streams are redirected to separate streams (in that case, the naive loop above could still deadlock).

假设您有一个编译过程

p = subprocess.Popen(["gcc","-c"]+mega_list_of_files,stdout=subprocess.PIPE,stderr=subprocess.PIPE)

现在，您想从此输出中获取输出，因此您可以这样做:

Now you want to get the output from this one, so you do:

output = p.stdout.read()

不幸的是，弹出了很多错误，在读取输出流时阻止了错误流:再次死锁.

unfortunately, a lot of errors pop up instead, blocking the error stream while you're reading the output stream: deadlock again.

尝试改为读取错误流，可能会发生完全相反的事情:大量的stdout输出会阻塞您的进程.

Try to read error stream instead, and the exact opposite could occur: lots of stdout output blocking your process.

communication 使用多线程能够处理输出&错误流同时将它们分开，没有阻塞的风险.唯一需要注意的是，您无法实时控制行输出/打印程序输出的实时输出:

communicate uses multithreading to be able to process output & error streams at the same time and keep them separated, without risk of blocking. Only caveat is that you cannot control the process output line by line / print program output in real time:

p = subprocess.Popen(["gcc","-c"]+mega_list_of_files,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
output,error = p.communicate()
return_code = p.wait()

这篇关于使用Popen.wait()时重现死锁的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！