Python Popen - 等待vs通信与CalledProcessError [英] Python Popen - wait vs communicate vs CalledProcessError
问题描述
继续从我以前的问题我看到，为了得到一个进程的错误代码，我通过Popen在python I中产生必须调用wait（）或communication（）（可以用于访问Popen stdout和stderr属性）：
app7z ='/path/to/7z.exe'
command = [app7z，'a'，dstFile.temp，-y，-r，os.path.join（src.Dir， *'）]
process = Popen（command，stdout = PIPE，startupinfo = startupinfo）
out = process.stdout
regCompressMatch = re.compile（'Compressing\s +（。+ '）.match
regErrMatch = re.compile（'Error：（。*）'）。match
errorLine = []
for line in out：
if len（errorLine ）或regErrMatch（行）：
errorLine.append（行）
如果regCompressMatch（行）：
＃更新进度条
result = process.wait（）＃HERE
如果结果：＃在希望7z返回0正确执行
dstFile.temp.remove（）
引发StateError（_（％s：压缩失败：\\\
％s）％（dstFile.s，
\\\
.join（errorLine）））
但是文档警告 wait（）可能会死锁（当stdout = PIPE，这是这种情况），而 communication（）可能会溢出。所以：
- 这里使用的是什么？请注意，我使用输出
-
我应该如何使用通信？会是：
process = Popen（command，stdout = PIPE，startupinfo = startupinfo）
out = process。 communication（）[0]
＃与以前一样...
result = process.returncode
如果结果：＃...
不知道阻塞和内存错误
- 任何更好/更多的pythonic处理问题的方式？我不认为
subprocess.CalledProcessError或子流程.check_call / check_output适用于我的案例 - 还是这样？
免责声明：我没有写代码，我是现在的维护者，所以问题3。
相关：
如果这样做有区别，我在windows上 - python 2.7.8
应该有一个 - 最好只有一个 - 显而易见的方法
- 关于死锁：使用
stdout = PIPE和wait（）一起 iff 。.communicate（）为您查阅和调用wait（） - 关于内存：如果输出可以是无限制的，那么你不应该使用在内存中累积所有输出的
.communicate（）。 >
这里使用的是什么？
要启动子进程，请逐行读取其输出并等待它退出：
$ p $ e
process = Popen（command，stdout = PIPE，bufsize = 1）
with process.stdout：
for row in iter（process.stdout.readline，b）：
句柄（行）
returncode =进程.wait（）
由于有限的OS管道缓冲区，此代码不会死锁。此外，代码支持无限输出的命令（如果单独的行适合内存）。
iter（）用于在子进程的stdout缓冲区刷新之后立即读取一行，以解决办法 Python中的预读错误2 。如果在等待缓冲区填写的情况下不需要读取行，或者子进程，则可以在process.stdout 中使用一个简单的结束请参阅 Python：从subprocess.communicate（）中读取流输入。
如果您在所有情况下都知道命令输出可以适合内存，那么您可以一次获取输出：
all_output = check_output（command）
/ pre>
如果命令返回非零退出状态，它将引发 CalledProcessError 。在内部， check_output（）使用 Popen（）和 .communicate（）
应该有一个 - 最好只有一个 - 显而易见的方式来实现
subprocess.Popen（）是在许多情况下起作用的主要API。有便利的功能/方法，例如 Popen.communicate（）， check_output（），对于常见的用例，check_call（）
有多种方法，功能有多种不同的用例。
Continuing from my previous question I see that to get the error code of a process I spawned via Popen in python I have to call either wait() or communicate() (which can be used to access the Popen stdout and stderr attributes):
app7z = '/path/to/7z.exe'
command = [app7z, 'a', dstFile.temp, "-y", "-r", os.path.join(src.Dir, '*')]
process = Popen(command, stdout=PIPE, startupinfo=startupinfo)
out = process.stdout
regCompressMatch = re.compile('Compressing\s+(.+)').match
regErrMatch = re.compile('Error: (.*)').match
errorLine = []
for line in out:
if len(errorLine) or regErrMatch(line):
errorLine.append(line)
if regCompressMatch(line):
# update a progress bar
result = process.wait() # HERE
if result: # in the hopes that 7z returns 0 for correct execution
dstFile.temp.remove()
raise StateError(_("%s: Compression failed:\n%s") % (dstFile.s,
"\n".join(errorLine)))
However the docs warn that wait() may deadlock (when stdout=PIPE, which is the case here) while communicate() might overflow. So:
- what is the proper thing to use here ? Note that I do use the output
how exactly should I use communicate ? Would it be:
process = Popen(command, stdout=PIPE, startupinfo=startupinfo)
out = process.communicate()[0]
# same as before...
result = process.returncode
if result: # ...
not sure about blocking and the memory errors
- Any better/more pythonic way of handling the problem ? I do not think that
subprocess.CalledProcessError or the subprocess.check_call/check_output apply in my case - or do they ?
DISCLAIMER: I did not write the code, I am the current maintainer, hence question 3.
Related:
- Python popen command. Wait until the command is finished
- Check a command's return code when subprocess raises a CalledProcessError exception
- wait process until all subprocess finish?
I am on windows if this makes a difference - python 2.7.8
There should be one-- and preferably only one --obvious way to do it
解决方案
- about the deadlock: It is safe to use
stdout=PIPE and wait() together iff you read from the pipe. .communicate() does the reading and calls wait() for you
- about the memory: if the output can be unlimited then you should not use
.communicate() that accumulates all output in memory.
what is the proper thing to use here ?
To start subprocess, read its output line by line and to wait for it to exit:
#!/usr/bin/env python
from subprocess import Popen, PIPE
process = Popen(command, stdout=PIPE, bufsize=1)
with process.stdout:
for line in iter(process.stdout.readline, b''):
handle(line)
returncode = process.wait()
This code does not deadlock due to a finite OS pipe buffer. Also, the code supports commands with unlimited output (if an individual line fits in memory).
iter() is used to read a line as soon as the subprocess' stdout buffer is flushed, to workaround the read-ahead bug in Python 2. You could use a simple for line in process.stdout if you don't need to read lines as soon as they are written without waiting for the buffer to fill or the child process to end. See Python: read streaming input from subprocess.communicate().
If you know that the command output can fit in memory in all cases then you could get the output all at once:
#!/usr/bin/env python
from subprocess import check_output
all_output = check_output(command)
It raises CalledProcessError if the command returns with a non-zero exit status. Internally, check_output() uses Popen() and .communicate()
There should be one-- and preferably only one --obvious way to do it
subprocess.Popen() is the main API that works in many many cases. There are convenience functions/methods such as Popen.communicate(), check_output(), check_call() for common use-cases.
There are multiple methods, functions because there are multiple different use-cases.
这篇关于Python Popen - 等待vs通信与CalledProcessError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
