python 2.7 Popen:"close_fds"有什么作用? [英] python 2.7 Popen: what does `close_fds` do?

问题描述

我有一个使用Python(2.7)的Web服务器，该服务器使用 Popen 将某些工作委托给子进程:

I have a web server in Python (2.7) that uses Popen to delegate some work to a child process:

url_arg = "http://localhost/index.html?someparam=somevalue"
call = ('phantomjs', 'some/phantom/script.js', url_arg)

imageB64data = tempfile.TemporaryFile()
errordata = tempfile.TemporaryFile()

p = Popen(call, stdout=imageB64data, stderr=errordata, stdin=PIPE)
p.communicate(input="")

我看到间歇性问题，其中发生了一定数量的这些 Popen (大约64个)，该进程用尽了文件描述符，无法运行-完全无响应，并且所有如果线程试图打开任何文件或套接字，它们似乎会永远阻塞.

I am seeing intermittent issues where after some number of these Popens have occurred (roughly 64), the process runs out of file descriptors and is unable to function -- it becomes completely unresponsive and all threads seem to block forever if they attempt to open any files or sockets.

(可能相关: phantomjs 子进程将URL调用加载回产生它的服务器中.)

(Possibly relevant: the phantomjs child process loads a URL calls back into the server that spawned it.)

基于此Python错误报告，我相信我需要设置 close_fds = True在服务器进程中所有 Popen 调用上，以减轻文件描述符的泄漏.但是，我不熟悉围绕 exec 进行子处理的机制以及文件描述符的继承，因此许多 Popen 文档以及前面提到的错误报告中的注释对我来说都不是很清楚.

Based on this Python bug report, I believe I need to set close_fds=True on all Popen calls from inside my server process in order to mitigate the leaking of file descriptors. However, I am unfamiliar with the machinery around exec-ing subprocesses and inheritance of file descriptors so much of the Popen documentation and the notes in the aforementioned bug report are unclear to me.

听起来，在执行子流程之前，它实际上会关闭我的流程中的所有打开的文件描述符(包括活动的请求套接字，日志文件句柄等).听起来这绝对比泄漏套接字更好，但仍然会导致错误.

It sounds like it would actually close all open file descriptors (which includes active request sockets, log file handles, etc.) in my process before executing the subprocess. This sounds like it would be strictly better than leaking the sockets, but would still result in errors.

但是，在实践中，当我在Web请求期间使用 close_fds = True 时，它似乎可以正常工作，到目前为止，我还无法构造出它实际上关闭任何其他请求套接字的方案，数据库请求等.

However, in practice, when I use close_fds=True during a web request, it seems to work fine and thus far I have been unable to construct a scenario where it actually closes any other request sockets, database requests, etc.

文档状态:

如果close_fds为true，则在执行子进程之前，将关闭除0、1和2以外的所有文件描述符.

If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is executed.

所以我的问题是:在多线程Python Web服务器中将 close_fds = True 传递给 Popen 是安全"还是正确"?或者如果其他请求同时执行文件/套接字IO，我应该期望这样做会有副作用吗?

So my question is: is it "safe" and "correct" to pass close_fds=True to Popen in a multithreaded Python web server? Or should I expect this to have side effects if other requests are doing file/socket IO at the same time?

推荐答案

我使用Python 3.2/3.3的 subprocess 的 subprocess32 反向端口尝试了以下测试:/p>

I tried the following test with the subprocess32 backport of Python 3.2/3.3's subprocess:

import tempfile
import subprocess32 as subprocess

fp = open('test.txt', 'w')

fp.write("some stuff")

echoed = tempfile.TemporaryFile()
p = subprocess.Popen(("echo", "this", "stuff"), stdout=echoed, close_fds=True)
p.wait()
echoed.seek(0)

fp.write("whatevs")
fp.write(echoed.read())
fp.close()

我在 test.txt 中得到了一些东西的预期结果.

所以看来 close_fds 中 close 的含义 not 并不意味着在父进程中打开文件(套接字等)执行子进程后将无法使用.

So it appears that the meaning of close in close_fds does not mean that open files (sockets, etc.) in the parent process will be unusable after executing a child process.

同样值得注意的是: subprocess32 在POSIX系统AFAICT上默认为 close_fds = True .在我看来，这并不像听起来那么危险.

Also worth noting: subprocess32 defaults close_fds=True on POSIX systems, AFAICT. This implies to me that it is not as dangerous as it sounds.

这篇关于python 2.7 Popen:"close_fds"有什么作用?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！