货叉:关闭所有打开的插座 [英] fork: close all open sockets
问题描述
我正在使用multiprocessing.Pool.map
,它分叉当前进程.
I am using multiprocessing.Pool.map
, which forks the current process.
我的理解是,默认情况下,分叉时会从主进程中复制所有文件描述符(包括套接字).主进程本身是一个Web服务器(使用 cherrypy ),因此这会对开放端口等造成严重破坏.分叉的进程实际上只在服务器所包含的一个库中做一些 CPU繁重的数字正在使用-与网络/插座部分无关.
My understanding is that by default, all file descriptors including sockets are copied from the master process when forking. The master process itself is a web server (using cherrypy), so this wreaks havoc with open ports etc. The forked processes are really only doing some CPU-heavy numerical stuff inside one of the libraries that the server is using -- nothing to do with the web/socket part.
是否有一种简单的方法来自动关闭新进程中的所有套接字?还是另一种避免派生CherryPy服务器问题的方法?
Is there an easy way to automatically close all sockets in the new processes? Or another way to avoid issues with forking a CherryPy server?
使用CherryPy 3.2.2,Python 2.7;必须在Linux和OS X上运行.
Using CherryPy 3.2.2, Python 2.7; must work on Linux and OS X.
推荐答案
POSIX不包括列出或关闭一系列文件描述符的明智方法.
POSIX does not include a sensible way to list or close a range of file descriptors.
因此,我们必须遍历整个范围(例如3到1023),一次关闭一个文件描述符.
So we have to loop over the full range (like from 3 to 1023), closing the file descriptors one at a time.
或者,如果我们具有/proc文件系统,则可以读取/proc/self/fd中打开的文件描述符的列表,然后仅关闭它们.这比关闭所有可能的文件描述符更快.
Or, if we have a /proc file system, we can read the list of open file descriptors in /proc/self/fd and close just those. This can be quicker than closing all possible file descriptors.
import os
def close_files(fd_min=3, fd_max=-1):
if os.path.exists('/proc/self/fd'):
close_files_with_procfs(fd_min, fd_max)
else:
close_files_exhaustively(fd_min, fd_max)
def close_files_exhaustively(fd_min=3, fd_max=-1):
import resource
fd_top = resource.getrlimit(resource.RLIMIT_NOFILE)[1] - 1
if fd_max == -1 or fd_max > fd_top:
fd_max = fd_top
for fd in range(fd_min, fd_max+1):
try:
os.close(fd)
except OSError:
pass
def close_files_with_procfs(fd_min=3, fd_max=-1):
for nm in os.listdir("/proc/self/fd"):
if nm.startswith('.'):
continue
fd = int(nm)
if fd >= fd_min and (fd_max == -1 or fd < fd_max):
try:
os.close(fd)
except OSError:
pass
def timereps(reps, func):
from time import time
start = time()
for i in range(0, reps):
func()
end = time()
return (end - start) / reps
print "close_files: %f" % timereps(100, lambda: close_files())
print "close_files_exhaustively: %f" % timereps(100, lambda: close_files_exhaustively())
print "close_files_with_procfs: %f" % timereps(1000, lambda: close_files_with_procfs())
在我的系统上:
$ python ./close_fds.py
close_files: 0.000094
close_files_exhaustively: 0.010151
close_files_with_procfs: 0.000039
这篇关于货叉:关闭所有打开的插座的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!