Tornado [Errno 24] 打开的文件太多 [英] Tornado [Errno 24] Too many open files
问题描述
我们在 RedHat 操作系统上运行 Tornado 3.0 服务并收到以下错误:
We are running a Tornado 3.0 service on a RedHat OS and getting the following error:
[E 140102 17:07:37 ioloop:660] Exception in I/O handler for fd 11
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 653, in start
self._handlers[fd](fd, events)
File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 241, in wrapped
callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tornado/netutil.py", line 136, in accept_handler
connection, address = sock.accept()
File "/usr/lib/python2.7/socket.py", line 202, in accept
error: [Errno 24] Too many open files
但我们无法弄清楚这意味着什么.
But we couldn't figure out what that means.
我们的 Tornado 代码如下:
Our Tornado code is as follows:
import sys
from tornado.ioloop import IOLoop
from tornado.options import parse_command_line, define, options
from tornado.httpserver import HTTPServer
from tornado.netutil import bind_sockets
import tornado
sys.path.append("..")
from tornado.web import RequestHandler, Application
from shared.bootstrap import *
from time import time
from clients import ClientFactory
from shared.configuration import Config
from shared.logger import Logger
from algorithms.neighborhood.application import NeighborhoodApplication
import traceback
define('port', default=8000, help="Run on the given port", type=int)
define('debug', default=True, help="Run application in debug mode", type=bool)
class WService(RequestHandler):
_clients = {}
def prepare(self):
self._start_time = time()
RequestHandler.prepare(self)
def get(self, algorithm = None):
self.add_header('Content-type', 'application/json')
response = {'skus' : []}
algorithm = 'neighborhood' if not algorithm else algorithm
try:
if not algorithm in self._clients:
self._clients[algorithm] = ClientFactory.get_instance(algorithm)
arguments = self.get_arguments_by_client(self._clients[algorithm].get_expected_arguments())
response['skus'] = app.get_manager().make_recommendations(arguments)
self.write(response)
except Exception as err:
self.write(response)
error("Erro: " + str(err))
def get_arguments_by_client(self, expected_arguments):
arguments = {}
for key in expected_arguments:
arguments[key] = self.get_argument(key, expected_arguments[key])
return arguments
def on_connection_close(self):
self.finish({'skus':[]})
RequestHandler.on_connection_close(self)
def on_finish(self):
response_time = 1000.0 *(time() - self._start_time)
log("%d %s %.2fms" % (self.get_status(), self._request_summary(), response_time))
RequestHandler.on_finish(self)
def handling_exception(signal, frame):
error('IOLoop blocked for %s seconds in\n%s\n\n' % ( io_loop._blocking_signal_threshold, ''.join(traceback.format_stack(frame)[-3:])))
if __name__ == "__main__":
configuration = Config()
Logger.configure(configuration.get_configs('logger'))
app = NeighborhoodApplication({
'application': configuration.get_configs('neighborhood'),
'couchbase': configuration.get_configs('couchbase'),
'stock': configuration.get_configs('stock')
})
app.run()
log("Neighborhood Matrices successfully created...")
log("Initiating Tornado Service...")
parse_command_line()
application = Application([
(r'/(favicon.ico)', tornado.web.StaticFileHandler, {"path": "./images/"}),
(r"/(.*)", WService)
], **{'debug':options.debug, 'x-headers' : True})
sockets = bind_sockets(options.port, backlog=1024)
server = HTTPServer(application)
server.add_sockets(sockets)
io_loop = IOLoop.instance()
io_loop.set_blocking_signal_threshold(.05, handling_exception)
io_loop.start()
这是一个非常基本的脚本，基本上它获取 URL，在 make_recommendation 函数中处理它并返回响应.
It's a very basic script, basically it gets the URL, process it in the make_recommendation function and sends back the response.
我们尝试通过 io_loop.set_blocking_signal_threshold 函数将龙卷风超时设置为 50 毫秒，因为有时处理 URL 可能需要这么长时间.
We've tried to set a tornado timeout of 50 ms through the io_loop.set_blocking_signal_threshold function as sometimes the processing of the URL might take this long.
系统每分钟收到大约 8000 个请求，并且它运行了大约 30 分钟，但之后它开始抛出文件过多错误"并崩溃了.一般来说，请求需要大约 20 毫秒来处理，但是当错误开始发生时，消耗的时间突然增加到秒.
The system receives around 8000 requests per minute and it worked fine for about 30 minutes, but after that it started throwing the "too many files error" and broke down. On general the requests were taking about 20 ms to get processed but when the error started happening the time consumed increased to seconds, all of a sudden.
我们尝试查看端口 8000 有多少个连接，并且它有几个打开的连接都处于ESTABLISHED"状态.
We tried to see how many connections the port 8000 had and it had several open connections all with the "ESTABLISHED" status.
我们的 Tornado 脚本有问题吗?我们认为我们的超时功能工作不正常，但就我们迄今为止的研究而言，一切似乎都没有问题.
Is there something wrong in our Tornado script? We believe our timeout function is not working properly, but for what we've researched so far everything seems to be ok.
如果您需要更多信息，请告诉我.
If you need more info please let me know.
提前致谢，
推荐答案
许多 Linux 发行版对每个进程的打开文件数设置了非常低的限制(例如 250).您可以使用ulimit -n"来查看系统上的当前值(确保在您的 Tornado 服务器运行的相同环境中发出此命令).要提高限制，您可以使用 ulimit 命令或修改 /etc/security/limits.conf(尝试将其设置为 50000).
Many linux distributions ship with very low limits (e.g. 250) for the number of open files per process. You can use "ulimit -n" to see the current value on your system (be sure to issue this command in the same environment that your tornado server runs as). To raise the limit you can use the ulimit command or modify /etc/security/limits.conf (try setting it to 50000).
Tornado 的 HTTP 服务器不会(从 3.2 版开始)关闭 Web 浏览器打开的连接，因此空闲连接可能会随着时间的推移而累积.这就是为什么建议在 Tornado 服务器前使用 nginx 或 haproxy 之类的代理的原因之一；这些服务器针对此问题和其他潜在的 DoS 问题进行了更严格的处理.
Tornado's HTTP server does not (as of version 3.2) close connections that a web browser has left open, so idle connections may accumulate over time. This is one reason why it is recommended to use a proxy like nginx or haproxy in front of a Tornado server; these servers are more hardened against this and other potential DoS issues.
这篇关于Tornado [Errno 24] 打开的文件太多的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
