龙卷风“@run_on_executor"正在阻塞 [英] Tornado '@run_on_executor' is blocking
问题描述
我想问一下 tornado.concurrent.run_on_executor(后来只是 run_on_executor)有效，因为我可能不明白如何运行同步任务来不阻塞主 IOLoop.
I would like to ask how tornado.concurrent.run_on_executor (later just run_on_executor) works, because
I probably do not understand how to run synchronous task to not block the main IOLoop.
我发现所有使用 run_on_executor 的示例都只使用 time 来阻止循环.使用 time 模块它工作正常，但是当我尝试使用 run_on_executor 进行一些时间密集计算时，任务会阻止 IOLoop.我能够看到该应用程序使用了多个线程，但它仍然处于阻塞状态.
All the examples using run_on_executor, which I found, are using just time to block the loop.
With time module it works fine, but when I try some time intesive calculations, using run_on_executor, the task blocks the IOLoop.
I am able to see that the app uses multiple threads, but it is still blocking.
我想使用 run_on_executor 来使用 bcrypt 对密码进行散列，但用此计算替换它以获得一些额外的测试时间.
I want to use run_on_executor for hashing passwords using bcrypt, but replaced it with this calculation to gain some extra time for testing.
这里我有一个小应用程序来展示我的困惑.
Here I have small app, to demonstrate my confusion.
from tornado.options import define, options
import tornado.web
import tornado.httpserver
from tornado import gen
from tornado.concurrent import run_on_executor
import tornado.httpclient
import tornado.escape
import time
import concurrent.futures
import urllib
executor = concurrent.futures.ThreadPoolExecutor(20)
define("port", default=8888, help="run on the given port", type=int)
# Should not be blocking ?
class ExpHandler(tornado.web.RequestHandler):
_thread_pool = executor
@gen.coroutine
def get(self, num):
i = int(num)
result = yield self.exp(2, i)
self.write(str(result))
self.finish()
@run_on_executor(executor="_thread_pool")
def exp(self, x, y):
result = x ** y
return(result)
class NonblockingHandler(tornado.web.RequestHandler):
@gen.coroutine
def get(self):
http_client = tornado.httpclient.AsyncHTTPClient()
try:
response = yield http_client.fetch("http://www.google.com/")
self.write(response.body)
except tornado.httpclient.HTTPError as e:
self.write(("Error: " + str(e)))
finally:
http_client.close()
self.finish()
class SleepHandler(tornado.web.RequestHandler):
_thread_pool = executor
@gen.coroutine
def get(self, sec):
sec = float(sec)
start = time.time()
res = yield self.sleep(sec)
self.write("Sleeped for {} s".format((time.time() - start)))
self.finish()
@run_on_executor(executor="_thread_pool")
def sleep(self, sec):
time.sleep(sec)
return(sec)
class Application(tornado.web.Application):
def __init__(self):
handlers = [
(r'/exp/(?P<num>[^\/]+)?', ExpHandler),
(r'/nonblocking/?', NonblockingHandler),
(r'/sleep/(?P<sec>[^\/]+)?',SleepHandler)
]
settings = dict(
debug=True,
logging="debug"
)
tornado.web.Application.__init__(self, handlers, **settings)
def main():
tornado.options.parse_command_line()
http_server = tornado.httpserver.HTTPServer(Application())
http_server.listen(options.port)
io_loop = tornado.ioloop.IOLoop.instance()
io_loop.start()
if __name__ == "__main__":
main()
如果能解释为什么在 executor 中运行的 ExpHandler 会阻塞循环，我将不胜感激.
I would be very grateful for any explanation why ExpHandler, running in executor is blocking the loop.
推荐答案
Python(至少在 CPython 实现中)具有全局解释器锁，可防止多个线程同时执行 Python 代码.特别是，在单个 Python 操作码中运行的任何东西都是不可中断的，除非它调用显式释放 GIL 的 C 函数.** 的大指数一直保持 GIL，从而阻塞所有其他 python 线程，而对 bcrypt() 的调用将释放 GIL，以便其他线程可以继续工作.
Python (at least in the CPython implementation) has a Global Interpreter Lock which prevents multiple threads from executing Python code at the same time. In particular, anything which runs in a single Python opcode is uninterruptible unless it calls a C function which explicitly releases the GIL. A large exponentation with ** holds the GIL the whole time and thus blocks all other python threads, while a call to bcrypt() will release the GIL so other threads can continue to work.
这篇关于龙卷风“@run_on_executor"正在阻塞的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
