Tornado Web 和线程 [英] Tornado Web and Threads

问题描述

我是 Tornado 和 Python 线程的新手.我想实现以下目标:我有一个 Tornado Web 服务器，它接受用户的请求.我想在本地存储一些数据并定期将其作为批量插入写入数据库.

I am new to Tornado and Python Threads. What I would like to achieve is the following: I have a Tornado web server which takes requests from users. I want to store some of the data locally and write it periodically to a data base as bulk-inserts.

import tornado.ioloop
import tornado.web
import threading

# Keep userData locally in memory
UserData = {}

def background(f):
    """
    a threading decorator
    use @background above the function you want to thread
    (run in the background)
    """
    def bg_f(*a, **kw):
        threading.Thread(target=f, args=a, kwargs=kw).start()
    return bg_f

@background
def PostRecentDataToDBThread(iter = -1):
    i = 0
    while iter == -1 or i < iter: 
        #send data to DB
        UserData = {}
        time.sleep(5*60)
        i = i + 1

class AddHandler(tornado.web.RequestHandler):
    def post(self):
        userID = self.get_argument('ui')
        Data = self.get_argument('data')

        UserData[userID] = Data 


if __name__ == "__main__":
    tornado.options.parse_command_line()

    print("start PostRecentDataToDBThread")
    ### Here we start a thread that periodically sends data to the data base.
    ### The thread is called every 5min. 
    PostRecentDataToDBThread(-1)

    print("Started tornado on port: %d" % options.port)

    application = tornado.web.Application([
        (r"/", MainHandler),
        (r"/add", AddHandler)
    ])
    application.listen(options.port)
    tornado.ioloop.IOLoop.instance().start()

这是实现我的目标的好方法吗?我想尽量减少服务器阻塞时间.还是我应该使用 gevent 或其他任何东西?我可以通过从 Tornado 和线程访问 UserData 来遇到问题吗?只要没有服务器崩溃，数据一致性在这里不是那么重要.

Is this a good way to achieve my goal? I would like to minimize the server blocking time. Or should I rather use gevent or anything else? Can I run into problems by accessing UserData both from Tornado and the thread? Data consistency is not so important here as long as there is no server crash.

推荐答案

Tornado 不适用于多线程.基于 epoll 在不同代码部分之间切换上下文.

Tornado is not intended to be used with multithreading. It's based on epoll to switch context between different parts of code.

总的来说，我会建议通过消息队列将数据发送到单独的工作进程(比如pika+RabbitMQ，它集成得很好与龙卷风).工作进程可以使用数据累积消息并将它们批量写入数据库，或者您可以使用此设置实现任何其他数据处理逻辑.

In general I would recommend sending the data to separate worker process via message queue (like pika+RabbitMQ, it integrates very well with Tornado). Worker process(es) can accumulate messages with data and write them to database in batch or you could implement any other logic of data processing with this setup.

或者，例如，您可以使用带有 brukva 的 Redis 将传入数据异步写入内存数据库，这在turn 将根据 Redis 配置异步将其转储到磁盘.

Alternatively you can use, for example, Redis with brukva to just asynchronously write incoming data to in-memory database, which in turn will asynchronously dump it to disk depending on Redis configuration.

这篇关于Tornado Web 和线程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！