在Flask中初始化全局数据连接的并发安全方法 [英] Concurrency-safe way to initialize global data connections in Flask
问题描述
Global variables are not thread-safe because a Flask application may be spawned in multiple processes, not just multiple threads.
但是，我需要打开与每个工作人员将使用的服务的连接，例如PubSub客户端或Cloud Storage客户端.似乎这些仍需要是全局的，以便应用程序中的任何功能都可以访问它们.为了延迟初始化它们，我检查变量是否为None，并且这需要是线程安全的.建议打开每个请求将使用的连接的方法是什么?我应该使用线程锁进行同步吗?
However, I need to open connections to services that each worker will use, such as a PubSub client or a Cloud Storage client. It seems like these still need to be global so that any function in the application can access them. To lazily initialize them, I check if the variable is None, and this needs to be thread-safe. What is the recommended approach for opening connections that each request will use? Should I use a thread lock to synchronize?
推荐答案
The question you linked is talking about data, not connections. Having multiple workers mutating global data is not good because you can't reason about where those workers are in a web application to keep them in sync.
该问题的解决方案是使用必须以某种方式连接的外部数据源，例如数据库.但是，您认为建立一个全局连接的想法并不安全，因为多个工作线程会同时与之交互，并且彼此混乱，或者一次等待一个来获取资源.解决此问题的最简单方法是在需要时在每个视图中建立连接.
The solution to that question is to use an external data source, like a database, which must be connected to somehow. Your idea to have one global connection is not safe though, since multiple worker threads would interact with it concurrently and either mess with each other's state or wait one at a time to acquire the resource. The simplest way to handle this is to establish a connection in each view when you need it.
此示例显示了如何为每个请求建立唯一的连接，而没有全局连接，如何在为请求建立连接后重新使用该连接. g对象看起来像一个全局对象，但在后台被实现为线程局部的，因此每个工作人员都可以获取自己的g实例，并且仅在一个请求期间将连接存储在该实例上.
This example shows how to have a unique connection per request, without globals, reusing the connection once it's established for the request. The g object, while it looks like a global, is implemented as a thread-local behind the scenes, so each worker gets it's own g instance and connection stored on it during one request only.
from flask import g
def get_conn():
"""Use this function to establish or get the already established
connection during a request. The connection is closed at the end
of the request. This avoids having a global connection by storing
the connection on the g object per request.
"""
if "conn" not in g:
g.conn = make_connection(...)
return g.conn
@app.teardown_request
def close_conn(e):
"""Automatically close the connection after the request if
it was opened.
"""
conn = g.pop("conn", None)
if conn is not None:
conn.close()
@app.route("/get_data")
def get_data():
# If something else has already used get_conn during the
# request, this will return the same connection. Anything
# that uses it after this will also use the same connection.
conn = get_conn()
data = conn.query(...)
return jsonify(data)
您最终可能会发现，一旦您有成千上万的并发请求，则每个请求的建立成本都太高了.一种解决方案是构建连接池以全局存储连接列表，并使用线程安全的方式在需要时获取和替换列表中的连接. SQLAlchemy(和Flask-SQLAlchemy)使用此技术.许多库已经提供了连接池的实现，因此请使用它们或将其用作自己的参考.
You might eventually find that establishing a new connection each request is too expensive once you have many thousands of concurrent requests. One solution is to build a connection pool to store a list of connections globally, with a thread-safe way to acquire and replace a connection in the list as needed. SQLAlchemy (and Flask-SQLAlchemy) uses this technique. Many libraries already provide connection pool implementations, so either use them or use them as a reference for your own.
这篇关于在Flask中初始化全局数据连接的并发安全方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
