如何使用pymongo在lambda函数中实现Mongodb连接池 [英] How to implement Mongodb connection pooling in lambda function using pymongo
问题描述
我正在使用aws lambda函数，python和monogdb地图集.我已经执行了以下代码.
I am using aws lambda function, python and monogdb atlas. I have executed the below code.
client = MongoClient('mongodb+srv://app_user:123456@accesdev-dxjpa.mongodb.net/test')
db = client.test
def lambda_handler(event, context):
print("Received event: " + json.dumps(event, indent=1))
user_profile = db.user_profile
Email = event['email']
res = Email.find('@')
if res == -1:
disuname = list(user_profile.find({"username" : Email},{"name": 1, "photo": 1, "bio": 1}))
uid = json.dumps(disuname, default=json_util.default)
return json.loads(uid)
else:
disuname = list(user_profile.find({"email": Email},{"name": 1, "photo": 1, "bio": 1}))
uid = json.dumps(disuname, default=json_util.default)
return json.loads(uid)
在上面的代码执行中，mongodb连接大小将增加，并且将达到最大大小.我听到了mongodb连接池的概念，但是我不明白如何在lambda函数中的pymongo中实现它.您能帮我解决问题吗?
In this above code execute, mongodb connections size will increase and it will reach maximum size. I heard the concept of mongodb connection pooling but i didn't understand how to implement it in pymongo in lambda function. Can you please help me the solution.
推荐答案
以下几行来自以下给出的链接，这应该是一个很好的解释，或者是一个更好的起点-请仔细阅读:
每个MongoClient实例在MongoDB拓扑中每个服务器都有一个内置的连接池.这些池按需打开套接字以支持您的多线程应用程序所需的并发MongoDB操作数.套接字没有线程关联性.
Every MongoClient instance has a built-in connection pool per server in your MongoDB topology. These pools open sockets on demand to support the number of concurrent MongoDB operations that your multi-threaded application requires. There is no thread-affinity for sockets.
每个连接池的大小上限为maxPoolSize，默认为100.如果与服务器的连接为maxPoolSize，并且所有连接都在使用中，则对该服务器的下一个请求将等待，直到其中一个连接可用为止.
The size of each connection pool is capped at maxPoolSize, which defaults to 100. If there are maxPoolSize connections to a server and all are in use, the next request to that server will wait until one of the connections becomes available.
该客户端实例在MongoDB拓扑中为每台服务器打开一个额外的套接字，以监视服务器的状态.
The client instance opens one additional socket per server in your MongoDB topology for monitoring the server’s state.
例如，连接到3节点副本集的客户端将打开3个监视套接字.它还会根据需要打开任意数量的套接字，以支持多线程应用程序在每台服务器上的并发操作，最大可达maxPoolSize. maxPoolSize为100时，如果应用程序仅使用主连接(默认设置)，则仅主连接池增长，并且总连接数最多为103.如果应用程序使用ReadPreference来查询第二个连接，则它们的连接池也将增长，连接总数可以达到303.
For example, a client connected to a 3-node replica set opens 3 monitoring sockets. It also opens as many sockets as needed to support a multi-threaded application’s concurrent operations on each server, up to maxPoolSize. With a maxPoolSize of 100, if the application only uses the primary (the default), then only the primary connection pool grows and the total connections is at most 103. If the application uses a ReadPreference to query the secondaries, their pools also grow and the total connections can reach 303.
可以使用minPoolSize设置与每个服务器的最小并发连接数，默认为0.连接池将使用此数量的套接字初始化.如果套接字由于任何网络错误而关闭，从而导致套接字总数(正在使用和空闲)下降到最小值以下，则将打开更多套接字，直到达到最小值为止.
It is possible to set the minimum number of concurrent connections to each server with minPoolSize, which defaults to 0. The connection pool will be initialized with this number of sockets. If sockets are closed due to any network errors, causing the total number of sockets (both in use and idle) to drop below the minimum, more sockets are opened until the minimum is reached.
可以使用maxIdleTime设置连接在池中保持空闲状态的最长时间(以毫秒为单位)，默认为无(无限制).
The maximum number of milliseconds that a connection can remain idle in the pool before being removed and replaced can be set with maxIdleTime, which defaults to None (no limit).
MongoClient的默认配置适用于大多数应用程序:
The default configuration for a MongoClient works for most applications:
client = MongoClient(host, port)
为每个进程创建一次此客户端，然后将其重新用于所有操作.为每个请求创建一个新的客户端是一个普遍的错误，这是非常低效的.
Create this client once for each process, and reuse it for all operations. It is a common mistake to create a new client for each request, which is very inefficient.
要在一个进程中支持大量并发MongoDB操作，请增加maxPoolSize:
To support extremely high numbers of concurrent MongoDB operations within one process, increase maxPoolSize:
client = MongoClient(host, port, maxPoolSize=200)
…或使其不受限制:
client = MongoClient(host, port, maxPoolSize=None)
一旦池达到最大大小，其他线程就必须等待套接字可用. PyMongo并不限制等待套接字可用的线程数，应用程序有责任在负载高峰期间将其线程池的大小限制为绑定队列.除非定义了waitQueueTimeoutMS，否则允许线程等待任何时间长度:
Once the pool reaches its maximum size, additional threads have to wait for sockets to become available. PyMongo does not limit the number of threads that can wait for sockets to become available and it is the application’s responsibility to limit the size of its thread pool to bound queuing during a load spike. Threads are allowed to wait for any length of time unless waitQueueTimeoutMS is defined:
client = MongoClient(host, port, waitQueueTimeoutMS=100)
等待套接字(在此示例中)超过100毫秒的线程会引发ConnectionFailure.如果限制负载高峰期间的操作持续时间比完成每个操作更重要，请使用此选项.
A thread that waits more than 100ms (in this example) for a socket raises ConnectionFailure. Use this option if it is more important to bound the duration of operations during a load spike than it is to complete every operation.
任何线程调用close()时，所有空闲套接字都将关闭，并且所有正在使用的套接字将在返回到池中时关闭.
When close() is called by any thread, all idle sockets are closed, and all sockets that are in use will be closed as they are returned to the pool.
参考: pymongo中的连接池
这篇关于如何使用pymongo在lambda函数中实现Mongodb连接池的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
