在线程中使用Django ORM并避免“太多客户端”例外使用BoundedSemaphore [英] Using Django ORM in threads and avoiding "too many clients" exception by using BoundedSemaphore

问题描述

我在manage.py命令上工作，它创建了大约200个线程来检查远程主机。我的数据库设置允许我使用120个连接，所以我需要使用某种池。我试过使用分离的线程，像这样

 类池（线程）：
 def __init __（self） 
 Thread .__ init __（self）
 self.semaphore = threading.BoundedSemaphore（10）
 
 def give（self，trackers）：
 self.semaphore.acquire ）
 data = ...一些ORM（不懒惰，这里查询触发）... 
 self.semaphore.release（）
返回数据
  

我将这个对象的实例传递给每个检查线程，但在init之后仍然得到OperationalError：FATAL：对不起，太多客户端 - 120个线程。
我预计只有10个数据库连接将被打开，线程将等待免费的信号量插槽。我可以通过评论release（）来检查信号量，在这种情况下，只有10个线程可以工作，而其他线程将等到应用程序终止。

据我所知，每个线程都打开与数据库的新连接，即使实际调用在不同的线程内，但为什么？有没有办法在一个线程内执行所有数据库查询？

解决方案

Django的ORM管理线程局部变量中的数据库连接。所以访问ORM的每个不同线程都将创建自己的连接。你可以看到在$ code> django / db / backends / __ init __。py 的前几行。

如果你要限制数据库连接的数量，您必须限制实际访问ORM的不同线程的数量。解决方案可能是实现将ORM请求委托给专用ORM线程池的服务。要将请求及其结果从其他线程传输到其他线程，您将必须实现某种消息传递机制。由于这是一个典型的生产者/消费者问题，关于线程的Python文档应该给出一些提示如何实现这一点。

编辑：我刚刚在django连接池中搜索。有很多人抱怨Django没有提供正确的连接池。其中一些设法集成了一个单独的池包。对于PostgreSQL，我将看看pgpool中间件。

I work on manage.py command which creates about 200 threads to check remote hosts. My database setup allows me to use 120 connections, so I need to use some kind of pooling. I've tried using separated thread, like this

class Pool(Thread):
    def __init__(self):
        Thread.__init__(self)        
        self.semaphore = threading.BoundedSemaphore(10)

    def give(self, trackers):
        self.semaphore.acquire()
        data = ... some ORM (not lazy, query triggered here) ...
        self.semaphore.release()
        return data

I pass instance of this object to every check-thread but still getting "OperationalError: FATAL: sorry, too many clients already" inside Pool object after init-ing 120 threads . I've expected that only 10 database connections will be opened and threads will wait for free semaphore slot. I can check that semaphore works by commenting "release()", in that case only 10 threads will work and other will wait till app termination.

As much as I understand, every thread is opening new connection to database even if actual call is inside different thread, but why? Is there any way to perform all database queries inside only one thread?

解决方案

Django's ORM manages database connections in thread-local variables. So each different thread accessing the ORM will create its own connection. You can see that in the first few lines of django/db/backends/__init__.py.

If you want to limit the number of database connections made, you must limit the number of different threads that actually access the ORM. A solution could be to implement a service that delegates ORM requests to a pool of dedicated ORM threads. To transmit the requests and their results from and to other threads you will have to implement some sort of message passing mechanism. Since this is a typical producer/consumer problem, the Python docs about threading should give some hints how to achieve this.

Edit: I've just googled for "django connection pooling". There are many people who complain that Django does not provide a proper connection pool. Some of them managed to integrate a separate pooling package. For PostgreSQL, I would take a look at the pgpool middleware.

这篇关于在线程中使用Django ORM并避免“太多客户端”例外使用BoundedSemaphore的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！