每台主机与twisted.web.client.Agent 的最大连接数 [英] Maximum number of connections per host with twisted.web.client.Agent

问题描述

我有以下代码，它使用 TwistedMatrix Python 框架创建一个 HTTPConnectionPool，以及一个用于 HTTP 请求的代理:

I have the following code which creates an HTTPConnectionPool using TwistedMatrix Python framework, and an Agent for HTTP requests:

    self.pool = HTTPConnectionPool(reactor, persistent=True)
    self.pool.retryAutomatically = False
    self.pool.maxPersistentPerHost = 1
    self.agent = Agent(reactor, pool=self.pool)

然后我创建连接到本地服务器的请求:

then I create requests to connect to a local server:

    d = self.agent.request(
        "GET",
         url,
         Headers({"Host": ["localhost:8333"]}),
         None)

问题是:当同时发出多个请求时，本地服务器有时会表现不正确，所以我想将同时请求的数量限制为 1.

The problem is: the local server sometimes behaves incorrectly when multiple simultaneous requests are made, so I would like to limit the number of simultaneous requests to 1.

额外的请求应该排队，直到待处理的请求完成.

The additional requests should be queued until the pending request completes.

我尝试过 self.pool.maxPersistentPerHost = 1 但它不起作用.

I've tried with self.pool.maxPersistentPerHost = 1 but it doesn't work.

twisted.web.client.Agent 和 HTTPConnectionPool 是否支持限制每个主机的最大连接数，还是我必须自己实现一个请求 FIFO 队列?

Does twisted.web.client.Agent with HTTPConnectionPool support limiting the maximum number of connections per host, or do I have to implement a request FIFO queue myself?

推荐答案

将 maxPersistentPerHost 设置为 1 没有帮助的原因是 maxPersistentPerHost> 用于控制每个主机缓存的最大持久连接数.它不会阻止打开额外的连接来为新请求提供服务，它只会在收到响应后立即关闭它们，前提是已达到最大缓存连接数.

The reason setting maxPersistentPerHost to 1 didn't help is that maxPersistentPerHost is for controlling the maximum number of persistent connections to cache per host. It does not prevent additional connections from being opened in order to service new requests, it will only cause them to be closed immediately after a response is received, if the maximum number of cached connections has already been reached.

您可以通过多种方式强制序列化.拥有FIFO 队列"的一种方法是使用 twisted.internet.defer.DeferredLock.像这样与 Agent 一起使用:

You can enforce serialization in a number of ways. One way to have a "FIFO queue" is with twisted.internet.defer.DeferredLock. Use it together with Agent like this:

lock = DeferredLock()
d1 = lock.run(agent.request, url, ...)
d2 = lock.run(agent.request, url, ...)

在第一个请求完成后，第二个请求才会运行.

The second request will not run until after the first as completed.

这篇关于每台主机与twisted.web.client.Agent 的最大连接数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！