requests + grequests:是“连接池已满，正在丢弃连接:"警告相关? [英] requests + grequests: is the "Connection pool is full, discarding connection:" warning relevant?

问题描述

我在本地主机上托管一个服务器，我想异步触发数百个 GET 请求.为此，我使用 grequests.一切似乎工作正常，但我反复收到警告:

<块引用>

警告:requests.packages.urllib3.connectionpool:连接池已满，丢弃连接:date.jsontest.com

搜索显示在 requests 中创建 Session() 时如何避免全池问题，例如此处.但是，有几点:

1. 即使我没有采取任何措施来避免警告，我似乎也始终如一地获得了预期的结果.如果我确实使用了该解决方法，则任何超过 pool_maxsize 数量的请求都会发出警告.
2. 如果请求数量超过池大小，链接的解决方法仍将导致警告.我认为会有某种限制以防止在任何时候超过池大小
3. 我似乎找不到禁用警告的方法.requests.packages.urllib3.disable_warnings() 似乎没有做任何事情.

所以我的问题是:

1. 这个警告实际上意味着什么?我的解释是，它只是从触发中删除请求，但似乎并非如此.
2. 这个警告是否真的与 grequests 库相关，尤其是当我采取措施限制池大小时?我是否会在测试中引发意外行为并未能达到预期的结果?
3. 有没有办法禁用它?

一些要测试的代码:

导入 grequests进口请求requests.packages.urllib3.disable_warnings() # 好像不行?session = requests.Session()# 散列以下将导致 105 个警告而不是 5 个适配器 = requests.adapters.HTTPAdapter(pool_connections=100,pool_maxsize=100)session.mount('http://', 适配器)# 测试查询query_list = ['http://date.jsontest.com/' for x in xrange(105)]rs = [grequests.get(item, session=session) for query_list 中的项目]响应 = grequests.map(rs)打印 len([item.json() 用于响应中的项目])

解决方案

1) 这个警告实际上是什么意思?我的解释是它只是从触发中删除请求，但似乎不是案例.

这对我来说实际上仍然不清楚.即使发出一个请求也足以得到警告，但仍然会给我预期的响应.

<块引用>

2) 这个警告是否真的与 grequests 库相关，特别是当我采取措施限制池大小时?我在邀请吗在我的测试中出现意外行为和我的预期结果?

最后一部分:是.我与之通信的服务器可以同时处理 10 个查询.使用以下代码，我可以在单个列表理解中发送 400 个左右的请求，并且一切正常(即我的服务器从未被淹没，所以它一定以某种方式进行了节流).在请求数量达到某个临界点之后，代码将停止触发任何请求并简单地给出一个 None 列表.它甚至没有尝试通过列表，它甚至没有触发第一个查询，它只是阻塞了.

sess = requests.Session()适配器 = requests.adapters.HTTPAdapter(pool_connections=10,pool_maxsize=10)sess.mount('http://', 适配器)# 启动约 500 个或更多请求会突然导致失败rs = [grequests.get(item[0], session=session) 用于查询中的项目]响应 = grequests.map(rs)

<块引用>

3) 有没有办法禁用它?

是的，如果您想成为像我一样的傻瓜并在源代码中将其散列.我找不到任何其他方法来让它安静下来，它又回来咬我.

解决方案

解决方案是轻松过渡到使用 requests-futures 代替.以下代码的行为完全符合预期，没有发出任何警告，并且到目前为止，可以扩展到我向其抛出的任意数量的查询.

from requests_futures.sessions import FuturesSessionsession = FuturesSession(max_workers = 10)fire_requests = [session.get(url) for url]响应 = [item.result() for item in fire_requests]

I'm hosting a server on localhost and I want to fire hundreds of GET requests asynchronously. For this I am using grequests. Everything appears to work fine but I repeatedly get the warning:

WARNING:requests.packages.urllib3.connectionpool:Connection pool is full, discarding connection: date.jsontest.com

A search shows how the full pool issue can be avoided when creating a Session() in requests e.g. here. However, a couple of things:

1. Even if I don't take any steps to avoid the warning, I appear to consistently get the expected results. If I do use the workaround, any requests over the number of the pool_maxsize will give a warning.
2. The linked workaround will still result in the warning if the number of requests exceeds the pool size. I assumed there would be some kind of throttling to prevent the pool size being exceeded at any one time
3. I can't seem to find a way to disable the warning. requests.packages.urllib3.disable_warnings() doesn't seem to do anything.

So my questions are:

1. What does this warning actually mean? My interpretation is that it is simply dropping the requests from firing, but it doesn't seem to be the case.
2. Is this warning actually relevant for the grequests library, especially when I take steps to limit the pool size? Am I inviting unexpected behaviour and fluking my expected result in my tests?
3. Is there a way to disable it?

Some code to test:

import grequests
import requests

requests.packages.urllib3.disable_warnings() # Doesn't seem to work?

session = requests.Session()

# Hashing the below will cause 105 warnings instead of 5
adapter = requests.adapters.HTTPAdapter(pool_connections=100, 
                                            pool_maxsize=100)
session.mount('http://', adapter)

# Test query
query_list = ['http://date.jsontest.com/' for x in xrange(105)]

rs = [grequests.get(item, session=session) for item in query_list]
responses = grequests.map(rs)
print len([item.json() for item in responses])

解决方案

1) What does this warning actually mean? My interpretation is that it is simply dropping the requests from firing, but it doesn't seem to be the case.

This is actually still unclear to me. Even firing one request was enough to get the warning but would still give me the expected response.

2) Is this warning actually relevant for the grequests library, especially when I take steps to limit the pool size? Am I inviting unexpected behaviour and fluking my expected result in my tests?

For the last part: yes. The server I was communicating with could handle 10 queries concurrently. With the following code I could send 400 or so requests in a single list comprehension and everything worked out fine (i.e. my server never got swamped so it must have been throttling in some way). After some tipping point in the number of requests, the code would stop firing any requests and simply give a list of None. It's not as though it even tried to get through the list, it didn't even fire the first query, it just blocks up.

sess = requests.Session()
adapter = requests.adapters.HTTPAdapter(pool_connections=10, 
                                            pool_maxsize=10)
sess.mount('http://', adapter)  

# Launching ~500 or more requests will suddenly cause this to fail
rs = [grequests.get(item[0], session=session) for item in queries]
responses = grequests.map(rs)

3) Is there a way to disable it?

Yes, if you want to be a doofus like me and hash it out in the source code. I couldn't find any other way to silence it, and it came back to bite me.

SOLUTION

The solution was a painless transition to using requests-futures instead. The following code behaves exactly as expected, gives no warnings and, thus far, scales to any number of queries that I throw at it.

from requests_futures.sessions import FuturesSession

session = FuturesSession(max_workers = 10)
fire_requests = [session.get(url) for url in queries]
responses = [item.result() for item in fire_requests]

这篇关于requests + grequests:是“连接池已满，正在丢弃连接:"警告相关?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！