请求/aiohttp:关闭响应对象 [英] Requests/aiohttp: closing response objects

问题描述

我对在 requests 和 aiohttp 中都需要 .close() 响应对象感到有点困惑.(请注意，这是一个与 session.close() 不同的实例方法——我说的是响应对象本身.)

• Response (requests) 或 ClientResponse (aiohttp) 是否需要明确调用 .close()?
• 如果不是，使用响应本身作为上下文管理器的目的是什么?(async with session.request('GET', 'https://www.pastebin.com') 下面.)如果如下所示隐式关闭，为什么要为此定义两个dunder方法?

一些简单的测试(如下)似乎暗示响应在会话上下文管理器中定义时会自动关闭.(它本身调用 self.close() 在 __exit__ 或 __aexit__ 中.但这是 Session 的关闭，而不是 Response 对象.)

示例 - 请求

<预><代码>>>>进口请求>>>>>>使用 requests.Session() 作为 s:... resp = s.request('GET', 'https://www.pastebin.com')... resp.raise_for_status()... print(resp.raw.closed) # `raw` 是 urllib3.response.HTTPResponse 对象...打印(resp.raw._pool)...打印(resp.raw._connection)... c = resp.text...真的HTTPSConnectionPool(host='pastebin.com', port=443)没有任何>>>>>>而 1:...打印(resp.raw.closed)...打印(resp.raw._pool)...打印(resp.raw._connection)... 休息...真的HTTPSConnectionPool(host='pastebin.com', port=443)没有任何

示例 - aiohttp

<预><代码>>>>导入异步>>>导入 aiohttp>>>>>>异步 def get():...与 aiohttp.ClientSession() 异步作为 s:... # 在这个 `with` 块之后响应已经关闭.... # 为什么需要将其用作上下文管理器?... resp = await s.request('GET', 'https://www.pastebin.com')...打印(resp._closed)...打印(resp._connection)...打印(resp._released)... c = 等待 resp.text()... 打印()...打印(resp._closed)...打印(resp._connection)...打印(resp._released)...返回 c...>>>c = asyncio.run(get()) # Python 3.7 +错误的Connection错误的真的没有任何错误的

<小时>

这是requests.models.Response 的来源.通常不需要显式调用"是什么意思?有哪些例外?

def close(self):"""释放连接回池.一旦这个方法被不能再次访问称为底层原始"对象.*注意:通常不需要显式调用.*"""如果不是 self._content_consumed:self.raw.close()release_conn = getattr(self.raw, 'release_conn', None)如果 release_conn 不是 None:release_conn()

解决方案

Requests:您无需显式调用 close().请求将在完成后自动关闭，因为它基于 urlopen(这就是为什么 resp.raw.closed 为 True)，这是我观看后的简化代码 session.py 和adapters.py:

from urllib3 import PoolManager导入时间经理 = 池经理(10)conn = manager.connection_from_host('host1.example.com')conn2 = manager.connection_from_host('host2.example.com')res = conn.urlopen(url="http://host1.example.com/",method="get")打印(len(经理.池))manager.clear()打印(len(经理.池))打印(res.关闭)#2#0#真的

那么 __exit__ 做了什么?它用于清除 PoolManager(self.poolmanager=PoolManager(...)) 和 proxy.

# session.pydef __exit__(self, *args): #line 423self.close()def close(self): #line 733对于 self.adapters.values() 中的 v:v.close()# 适配器.py# v.close()def close(self): #line 307self.poolmanager.clear()对于 self.proxy_manager.values() 中的代理:代理清除()

那么什么时候需要使用 close() ，正如注释所说将连接释放回池，因为DEFAULT_POOLSIZE = 10(http/https 是独立的).这意味着如果您想在一个会话中访问 10 个以上的网站，您可以选择关闭一些您不需要的网站，否则当您有一个时，管理器将关闭从第一个到最新的连接.但实际上你不必关心这个，你可以指定池大小，这样就不会浪费太多时间重建连接

aiohttp aiohttp.ClientSession() 为所有请求使用一个 TCPConnector.当它触发 __aexit__ 时， self._connector 将被关闭.

s.request() 已建立来自主机的连接，但未收到响应.await resp.text() 只能在收到响应后执行，如果你没有做这一步(等待响应)，你将无响应退出.

如果连接器为无:#line 132连接器 = TCPConnector(循环=循环)...self._connector = 连接器#line 151# 连接超时尝试:使用 CeilTimeout(real_timeout.connect,loop=self._loop):断言 self._connector 不是 Noneconn = 等待 self._connector.connect(请求，痕迹=痕迹，超时=real_timeout)...async def close(self) ->没有任何:如果不是 self.closed:如果 self._connector 不是 None 和 self._connector_owner:self._connector.close()self._connector = 无...异步定义 __aexit__(self,...) ->没有任何:等待 self.close()

这是显示我所说内容的代码

导入异步导入 aiohttp导入时间异步 def get():与 aiohttp.ClientSession() 异步作为 s:# 在这个 `with` 块之后响应已经关闭.# 为什么需要将其用作上下文管理器?resp = await s.request('GET', 'https://www.stackoverflow.com')resp2 = await s.request('GET', 'https://www.github.com')打印(响应:"，响应._关闭)打印(响应:"，响应._connection)打印(resp2:"，resp2._closed)打印(resp2:"，resp2._connection)s.close()打印(s.关闭)c = 等待 resp.text()d = 等待 resp2.text()打印()打印(s._connector)打印(响应:"，响应._关闭)打印(响应:"，响应._connection)打印(resp2:"，resp2._closed)打印(resp2:"，resp2._connection)loop = asyncio.get_event_loop()loop.run_until_complete(get()) # Python 3.5 +#死循环

I'm a bit confused about the need to .close() a response object in both requests and aiohttp. (Note that this is a separate instance method than session.close()--I'm talking about the response object itself.)

• Does Response (requests) or ClientResponse (aiohttp) ever need explicitly call .close()?
• If not, what is the purpose of using the response itself as a context manager? (async with session.request('GET', 'https://www.pastebin.com') below.) Why define the two dunder methods for this if it gets closed implicitly as shown below?

Some simple tests (below) seem to imply that responses are closed automatically when they are defined inside of a Session context manager. (Which itself calls self.close() in __exit__ or __aexit__. But this is the closing of the Session, not the Response object.)

Example - requests

>>> import requests
>>> 
>>> with requests.Session() as s:
...     resp = s.request('GET', 'https://www.pastebin.com')
...     resp.raise_for_status()
...     print(resp.raw.closed)  # `raw` is urllib3.response.HTTPResponse object
...     print(resp.raw._pool)
...     print(resp.raw._connection)
...     c = resp.text
... 
True
HTTPSConnectionPool(host='pastebin.com', port=443)
None
>>>
>>> while 1:
...     print(resp.raw.closed)
...     print(resp.raw._pool)
...     print(resp.raw._connection)
...     break
... 
True
HTTPSConnectionPool(host='pastebin.com', port=443)
None

Example - aiohttp

>>> import asyncio
>>> import aiohttp
>>>
>>> async def get():
...     async with aiohttp.ClientSession() as s:
...         # The response is already closed after this `with` block.
...         # Why would it need to be used as a context manager?
...         resp = await s.request('GET', 'https://www.pastebin.com')
...         print(resp._closed)
...         print(resp._connection)
...         print(resp._released)
...         c = await resp.text()
...     print()
...     print(resp._closed)
...     print(resp._connection)
...     print(resp._released)
...     return c
... 
>>> c = asyncio.run(get())  # Python 3.7 +
False
Connection<ConnectionKey(host='pastebin.com', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=None)>
False

True
None
False

---

Here's the source for requests.models.Response. What does "Should not normally need to be called explicitly" mean? What are the exceptions?

def close(self):
    """Releases the connection back to the pool. Once this method has been
    called the underlying ``raw`` object must not be accessed again.
    *Note: Should not normally need to be called explicitly.*
    """
    if not self._content_consumed:
        self.raw.close()

    release_conn = getattr(self.raw, 'release_conn', None)
    if release_conn is not None:
        release_conn()

解决方案

Requests: You need not explicitly call close(). request will automatically close after finished because it bases on urlopen (this is why resp.raw.closed is True), This is the simplified code after i watched session.py and adapters.py:

from urllib3 import PoolManager
import time
manager = PoolManager(10)
conn = manager.connection_from_host('host1.example.com')
conn2 = manager.connection_from_host('host2.example.com')
res = conn.urlopen(url="http://host1.example.com/",method="get")
print(len(manager.pools))
manager.clear()
print(len(manager.pools))
print(res.closed)

#2
#0
#True

Then what did the __exit__ do? It uses to clear PoolManager(self.poolmanager=PoolManager(...)) and proxy.

# session.py
def __exit__(self, *args): #line 423
    self.close()
def close(self): #line 733
    for v in self.adapters.values():
        v.close()

# adapters.py
# v.close()
def close(self): #line 307
        self.poolmanager.clear()
        for proxy in self.proxy_manager.values():
            proxy.clear()

So when should you need to use close() , as the note said Releases the connection back to the pool, because DEFAULT_POOLSIZE = 10(http/https are independent). That means if you want to access more than 10 website with one session , you can chose to close some you do not need otherwise manager will close connection from the first to the newest when you have one more. But actually you need not to care about this , you can specify pool size and it would not waste much time to rebuild connection

aiohttp aiohttp.ClientSession() is using one TCPConnector for all requests. When it triggered __aexit__ , self._connector will be closed.

Edit: s.request() is set up a connection from host but it did not get response. await resp.text() can only be done after got response, if you did not do such step(wait for response), you will exit without having response.

if connector is None: #line 132
    connector = TCPConnector(loop=loop)
...
self._connector = connector #line 151
# connection timeout
try:
    with CeilTimeout(real_timeout.connect,loop=self._loop):
    assert self._connector is not None
    conn = await self._connector.connect(
        req,
        traces=traces,
        timeout=real_timeout
        )
...
async def close(self) -> None:
        if not self.closed:
            if self._connector is not None and self._connector_owner:
                self._connector.close()
            self._connector = None
...
async def __aexit__(self,
                       ...) -> None:
        await self.close()

This is code to show what i said

import asyncio
import aiohttp
import time

async def get():
    async with aiohttp.ClientSession() as s:
        # The response is already closed after this `with` block.
        # Why would it need to be used as a context manager?
        resp = await s.request('GET', 'https://www.stackoverflow.com')
        resp2 = await s.request('GET', 'https://www.github.com')
        print("resp:",resp._closed)
        print("resp:",resp._connection)
        print("resp2:",resp2._closed)
        print("resp2:",resp2._connection)
        s.close()
        print(s.closed)
        c = await resp.text()
        d = await resp2.text()

    print()
    print(s._connector)
    print("resp:",resp._closed)
    print("resp:",resp._connection)
    print("resp2:",resp2._closed)
    print("resp2:",resp2._connection)

loop = asyncio.get_event_loop()
loop.run_until_complete(get())  # Python 3.5 +

#dead loop

这篇关于请求/aiohttp:关闭响应对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！