是否可以使grequests和request_cache一起工作？ [英] Is it possible to make grequests and requests_cache work together?

问题描述

看下面的代码：

import requests
import grequests
import requests_cache
requests_cache.install_cache('bla')

urls = [
    'http://www.heroku.com',
    'http://python-tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://kennethreitz.com'
]

rs = (grequests.get(u) for u in urls)
results = grequests.map(rs)

我希望执行完此操作后，我会在当前目录中找到 bla.sqlite 文件并执行

I would expect that after executing this I will find bla.sqlite file in the current directory and executing

results = grequests.map(rs)

将更快，因为数据将从sqlite缓存中获取。不幸的是，这不是真的，根本没有创建文件，也没有加速。当我使用grequests insetead的请求时，一切正常。所以问题是标题说：是否可以使grequests和requests_cache一起工作？，如果可以，怎么办？

will be MUCH faster because data will be taken from sqlite cache. Unfortunately this is not true, file wasn't created at all and there is no speedup. When I use requests insetead of grequests everything works fine. So the question is a title says: Is it possible to make grequests and requests_cache work together? and if yes, how?

推荐答案

requests_cache.install_cache（）函数修补了 requests.Session ，但您已经导入了 grequests ，它用于：

The requests_cache.install_cache() function patches requests.Session, but you already imported grequests, which used:

from requests import Session

因此， grequests 从未使用修补的会话对象。

As such, grequests never uses the patched session object.

在安装缓存后将导入移动到 ：

Move the import to after you installed the cache:

import requests_cache
requests_cache.install_cache('bla')
import grequests

或者，创建一个 CachedSession 对象，并将其作为传递给 grequests.get（）（及相关）方法会话参数：

Alternatively, create a CachedSession object and pass that in to the grequests.get() (and related) methods as the session parameter:

import grequests
import requests_cache

session = requests_cache.CachedSession('bla')

urls = [
    'http://www.heroku.com',
    'http://python-tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://kennethreitz.com'
]

rs = (grequests.get(u, session=session) for u in urls)
results = grequests.map(rs)

请注意，缓存存储后端可能无法安全处理并发访问。例如， sqlite 后端使用线程锁，这很可能会发生冲突。

Take into account that the cache storage backend may not be able to handle the corcurrent access safely. The sqlite backend uses a thread lock, for example, which might well clash.

这篇关于是否可以使grequests和request_cache一起工作？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！