在 Python 中对 grequest 应用重试 [英] Applying retry on grequests in Python

问题描述

我有一个 URL 列表，我想使用 Python 的 grequests 模块对其进行 HTTP Get 请求.

I have a list of URLs for which I want to do a HTTP Get request using Python's grequests module.

有些 URL 没有返回 OK 状态，在这种情况下，我想重试该 URL.

Some of the URLs don't return an OK status, and in that case, I want to retry that URL.

我可以使用一个队列来做到这一点，该队列存储所有尚未尝试过的 URL，或者在之前的尝试中没有返回 200 的 URL，并继续批量发送请求.不过，我正在为此寻找一个更清洁/更Pythonic"的实现.

I can do this using a Queue which stores all such URLs that have not been tried yet, or the ones that didn't return 200 in previous tries, and keep sending requests in batches. I am looking for a cleaner/more "Pythonic" implementation for this though.

我之前使用过 retrying 模块进行重试，它非常简洁.我想知道是否有一些类似的、简洁的实现来重试 grequests 发送的请求.

I have used the retrying module for retries before and it is pretty neat. I'm wondering if there is some similar, concise implementation for retrying requests sent by grequests.

推荐答案

您可以将自己的请求会话传递给 grequests 并在该会话上设置重试，即.

You can pass in your own requests session to grequests and set retrying on that session ie.

    s = requests.Session()
    retries = Retry(total=5, backoff_factor=0.2, status_forcelist=[500, 502, 503, 504], raise_on_redirect=True,
                    raise_on_status=True)
    s.mount('http://', HTTPAdapter(max_retries=retries))
    s.mount('https://', HTTPAdapter(max_retries=retries))
    reqs = (grequests.get(url, session=s) for url in urls)
    for resp in grequests.imap(reqs, stream=False):
        ...

如果你想流式传输，你可以设置 stream=True.

You can put stream=True if you want streaming.

这篇关于在 Python 中对 grequest 应用重试的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！