如何在python请求库中实现重试机制? [英] How to implement retry mechanism into python requests library?

问题描述

我想在python请求库中添加一个重试机制，因此正在使用该脚本的脚本将重试非致命错误.

I would like to add a retry mechanism to python request library, so scripts that are using it will retry for non fatal errors.

目前，我确实认为三种错误是可以恢复的:

At this moment I do consider three kind of errors to be recoverable:

• HTTP返回码502、503、504
• 找不到主机(现在不重要了)
• 请求超时

在第一阶段，我想每分钟重试指定的5xx请求.

At the first stage I do want to retry specified 5xx requests every minute.

我希望能够透明地添加此功能，而不必为从这些使用python-requests的脚本或库中进行的每个HTTP调用手动实现恢复.

I want to be able to add this functionality transparently, without having to manually implement recovery for each HTTP call made from inside these scripts or libraries that are using python-requests.

推荐答案

通过扩展requests.Session类，我可以获得所需的可靠性.

I was able to obtain the desired level of reliability by extending requests.Session class.

这是代码 https://bitbucket.org/bspeakmon/jira-python/src/a7fca855394402f58507ca4056de87ccdbd6a213/jira/resilientsession.py?at=master

编辑，该代码是:

from requests import Session
from requests.exceptions import ConnectionError
import logging
import time


class ResilientSession(Session):

    """
    This class is supposed to retry requests that do return temporary errors.

    At this moment it supports: 502, 503, 504
    """

    def __recoverable(self, error, url, request, counter=1):
        if hasattr(error,'status_code'):
            if error.status_code in [502, 503, 504]:
                error = "HTTP %s" % error.status_code
            else:
                return False
        DELAY = 10 * counter
        logging.warn("Got recoverable error [%s] from %s %s, retry #%s in %ss" % (error, request, url, counter, DELAY))
        time.sleep(DELAY)
        return True


    def get(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).get(url, **kwargs)
            except ConnectionError as e:
                r = e.message
            if self.__recoverable(r, url, 'GET', counter):
                continue
            return r

    def post(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).post(url, **kwargs)
            except ConnectionError as e:
                r = e.message
            if self.__recoverable(r, url, 'POST', counter):
                continue
            return r

    def delete(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).delete(url, **kwargs)
            except ConnectionError as e:
                r = e.message
            if self.__recoverable(r, url, 'DELETE', counter):
                continue
            return r

    def put(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).put(url, **kwargs)
            except ConnectionError as e:
                r = e.message

            if self.__recoverable(r, url, 'PUT', counter):
                continue
            return r

    def head(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).head(url, **kwargs)
            except ConnectionError as e:
                r = e.message
            if self.__recoverable(r, url, 'HEAD', counter):
                continue
            return r

    def patch(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).patch(url, **kwargs)
            except ConnectionError as e:
                r = e.message

            if self.__recoverable(r, url, 'PATCH', counter):
                continue
            return r

    def options(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).options(url, **kwargs)
            except ConnectionError as e:
                r = e.message

            if self.__recoverable(r, url, 'OPTIONS', counter):
                continue
            return r

这篇关于如何在python请求库中实现重试机制?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！