使用request / urllib3在每次重试尝试中添加回调函数 [英] Adding callback function on each retry attempt using requests/urllib3

问题描述

我已经使用 urllib3.util.retry 都建议此处和。

I've implemented a retry mechanism to requests session using urllib3.util.retry as suggested both here and here.

现在，我想弄清楚添加每次重试都会调用的回调函数的最佳方法是什么。

Now, I am trying to figure out what is the best way to add a callback function that will be called on every retry attempt.

如果 Retry 对象或请求 get 方法具有更多内容，请进一步说明自己一种添加回调函数的方法，这会很棒。也许是这样的：

To explain myself even more, if either the Retry object or the requests get method had a way to add a callback function, it would have been great. Maybe something like:

import requests
from requests.packages.urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

def retry_callback(url):
    print url   

s = requests.Session()
retries = Retry(total=5, status_forcelist=[ 500, 502, 503, 504 ])
s.mount('http://', HTTPAdapter(max_retries=retries))

url = 'http://httpstat.us/500'
s.get(url, callback=retry_callback, callback_params=[url])

我知道我可以使用日志记录来打印url，但这只是一个更复杂的用法的简单示例。

如果不是最好的python编码，请原谅。我希望它已经足够清楚了。

I know that for printing url I can use the logging, but this is only a simple example for a more complex use.
Please excuse me if it's not the best python coding, but I hope it is clear enough.

谢谢。

推荐答案

您可以将 Retry 类子类化以添加该功能。

You can subclass the Retry class to add that functionality.

这是与<$ c $的完整交互流程c>重试实例以进行给定的连接尝试：

This is the full interaction flow with the Retry instance for a given connection attempt:

• Retry.increment（）用当前方法，URL，响应对象调用（如果有一个）和异常（如果引发了一个异常），或者在引发异常，返回30x重定向响应或 Retry.is_retry（）方法时返回true。
  
  
  
  
  • .increment（）将重新引发错误（如果存在）和对象配置为不重试特定类型的错误。
  
  
  • .increment（）调用 Retry.new（ ）创建一个更新的实例，其中所有相关计数器都已更新，并且 history 属性用新的 RequestHistory（）实例（一个命名的元组）。
  
  
  • .increment（）将引发 MaxRetryError Retry.new（）的返回值上调用 Retry.is_exhausted（）为true，则$ c>异常。 is_exhausted（）当它跟踪的任何计数器降到0以下时，返回true（忽略设置为 None 的计数器） 。
  
  
  • .increment（）返回新的 Retry 实例。 li>
    
  • Retry.increment() is called with the current method, url, response object (if there is one), and exception (if one was raised) whenever an exception is raised, or a 30x redirection response was returned, or the Retry.is_retry() method returns true.
    • .increment() will re-raise the error (if there was one) and the object was configured not to retry that specific class of errors.
    • .increment() calls Retry.new() to create an updated instance, with any relevant counters updated and the history attribute amended with a new RequestHistory() instance (a named tuple).
    • .increment() will raise a MaxRetryError exception if Retry.is_exhausted() called on the return value of Retry.new() is true. is_exhausted() returns true when any of the counters it tracks has dropped below 0 (counters set to None are ignored).
    • .increment() returns the new Retry instance.

    此给你3个不错的回调点；在 .increment（）的开始处，在创建新的 Retry 实例时，以及在<$附近的上下文管理器中c $ c> super（）。increment（）允许回调否决异常或在退出时更新返回的重试策略。

    This gives you 3 good callback points; at the start of .increment(), when creating the new Retry instance, and in a context manager around super().increment() to let a callback veto an exception or update the returned retry policy on exit.

    此是 .increment（）开头的样子：

    import logging
    
    logger = getLogger(__name__)
    
    class CallbackRetry(Retry):
        def __init__(self, *args, **kwargs):
            self._callback = kwargs.pop('callback', None)
            super(CallbackRetry, self).__init__(*args, **kwargs)
        def new(self, **kw):
            # pass along the subclass additional information when creating
            # a new instance.
            kw['callback'] = self._callback
            return super(CallbackRetry, self).new(**kw)
        def increment(self, method, url, *args, **kwargs):
            if self._callback:
                try:
                    self._callback(url)
                except Exception:
                    logger.exception('Callback raised an exception, ignoring')
            return super(CallbackRetry, self).increment(method, url, *args, **kwargs)
    

    请注意， url 参数实际上只是 URL路径，请求的净位置部分被省略了（您可以d必须从 _pool 参数中提取出它具有 .scheme ， .host 和 .port 属性）。

    Note, the url argument is really only the URL path, the net location portion of the request is omitted (you'd have to extract that from the _pool argument, it has .scheme, .host and .port attributes).

    演示：

    >>> def retry_callback(url):
    ...     print('Callback invoked with', url)
    ...
    >>> s = requests.Session()
    >>> retries = CallbackRetry(total=5, status_forcelist=[500, 502, 503, 504], callback=retry_callback)
    >>> s.mount('http://', HTTPAdapter(max_retries=retries))
    >>> s.get('http://httpstat.us/500')
    Callback invoked with /500
    Callback invoked with /500
    Callback invoked with /500
    Callback invoked with /500
    Callback invoked with /500
    Callback invoked with /500
    Traceback (most recent call last):
      File "/.../lib/python3.6/site-packages/requests/adapters.py", line 440, in send
        timeout=timeout
      File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 732, in urlopen
        body_pos=body_pos, **response_kw)
      File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 732, in urlopen
        body_pos=body_pos, **response_kw)
      File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 732, in urlopen
        body_pos=body_pos, **response_kw)
      [Previous line repeated 1 more times]
      File "/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 712, in urlopen
        retries = retries.increment(method, url, response=response, _pool=self)
      File "<stdin>", line 8, in increment
      File "/.../lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='httpstat.us', port=80): Max retries exceeded with url: /500 (Caused by ResponseError('too many 500 error responses',))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/.../lib/python3.6/site-packages/requests/sessions.py", line 521, in get
        return self.request('GET', url, **kwargs)
      File "/.../lib/python3.6/site-packages/requests/sessions.py", line 508, in request
        resp = self.send(prep, **send_kwargs)
      File "/.../lib/python3.6/site-packages/requests/sessions.py", line 618, in send
        r = adapter.send(request, **kwargs)
      File "/.../lib/python3.6/site-packages/requests/adapters.py", line 499, in send
        raise RetryError(e, request=request)
    requests.exceptions.RetryError: HTTPConnectionPool(host='httpstat.us', port=80): Max retries exceeded with url: /500 (Caused by ResponseError('too many 500 error responses',))
    

    在 .new（）方法中添加一个钩子可以让您调整下次尝试使用的策略，以及让您自省 .history 属性，但又不能避免避免重新引发异常。

    Putting a hook in the .new() method would let you adjust the policy for a next attempt, as well as let you introspect the .history attribute, but would not let you avoid the exception re-raising.

    这篇关于使用request / urllib3在每次重试尝试中添加回调函数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！