python - scrapy的异常处理一般怎么做?

问题描述

问 题

发现在运行过程中有少量的请求有异常，或者请求响应了404 500之类的，又或者代码本身运行抛异常， 怎么记录这些异常呢。

解决方案

我目前的做法是用下载中间件去捕捉失败的请求，仅供大家参考

class CustomFaillogMiddleware(object):

    @classmethod
    def from_crawler(cls, crawler):
        return cls()

    def process_response(self, request, response, spider):
        if response.status >= 400:
            reason = response_status_message(response.status)
            self._faillog(request, u'HTTPERROR',reason, spider)
        return response

    def process_exception(self, request, exception, spider):
        self._faillog(request, u'EXCEPTION', exception, spider)
        return request

    def _faillog(self, request, errorType, reason, spider):
        with codecs.open('log/faillog.log', 'a', encoding='utf-8') as file:
            file.write("%(now)s [%(error)s] %(url)s reason: %(reason)s \r\n" %
                       {'now':datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
                        'error': errorType,
                        'url': request.url,
                        'reason': reason})

这篇关于python - scrapy的异常处理一般怎么做?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！