遇到第一个异常时如何使scrapy crawl中断并退出? [英] How can I make scrapy crawl break and exit when encountering the first exception?

问题描述

出于开发目的，我想在第一个异常(在蜘蛛或管道中)发生时立即停止所有爬虫爬行活动.

For development purposes, I would like to stop all scrapy crawling activity as soon a first exception (in a spider or a pipeline) occurs.

有什么建议吗?

推荐答案

从 0.11 开始，有 CLOSESPIDER_ERRORCOUNT:

Since 0.11, there is CLOSESPIDER_ERRORCOUNT:

一个整数，指定在关闭蜘蛛之前接收的最大错误数.如果蜘蛛产生的错误数超过该数量，它将以 closespider_errorcount 为由关闭.如果为零(或未设置)，蜘蛛将不会因错误次数而关闭.

An integer which specifies the maximum number of errors to receive before closing the spider. If the spider generates more than that number of errors, it will be closed with the reason closespider_errorcount. If zero (or non set), spiders won’t be closed by number of errors.

如果设置为1，蜘蛛将在第一个异常时关闭.

If it is set to 1, the spider will be closed on the first exception.

这篇关于遇到第一个异常时如何使scrapy crawl中断并退出?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！