遇到第一个例外时，怎样才能使抓取物爬行并退出？ [英] How can I make scrapy crawl break and exit when encountering the first exception?

问题描述

出于开发目的，我想尽快停止所有的抓取抓取活动（第一个例外（蜘蛛或管道））。

For development purposes, I would like to stop all scrapy crawling activity as soon a first exception (in a spider or a pipeline) occurs.

任何建议？ / p>

Any advice?

推荐答案

自0.11起，有 CLOSESPIDER_ERRORCOUNT ：

Since 0.11, there is CLOSESPIDER_ERRORCOUNT:

一个整数，它指定在关闭蜘蛛之前要接收的最大错误数。如果蜘蛛生成的错误数量超过了这个数量，那么将关闭它的原因是关闭这个错误。如果为零（或未设置），则蜘蛛不会被错误的数量关闭。

An integer which specifies the maximum number of errors to receive before closing the spider. If the spider generates more than that number of errors, it will be closed with the reason closespider_errorcount. If zero (or non set), spiders won’t be closed by number of errors.

如果设置为 1 ，蜘蛛将在第一个例外关闭。

If it is set to 1, the spider will be closed on the first exception.

这篇关于遇到第一个例外时，怎样才能使抓取物爬行并退出？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！