Scrapy:如何限制在 SitemapSpider 中抓取的 url 数量 [英] Scrapy: How to limit number of urls scraped in SitemapSpider

问题描述

我正在研究站点地图蜘蛛.该蜘蛛获取一个站点地图网址并抓取此站点地图中的所有网址.我想将 url 的数量限制为 100.

I'm working on a sitemap spider. This spider gets one sitemap url and scrape all urls in this sitemap. I want to limit the number of urls to 100.

我不能使用 CLOSESPIDER_PAGECOUNT，因为我使用 XML 导出管道.似乎当scrapy 达到pagecount 时，它会停止一切，包括XML 生成.所以 XML 文件没有关闭等等，它是无效的.

I can't use CLOSESPIDER_PAGECOUNT because I use XML export pipeline. It seems that when scrapy gets to the pagecount, it stops everything including XML generating. So the XML file is not closed etc. it's invalid.

class MainSpider(SitemapSpider):
    name = 'main_spider'
    allowed_domains = ['doman.com']
    sitemap_urls = ['http://doman.com/sitemap.xml']

    def start_requests(self):
        for url in self.sitemap_urls:
            yield Request(url, self._parse_sitemap)


    def parse(self, response):
        print u'URL: {}'.format(response.url)
        if self._is_product(response):
            URL = response.url
            ITEM_ID = self._extract_code(response)

    ...

你知道该怎么做吗?

推荐答案

使用 return 对我来说还不够，但您可以将其与 CloseSpider 异常结合使用:

Using on return was not enough for me, but you can combine it with the CloseSpider exception :

# To import it :
from scrapy.exceptions import CloseSpider


#Later to use it:
raise CloseSpider('message')

我在这里在stackoverflow上发布了整个代码

这篇关于Scrapy:如何限制在 SitemapSpider 中抓取的 url 数量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！