在Scrapy项目中使用多个spider [英] Using multiple spiders at in the project in Scrapy

问题描述

我想知道是否可以在同一个项目中一起使用多个蜘蛛.其实我需要2只蜘蛛.第一个收集第二个蜘蛛应该抓取的链接.他们都在同一个网站上工作，所以域相似.这可能吗?如果是，你能给我举个例子吗?谢谢

I wanna know if it is possible to use multiple spiders within the same project together. Actually I need 2 spiders. The first one gathers the links on which the second spider should scrape. They both work on the same website, so the domain is similar.Is it possible? If yes can you give me an example? Thanks

推荐答案

也许这就是您要找的:

def parse(self, response):
    # parse the links (aka your first spider)
    for link in hxs('//XPATH'):
        yield Request(link.extract(), callback=self.parse_link)

def parse_link(self, response):
    # continue parsing (aka your second spider)

希望对你有帮助:)

这篇关于在Scrapy项目中使用多个spider的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！