Scrapy:如何一个接一个地运行两个爬虫? [英] Scrapy: how to run two crawlers one after another?

问题描述

我在同一个项目中有两个蜘蛛.其中一个取决于另一个先运行.他们使用不同的管道.如何确保它们按顺序运行?

I have two spiders within the same project. One of them depends on the other running first. They use different pipelines. How can I make sure they are run sequentially?

推荐答案

仅来自文档:https://doc.scrapy.org/en/1.2/topics/request-response.html

相同的例子，但通过链接延迟顺序运行蜘蛛:

Same example but running the spiders sequentially by chaining the deferreds:

from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging

class MySpider1(scrapy.Spider):
    # Your first spider definition
    ...

class MySpider2(scrapy.Spider):
    # Your second spider definition
    ...

configure_logging()
runner = CrawlerRunner()

@defer.inlineCallbacks
def crawl():
    yield runner.crawl(MySpider1)
    yield runner.crawl(MySpider2)
    reactor.stop()

crawl()
reactor.run() # the script will block here until the last crawl call is finished

这篇关于Scrapy:如何一个接一个地运行两个爬虫?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！