在scrapy中为1个网站并行运行多个蜘蛛? [英] Running Multiple spiders in scrapy for 1 website in parallel?

问题描述

我想抓取一个包含 2 个部分的网站，但我的脚本没有我需要的那么快.

I want to crawl a website with 2 parts and my script is not as fast as I need.

是否可以发射 2 个蜘蛛，一个用于抓取第一部分，第二个用于第二部分?

Is it possible to launch 2 spiders, one for scraping the first part and the second one for the second part?

我尝试有 2 个不同的类，然后运行它们

I tried to have 2 different classes, and run them

scrapy crawl firstSpider
scrapy crawl secondSpider

但我认为它不聪​​明.

but i think that it is not smart.

我阅读了scrapyd 的文档，但我不知道这对我的情况有好处.

I read the documentation of scrapyd but I don't know if it's good for my case.

推荐答案

我认为您正在寻找的是这样的:

I think what you are looking for is something like this:

import scrapy
from scrapy.crawler import CrawlerProcess

class MySpider1(scrapy.Spider):
    # Your first spider definition
    ...

class MySpider2(scrapy.Spider):
    # Your second spider definition
    ...

process = CrawlerProcess()
process.crawl(MySpider1)
process.crawl(MySpider2)
process.start() # the script will block here until all crawling jobs are finished

您可以在以下位置阅读更多信息:在同一进程中运行多个蜘蛛.

You can read more at: running-multiple-spiders-in-the-same-process.

这篇关于在scrapy中为1个网站并行运行多个蜘蛛?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！