有没有什么方法可以为每个蜘蛛使用单独的scrapy管道? [英] Is there any method to using seperate scrapy pipeline for each spider?

问题描述

我想获取不同域下的网页，这意味着我必须在scrapy crawl myspider"命令下使用不同的蜘蛛.但是，由于网页的内容不同，我必须使用不同的管道逻辑将数据放入数据库.但是对于每个蜘蛛，它们都必须通过 settings.py 中定义的所有管道.是否有其他优雅的方法可以为每个蜘蛛使用单独的管道?

I wanna to fetch web pages under different domain, that means I have to use different spider under the command "scrapy crawl myspider". However, I have to use different pipeline logic to put the data into database since the content of web pages are different. But for every spider, they have to go through all of the pipelines which defined in settings.py. Is there have other elegant method to using seperate pipelines for each spider?

推荐答案

ITEM_PIPELINES 设置是在引擎启动期间为项目中的所有蜘蛛全局定义的.每个蜘蛛都不能即时更改.

ITEM_PIPELINES setting is defined globally for all spiders in the project during the engine start. It cannot be changed per spider on the fly.

以下是一些需要考虑的选项:

Here are some options to consider:

• 更改管道的代码.在管道的 process_item 方法中跳过/继续处理蜘蛛返回的项目，例如:

• Change the code of pipelines. Skip/continue processing items returned by spiders in the process_item method of your pipeline, e.g.:

def process_item(self, item, spider): 
    if spider.name not in ['spider1', 'spider2']: 
        return item  

    # process item

改变你开始爬行的方式.从脚本，根据作为参数传递的蜘蛛名称，在调用 crawler.configure() 之前覆盖您的 ITEM_PIPELINES 设置.

Change the way you start crawling. Do it from a script, based on spider name passed as a parameter, override your ITEM_PIPELINES setting before calling crawler.configure().

另见:

• Scrapy.开始爬行后如何更改蜘蛛设置?
• 我可以使用蜘蛛专用设置吗?
• 为多个网站使用一个 Scrapy 蜘蛛
• 相关答案

希望有所帮助.

这篇关于有没有什么方法可以为每个蜘蛛使用单独的scrapy管道?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！