将scrapy项目导出到不同的文件 [英] Export scrapy items to different files

问题描述

我正在从 moocs 中抓取这样的评论 一个

I'm scraping review from moocs likes this one

从那里我获得了所有课程的详细信息，每个评论本身有 5 个项目和另外 6 个项目.

From there I'm getting all the course details, 5 items and another 6 items from each review itself.

这是我的课程详情代码:

This is the code I have for the course details:

def parse_reviews(self, response):
    l = ItemLoader(item=MoocsItem(), response=response)
    l.add_xpath('course_title', '//*[@class="course-header-ng__main-info__name__title"]//text()')
    l.add_xpath('course_description', '//*[@class="course-info__description"]//p/text()')
    l.add_xpath('course_instructors', '//*[@class="course-info__instructors__names"]//text()')
    l.add_xpath('course_key_concepts', '//*[@class="key-concepts__labels"]//text()')
    l.add_value('course_link', response.url)
    return l.load_item()

现在我想包括评论详细信息，每个评论另外 5 个项目.由于课程数据对于所有评论都是通用的，我想将其存储在不同的文件中，然后使用课程名称/ID 关联数据.

Now I want to include the review details, another 5 items for each review. Since the course data is common for all the reviews I want to store it in a different file and use course name/id to relate the data afterward.

这是我用于评论项目的代码:

This is the code I have for the review's items:

for review in response.xpath('//*[@class="review-body"]'):
    review_body = review.xpath('.//div[@class="review-body__content"]//text()').extract()
    course_stage =  review.xpath('.//*[@class="review-body-info__course-stage--completed"]//text()').extract()
    user_name =  review.xpath('.//*[@class="review-body__username"]//text()').extract()
    review_date =  review.xpath('.//*[@itemprop="datePublished"]/@datetime').extract()
    score =  review.xpath('.//*[@class="sr-only"]//text()').extract()

我尝试使用临时解决方案，返回每个案例的所有项目，但也不起作用:

I tried to work with a temporary solution, returning all the items for each case but is not working either:

def parse_reviews(self, response):
    #print response.body
    l = ItemLoader(item=MoocsItem(), response=response)
    #l = MyItemLoader(selector=response)
    l.add_xpath('course_title', '//*[@class="course-header-ng__main-info__name__title"]//text()')
    l.add_xpath('course_description', '//*[@class="course-info__description"]//p/text()')
    l.add_xpath('course_instructors', '//*[@class="course-info__instructors__names"]//text()')
    l.add_xpath('course_key_concepts', '//*[@class="key-concepts__labels"]//text()')
    l.add_value('course_link', response.url)

    for review in response.xpath('//*[@class="review-body"]'):
        l.add_xpath('review_body', './/div[@class="review-body__content"]//text()')
        l.add_xpath('course_stage', './/*[@class="review-body-info__course-stage--completed"]//text()')
        l.add_xpath('user_name', './/*[@class="review-body__username"]//text()')
        l.add_xpath('review_date', './/*[@itemprop="datePublished"]/@datetime')
        l.add_xpath('score', './/*[@class="sr-only"]//text()')

        yield l.load_item()

该脚本的输出文件已损坏，单元格被替换，字段大小不正确.

The output file for that script is corrupted, cells are displaced and the size of the fields is not correct.

我想在输出中有两个文件:

I want to have two files at the output:

第一个包含:

course_title,course_description,course_instructors,course_key_concepts,course_link

第二个:

course_title,review_body,course_stage,user_name,review_date,score

推荐答案

问题是您将所有内容混合到一个项目中，这不是正确的做法.您应该创建两个项目:MoocsItem 和 MoocsReviewItem.

The issue is you are mixing everything up into a single item, which is not the right way to do it. You should create two items: MoocsItem and MoocsReviewItem.

然后更新如下代码

def parse_reviews(self, response):
    #print response.body
    l = ItemLoader(item=MoocsItem(), response=response)
    l.add_xpath('course_title', '//*[@class="course-header-ng__main-info__name__title"]//text()')
    l.add_xpath('course_description', '//*[@class="course-info__description"]//p/text()')
    l.add_xpath('course_instructors', '//*[@class="course-info__instructors__names"]//text()')
    l.add_xpath('course_key_concepts', '//*[@class="key-concepts__labels"]//text()')
    l.add_value('course_link', response.url)

    item = l.load_item()

    for review in response.xpath('//*[@class="review-body"]'):
        r = ItemLoader(item=MoocsReviewItem(), response=response, selector=review)
        r.add_value('course_title', item['course_title'])
        r.add_xpath('review_body', './/div[@class="review-body__content"]//text()')
        r.add_xpath('course_stage', './/*[@class="review-body-info__course-stage--completed"]//text()')
        r.add_xpath('user_name', './/*[@class="review-body__username"]//text()')
        r.add_xpath('review_date', './/*[@itemprop="datePublished"]/@datetime')
        r.add_xpath('score', './/*[@class="sr-only"]//text()')

        yield r.load_item()

    yield item

现在您想要的是不同的项目类型出现在不同的 csv 文件中，这就是以下 SO 线程的答案:

Now what you want is that different item type goes in different csv files, which is what the below SO thread answers:

如何将scrapy导出项目分开每个项目的 csv 文件

我没有测试过下面的代码，但代码看起来像这样:

I have not tested the below, but the code will look something like this:

from scrapy.exporters import CsvItemExporter
from scrapy import signals
from scrapy.xlib.pydispatch import dispatcher


def item_type(item):
    return type(item).__name__.replace('Item','').lower()  # TeamItem => team

class MultiCSVItemPipeline(object):
    SaveTypes = ['moocs','moocsreview']
    
    def __init__(self):
        dispatcher.connect(self.spider_opened, signal=signals.spider_opened)
        dispatcher.connect(self.spider_closed, signal=signals.spider_closed)

    def spider_opened(self, spider):
        self.files = dict([ (name, open(CSVDir+name+'.csv','w+b')) for name in self.SaveTypes ])
        self.exporters = dict([ (name,CsvItemExporter(self.files[name])) for name in self.SaveTypes])
        [e.start_exporting() for e in self.exporters.values()]

    def spider_closed(self, spider):
        [e.finish_exporting() for e in self.exporters.values()]
        [f.close() for f in self.files.values()]

    def process_item(self, item, spider):
        what = item_type(item)
        if what in set(self.SaveTypes):
            self.exporters[what].export_item(item)
        return item

您需要确保更新 ITEM_PIPELINES 以使用此 MultiCSVItemPipeline 类

You need to make sure the ITEM_PIPELINES is updated to use this MultiCSVItemPipeline class

ITEM_PIPELINES = {
    'mybot.pipelines.MultiCSVItemPipeline': 300,
}

这篇关于将scrapy项目导出到不同的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！