如何将scrapy导出项目以将每个项目的csv文件分开 [英] How can scrapy export items to separate csv files per item

问题描述

我正在抓取一个足球网站，蜘蛛(一只蜘蛛)从网站页面中获取了多种项目:团队、比赛、俱乐部等.我正在尝试使用 CSVItemExporter 将这些项目存储在单独的 csv 文件中，team.csv、matches.csv、clubs.csv 等

I am scraping a soccer site and the spider (a single spider) gets several kinds of items from the site's pages: Team, Match, Club etc. I am trying to use the CSVItemExporter to store these items in separate csv files, teams.csv, matches.csv, clubs.csv etc.

我不确定这样做的正确方法是什么.到目前为止，我想到的唯一方法是像示例中一样创建自己的自定义管道http://doc.scrapy.org/en/0.14/topics/exporters.html 并在 spider_opened 方法中打开所有需要的 csv 文件，即为每个 csv 文件创建一个 csv 导出器，并在 process_item 中放置代码以找出item"参数是什么类型的项目，然后将其发送到相应的导出对象.

I am not sure what is the right way to do this. The only way I have thought so far is to create my own custom pipeline like in the example http://doc.scrapy.org/en/0.14/topics/exporters.html and there open all needed csv files in the spider_opened method, ie create a csv exporter for each csv file and in the process_item put code to figure out what kind of item is the "item" parameter and then send it to the corresponding exporter object.

无论如何，我还没有找到任何在 scrapy 中处理多个 csv 文件(每个项目类型)的示例，所以我担心我以一种不应该使用的方式使用它.(这是我第一次接触 Scrapy).

Anyway I haven't found any examples of handling multiple csv files (per item type) in scrapy so I am worrying that I am using it in a way that is not meant to be used. (this is my first experience with Scrapy).

diomedes

推荐答案

我觉得你的方法没问题.管道是 Scrapy 的一项重要功能，是 IMO 为类似您的方法而构建的.

You approach seems fine to me. Piplines are a great feature of Scrapy and are IMO build for something like your approach.

您可以创建多个项目(例如 SoccerItem、MatchItem)，并在您的 MultiCSVItemPipeline 中通过检查项目类将每个项目委派给其自己的 CSV 类.

You could create multiple items (e.g. SoccerItem, MatchItem) and in your MultiCSVItemPipeline just delegate each item to its own CSV class by checking the item class.

这篇关于如何将scrapy导出项目以将每个项目的csv文件分开的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！