将大型XML文件拆分为Hadoop的易管理部分 [英] Splitting large XML files into manageble sections for Hadoop

问题描述

是否有一个输入类根据Hadoop中的树结构处理[多个]大型XML文件？我有一组具有相同模式的XML文件，但我需要将它们拆分为数据段，而不是将这些段分开。

例如，XML文件应该是：

 < root> ; 
< parent>资料< / parent> 
< parent>更多资料< / parent> 
< parent>更多数据< / parent> 
< / root> 
  

我将每个部分定义为：
/ root / parent。

我问的是：Hadoop中是否有一个记录输入阅读器可以执行此操作？

解决方案

我认为 Cloud9项目在UMD 可能会帮助你。

图书馆提供了 XMLInputFormat class 可能有用。

Cloud9文档中的这个页面是关于如何处理维基百科在MapReduce中的XML转储。

Is there a input class to deal with [multiple] large XML files based on their tree structure in Hadoop? I have a set of XML files that are of the same schema, but I need to split them into sections of data, as opposed to breaking the sections up.

For example the XML file would be:

<root>
  <parent> data </parent>
  <parent> more data</parent>
  <parent> even more data</parent>
</root>

I would define each section as: /root/parent.

What I'm asking is: Is there a record input reader already included for Hadoop to do this?

解决方案

I think the Cloud9 project at UMD might help you with this.

The library provides has an XMLInputFormat class which might be of use.

Also of interest is this page in the Cloud9 documentation which looks at how you can deal with an XML dump of Wikipedia in MapReduce.

这篇关于将大型XML文件拆分为Hadoop的易管理部分的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！