Hadoop自定义分割TextFile [英] Hadoop custom split of TextFile

问题描述

我有一个相当大的文本文件，我想将其转换为SequenceFile。不幸的是，该文件由Python代码组成，逻辑行通过几条物理线路运行。例如，

打印Blah Blah \

... blah blah

每条逻辑行都以NEWLINE结尾。有人可以澄清我怎么可能在Map-Reduce中生成Key，Value对，其中每个Value都是整个逻辑行？

I have a fairly large text file that I would like to convert into a SequenceFile. Unfortunately, the file consists of Python code with logical lines running over several physical lines. For example,
print "Blah Blah\
... blah blah"
Each logical line is terminated by a NEWLINE. Could someone clarify how I could possibly generate Key, Value pairs in Map-Reduce where each Value is the entire logical line?

推荐答案

您应该在TextInputFormat上创建自己的变体。在那里你创建一个新的RecordReader，它跳过行，直到它看到一条逻辑行的开始。

You should create your own variation on TextInputFormat. In there you make a new RecordReader that skips lines until it sees the start of a logical line.

这篇关于Hadoop自定义分割TextFile的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！