如何阅读由特殊字符分隔的hadoop中的文本源 [英] How to read text source in hadoop separated by special character

问题描述

我的数据格式使用\0而不是换行。所以默认的hadoop textLine阅读器无法工作。
如何配置它读取由特殊字符分隔的行？

如果不可能配置LineReader，也许可以应用特定的流处理器（ tr\0\\\
）不知道如何做到这一点。 你可以编写自己的InputFormat类，它将数据分割为 \ 0 而不是 \\\
。有关如何做到这一点的演练，请点击此处： http://developer.yahoo .com / hadoop / tutorial / module5.html＃fileformat

它的要点是你需要继承默认的 InputFormat 类或其任何子类，并定义您的拥有自定义规则的 RecordReader 。有关更多信息，请参阅InputFormat文档。

My data format uses \0 instead of new line. So default hadoop textLine reader dosn't work. How can I configure it to read lines separated by special character?

If it is impossible to configure LineReader, Maybe it is possible to apply specic stream processor(tr "\0" "\n") not sure how to do this.

解决方案

You can write your own InputFormat class that splits data on \0 instead of \n. For a walkthrough on how to do that, check here: http://developer.yahoo.com/hadoop/tutorial/module5.html#fileformat

The gist of it is that you need to subclass the default InputFormat class, or any of its subclasses, and define your own RecordReader with custom rules. For more on that, you can refer to the InputFormat documentation.

这篇关于如何阅读由特殊字符分隔的hadoop中的文本源的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！