Hadoop Streaming作业与二进制输入？ [英] Hadoop Streaming Job with binary input?

问题描述

我希望将一种格式的二进制文件转换为SequenceFile。

我有一个Python脚本，它可以在stdin上采用这种格式，并可以输出任何我想要的内容。

输入格式不是基于行的。各个记录是二进制本身，因此输出格式不能\t分隔或破碎成与\\\
线。

可否使用的Hadoop流接口消耗二进制格式？如何生成二进制输出格式？

我假设答案是否，除非我另有说明。

I wish to convert a binary file in one format to a SequenceFile.

I have a Python script that takes that format on stdin and can output whatever I want.

The input format is not line-based. The individual records are binary themselves, hence the output format cannot be \t delimited or broken into lines with \n.

Can I use the Hadoop Streaming interface to consume a binary format? How do I produce a binary output format?

I assume the answer is "No" unless I hear otherwise.

解决方案

You may consider using NullWritable as output, and generating the SequenceFile directly inside of your python script. You can look up the hadoop-python project in github to see candidate code: though it is admittedly bit large-ish/heavy it does handle the sequencefile generation.

这篇关于Hadoop Streaming作业与二进制输入？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！