在Hadoop作业中保存和读取复杂的可写值 [英] Save and read complicated Writable value in Hadoop job

问题描述

我需要将复杂的值(可写实现)从第一个map-reduce作业的输出移至其他map-reduce作业的输入.第一份工作的结果已保存到文件中.文件可以存储文本数据或BytesWritable(具有默认输出\输入格式).因此，我需要一些简单的方法来将Writable转换为Text或BytesWritable并从中转换.是否存在?还有其他替代方法吗? 非常感谢

I need to move complicated value (implements Writable) from output of 1st map-reduce job to input of other map-reduce job. Results of 1st job saved to file. File can store Text data or BytesWritable (with default output \ input formats). So I need some simple way to convert my Writable to Text or To BytesWritable and from it. Does it exists? Any alternative way to do this? Thanks a lot

推荐答案

用户irW是正确的，请使用SequenceFileOutputFormat. SequenceFile解决了这个确切的问题，而无需转换为Text Writable.设置作业时，请使用job.setOutputKeyClass和job.setOutputValueClass设置您正在使用的Writable子类:

User irW is correct, use SequenceFileOutputFormat. SequenceFile solves this exact problem, without converting to Text Writable. When setting up your job, use job.setOutputKeyClass and job.setOutputValueClass to set the Writable subclasses you are using:

job.setOutputKeyClass(MyWritable1.class);
job.setOutputValueClass(MyWritable2.class);
job.setOutputFormatClass(SequenceFileOutputFormat.class);

这将使用Hadoop SequenceFile格式存储您的可写文件.然后在下一个作业中，使用SequenceFileInputFormat:

This will use the Hadoop SequenceFile format to store your Writables. Then in your next job, use SequenceFileInputFormat:

job.setInputFormatClass(SequenceFileInputFormat.class);

然后，此作业中映射器的输入键和值将是您最初在上一个作业中指定为输出的两个Writable类.

Then the input key and value for the mapper in this job will be the two Writable classes you originally specified as output in the previous job.

请注意，正确实现复杂的Writable子类至关重要.除了必须有一个空的构造函数外，还必须实现write和readFields方法，以使该类中的所有Writable字段也可以写入和读取其信息.

Note, it is crucial that your complex Writable subclass is implemented correctly. Beyond the fact that you must have an empty constructor, the write and readFields methods must be implemented such that any Writable fields in the class also write and read their information.

这篇关于在Hadoop作业中保存和读取复杂的可写值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！