键入来自map的键不匹配：expected .. Text，received ... LongWritable [英] Type mismatch in key from map: expected .. Text, received ... LongWritable
本文介绍了键入来自map的键不匹配：expected .. Text，received ... LongWritable的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我有一个简单的hadoop应用程序，它获得一个CSV文件，然后用，分隔条目，然后计算第一项。
以下是我的代码。
package com.bluedolphin;
import java.io.IOException;
import java.util.Iterator;
导入org.apache.hadoop.conf.Configuration;
导入org.apache.hadoop.conf.Configured;
导入org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class MyJob extends Configured implements Tool {
private final static LongWritable one = new LongWritable（1）;
public static class MapClass extends Mapper< Object，Text，Text，LongWritable> {
private text = new Text（）;
public void map（Object key，
Text value，
OutputCollector< Text，LongWritable> output，
Reporter reporter）throws IOException，InterruptedException {
String [] citation = value.toString（）。split（，）;
word.set（引用[0]）;
output.collect（word，one）;
}
}
public static class Reduce extends Reducer< Text，LongWritable，Text，LongWritable> {
public void reduce（
Text key，
Iterator< LongWritable> values，
OutputCollector< Text，LongWritable> output，
Reporter reporter）throws IOException，InterruptedException {
int sum = 0; $（b
$ b while（values.hasNext（））{
sum + = values.next（）。get（）;
}
output.collect（key，new LongWritable（sum））;
}
}
public static class Combiner扩展了Reducer< Text，IntWritable，Text，LongWritable> {
public void reduce（
Text key，
Iterator< LongWritable> values，
OutputCollector< Text，LongWritable> output，
Reporter reporter）throws IOException，InterruptedException {
int sum = 0; $（b
$ b while（values.hasNext（））{
sum + = values.next（）。get（）;
}
output.collect（key，new LongWritable（sum））;
$ b public int run（String [] args）throws Exception {
Configuration conf = getConf（）;
工作职位=新职位（conf，MyJob）;
job.setJarByClass（MyJob.class）;
Path in = new Path（args [0]）;
Path out = new Path（args [1]）;
FileInputFormat.setInputPaths（job，in）;
FileOutputFormat.setOutputPath（job，out）;
job.setMapperClass（MapClass.class）;
// job.setCombinerClass（Combiner.class）;
job.setReducerClass（Reduce.class）;
// job.setInputFormatClass（KeyValueInputFormat.class）;
job.setInputFormatClass（TextInputFormat.class）;
// job.setOutputFormatClass（KeyValueOutputFormat.class）;
job.setOutputKeyClass（Text.class）;
job.setOutputValueClass（LongWritable.class）;
System.exit（job.waitForCompletion（true）？0：1）;
返回0;
}
public static void main（String args []）throws Exception {
int res = ToolRunner.run（new Configuration（），new MyJob（），args）;
System.exit（res）;
}
}
这是错误：
11/12/16 22:16:58信息mapred.JobClient：任务ID：attempt_201112161948_0005_m_000000_0，状态：FAILED
java.io.IOException：键入来自map的键不匹配：expected org.apache.hadoop.io.Text，收到org.apache.hadoop.io.LongWritable $ b $在org.apache.hadoop.mapred.MapTask $ MapOutputBuffer.collect（MapTask.java ：1013）
at org.apache.hadoop.mapred.MapTask $ NewOutputCollector.write（MapTask.java:690）
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write（TaskInputOutputContext.java:80 ）
at org.apache.hadoop.mapreduce.Mapper.map（Mapper.java:124）
at org.apache.hadoop.mapreduce.Mapper.run（Mapper.java:144）
在org.apache.hadoop.mapred.MapTask.runNewMapper（MapTask.java:763）
at org.apache.hadoop.mapred.MapTask.run（MapTask.java:369）
at org。 apache.hadoop.mapred.Child $ 4.run（Child.java:259）$ java.util.AccessCon $ b $ troller.doPrivileged（Native方法）
在javax.security.auth.Subject.doAs（Subject.java:416）
在org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1059 ）
at org.apache.hadoop.mapred.Child.main（Child.java:253）
解决方案
在代码中修复一些事情
- 旧的（oahmapred）和新的API（oahmapreduce）不兼容，所以不应混用。
b
$ b
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
-
确保输入/输出mappers / redurs的类型是oahio.Writable。 Mapper的输入键是Object，使它成为LongWritable。
看起来Combiner和Reducer的功能是一样的，所以你不需要重复它。
job.setCombinerClass（Reducer.class）;
另外，您可以使用 WordCount 示例，您的要求与WordCount示例之间没有太大区别。
I have a simple hadoop application, which get one CSV file, then split the entry by ",", then count the first items.
The following is my code.
package com.bluedolphin;
import java.io.IOException;
import java.util.Iterator;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class MyJob extends Configured implements Tool {
private final static LongWritable one = new LongWritable(1);
public static class MapClass extends Mapper<Object, Text, Text, LongWritable> {
private Text word = new Text();
public void map(Object key,
Text value,
OutputCollector<Text, LongWritable> output,
Reporter reporter) throws IOException, InterruptedException {
String[] citation = value.toString().split(",");
word.set(citation[0]);
output.collect(word, one);
}
}
public static class Reduce extends Reducer<Text, LongWritable, Text, LongWritable> {
public void reduce(
Text key,
Iterator<LongWritable> values,
OutputCollector<Text, LongWritable> output,
Reporter reporter) throws IOException, InterruptedException {
int sum = 0;
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new LongWritable(sum));
}
}
public static class Combiner extends Reducer<Text, IntWritable, Text, LongWritable> {
public void reduce(
Text key,
Iterator<LongWritable> values,
OutputCollector<Text, LongWritable> output,
Reporter reporter) throws IOException, InterruptedException {
int sum = 0;
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new LongWritable(sum));
}
}
public int run(String[] args) throws Exception {
Configuration conf = getConf();
Job job = new Job(conf, "MyJob");
job.setJarByClass(MyJob.class);
Path in = new Path(args[0]);
Path out = new Path(args[1]);
FileInputFormat.setInputPaths(job, in);
FileOutputFormat.setOutputPath(job, out);
job.setMapperClass(MapClass.class);
// job.setCombinerClass(Combiner.class);
job.setReducerClass(Reduce.class);
// job.setInputFormatClass(KeyValueInputFormat.class);
job.setInputFormatClass(TextInputFormat.class);
// job.setOutputFormatClass(KeyValueOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(LongWritable.class);
System.exit(job.waitForCompletion(true) ? 0 : 1);
return 0;
}
public static void main(String args[]) throws Exception {
int res = ToolRunner.run(new Configuration(), new MyJob(), args);
System.exit(res);
}
}
This is the error:
11/12/16 22:16:58 INFO mapred.JobClient: Task Id : attempt_201112161948_0005_m_000000_0, Status : FAILED
java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1013)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:690)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
解决方案
Couple of things to be fixed in the code
- The old (o.a.h.mapred) and the new API (o.a.h.mapreduce) are not compatible, so they should not be mixed.
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
Make sure the input/output for the mappers/reducers are of the type o.a.h.io.Writable. The input key for the Mapper is Object, make it LongWritable.
Looks like the Combiner and the Reducer functionality is the same, so you need not repeat it.
job.setCombinerClass(Reducer.class);
Also, you could use the WordCount example, there is not much difference between your requirement and the WordCount example.
这篇关于键入来自map的键不匹配：expected .. Text，received ... LongWritable的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
