键入来自map的键不匹配：expected org.apache.hadoop.io.Text，recieved org.apache.hadoop.io.LongWritable [英] Type mismatch in key from map: expected org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable

问题描述

我试图在java中运行map / reducer。以下是我的文件

WordCount.java

  package counter; 
 
 
 public class WordCount extends Configured implements Tool {
 $ b $ public int run（String [] arg0）throws Exception {
 Configuration conf = new Configuration（ ）; 
 
工作职位=新职位（conf，wordcount）; 
 
 job.setOutputKeyClass（Text.class）; 
 job.setOutputValueClass（IntWritable.class）; 
 
 job.setMapperClass（WordCountMapper.class）; 
 job.setReducerClass（WordCountReducer.class）; 
 
 job.setInputFormatClass（TextInputFormat.class）; 
 job.setOutputFormatClass（TextOutputFormat.class）; 
 
 FileInputFormat.addInputPath（job，new Path（counterinput））; 
 //擦除上一次运行输出（如果有的话）
 FileSystem.get（conf）.delete（new Path（counteroutput），true）; 
 FileOutputFormat.setOutputPath（job，new Path（counteroutput））; 
 
 job.waitForCompletion（true）; 
返回0; 
 
 $ b $ public static void main（String [] args）throws Exception {
 int res = ToolRunner.run（new Configuration（），new WordCount（），args）; 
 System.exit（res）; 
 
 
 $ b WordCountMapper.java 
  public class WordCountMapper extends 
 Mapper< LongWritable，Text，Text，IntWritable> {
 private static static IntWritable one = new IntWritable（1）; 
私人文字=新文字（）; 
 
 public void map（LongWritable key，Text value，OutputCollector< Text，IntWritable> output，Reporter reporter）
 throws IOException，InterruptedException {
 System.out.println（hi ）; 
 String line = value.toString（）; 
 StringTokenizer tokenizer = new StringTokenizer（line）; 
 while（tokenizer.hasMoreTokens（））{
 word.set（tokenizer.nextToken（））; 
 output.collect（word，one）; 
} 
} 
} 
  
 WordCountReducer.java 
  public class WordCountReducer扩展了Reducer< Text，IntWritable，Text，IntWritable> {
 public void reduce（Text key，Iterator< IntWritable> values，
 OutputCollector< Text，IntWritable> output，Reporter reporter）throws IOException，InterruptedException {
 System.out.println（hello ）; 
 int sum = 0; 
 while（values.hasNext（））{
 sum + = values.next（）。get（）; 
} 
 output.collect（key，new IntWritable（sum））; 
} 
} 
  
我收到错误消息 
 
 
  13/06/23 23:13:25 INFO jvm.JvmMetrics：使用
初始化JVM度量标准processName = JobTracker，sessionId = 
 
 13/06/23 23:13:25警告mapred.JobClient：使用GenericOptionsParser解析
参数。应用程序应该实现相同的工具。 
 13/06/23 23:13:26 INFO input.FileInputFormat：要输入的总输入路径：1 
 13/06/23 23:13:26信息mapred.JobClient：正在运行的作业：job_local_0001 
 13/06/23 23:13:26 INFO input.FileInputFormat：要输入的总输入路径：1 
 13/06/23 23:13:26信息mapred.MapTask：io.sort.mb = 100 
 13/06/23 23:13:26信息mapred.MapTask：data buffer = 79691776/99614720 
 13/06/23 23:13:26 INFO mapred.MapTask：record buffer = 262144 / 327680 
 13/06/23 23:13:26警告mapred.LocalJobRunner：job_local_0001 
 java.io.IOException：类型与映射关键字不匹配：expected org.apache.hadoop.io.Text，
收到org.apache.hadoop.io.LongWritable $ b $在org.apache.hadoop.mapred.MapTask $ MapOutputBuffer.collect（MapTask.java:845）
在org.apache.hadoop.mapred .MapTask $ NewOutputCollector.write（MapTask.java:541）
 at org。 
 apache.hadoop.mapreduce.TaskInputOutputContext.write（TaskInputOutputContext.java:80）
 at org.apache.hadoop.mapreduce.Mapper.map（Mapper.java:124）
 at org。 apache.hadoop.mapreduce.Mapper.run（Mapper.java:144）
 at org.apache.hadoop.mapred.MapTask.runNewMapper（MapTask.java:621）
 at org.apache.hadoop。 mapred.MapTask.run（MapTask.java:305）
 at org.apache.hadoop.mapred.LocalJobRunner $ Job.run（LocalJobRunner.java:177）
 13/06/23 23:13： 27信息mapred.JobClient：map 0％reduce 0％
 13/06/23 23:13:27信息mapred.JobClient：工作完成：job_local_0001 
 13/06/23 23:13:27信息mapred.JobClient：计数器：0 
  
我认为它无法找到Mapper和reducer类。我在主类中编写了代码
它获取了默认的Mapper和Reducer类。
 
解决方案
将这2个在你的代码行：
  job.setMapOutputKeyClass（Text.class）; 
 job.setMapOutputValueClass（IntWritable.class）; 
  
您正在使用发出LongWritable键的 TextOutputFormat 和文本值默认情况下，但您正在发送文本作为键和IntWritable作为值。你需要告诉这个名誉。
 
 
 
 HTH 
 
I am trying to run a map/reducer in java. Below are my files

WordCount.java
package counter;


public class WordCount extends Configured implements Tool {

public int run(String[] arg0) throws Exception {
    Configuration conf = new Configuration();

    Job job = new Job(conf, "wordcount");

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    job.setMapperClass(WordCountMapper.class);
    job.setReducerClass(WordCountReducer.class);

    job.setInputFormatClass(TextInputFormat.class);
    job.setOutputFormatClass(TextOutputFormat.class);

    FileInputFormat.addInputPath(job, new Path("counterinput"));
    // Erase previous run output (if any)
    FileSystem.get(conf).delete(new Path("counteroutput"), true);
    FileOutputFormat.setOutputPath(job, new Path("counteroutput"));

    job.waitForCompletion(true);
    return 0;
}   

public static void main(String[] args) throws Exception {
    int res = ToolRunner.run(new Configuration(), new WordCount(), args);
    System.exit(res);

    }
}
WordCountMapper.java
public class WordCountMapper extends
Mapper<LongWritable, Text, Text, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(LongWritable key, Text value, OutputCollector<Text,IntWritable> output, Reporter reporter)
    throws IOException, InterruptedException {
        System.out.println("hi");
    String line = value.toString();
    StringTokenizer tokenizer = new StringTokenizer(line);
    while (tokenizer.hasMoreTokens()) {
        word.set(tokenizer.nextToken());
        output.collect(word, one);
        }
    }
}
WordCountReducer.java
public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
    public void reduce(Text key, Iterator<IntWritable> values,
        OutputCollector<Text,IntWritable> output, Reporter reporter) throws IOException, InterruptedException {
        System.out.println("hello");
        int sum = 0;
        while (values.hasNext()) {
            sum += values.next().get();
        }
        output.collect(key, new IntWritable(sum));
    }
}
I am getting following error
13/06/23 23:13:25 INFO jvm.JvmMetrics: Initializing JVM Metrics with  
processName=JobTracker, sessionId=

13/06/23 23:13:25 WARN mapred.JobClient: Use GenericOptionsParser for parsing the 
arguments. Applications should implement Tool for the same.
13/06/23 23:13:26 INFO input.FileInputFormat: Total input paths to process : 1
13/06/23 23:13:26 INFO mapred.JobClient: Running job: job_local_0001
13/06/23 23:13:26 INFO input.FileInputFormat: Total input paths to process : 1
13/06/23 23:13:26 INFO mapred.MapTask: io.sort.mb = 100
13/06/23 23:13:26 INFO mapred.MapTask: data buffer = 79691776/99614720
13/06/23 23:13:26 INFO mapred.MapTask: record buffer = 262144/327680
13/06/23 23:13:26 WARN mapred.LocalJobRunner: job_local_0001
java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, 
recieved org.apache.hadoop.io.LongWritable
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:845)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:541)
at org.
apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
13/06/23 23:13:27 INFO mapred.JobClient:  map 0% reduce 0%
13/06/23 23:13:27 INFO mapred.JobClient: Job complete: job_local_0001
13/06/23 23:13:27 INFO mapred.JobClient: Counters: 0
I think it is not able to find Mapper and reducer class. I have written the code in main class, 
It is getting default Mapper and reducer class.
解决方案
Add these 2 lines in your code :
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
You are using TextOutputFormat which emits LongWritable key and Text value by default, but you are emitting Text as key and IntWritable as value. You need to tell this to the famework.

HTH

                        
这篇关于键入来自map的键不匹配：expected org.apache.hadoop.io.Text，recieved org.apache.hadoop.io.LongWritable的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！