Hadoop类型与映射期望值中的键不匹配文本接收值LongWritable [英] Hadoop type mismatch in key from map expected value Text received value LongWritable
本文介绍了Hadoop类型与映射期望值中的键不匹配文本接收值LongWritable的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
任何人都有任何想法，为什么我会得到这个错误？我看了很多其他类似的帖子，但其中大多数并不适用于我，我也尝试了几个适用于我的解决方案，但他们没有工作，我确信我只是错过了一些愚蠢的东西，感谢您的帮助
chris @ chrisUHadoop：/usr/local/hadoop-1.0.3/build$ hadoop MaxTemperature 1901 output4
12/07/03 17:23:08警告mapred.JobClient：使用GenericOptionsParser解析参数。应用程序应该实现相同的工具。
12/07/03 17:23:08 INFO input.FileInputFormat：要输入的总输入路径：1
12/07/03 17:23:08 INFO util.NativeCodeLoader：加载native-hadoop library
12/07/03 17:23:08 WARN snappy.LoadSnappy：Snappy本地库未加载
12/07/03 17:23:09信息mapred.JobClient：正在运行的作业：job_201207031642_0005
12/07/03 17:23:10信息mapred.JobClient：map 0％reduce 0％
12/07/03 17:23:28信息mapred.JobClient：Task Id：attempt_201207031642_0005_m_000000_0，Status：FAILED
java.io.IOException：键入来自map的键不匹配：expected org.apache.hadoop.io.Text，收到org.apache.hadoop.io.LongWritable $ b $ org.apache.hadoop.mapred .MapTask $ MapOutputBuffer.collect（MapTask.java:1014）
at org.apache.hadoop.mapred.MapTask $ NewOutputCollector.write（MapTask.java:691）
at org.apache.hadoop.mapreduce .TaskInputOutputContext.write（TaskInputOutputContext.java:80）
at org.apache.hadoop.mapreduce.Mapper.map（Mapper.java:124）
at org.apache.hadoop.mapred uce.Mapper.run（Mapper.java:144）
at org.apache.hadoop.mapred.MapTask.runNewMapper（MapTask.java:764）
at org.apache.hadoop.mapred.MapTask。运行（MapTask.java:370）
在org.apache.hadoop.mapred.Child $ 4.run（Child.java:255）
在java.security.AccessController.doPrivileged（本地方法）
at javax.security.auth.Subject.doAs（Subject.java:415）
at org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1121）
at org.apache .hadoop.mapred.Child.main（Child.java:249）
程序：
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
public class MaxTemperatureMapper扩展了Mapper< LongWritable，Text，Text，IntWritable> {
private static final int MISSING = 9999;
@Override
public void map（LongWritable key，Text value，Context context）throws IOException，InterruptedException
{
String line = value.toString（）;
String year = line.substring（15,19）;
int airTemperature;
if（line.charAt（87）=='+'）
{
airTemperature = Integer.parseInt（line.substring（88,92））;
}
else
{
airTemperature = Integer.parseInt（line.substring（87,92））;
}
字符串质量= line.substring（92,93）; （airTemperature！= MISSING& amp; quality.matches（[01459]））
{
context.write（new Text（year），new IntWritable（airTemperature）） ;
}
}
}
import java.io.IOException;
导入org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
public class MaxTemperatureReducer扩展了Reducer< Text，IntWritable，Text，IntWritable>
{
@Override
public void reduce（Text key，Iterable< IntWritable> values，Context context）throws IOException，InterruptedException
{
int maxValue = Integer.MIN_VALUE ; （IntWritable value：values）
{
maxValue = Math.max（maxValue，value.get（））;
}
context.write（key，new IntWritable（maxValue））;
}
}
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class MaxTemperature
{
public static void main（String [] args）throws Exception
{
if（args.length ！= 2）
{
System.out.println（Usage：MaxTemperature< input path>< output path>）;
System.exit（-1）;
}
Job job = new Job（）;
job.setJarByClass（MaxTemperature.class）;
job.setJobName（Max temperature）;
FileInputFormat.addInputPath（job，new Path（args [0]））;
FileOutputFormat.setOutputPath（job，new Path（args [1]））;
job.setOutputKeyClass（Text.class）;
job.setOutputValueClass（IntWritable.class）;
System.exit（job.waitForCompletion（true）？0：1）;
$ b
解决方案
您似乎缺少一些配置属性：
你的具体错误信息是因为身份映射器只是输出它传入的相同键/值类型，在这种情况下可能是LongWritable类型的键和Text类型的值（因为您没有定义输入格式，默认值可能是TextInputFormat）。在您的配置中，您已将输出键类型定义为文本，但映射器正在输出LongWritable，因此会显示错误消息。
Anyone have any idea why I would be getting this error? I have looked at alot of other similar posts but most of them did not apply to me, I also tried the few solutions that were posted that did apply to me but they did not work, I'm sure I'm just missing something stupid, thanks for the help
chris@chrisUHadoop:/usr/local/hadoop-1.0.3/build$ hadoop MaxTemperature 1901 output4
12/07/03 17:23:08 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/07/03 17:23:08 INFO input.FileInputFormat: Total input paths to process : 1
12/07/03 17:23:08 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/07/03 17:23:08 WARN snappy.LoadSnappy: Snappy native library not loaded
12/07/03 17:23:09 INFO mapred.JobClient: Running job: job_201207031642_0005
12/07/03 17:23:10 INFO mapred.JobClient: map 0% reduce 0%
12/07/03 17:23:28 INFO mapred.JobClient: Task Id : attempt_201207031642_0005_m_000000_0, Status : FAILED
java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1014)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Program:
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
public class MaxTemperatureMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
private static final int MISSING = 9999;
@Override
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException
{
String line = value.toString();
String year = line.substring(15,19);
int airTemperature;
if (line.charAt(87) == '+')
{
airTemperature = Integer.parseInt(line.substring(88,92));
}
else
{
airTemperature = Integer.parseInt(line.substring(87,92));
}
String quality = line.substring(92,93);
if (airTemperature != MISSING && quality.matches("[01459]"))
{
context.write(new Text(year), new IntWritable(airTemperature));
}
}
}
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
public class MaxTemperatureReducer extends Reducer<Text, IntWritable, Text, IntWritable>
{
@Override
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException
{
int maxValue = Integer.MIN_VALUE;
for (IntWritable value : values)
{
maxValue = Math.max(maxValue, value.get());
}
context.write(key, new IntWritable(maxValue));
}
}
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class MaxTemperature
{
public static void main(String[] args) throws Exception
{
if (args.length != 2)
{
System.out.println("Usage: MaxTemperature <input path> <output path>");
System.exit(-1);
}
Job job = new Job();
job.setJarByClass(MaxTemperature.class);
job.setJobName("Max temperature");
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
解决方案
You appear to be missing a number of configuration properties:
- Mapper and Reducer classes? - if not defined, you'll be defaulted to the 'Identity' Mapper / Reducer
Your specific error message is because the identity mapper just outputs the same key / value types it was passed in, in this case probably a key of type LongWritable and value of type Text (as you haven't defined an Input format, the default is probably TextInputFormat). In your configuration you have defined the output key type as Text, but the mapper is outputting LongWritable, hence the error message.
这篇关于Hadoop类型与映射期望值中的键不匹配文本接收值LongWritable的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
