Hadoop无法找到映射器类 [英] Hadoop can not find the mapper class
本文介绍了Hadoop无法找到映射器类的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我是Hadoop的新手，我想运行MapReduce作业。但是，我得到了hadoop无法找到映射器类的错误。这是错误：
信息mapred.JobClient：任务ID：attempt_201608292140_0023_m_000000_0，状态：FAILED
java.lang。 RuntimeException：java.lang.ClassNotFoundException：TransMapper1
at org.apache.hadoop.conf.Configuration.getClass（Configuration.java:857）
at org.apache.hadoop.mapreduce.JobContext.getMapperClass（JobContext .java：199）
at org.apache.hadoop.mapred.MapTask.runNewMapper（MapTask.java:718）
at org.apache.hadoop.mapred.MapTask.run（MapTask.java:364 ）
at org.apache.hadoop.mapred.Child $ 4.run（Child.java:255）
at java.security.AccessController.doPrivileged（Native Method）
at javax.security。 auth.Subject.doAs（Subject.java:422）
我检查了我的jar文件的权限，没事。这里是jar文件的许可：
-rwxrwxrwx。
这是启动mapreduce作业的代码的代码：
import java.io.File;
导入org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class mp {
public static void main（String [] args）throws Exception {
Job job1 = new Job（）;
job1.setJarByClass（mp.class）;
FileInputFormat.addInputPath（job1，new Path（args [0]））;
String oFolder = args [0] +/ output;
FileOutputFormat.setOutputPath（job1，new Path（oFolder））;
job1.setMapperClass（TransMapper1.class）;
job1.setReducerClass（TransReducer1.class）;
job1.setMapOutputKeyClass（LongWritable.class）;
job1.setMapOutputValueClass（DnaWritable.class）;
job1.setOutputKeyClass（LongWritable.class）;
job1.setOutputValueClass（Text.class）;
$ b $ p
$ b 这里是映射类（TransMapper1）：
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
public class TransMapper1扩展了Mapper< LongWritable，Text，LongWritable，DnaWritable> {
$ b @Override
public void map（LongWritable key，Text value，Context context）throws IOException，InterruptedException {
String line = value.toString（）;
StringTokenizer tokenizer = new StringTokenizer（line）;
LongWritable bamWindow = new LongWritable（Long.parseLong（tokenizer.nextToken（）））;
LongWritable read = new LongWritable（Long.parseLong（tokenizer.nextToken（）））;
LongWritable refWindow = new LongWritable（Long.parseLong（tokenizer.nextToken（）））;
IntWritable chr = new IntWritable（Integer.parseInt（tokenizer.nextToken（）））;
DoubleWritable dist = new DoubleWritable（Double.parseDouble（tokenizer.nextToken（）））;
DnaWritable dnaW = new DnaWritable（bamWindow，read，refWindow，chr，dist）;
context.write（bamWindow，dnaW）;
$ / code>
我使用以下命令编译包： p>
javac -classpath $ MR_HADOOPJAR $ {rootPath} mp / src / *。java
jar cvfm $ mpJar $ MR_MANIFEST $ { rootPath} mp / src / *。class
这是jar -tf mp / src / mp.jar命令：
META-INF /
META-INF / MANIFEST.MF
mnt / miczfs / tide / mp / src / DnaWritable.class
mnt / miczfs / tide / mp / src / mp.class
mnt / miczfs / tide / mp / src / TransMapper1.class
mnt /miczfs/tide/mp/src/TransMapper2.class
mnt / miczfs / tide / mp / src / TransReducer1.class
mnt / miczfs / tide / mp / src / TransReducer2.class
我正在运行这项工作：
mpJar = $ {rootPath} mp / src / mp.jar
mp_exec = mp
export HADOOP_CLASSPATH = $ mpJar
hadoop $ mp_exec< input path>
另外，我也尝试了这个命令：
hadoop jar $ mp_exec<输入路径>
我改变了创建jar文件到这个命令的方式：
jar cf $ mpJar $ MR_MANIFEST $ {rootPath} mp / src / *。class
在这种变化下，错误已经改变为：
线程main中的异常java.lang.ClassNotFoundException：在java.net.URLClassLoader.findClass处的mp
（URLClassLoader.java:381）$ java.util.ClassLoader.loadClass处的
（ClassLoader.java:424 ）在java.lang.ClassLoader.loadClass中的
（ClassLoader.java:357）在java.lang.Class.forName0中的
（本地方法）$在java.lang.Class.forName中的
.java：348）
at org.apache.hadoop.util.RunJar.main（RunJar.java:153）
所以，在我的问题出现之前，程序找不到映射类，现在找不到主类!!!任何想法？
谢谢大家
解决方案 HADOOP_CLASSPATH必须指定JAR文件所在的文件夹，因此无法找到类定义。
I am new to Hadoop and I want to run a MapReduce job. However, I've got the error that the hadoop can not find the mapper class. This is the error:
INFO mapred.JobClient: Task Id : attempt_201608292140_0023_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: TransMapper1
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:718)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
I checked the permission of my jar file and it is fine. Here it is the permission of the jar file:
-rwxrwxrwx.
Here it is the code for the code that launches the mapreduce job:
import java.io.File;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class mp{
public static void main(String[] args) throws Exception {
Job job1 = new Job();
job1.setJarByClass(mp.class);
FileInputFormat.addInputPath(job1, new Path(args[0]));
String oFolder = args[0] + "/output";
FileOutputFormat.setOutputPath(job1, new Path(oFolder));
job1.setMapperClass(TransMapper1.class);
job1.setReducerClass(TransReducer1.class);
job1.setMapOutputKeyClass(LongWritable.class);
job1.setMapOutputValueClass(DnaWritable.class);
job1.setOutputKeyClass(LongWritable.class);
job1.setOutputValueClass(Text.class);
}
}
And here it is the mapper class (TransMapper1):
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
public class TransMapper1 extends Mapper<LongWritable, Text, LongWritable, DnaWritable> {
@Override
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
LongWritable bamWindow = new LongWritable(Long.parseLong(tokenizer.nextToken()));
LongWritable read = new LongWritable(Long.parseLong(tokenizer.nextToken()));
LongWritable refWindow = new LongWritable(Long.parseLong(tokenizer.nextToken()));
IntWritable chr = new IntWritable(Integer.parseInt(tokenizer.nextToken()));
DoubleWritable dist = new DoubleWritable(Double.parseDouble(tokenizer.nextToken()));
DnaWritable dnaW = new DnaWritable(bamWindow,read,refWindow,chr,dist);
context.write(bamWindow,dnaW);
}
}
I am compiling the package using following commands:
javac -classpath $MR_HADOOPJAR ${rootPath}mp/src/*.java
jar cvfm $mpJar $MR_MANIFEST ${rootPath}mp/src/*.class
This is the results of jar -tf mp/src/mp.jar command:
META-INF/
META-INF/MANIFEST.MF
mnt/miczfs/tide/mp/src/DnaWritable.class
mnt/miczfs/tide/mp/src/mp.class
mnt/miczfs/tide/mp/src/TransMapper1.class
mnt/miczfs/tide/mp/src/TransMapper2.class
mnt/miczfs/tide/mp/src/TransReducer1.class
mnt/miczfs/tide/mp/src/TransReducer2.class
And I am running the job with this:
mpJar=${rootPath}mp/src/mp.jar
mp_exec=mp
export HADOOP_CLASSPATH=$mpJar
hadoop $mp_exec <input path>
Also, I tried this command too:
hadoop jar $mp_exec <input path>
I changed the way I create the jar file into this command:
jar cf $mpJar $MR_MANIFEST ${rootPath}mp/src/*.class
And with this change, the the error has been changed to this:
Exception in thread "main" java.lang.ClassNotFoundException: mp
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
So, before my problem was that the program can not find the mapper class, right now it can not find the main class!!! any thoughts??
Thank you guys
解决方案 HADOOP_CLASSPATH must specify the folder where the JAR file is located, due to which it is unable to find the class definition.
这篇关于Hadoop无法找到映射器类的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
