如何在hadoop中运行jar文件？ [英] How to run a jar file in hadoop?

问题描述

我使用此博客中的java文件创建了一个jar文件以下语句

javac -classpath /usr/local/hadoop/hadoop-core-1.0.3.jar -d / home / hduser / dir Dictionary.java

/usr/lib/jvm/jdk1.7.0_07/bin/jar cf Dictionary.jar / home / hduser / dir

现在我已经尝试在hadoop中运行这个jar，并命令和试用各种命令。
$ b <1> code> hduser @ ubuntu：〜$ / usr / local / hadoop / bin / hadoop jar Dictionary.jar

输出：

 警告：$ HADOOP_HOME已弃用。 
 
 RunJar jarFile [mainClass] args ... 
  

2。 hduser @ ubuntu：〜$ / usr / local / hadoop / bin / hadoop jar Dictionary.jar Dictionary

$ b

输出：

 警告：$ HADOOP_HOME已弃用。 
 
线程main中的异常java.lang.ClassNotFoundException：在java.net.URLClassLoader中的字典
 $ 1.run（URLClassLoader.java:366）$ java 
。 $ URBlassLoader $ 1.run（URLClassLoader.java:355）
在java.security.AccessController.doPrivileged（本地方法）$ b $在java.net.URLClassLoader.findClass（URLClassLoader.java:354）
在java.lang.ClassLoader.loadClass（ClassLoader.java:423）$ b $在java.lang.ClassLoader.loadClass（ClassLoader.java:356）
在java.lang.Class.forName0（本地方法） 
在java.lang.Class.forName（Class.java:264）
在org.apache.hadoop.util.RunJar.main（RunJar.java:149）
  

我如何在hadoop中运行jar？我有我的程序需要的正确的DFS位置。

问题在于你创建jar的地方。

基本上，打包到jar中的目录会在定位主类文件时混淆jar文件。相反，如果你尝试这样做：

  /usr/lib/jvm/jdk1.7.0_07/bin/jar cf Dictionary.jar / home / hduser / dir / Dictionary.class 
  

即将类文件打包到jar中，然后运行：

  / usr / local / hadoop / bin / hadoop jar Dictionary.jar字典
  

只要在您的类中有一个名为Dictionary的主函数，它就可以正常工作。

问题在于当你将一个完整的目录打包到一个jar文件中时，jar还需要知道目录结构来定位这个类文件。为此，我们需要有一个定义良好的包层次来定义类的位置。因此，在将 / home / hduser / dir / 打包到jar中时，jar并不知道位于此目录结构深处的类文件的位置。为此，您需要根据目录结构将包名称添加到 .java 文件中，例如 home.hduser.dir 并在运行 hadoop jar 命令的同时使用包结构指定类名称，例如 home.hduser.dir.Dictionary 。

I have created a jar file using the java file from this blog using following statements

javac -classpath /usr/local/hadoop/hadoop-core-1.0.3.jar -d /home/hduser/dir Dictionary.java

/usr/lib/jvm/jdk1.7.0_07/bin/jar cf Dictionary.jar /home/hduser/dir

Now i have tried running this jar in hadoop by hit and trial of various commands

1hduser@ubuntu:~$ /usr/local/hadoop/bin/hadoop jar Dictionary.jar

Output:

Warning: $HADOOP_HOME is deprecated.

RunJar jarFile [mainClass] args...  

2.hduser@ubuntu:~$ /usr/local/hadoop/bin/hadoop jar Dictionary.jar Dictionary

Output:

Warning: $HADOOP_HOME is deprecated.

Exception in thread "main" java.lang.ClassNotFoundException: Dictionary
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)

How can i run the jar in hadoop? I have the right DFS Locations as per needed by my program.

解决方案

I was able to reproduce your problem. The problem is where you are creating the jar.

Basically, the directory that you are packaging into the jar is confusing the jar file in locating the main class file. Instead if you try doing :

/usr/lib/jvm/jdk1.7.0_07/bin/jar cf Dictionary.jar /home/hduser/dir/Dictionary.class

i.e. package the class file specifically into the jar and then run:

/usr/local/hadoop/bin/hadoop jar Dictionary.jar Dictionary

It just works fine provided that you have a main function in your class called Dictionary.

The problem is when you package a full directory inside a jar then the jar also needs to be aware of the directory structure to locate the class file. For this, we need to have a well defined package hierarchy to define the class location. So, when you are packaging /home/hduser/dir/ into the jar, the jar is not aware of the location of the class file which is located deep inside this directory structure. For this you need to add a package name to your .java file according to the directory structure , for example home.hduser.dir and while running the hadoop jar command specify the class name with the package structure, for example home.hduser.dir.Dictionary.

这篇关于如何在hadoop中运行jar文件？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！