如何从HDFS FileSystem执行hadoop jar？ [英] how to execute hadoop jar from HDFS FileSystem?

问题描述

我总是从本地文件系统执行作业，如下所示：

  hadoop jar / home / usr / jar / myjar。 jar com.test.TestMain 
  

如何执行相同的命令，但myjar.jar会在hdfs中？

类似于

  hadoop jar hdfs：/// home / usr / jar / myjar.jar com.test.TestMain 
  

但这当然不起作用。 ..

解决方案> 查看Hadoop的 bin / hadoop 执行一个jar文件它使用这个条件 -

  elif [$ COMMAND=jar];然后
 CLASS = org.apache.hadoop.util.RunJar 
  

这就是说它使用 RunJar 类来执行jar。
如果您看到RunJar类，则支持的参数是 -

  RunJar jarFile [mainClass] args ... 
  

其中，使用RunJar类中的以下代码访问jarFile

  int firstArg = 0; 
 String fileName = args [firstArg ++]; 
档案档案=新档案（档案名称）; 
  

因此， fileName 指向 jarFile ，所以我的视图中的 fileName 不能是上面所看到的HDFS路径，除非您能以某种方式在您的文件系统中挂载HDFS，以便可以通过java File class。

I'm always executing job from local File system as following

hadoop jar /home/usr/jar/myjar.jar com.test.TestMain 

How can I execute same command but myjar.jar will be in hdfs ?

Something like

hadoop jar hdfs:///home/usr/jar/myjar.jar com.test.TestMain 

But of course this doesn't work...

解决方案

Looking at bin/hadoop script for Hadoop, for executing a jar file it uses this condition --

elif [ "$COMMAND" = "jar" ] ; then
  CLASS=org.apache.hadoop.util.RunJar

This says it uses RunJar class to execute the jar. If you see the RunJar class, the arguments supported are --

RunJar jarFile [mainClass] args...

where, jarFile is accessed using the following code inside RunJar Class--

int firstArg = 0;
String fileName = args[firstArg++];
File file = new File(fileName);

Hence, fileName points to jarFile, so fileName in my view cannot be a HDFS path as seen above, unless you are somehow able to mount the HDFS in your file system, so that it becomes accessible by the java File class.

这篇关于如何从HDFS FileSystem执行hadoop jar？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！