Hadoop外部罐子 [英] Hadoop external jars

问题描述

我试图在服务器上运行hadoop作业。该版本是0.20.2

我有大量的罐子，我正在运行

hadoop jar GenData.jar -libjars / path / jar1，path / jar2，...

下面的错误，相应的类在罐子里面。任何想法？

提前致谢

Altober

<在GenerateTrainningData.main（GenerateTrainningData.java:256）处
在sun.reflect.NativeMethodAccessorImpl.invoke0处
（Native Method）
at sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:57）
at sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）
at java.lang .reflect.Method.invoke（Method.java:606）
at org.apache.hadoop.util.RunJar.main（RunJar.java:197）
引起：java.lang.ClassNotFoundException：org .apache.avro.mapreduce.AvroKeyInputFormat $ b $在java.net.URLClassLoader $ 1.run（URLClassLoader.java:366）
在java.net.URLClassLoader $ 1.run（URLClassLoader.java:355）$ b $ java.util.AccessController.doPrivileged（Native Method）$ b $ java.net.URLClassLoader.findClass（URLC在java.lang.ClassLoader.loadClass中使用
（ClassLoader.java:425）$ java.util.ClassLoader.loadClass中的
（ClassLoader.java:358）

解决方案

看起来您从Hadoop客户端获得此异常，Mapreduce驱动程序代码执行发生在Client JVM中。在hadoop -libjars是一个通用的选项，用于添加依赖瓶到映射器/减速器。在你将Jars添加到Client集的情况下，你可以在执行hadoop命令之前设置以下环境变量。

  export HADOOP_CLASSPATH =< PATH_to_jar> / Jar1：< PATH_to_jar> / Jar2; 
  

（冒号：可用于指定多个罐子，在您的情况下，您可以添加包含类 org.apache.avro.mapreduce.AvroKeyInputFormat 的Jar）。

新编辑

首先，您需要找到包含jar类 org.apache.avro.mapreduce.AvroKeyInputFormat 。您可以在jar中找到该类 avro-mapred * .jar （获取 avro-mapred-version.jar的兼容版本来自internet）使用上面的命令在您的类路径中包含相同的内容。

I am trying to run a hadoop job on a server. The version is 0.20.2

I have a big amount of jars, I am running

hadoop jar GenData.jar -libjars /path/jar1,path/jar2,...

I am getting the error below, the corresponding classes are there inside the jars. Any idea?

thanks in advance

Altober

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/mapreduce/AvroKeyInputFormat at GenerateTrainningData.main(GenerateTrainningData.java:256) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) Caused by: java.lang.ClassNotFoundException: org.apache.avro.mapreduce.AvroKeyInputFormat at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

解决方案

Looks like you are getting this exception from Hadoop client side, Mapreduce driver code execution happens in Client JVM. In hadoop -libjars is a generic option which is used for adding dependent jars to mapper/reducer. In your case for adding Jars to Client set you may set the following environment variable,before executing the hadoop command.

export HADOOP_CLASSPATH=<PATH_to_jar>/Jar1:<PATH_to_jar>/Jar2;    

(colon ":" can be used for specifying more than 1 jars, In your case you may add the Jar that contains the class org.apache.avro.mapreduce.AvroKeyInputFormat).

New edits

Here first of all you need to find the jar containing the class org.apache.avro.mapreduce.AvroKeyInputFormat. You can find the class inside the jar avro-mapred*.jar (Get the compatible version of avro-mapred-version.jar from internet ) include the same in your classpath using the above command.

这篇关于Hadoop外部罐子的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！