Hadoop Maven依赖错误 [英] Hadoop Maven Dependency Error

问题描述

我正在尝试使用Maven构建一个Hadoop作业。当我不使用Maven并直接将Hadoop Jar依赖关系导入到eclipse中时，这个工作效果很好。

我还可以构建一个简单的jar（hello world type ）使用没有Hadoop依赖关系的Maven。

但是当我添加Hadoop依赖关系并运行jar时，我收到这个错误：

 线程main中的dataException java.lang.NoClassDefFoundError：org / apache / hadoop / conf / Configuration 
在org.graphhadoop.CreateAdjacency.Adjacency（CreateAdjacency .java：74）
在org.graphhadoop.Driver.main（Driver.java:12）
引起的：java.lang.ClassNotFoundException：org.apache.hadoop.conf.Configuration 
在java.net.URLClassLoader $ 1.run（URLClassLoader.java:202）
在java.security.AccessController.doPrivileged（本机方法）
在java.net.URLClassLoader.findClass（URLClassLoader.java:190 ）
在java.lang.ClassLoader.loadClass（ClassLoader.java:306）
 at sun.misc.Launcher $ AppClassLoader.loadClass（Launche r.java:301）
在java.lang.ClassLoader.loadClass（ClassLoader.java:247）
 ... 2更多
  

pom文件更改是这些：

 < dependency> 
< groupId> junit< / groupId> 
< artifactId> junit< / artifactId> 
< version> 3.8.1< / version> 
< scope> test< / scope> 
< / dependency> 
<依赖关系> 
< groupId> org.apache.hadoop< / groupId> 
< artifactId> hadoop-core< / artifactId> 
< version> 0.20.203.0< / version> 
< / dependency> 
< / dependencies> 
 
< build> 
< plugins> 
< plugin> 
< groupId> org.apache.maven.plugins< / groupId> 
< artifactId> maven-jar-plugin< / artifactId> 
< configuration> 
< archive> 
< manifest> 
< addClasspath> true< / addClasspath> 
< mainClass> org.graphhadoop.Driver< / mainClass> 
< / manifest> 
< / archive> 
< manifestEntries> 
< Class-Path> ../ target / classes< / Class-Path> 
< / manifestEntries> 
< / configuration> 
< / plugin> 
< / plugins> 
< / build> 
  

我哪里错了？

编辑：

现在，我在本地运行Jar - 不是在Hadoop上运行，而是在本地数据。

  java -jar jarname.jar 
  

解决方案

作为解决方案，有人遇到同样的问题，以下是解决方法：

这些依赖关系需要添加到pom.xml中：

 < dependency> 
< groupId> org.codehaus.jackson< / groupId> 
< artifactId> jackson-mapper-asl< / artifactId> 
< version> 1.5.2< / version> 
< / dependency> 
 
< dependency> 
< groupId> net.sf.json-lib< / groupId> 
< artifactId> json-lib< / artifactId> 
< version> 2.3< / version> 
< classifier> jdk15< / classifier> 
< / dependency> 
 
< dependency> 
< groupId> org.apache.hadoop< / groupId> 
< artifactId> hadoop-core< / artifactId> 
< version> 0.20.203.0< / version> 
< / dependency> 
< / dependencies> 
  

这些构建插件：

 <建立> 
< plugins> 
< plugin> 
< groupId> org.apache.maven.plugins< / groupId> 
< artifactId> maven-assembly-plugin< / artifactId> 
<执行> 
< execution> 
< goals> 
< goal> attached< / goal> 
< / goals> 
< phase> package< / phase> 
< configuration> 
< descriptorRefs> 
< descriptorRef> jar-with-dependencies< / descriptorRef> 
< / descriptorRefs> 
< archive> 
< manifest> 
< addClasspath> true< / addClasspath> 
< mainClass> org.lll ..< / mainClass> 
< / manifest> 
< manifestEntries> 
< Class-Path> ../ target / classes< / Class-Path> 
< / manifestEntries> 
< / archive> 
< / configuration> 
< / execution> 
< / executions> 
< / plugin> 
< plugin> 
< groupId> org.apache.maven.plugins< / groupId> 
< artifactId> maven-jar-plugin< / artifactId> 
< configuration> 
< archive> 
< manifest> 
< addClasspath> true< / addClasspath> 
< mainClass> org ...< / mainClass> 
< / manifest> 
< / archive> 
< manifestEntries> 
< Class-Path> ../ target / classes< / Class-Path> 
< / manifestEntries> 
< / configuration> 
< / plugin> 
< / plugins> 
< / build> 
  

I am trying to build a Hadoop job using Maven. This job works well when I don't use Maven and directly import the Hadoop Jar dependency into eclipse.

I'm also able to build a simple jar (hello world type) using Maven without the Hadoop dependency.

But when I add the Hadoop dependency, and run the jar, I get this error:

dataException in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
    at org.graphhadoop.CreateAdjacency.Adjacency(CreateAdjacency.java:74)
    at org.graphhadoop.Driver.main(Driver.java:12)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    ... 2 more

The pom file changes are these:

    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
        </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>0.20.203.0</version>
        </dependency>
  </dependencies>

<build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <configuration>
          <archive>
            <manifest>
              <addClasspath>true</addClasspath>
              <mainClass>org.graphhadoop.Driver</mainClass>
            </manifest>
          </archive>
          <manifestEntries>
          <Class-Path>../target/classes</Class-Path>
        </manifestEntries>
        </configuration>
      </plugin>
    </plugins>
  </build>

Where am I going wrong?

Edit:

For now, I am running the Jar locally - not on Hadoop, but on local data.

java -jar jarname.jar

解决方案

Just as a solution to somebody encountering the same problem, here's what resolved it:

These dependencies need to be added to pom.xml:

        <dependency>
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-mapper-asl</artifactId>
        <version>1.5.2</version>
        </dependency>

        <dependency>
        <groupId>net.sf.json-lib</groupId>
        <artifactId>json-lib</artifactId>
        <version>2.3</version>
        <classifier>jdk15</classifier>
        </dependency>

        <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>0.20.203.0</version>
        </dependency>
  </dependencies>

and these build plugins:

<build>
    <plugins>
    <plugin>
     <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-assembly-plugin</artifactId>
        <executions>
        <execution>
         <goals>
           <goal>attached</goal>
         </goals>
         <phase>package</phase>
         <configuration>
           <descriptorRefs>
             <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
          <archive>
            <manifest>
              <addClasspath>true</addClasspath>
              <mainClass>org.lll..</mainClass>
            </manifest>
      <manifestEntries>
          <Class-Path>../target/classes</Class-Path>
            </manifestEntries>
          </archive>
        </configuration>
        </execution>
    </executions>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <configuration>
          <archive>
            <manifest>
              <addClasspath>true</addClasspath>
          <mainClass>org...</mainClass>
            </manifest>
          </archive>
      <manifestEntries>
          <Class-Path>../target/classes</Class-Path>
        </manifestEntries>
        </configuration>
      </plugin>
    </plugins>  
</build>

这篇关于Hadoop Maven依赖错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！