CDH4：版本冲突：找到接口org.apache.hadoop.mapreduce.Counter，但是应该使用类 [英] CDH4: Version conflict: Found interface org.apache.hadoop.mapreduce.Counter, but class was expected

问题描述

我正在尝试从CDH3升级到CDH4，并且从编译到运行时遇到版本冲突。
我遇到此错误：
线程 main中的异常java.lang.IncompatibleClassChangeError：找到了接口org.apache.hadoop.mapreduce.Counter，但期望使用类

I'm trying to upgrade from CDH3 to CDH4 and am getting a version conflict from compile to run time. I'm getting this error: Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.Counter, but class was expected

从谷歌搜索看来，我的代码似乎是针对Hadoop 1.x编译的，并且正在Hadoop 2.0上运行。
我正在同一Hadoop客户端上编译和运行该应用程序，因此应该全部是Hadoop 2.0。
这是我在客户端或此测试集群中的任何其他节点上运行 hadoop版本所得到的：
Hadoop 2.0.0-cdh4.4.0
Subversion文件： ///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.4.0/src/hadoop-common-project/hadoop-common -r c0eba6cd38c984557e96a16ccd7356b7de835e79
由jenkins于2013年9月3日星期二19:33:17编译
来自带有校验和的源代码ac7e170aa709b3ace13dc5f775487180
此命令使用/usr/lib/hadoop/hadoop-common-2.0.0-运行cdh4.4.0.jar

From googling it seems that my code is being compiled against Hadoop 1.x and is running on Hadoop 2.0. I'm compiling and running the app on the same Hadoop client, so it should all be Hadoop 2.0. Here's what I get from running "hadoop version" on the client or any of the other nodes in this test cluster: Hadoop 2.0.0-cdh4.4.0 Subversion file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.4.0/src/hadoop-common-project/hadoop-common -r c0eba6cd38c984557e96a16ccd7356b7de835e79 Compiled by jenkins on Tue Sep 3 19:33:17 PDT 2013 From source with checksum ac7e170aa709b3ace13dc5f775487180 This command was run using /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.4.0.jar

这是我pom.xml中的相关依赖项和存储库：

Here's the relevant dependencies and repos from my pom.xml:

<dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>2.0.0-mr1-cdh4.4.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.0.0-mr1-cdh4.4.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.mrunit</groupId>
            <artifactId>mrunit</artifactId>
            <version>0.9.0-incubating</version>
            <classifier>hadoop1</classifier>
 </dependency>
...
 <repository>
            <id>cloudera</id>
            <name>cloudera</name>
            <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
        </repository>

在编译我的应用程序之后，该目录存在于我的Maven存储库中（我不希望这样做，似乎是我的问题）：
.m2 / repository / org / apache / hadoop / hadoop-core / 0.20.2-cdh3u0

And after compiling my app, this directory exists in my maven repo (which I wouldn't expect and seems like my problem): .m2/repository/org/apache/hadoop/hadoop-core/0.20.2-cdh3u0

但是我搜索了应用程序mavendependency：tree，并且对hadoop-core的唯一引用是：
org.apache.hadoop：hadoop-core：jar：2.0.0-mr1-cdh4.4.0：编译
仅供参考-我也尝试过排除MRunit依赖性和所有测试代码，但结果相同。

But I searched the apps maven dependency:tree and the only reference to hadoop-core is: org.apache.hadoop:hadoop-core:jar:2.0.0-mr1-cdh4.4.0:compile FYI - I've also tried excluding the MRunit dependency and all my test code but same results.

有人可以建议我m使我的版本从编译过渡到运行时（为什么在构建应用程序时hadoop-core / 0.20.2-cdh3u0被拉低）？
我相信，自从我安装了hadoop MRv1以来，它应该是在编译和运行时使用的Hadoop 2.0代码……但是我仍然对MRv1中的版本感到困惑。

Can anyone advise how I'm getting my version crossed from compile to run time (why is hadoop-core/0.20.2-cdh3u0 getting pulled down when I build the app) ? I believe since I've installed hadoop MRv1, it should be Hadoop 2.0 code at compile and run time ... but I'm still somewhat confused by the versioning in MRv1.

推荐答案

弄清楚了-
mvn依赖项：tree 本身并没有向我显示要求cdh3的依赖项，但 mvn依赖项：tree -Dverbose 显示了相互冲突的依赖项。
从那里我在pom中添加了一个排除项，现在工作就可以运行了。

Figured it out - mvn dependency:tree on it's own did not show me the dependency that was calling out for cdh3, but mvn dependency:tree -Dverbose revealed the conflicting dependency. From there I added an exclusion to the pom and now the job runs.

这篇关于CDH4：版本冲突：找到接口org.apache.hadoop.mapreduce.Counter，但是应该使用类的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！