版本与cloudera hadoop之间的版本不兼容 [英] Version Incompatibility between Spring batch and cloudera hadoop

查看：156
本文介绍了版本与cloudera hadoop之间的版本不兼容的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我正在尝试Spring批量字数统计程序，并面临如下版本问题： 

  ERROR [org.springframework。 batch.core.step.AbstractStep]  - <遇到执行步骤的错误> 
 java.lang.IncompatibleClassChangeError：找到接口org.apache.hadoop.mapreduce.Counter，但期望类

我使用Cloudera Hadoop2 cdh4.5.0 和Spring hadoop版本 1.0.1.RELEASE 。由于Spring批处理与 hadoop cdh4 兼容，因此无法确定确切的问题。



我的依赖关系树如下所示： 

  [信息] org.springframework.data:batch-wordcount:jar:0.0.1 
 [INFO] +  -  org.springframework：spring-context：jar：3.0.5.RELEASE：compile 
 [INFO] | +  -  org.springframework：spring-aop：jar：3.0.5.RELEASE：compile 
 [INFO] | +  -  org.springframework：spring-beans：jar：3.0.5.RELEASE：compile 
 [INFO] | +  -  org.springframework：spring-expression：jar：3.0.5.RELEASE：compile 
 [INFO] | \- org.springframework：spring-asm：jar：3.0.5.RELEASE：compile 
 [INFO] +  -  org.springframework：spring-core：jar：3.0.5.RELEASE：compile 
 [INFO] | \- commons-logging：commons-logging：jar：1.1.1：compile 
 [INFO] +  -  org.springframework：spring-tx：jar：3.0.5.RELEASE：compile 
 [INFO ] | +  -  aopalliance：aopalliance：jar：1.0：编译
 [INFO] +  -  org.springframework.batch：spring-batch-core：jar：2.2.0.RELEASE：compile 
 [INFO] | +  -  com.thoughtworks.xstream：xstream：jar：1.3：编译
 [INFO] | | \- xpp3：xpp3_min：jar：1.1.4c：编译
 [INFO] | +  -  org.codehaus.jettison：jettison：jar：1.1：编译
 [INFO] +  -  org.springframework.batch：spring-batch-infrastructure：jar：2.2.0.RELEASE：compile 
 [ INFO] | \- org.springframework.retry：spring-retry：jar：1.0.2.RELEASE：compile 
 [INFO] +  -  org.springframework.batch：spring-batch-test：jar：2.2.0.RELEASE ：编译
 [INFO] | +  - （junit：junit：jar：4.10：编译 - 与4.11冲突省略）
 [INFO] | +  - （org.springframework：spring-test：jar：3.2.0.RELEASE：compile  - 与3.0.5.RELEASE冲突时省略）
 [INFO] | +  -  commons-io：commons-io：jar：1.4：编译
 [INFO] | +  -  commons-dbcp：commons-dbcp：jar：1.2.2：编译
 [INFO] | | \- commons-pool：commons-pool：jar：1.3：编译
 [INFO] | \- commons-collections：commons-collections：jar：3.2：编译
 [INFO] +  -  junit：junit：jar：4.11：test（作用域未更新为编译）
 [信息] \- org.hamcrest：hamcrest-core：jar：1.3：test 
 [INFO] +  -  org.springframework.data:spring-data-hadoop:jar:1.0.2.RELEASE:compile 
 [INFO] +  -  org.apache.hadoop：hadoop-common：jar：2.0.0-cdh4.5.0：compile 
 [INFO] | +  -  org.apache.hadoop：hadoop-annotations：jar：2.0.0-cdh4.5.0：compile 
 [INFO] | | \ -jdk.tools:jdk.tools:jar:1.6:system 
 [INFO] | +  -  com.google.guava：guava：jar：11.0.2：compile 
 [INFO] | +  -  commons-net：commons-net：jar：3.1：编译
 [INFO] | +  -  javax.servlet：servlet-api：jar：2.5：编译
 [INFO] | +  -  org.mortbay.jetty：jetty：jar：6.1.26.cloudera.2：编译
 [INFO] | +  -  org.mortbay.jetty：jetty-util：jar：6.1.26.cloudera.2：compile 
 [INFO] | +  -  com.sun.jersey：jersey-core：jar：1.8：编译
 [INFO] | +  -  com.sun.jersey：jersey-json：jar：1.8：编译
 [INFO] | | +  -  com.sun.xml.bind：jaxb-impl：jar：2.2.3-1：compile 
 [INFO] | | | \ -javax.xml.bind：jaxb-api：jar：2.2.2：compile 
 [INFO] | | | \- javax.activation：activation：jar：1.1：编译
 [INFO] | | \-（com.sun.jersey：jersey-core：jar：1.8：compile  - 重复时省略）
 [INFO] | +  -  log4j：log4j：jar：1.2.17：编译
 [INFO] | +  -  org.apache.hadoop：cloudera-jets3t：jar：2.0.0-cdh4.5.0：compile 
 [INFO] | | \-（net.java.dev.jets3t：jets3t：jar：0.9.0：compile  - 与0.6.1冲突时省略）
 [INFO] | +  -  net.java.dev.jets3t：jets3t：jar：0.6.1：compile 
 [INFO] | | +  - （commons-codec：commons-codec：jar：1.3：compile  - 省略与1.4冲突）
 [INFO] | | +  - （commons-logging：commons-logging：jar：1.1.1：compile  - 重复时省略）
 [INFO] | | \-（commons-httpclient：commons-httpclient：jar：3.1：compile  - 重复时省略）
 [INFO] | +  - （junit：junit：jar：4.8.2：编译 - 与4.11冲突省略）
 [INFO] | +  -  commons-lang：commons-lang：jar：2.5：编译
 [INFO] | +  -  commons-configuration：commons-configuration：jar：1.6：编译
 [INFO] | | +  -  commons-digester：commons-digester：jar：1.8：编译
 [INFO] | | | +  -  commons-beanutils：commons-beanutils：jar：1.7.0：compile 
 [INFO] | | \- commons-beanutils：commons-beanutils-core：jar：1.8.0：compile 
 [INFO] | +  -  org.slf4j：slf4j-api：jar：1.6.1：编译
 [信息] | +  -  org.slf4j：slf4j-log4j12：jar：1.6.1：runtime 
 [信息] | +  -  org.codehaus.jackson：jackson-core-asl：jar：1.8.8：compile 
 [INFO] | +  -  org.codehaus.jackson：jackson-mapper -asl：jar：1.8.8：compile 
 [INFO] | +  -  net.sf.kosmosfs：kfs：jar：0.3：编译
 [INFO] | +  -  com.google.protobuf：protobuf-java：jar：2.4.0a：compile 
 [INFO] | +  -  org.apache.hadoop：hadoop-auth：jar：2.0.0-cdh4.5.0：compile 
 [INFO] | +  -  com.jcraft：jsch：jar：0.1.42：编译
 [INFO] | \- org.apache.zookeeper：zookeeper：jar：3.4.5-cdh4.5.0：compile 
 [INFO] | \- jline：jline：jar：0.9.94：compile 
 [INFO] +  -  org.apache.hadoop：hadoop-hdfs：jar：2.0.0-cdh4.5.0：compile 
 [INFO ] | +  -  tomcat：jasper-runtime：jar：5.5.23：编译
 [INFO] | | \- commons-el：commons-el：jar：1.0：编译
 [信息] +  -  org.apache.hadoop：hadoop-tools：jar：2.0.0-mr1-cdh4.5.0：compile 
 [INFO] | +  -  com.cloudera.cdh：hadoop-ant：pom：2.0.0-mr1-cdh4.5.0：compile 
 [INFO] | \- org.apache.hadoop：hadoop-core：jar：2.0.0-mr1-cdh4.5.0：compile 
 [INFO] | +  -  hsqldb：hsqldb：jar：1.8.0.10：compile 
 [INFO] | +  -  org.eclipse.jdt：core：jar：3.1.1：compile 
 [INFO] +  -  org.hamcrest：hamcrest-all：jar：1.1：compile 
 [INFO] +  -  org。 apache.hadoop：hadoop-client：jar：2.0.0-mr1-cdh4.5.0：compile 
 [INFO] +  -  org.apache.hadoop：hadoop-streaming：jar：2.0.0-mr1-cdh4。 5.0：compile 
 [INFO] \- org.springframework：spring-test：jar：3.0.5.RELEASE：compile

另外，重复依赖关系是否会导致任何错误？解析方案

您应该使用1.0 .2.RELEASE-cdh4版本与Cloudera CDH4 - 这些发行版的具体版本可在Spring存储库中找到： 

 < repository> 
< id> spring-releases< / id> 
< name> Spring版本库< /名称> 
< url> http：//repo.spring.io/libs-release< / url> 
< / repository>

更多信息： https://github.com/spring-projects/spring-hadoop/wiki/Build-with-Cloudera-CDH4


I was trying the spring batch word count program and faced a version issue like this : 

ERROR [org.springframework.batch.core.step.AbstractStep] - <Encountered an error executing the step>
java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.Counter, but class was expected

I use Cloudera Hadoop2 cdh4.5.0 and Spring hadoop version 1.0.1.RELEASE. I cant identify the exact problem as Spring batch is compatible with hadoop cdh4.

My dependency tree is as shown below.

[INFO] org.springframework.data:batch-wordcount:jar:0.0.1
[INFO] +- org.springframework:spring-context:jar:3.0.5.RELEASE:compile
[INFO] |  +- org.springframework:spring-aop:jar:3.0.5.RELEASE:compile
[INFO] |  +- org.springframework:spring-beans:jar:3.0.5.RELEASE:compile
[INFO] |  +- org.springframework:spring-expression:jar:3.0.5.RELEASE:compile
[INFO] |  \- org.springframework:spring-asm:jar:3.0.5.RELEASE:compile
[INFO] +- org.springframework:spring-core:jar:3.0.5.RELEASE:compile
[INFO] |  \- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] +- org.springframework:spring-tx:jar:3.0.5.RELEASE:compile
[INFO] |  +- aopalliance:aopalliance:jar:1.0:compile
[INFO] +- org.springframework.batch:spring-batch-core:jar:2.2.0.RELEASE:compile
[INFO] |  +- com.thoughtworks.xstream:xstream:jar:1.3:compile
[INFO] |  |  \- xpp3:xpp3_min:jar:1.1.4c:compile
[INFO] |  +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] +- org.springframework.batch:spring-batch-infrastructure:jar:2.2.0.RELEASE:compile
[INFO] |  \- org.springframework.retry:spring-retry:jar:1.0.2.RELEASE:compile
[INFO] +- org.springframework.batch:spring-batch-test:jar:2.2.0.RELEASE:compile
[INFO] |  +- (junit:junit:jar:4.10:compile - omitted for conflict with 4.11)
[INFO] |  +- (org.springframework:spring-test:jar:3.2.0.RELEASE:compile - omitted for conflict with 3.0.5.RELEASE)
[INFO] |  +- commons-io:commons-io:jar:1.4:compile
[INFO] |  +- commons-dbcp:commons-dbcp:jar:1.2.2:compile
[INFO] |  |  \- commons-pool:commons-pool:jar:1.3:compile
[INFO] |  \- commons-collections:commons-collections:jar:3.2:compile
[INFO] +- junit:junit:jar:4.11:test (scope not updated to compile)
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.springframework.data:spring-data-hadoop:jar:1.0.2.RELEASE:compile
[INFO] +- org.apache.hadoop:hadoop-common:jar:2.0.0-cdh4.5.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:2.0.0-cdh4.5.0:compile
[INFO] |  |  \- jdk.tools:jdk.tools:jar:1.6:system
[INFO] |  +- com.google.guava:guava:jar:11.0.2:compile
[INFO] |  +- commons-net:commons-net:jar:3.1:compile
[INFO] |  +- javax.servlet:servlet-api:jar:2.5:compile
[INFO] |  +- org.mortbay.jetty:jetty:jar:6.1.26.cloudera.2:compile
[INFO] |  +- org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.2:compile
[INFO] |  +- com.sun.jersey:jersey-core:jar:1.8:compile
[INFO] |  +- com.sun.jersey:jersey-json:jar:1.8:compile
[INFO] |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] |  |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] |  |  |     \- javax.activation:activation:jar:1.1:compile
[INFO] |  |  \- (com.sun.jersey:jersey-core:jar:1.8:compile - omitted for duplicate)
[INFO] |  +- log4j:log4j:jar:1.2.17:compile
[INFO] |  +- org.apache.hadoop:cloudera-jets3t:jar:2.0.0-cdh4.5.0:compile
[INFO] |  |  \- (net.java.dev.jets3t:jets3t:jar:0.9.0:compile - omitted for conflict with 0.6.1)
[INFO] |  +- net.java.dev.jets3t:jets3t:jar:0.6.1:compile
[INFO] |  |  +- (commons-codec:commons-codec:jar:1.3:compile - omitted for conflict with 1.4)
[INFO] |  |  +- (commons-logging:commons-logging:jar:1.1.1:compile - omitted for duplicate)
[INFO] |  |  \- (commons-httpclient:commons-httpclient:jar:3.1:compile - omitted for duplicate)
[INFO] |  +- (junit:junit:jar:4.8.2:compile - omitted for conflict with 4.11)
[INFO] |  +- commons-lang:commons-lang:jar:2.5:compile
[INFO] |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO] |  |  |  +- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.6.1:compile
[INFO] |  +- org.slf4j:slf4j-log4j12:jar:1.6.1:runtime
[INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
[INFO] |  +- net.sf.kosmosfs:kfs:jar:0.3:compile
[INFO] |  +- com.google.protobuf:protobuf-java:jar:2.4.0a:compile
[INFO] |  +- org.apache.hadoop:hadoop-auth:jar:2.0.0-cdh4.5.0:compile
[INFO] |  +- com.jcraft:jsch:jar:0.1.42:compile
[INFO] |  \- org.apache.zookeeper:zookeeper:jar:3.4.5-cdh4.5.0:compile
[INFO] |     \- jline:jline:jar:0.9.94:compile
[INFO] +- org.apache.hadoop:hadoop-hdfs:jar:2.0.0-cdh4.5.0:compile
[INFO] |  +- tomcat:jasper-runtime:jar:5.5.23:compile
[INFO] |  |  \- commons-el:commons-el:jar:1.0:compile
[INFO] +- org.apache.hadoop:hadoop-tools:jar:2.0.0-mr1-cdh4.5.0:compile
[INFO] |  +- com.cloudera.cdh:hadoop-ant:pom:2.0.0-mr1-cdh4.5.0:compile
[INFO] |  \- org.apache.hadoop:hadoop-core:jar:2.0.0-mr1-cdh4.5.0:compile
[INFO] |     +- hsqldb:hsqldb:jar:1.8.0.10:compile
[INFO] |     +- org.eclipse.jdt:core:jar:3.1.1:compile
[INFO] +- org.hamcrest:hamcrest-all:jar:1.1:compile
[INFO] +- org.apache.hadoop:hadoop-client:jar:2.0.0-mr1-cdh4.5.0:compile
[INFO] +- org.apache.hadoop:hadoop-streaming:jar:2.0.0-mr1-cdh4.5.0:compile
[INFO] \- org.springframework:spring-test:jar:3.0.5.RELEASE:compile

Also, Does ommited for duplicate dependencies cause any error ?

解决方案

You should use the 1.0.2.RELEASE-cdh4 version with Cloudera CDH4 - these distro specific releases are available in the Spring repository:

<repository>
  <id>spring-releases</id>
  <name>Spring Release Repository</name>
  <url>http://repo.spring.io/libs-release</url>
</repository>

More info here: https://github.com/spring-projects/spring-hadoop/wiki/Build-with-Cloudera-CDH4

这篇关于版本与cloudera hadoop之间的版本不兼容的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
分布式计算/Hadoop最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆