Hadoop库在mapreduce时间冲突 [英] Hadoop library conflict at mapreduce time

问题描述

我有一个使用Hadoop API的jar来启动各种远程mapreduce作业（即，im不使用命令行来启动作业）。执行各种作业的服务jar是使用maven的jar-with-dependencies构建的。

我的作业都运行良好，除了使用commons-codec 1.7，I get：
$ b

FATAL org.apache.hadoop.mapred.Child：运行child的错误：java.lang.NoSuchMethodError：org.apache.commons .codec.binary.Base64.encodeAsString（[B] Ljava / lang / String;

我认为这是因为我的jar包含commons-codec 1.7，而我的Hadoop安装的lib有commons-codec 1.4 ...

是他们指示Hadoop使用分布式commons-codec 1.7的任何方式这是作为一个工作依赖项分发的），而不是hadoop 1.0.3核心库中的commons-codec 1.4？

非常感谢！

注意：从我的Hadoop库文件夹中删除commons-codec-1.4.jar可以解决问题，但看起来并不太理智。希望有更好的选择。

解决方案

两种方法：

• 您应该能够从hadoop依赖项中排除commons-codec，并为commons-codec添加另一个显式依赖项。尝试将范围设置为的hadoop罐子包括在内。这假定这些罐子将在运行时类路径中。

I have a jar that uses the Hadoop API to launch various remote mapreduce jobs (ie, im not using the command-line to initiate the job). The service jar that executes the various jobs is built with maven's "jar-with-dependencies".

My jobs all run fine except one that uses commons-codec 1.7, I get:

FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.NoSuchMethodError: org.apache.commons.codec.binary.Base64.encodeAsString([B)Ljava/lang/String;

I think this is because my jar is including commons-codec 1.7 whereas my Hadoop install's lib has commons-codec 1.4 ...

Is their any way to instruct Hadoop to use the distributed commons-codec 1.7 (I assume this is distributed as a job dependency) rather than the commons-codec 1.4 in the hadoop 1.0.3 core lib?

Many thanks!

Note: Removing commons-codec-1.4.jar from my Hadoop library folder does solve the problem, but doesn't seem too sane. Hopefully there is a better alternative.

解决方案

Two approaches:

• You should be able to exclude commons-codec from within the hadoop dependency and add another explicit dependency for commons-codec
• Try setting the scope to provided so that none of the hadoop jars get included. This assumes that those jars would be in the runtime class path.

这篇关于Hadoop库在mapreduce时间冲突的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！