Hadoop NoSuchMethodError apache.commons.cli [英] Hadoop NoSuchMethodError apache.commons.cli

问题描述

我正在使用 hadoop-2.7.2 ，并用IntelliJ做了一个MapReduceJob。在我的工作中，我使用 apache.commons.cli-1.3.1 ，并将该lib放入jar中。

当我在我的Hadoop集群上使用MapReduceJob时，我有一个 NoSuchMethodError ：

线程main中的异常java.lang.NoSuchMethodError：
org.apache.commons.cli.Option.builder（Ljava / lang / String;）Lorg / apache / commons / cli / Option $ Builder;

我不明白，因为该方法存在于类选项和类选项从 commons-cli.jar 中提取我的应用程序jar。此外，我没有与我的其他图书馆有这个问题。

谢谢你的时间。

解决方案

这个问题似乎与如何加载 classloader 类。由于静态生成器类在 common-cli 1.4 中，而一些 hadoop 依赖关系仍然指旧版本 - 发生问题。

在我的情况下，通过更改jar文件添加顺序来解决问题shell脚本中的classpath负责在程序执行前设置环境。之前，我将jar添加到类路径中，如

  CLASSPATH =< Hadoop Jars>：< Common CLI jar>：$ CLASSPATH 
  

更改为

  CLASSPATH =< Common CLI jar>：< Hadoop Jars>：$ CLASSPATH 
  

它解决了问题。

I'm using hadoop-2.7.2 and I did a MapReduceJob with IntelliJ. In my job, I'm using apache.commons.cli-1.3.1 and I put the lib in the jar.

When I use the MapReduceJob on my Hadoop cluster I have a NoSuchMethodError:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;

I don't understand because the method exist in the class Option and the class Option is extracted from the commons-cli.jar to my application jar. Moreover, I don't have this issue with my others libraries.

Thank you for your time.

解决方案

The problem seems to be related to how the classloader is loading the classes. Because the static Builder class was in common-cli 1.4, while some of the hadoop dependencies were still referring to older version - the issue occurred.

In my case, the issue resolved by changing the sequence of jar file addition into the classpath in the shell script responsible for setting up environment before program execution. Earlier, I was adding the jar into classpath like

CLASSPATH=<Hadoop Jars>:<Common CLI jar>:$CLASSPATH

is changed to

CLASSPATH=<Common CLI jar>:<Hadoop Jars>:$CLASSPATH

and it fixed the problem.

这篇关于Hadoop NoSuchMethodError apache.commons.cli的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！