Oozie继续将旧版本的httpcore jar添加到classpath [英] Oozie keep adding a old version of httpcore jar to classpath

问题描述

由于Oozie将错误版本的httpcore jar添加到类路径中，所以我一直遇到异常.我尝试了其他选项，例如

I'm keep getting an exception because Oozie add a wrong version of httpcore jar to classpath. I tryed different options such as

oozie.launcher.mapreduce.task.classpath.user.precedence
oozie.launcher.mapreduce.user.classpath.first

oozie.launcher.mapreduce.task.classpath.user.precedence完全不响应，当我使用oozie.launcher.mapreduce.user.classpath.first时，应用程序甚至无法加载一个类.

oozie.launcher.mapreduce.task.classpath.user.precedence does not respond at all and when I use oozie.launcher.mapreduce.user.classpath.first, application cannot load even one class.

在类路径中，我可以看到http-core的两个版本.

In class path I can see two versions of http-core.

httpcore-4.4.1.jar
httpcore-4.2.4.jar

当应用程序以独立模式运行时，我没有收到该异常.

When application runs in stand alone mode, I'm not getting that exception.

例外:

    Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, java.lang.NoSuchFieldError: INSTANCE
org.apache.oozie.action.hadoop.JavaMainException: java.lang.NoSuchFieldError: INSTANCE
    at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
    at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NoSuchFieldError: INSTANCE
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.<clinit>(SSLConnectionSocketFactory.java:144)
    at microsoft.exchange.webservices.data.core.ExchangeServiceBase.createConnectionSocketFactoryRegistry(ExchangeServiceBase.java:244)
    at microsoft.exchange.webservices.data.core.ExchangeServiceBase.initializeHttpClient(ExchangeServiceBase.java:198)
    at microsoft.exchange.webservices.data.core.ExchangeServiceBase.<init>(ExchangeServiceBase.java:174)
    at microsoft.exchange.webservices.data.core.ExchangeServiceBase.<init>(ExchangeServiceBase.java:179)
    at microsoft.exchange.webservices.data.core.ExchangeService.<init>(ExchangeService.java:3729)
    at com.sonasoft.sonacloud.email.dispatcher.conn.EwsConnection.getConnection(EwsConnection.java:16)
    at com.sonasoft.sonacloud.email.dispatcher.conn.EwsConnection.getConnection(EwsConnection.java:10)
    at com.sonasoft.sonacloud.email.dispatcher.utils.EwsOperations.<init>(EwsOperations.java:47)
    at com.sonasoft.sonacloud.email.dispatcher.utils.EwsOperations.getInstance(EwsOperations.java:53)
    at com.sonasoft.sonacloud.email.dispatcher.main.MainClass.main(MainClass.java:41)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
    ... 15 more

Oozie客户端构建版本:4.2.0.2.3.2.0-2950

Oozie client build version: 4.2.0.2.3.2.0-2950

感谢您的帮助.

推荐答案

我们在HortonWorks Distro 2.3.2版中遇到了一个令人讨厌的问题(对他们感到羞耻):

We have had this nasty issue with HortonWorks distro 2.3.2 (shame on them):

• Oozie的启动器"工作始终在以下位置获得 httpcore 和 httpclient CLASSPATH作为Hadoop客户端的一部分
• Oozie的启动器"工作总是获得 httpcore 和 httpclient 捆绑在"Oozie" ShareLib中
• Hive/Hive2共享库在一个目录中包含 httpcore 和 httpclient 最新版本
• 从Hadoop的角度来看， user.classpath.first 适用于两者ShareLibs，因此有50/50的机会获得每个订单的正确订单JAR(总的机会是25/75)

• the Oozie "launcher" job always gets httpcore and httpclient in the CLASSPATH as part of the Hadoop client
• the Oozie "launcher" job always gets httpcore and httpclient as bundled in the "Oozie" ShareLib
• the Hive/Hive2 Sharelibs contain httpcore and httpclient in a more recent version
• from Hadoop point of view, user.classpath.first applies to both ShareLibs so it's a 50/50 chance of getting the right order for each JAR (so a 25/75 chance overall)

底线:我们必须

从"Oozie" ShareLib HDFS中
1. 删除 httpcore 和 httpclient dir(duh！)
为依赖Hive JARS 的所有动作
2. raise oozie.launcher.mapreduce.job.user.classpath.first 标志(即Hive动作，Hive2动作，Shell动作以某种方式调用JDBC驱动程序)等)

1. remove httpcore and httpclient from the "Oozie" ShareLib HDFS dir (duh!)
2. raise oozie.launcher.mapreduce.job.user.classpath.first flag for all actions relying on Hive JARS (i.e. Hive action, Hive2 action, Shell action calling the JDBC driver somehow, etc.)

后脚本-Oozie服务器在内存中保留了每个ShareLib中的JAR列表，因此在服务器运行时删除JAR会触发新作业中的错误.如果您不想停止Oozie服务器，则更新实时ShareLib的正确方法"是(a)在带有时间戳的新目录中创建新版本.[检查文档...] 和(b)告诉服务器使用 oozie admin -sharelibupdate

Post-scriptum -- the Oozie server keeps in memory a list of the JARs in each ShareLib, so that removing a JAR while the server is running will trigger errors in new jobs. If you don't want to stop the Oozie server, then the "proper way" to update a live ShareLib is to (a) create a new version in a new, time-stamped directory [check the documentation...] and (b) tell the server to resync on the newer libs with oozie admin -sharelibupdate

这篇关于Oozie继续将旧版本的httpcore jar添加到classpath的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！