在Oozie中使用Hive上下文失败的Spark作业 [英] Spark job that use hive context failing in oozie
问题描述
在我们的管道之一中，我们使用spark(java)进行聚合，并使用oozie对其进行编排. 该管道使用以下几行将汇总的数据写入ORC文件.
In one of our pipelines we are doing aggregation using spark(java) and it is orchestrated using oozie. This pipelines writes the aggregated data to an ORC file using the following lines.
HiveContext hc = new HiveContext(sc);
DataFrame modifiedFrame = hc.createDataFrame(aggregateddatainrdd, schema);
modifiedFrame.write().format("org.apache.spark.sql.hive.orc").partitionBy("partition_column_name").save(output);
当oozie作业中的spark动作被触发时，它会引发以下异常
When the spark action in the oozie job gets triggered it throws the following exception
Oozie Launcher失败，主班 [org.apache.oozie.action.hadoop.SparkMain]，main()抛出异常， org.apache.hadoop.hive.shims.HadoopShims.isSecurityEnabled()Z java.lang.NoSuchMethodError: org.apache.hadoop.hive.shims.HadoopShims.isSecurityEnabled()Z
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, org.apache.hadoop.hive.shims.HadoopShims.isSecurityEnabled()Z java.lang.NoSuchMethodError: org.apache.hadoop.hive.shims.HadoopShims.isSecurityEnabled()Z
但是多次重新运行工作流程后，同样成功了.
But the same is getting succeeded after rerunning the workflow multiple times.
在运行时和编译时，所有必需的jar都就位.
All the necessary jars are in place both during run time and compile time.
这是我的第一个spark应用程序，我无法理解该问题.
This is my first spark app and i am not able understand the issue.
有人可以帮助我更好地理解该问题以及针对该问题的可能解决方案.
Could someone help me in understanding the issue better and possible solution for the same.
推荐答案
多次重新运行工作流程后，同样成功"
听起来，您的Spark作业已与运行集群的 版本不同/与Hadoop客户端捆绑在一起；结果，CLASSPATH中存在冲突的JAR，并且您的作业随机失败，具体取决于首先选择的JAR.
Sounds like you have compiled / bundled your Spark job with a Hadoop client in a different version than the one running the cluster; as a result there are conflicting JARs in the CLASSPATH, and your job fails randomly depending on which JAR is picked up first.
可以肯定的是，选择一个成功的Oozie作业和一个失败的作业，获取动作的外部ID"(标记为job_*******_****，但指的是YARN ID application_******_****)并检查两个作业的YARN日志.您应该在Java CLASSPATH中看到JAR实际顺序的差异.
To be sure, choose one Oozie job that succeeded and one job that failed, get the "external ID" of the action (which is labeled job_*******_**** but refers to the YARN ID application_******_****) and inspect the YARN logs for both jobs. You should see a differenece in the actual order of JARs in the Java CLASSPATH.
如果确实如此，请尝试组合使用
If that's indeed the case, then try a combination of
- 在Oozie操作中，将属性
oozie.launcher.mapreduce.user.classpath.first设置为true(对于Spark驱动程序) - 在Spark配置中，将属性
spark.yarn.user.classpath.first设置为true(对于执行者)
- in Oozie action, set property
oozie.launcher.mapreduce.user.classpath.firsttotrue(for the Spark driver) - in Spark config, set property
spark.yarn.user.classpath.firsttotrue(for the executors)
您可以猜测 user.classpath.first 的含义是什么！！
You can guess what the user.classpath.first implies...!
但是，如果有冲突的JAR实际上不在Hadoop客户端中，而是在Oozie ShareLib中，则可能不起作用.从YARN的角度来看，Oozie是客户端"，您不能在Oozie从其ShareLib交付的内容与从Spark作业中交付的内容之间设置优先级.
But, it might not work, if the conflicting JARs are actually not in the Hadoop client but in the Oozie ShareLib. And from YARN point of view, Oozie is the "client", you cannot set a precedence between what Oozie ships from its ShareLib and what it ships from your Spark job.
在这种情况下，您将不得不在Java项目中使用适当的依赖项，并匹配将要运行的Hadoop版本-这只是常识，您认为吗? ?
In that case you would have to use the proper dependencies in your Java project, and match the Hadoop version you will be running against -- that's just common sense, don't you think?!?
这篇关于在Oozie中使用Hive上下文失败的Spark作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
