示例Oozie作业可通过Hue进行，但不能通过命令行进行操作:未找到SparkMain [英] Example Oozie job works from Hue, but not from command line: SparkMain not found

问题描述

我已经成功运行了Hue Oozie工作流编辑器(在Cloudera 5.5.1 QuickStart VM中)提供的示例Spark工作流(通过启动Spark Java程序复制文件").

I've successfully run the example Spark workflow ("Copy a file by launching a Spark Java program") provided in the Hue Oozie workflow editor (in the Cloudera 5.5.1 QuickStart VM).

我现在正尝试使用oozie命令行工具手动运行它:

I'm now trying to run it manually using the oozie commandline tool:

oozie job -oozie http://localhost:11000/oozie -config job.properties -run

工作流XML基本不变-我已将其复制到HDFS并具有以下job.properties:

The workflow XML is basically unchanged - I have copied it to HDFS and have the following job.properties:

nameNode=hdfs://localhost:8020
jobTracker=localhost:8032
oozie.wf.application.path=/user/cloudera/workflows/spark-scala/spark-scala.xml
input=/user/hue/oozie/workspaces/data/sonnets.txt
output=here

该作业被接受并显示在Hue Web仪表板上，但在几秒钟后被杀死，并且日志报告:

The job is accepted and appears in the Hue web dashboard, but is killed after a few seconds, and the logs report:

发射器异常:java.lang.ClassNotFoundException:未找到类org.apache.oozie.action.hadoop.SparkMain

Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found

这是什么问题?

推荐答案

Oozie不包含Spark动作的库，默认情况下-您需要将以下内容添加到job.properties:

Oozie doesn't include the libraries for the Spark action, by default - you need to add the following to the job.properties:

oozie.use.system.libpath=true

(单击"Hue仪表板"中以前成功的Hue工作流程，可以选择配置"选项卡以查看Hue提供的属性)

(Clicking on the previously successful Hue workflow in the Hue Dashboard, you can select the Configuration tab to see the properties that Hue has provided)

这篇关于示例Oozie作业可通过Hue进行，但不能通过命令行进行操作:未找到SparkMain的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！