AWS EMR Spark:错误:无法从JAR加载主类 [英] AWS EMR Spark: Error: Cannot load main class from JAR
问题描述
我正在尝试使用AWS控制台将Spark作业提交到AWS EMR集群.却失败了:
I am trying to submit a spark job to AWS EMR cluster using AWS console. But it fails with:
Cannot load main class from JAR
.当我在AWS EMR控制台->添加步骤的Arguments
选项中将主类指定为--class
时,作业成功运行.
Cannot load main class from JAR
. The job runs successfully when I specify main class as --class
in Arguments
option in AWS EMR Console-> Add Step.
在本地计算机上,当未按以下方式指定主类时,该作业似乎可以正常工作:
On the local machine, the job seems to work perfectly fine when no main class is specified as below:
./spark-submit /home/astro/spark-programs/SpotEMR/MyJob.jar
我已经使用运行配置将主类设置为jar.避免将主类传递为--class
的主要原因是,我必须使用EMRAcivity在AWS Datapipeline中运行此作业.在AWS Datapipeline中,当前无法为正在提交的作业指定主类.
I have set main class to jar using run configuration. The main reason to avoid passing main class as --class
is, I have to run this job in AWS Datapipeline using EMRAcivity. In AWS Datapipeline, currently there is no way to specify a main class to a job being submitted.
任何帮助将不胜感激.
推荐答案
实际上,您可以 通过EMRActivity和AWS Datapipeline传递作业的主类.
Actually, you can pass the job's main class with EMRActivity and AWS Datapipeline.
请参阅
https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-emractivity.html 使用step
启动EMRActivity
.
以及
https://docs.aws.amazon. com/emr/latest/ReleaseGuide/emr-spark-submit-step.html ,以使用主类的EMR step
提交火花作业.
as well as
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-submit-step.html to submit a spark job using an EMR step
with a main class.
该步骤如下所示:
command-runner.jar,spark-submit,--class,org.apache.spark.examples.SparkPi
这篇关于AWS EMR Spark:错误:无法从JAR加载主类的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!