AWS EMR Spark:错误:无法从JAR加载主类 [英] AWS EMR Spark: Error: Cannot load main class from JAR

问题描述

我正在尝试使用AWS控制台将Spark作业提交到AWS EMR集群.却失败了:

I am trying to submit a spark job to AWS EMR cluster using AWS console. But it fails with:

Cannot load main class from JAR.当我在AWS EMR控制台->添加步骤的Arguments选项中将主类指定为--class时，作业成功运行.

Cannot load main class from JAR. The job runs successfully when I specify main class as --class in Arguments option in AWS EMR Console-> Add Step.

在本地计算机上，当未按以下方式指定主类时，该作业似乎可以正常工作:

On the local machine, the job seems to work perfectly fine when no main class is specified as below:

 ./spark-submit /home/astro/spark-programs/SpotEMR/MyJob.jar

我已经使用运行配置将主类设置为jar.避免将主类传递为--class的主要原因是，我必须使用EMRAcivity在AWS Datapipeline中运行此作业.在AWS Datapipeline中，当前无法为正在提交的作业指定主类.

I have set main class to jar using run configuration. The main reason to avoid passing main class as --class is, I have to run this job in AWS Datapipeline using EMRAcivity. In AWS Datapipeline, currently there is no way to specify a main class to a job being submitted.

任何帮助将不胜感激.

推荐答案

实际上，您可以 通过EMRActivity和AWS Datapipeline传递作业的主类.

Actually, you can pass the job's main class with EMRActivity and AWS Datapipeline.

请参阅 https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-emractivity.html 使用step启动EMRActivity.

以及 https://docs.aws.amazon. com/emr/latest/ReleaseGuide/emr-spark-submit-step.html ，以使用主类的EMR step提交火花作业.

as well as https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-submit-step.html to submit a spark job using an EMR step with a main class.

该步骤如下所示:

command-runner.jar,spark-submit,--class,org.apache.spark.examples.SparkPi

这篇关于AWS EMR Spark:错误:无法从JAR加载主类的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！