为什么火花提交失败并显示"IllegalArgumentException:缺少应用程序资源"? [英] Why does spark-submit fail with "IllegalArgumentException: Missing application resource."?

问题描述

当我创建Spark应用程序的jar并尝试使用spark-submit运行它时，出现以下错误.

这是我以前运行的命令.

spark-submit --executor-memory 1g --jars s3://test-data-lab-users/spachari/test/test_2.10-1.0.jar

这是我遇到的错误.这是否表示我未在spark-submit中传递正确的参数?

Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.
    at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:276)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
    at org.apache.spark.launcher.Main.main(Main.java:86)
Command exiting with ret '1'

解决方案

tl; dr 删除--jars选项并重新开始.

java.lang.IllegalArgumentException:缺少应用程序资源.

您错过了该消息称为应用程序资源" 的...好...火花应用程序.

当您执行spark-submit并查看不同的命令行选项及其含义时，这一点更加明显.

./bin/spark-submit
Usage: spark-submit [options] <app jar | python file | R file> [app arguments]

这部分<app jar | python file | R file>是您错过的.

要重现问题，只需使用--jars选项执行spark-submit，而无需指定Spark应用程序的主jar或类.

$ ./bin/spark-submit --jars target/spark-parent_2.11-2.3.0-SNAPSHOT-tests.jar
Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.
    at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:274)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
    at org.apache.spark.launcher.Main.main(Main.java:86)

引用spark-submit --help，--jars是...

-jars JARS 以逗号分隔的jar列表，​​包括在驱动程序和执行程序的类路径中.

--jars在Spark应用程序依赖于其他jar文件(又称为依赖项)(即mysql-connect.jar您无法(或很可能不想)将其组装到您的超级jar)中时非常有用./p>

When I create the jar of my Spark application and try to run it using spark-submit, I am getting the following error.

This is the command I used to run.

spark-submit --executor-memory 1g --jars s3://test-data-lab-users/spachari/test/test_2.10-1.0.jar

This is the error i am getting. Does this mean I have not passed correct parameters in my spark-submit?

Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.
    at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:276)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
    at org.apache.spark.launcher.Main.main(Main.java:86)
Command exiting with ret '1'

解决方案

tl;dr Remove --jars option and start over.

java.lang.IllegalArgumentException: Missing application resource.

You missed your...well...Spark application that the message refers to as "application resource".

That's more obvious when you execute spark-submit and see the different command-line options and their meanings.

./bin/spark-submit
Usage: spark-submit [options] <app jar | python file | R file> [app arguments]

That part <app jar | python file | R file> is what you missed.

To reproduce your issue you can simply execute spark-submit with --jars options without specifying the main jar or class of a Spark application.

$ ./bin/spark-submit --jars target/spark-parent_2.11-2.3.0-SNAPSHOT-tests.jar
Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.
    at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:274)
    at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
    at org.apache.spark.launcher.Main.main(Main.java:86)

Quoting spark-submit --help, --jars is...

--jars JARS Comma-separated list of jars to include on the driver and executor classpaths.

--jars can be very helpful when a Spark application depends on additional jar files (aka dependencies), i.e. mysql-connect.jar that you cannot (or most likely don't want to) "assembly" to your uber jar.

这篇关于为什么火花提交失败并显示"IllegalArgumentException:缺少应用程序资源"?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！