在EMR Spark应用程序中加载道具文件 [英] Load props file in EMR Spark Application

问题描述

我正在尝试使用:-

command-runner.jar,spark-submit,--deploy-mode,cluster,--properties-file,s3://spark-config-test/myprops.conf,--num-executors,5,--executor-cores,2,--class,com.amazon.Main,#{input.directoryPath}/SWALiveOrderModelSpark-1.0-super.jar

但是，我收到以下异常:-

However, I am getting the following exception:-

线程主"中的异常java.lang.IllegalArgumentException:无效 属性文件's3://spark-config-test/myprops.conf''.在 org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241) 在 org.apache.spark.launcher.AbstractCommandBuilder.loadPropertiesFile(AbstractCommandBuilder.java:284) 在 org.apache.spark.launcher.AbstractCommandBuilder.getEffectiveConfig(AbstractCommandBuilder.java:264) 在 org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:233) 在组织

Exception in thread "main" java.lang.IllegalArgumentException: Invalid properties file 's3://spark-config-test/myprops.conf''. at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241) at org.apache.spark.launcher.AbstractCommandBuilder.loadPropertiesFile(AbstractCommandBuilder.java:284) at org.apache.spark.launcher.AbstractCommandBuilder.getEffectiveConfig(AbstractCommandBuilder.java:264) at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:233) at org

这是从S3加载文件的正确方法吗?

Is this the correct way to load file from S3?

推荐答案

您不能直接从S3加载属性文件.相反，您需要将属性文件下载到主节点的某个位置，然后提交引用该节点上的本地路径的spark作业.您可以使用command-runner.jar来运行aws cli实用程序进行下载.

You can't load a properties file directly from S3. Instead you will need to download the properties file to your master node somewhere, then submit the spark job referencing the local path on that node. You can do the download by using command-runner.jar to run the aws cli utility.

这篇关于在EMR Spark应用程序中加载道具文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！