在Mesos集群上运行Spark-Streaming时缺少Java系统属性 [英] Missing java system properties when running spark-streaming on Mesos cluster

问题描述

我将spark应用程序提交给mesos集群(以集群模式运行)，并通过"--drive-java-options=-Dkey=value -Dkey=value"传递java系统属性，但是这些系统属性在运行时不可用，似乎未设置. --conf "spark.driver.extraJavaOptions=-Dkey=value"也不起作用

I submit a spark app to mesos cluster(running in cluster mode), and pass java system property through "--drive-java-options=-Dkey=value -Dkey=value", however these system properties are not available at runtime, seems they are not set. --conf "spark.driver.extraJavaOptions=-Dkey=value" doesn't work either

更多详细信息: 该命令是

More details: the command is

bin/spark-submit --master mesos://10.3.101.119:7077 --deploy-mode cluster --class ${classname} --driver-java-options "-Dconfiguration.http=http://10.3.101.119:9090/application.conf" --conf "spark.executor.extraJavaOptions=-Dconfiguration.http=http://10.3.101.119:9090/application.conf" ${jar file}

我有一个由两个节点组成的mesos集群，一个节点同时运行主节点和从节点，另一个节点仅运行从节点.我在主节点上提交了spark应用程序.

I have a two-node mesos cluster, one node both runs master and slave, and the other runs slave only. I submit the spark application on master node.

内部，应用程序希望从java系统属性"configuration.http"中读取配置文件，如果该属性不可用，则应用程序将从类路径的根目录中加载默认文件.当我提交应用程序时，从日志中，我看到已加载默认配置文件.

Internally, the application hopes to read a configuration file from java system property "configuration.http", if the property is not available, the application will load a default file from the root of the classpath. When I submit the application, from the logs, i saw the default configuration file is loaded.

运行该应用程序的实际命令是

And the actual command to run the application is

"sh -c '/home/ubuntu/spark-1.6.0/bin/spark-submit --name ${appName} --master mesos://zk://10.3.101.184:2181/mesos/grant --driver-cores 1.0 --driver-memory 1024M --class ${classname} ./${jar file} '"

从这里您可以看到系统属性丢失

from here you can see the system property is lost

推荐答案

您可以查看

You might have a look at this blog post which recommends using an external properties file for this purpose:

$ vi app.properties
spark.driver.extraJavaOptions   -Dconfiguration.http=http://10.3.101.119:9090/application.conf
spark.executor.extraJavaOptions –Dconfiguration.http=http://10.3.101.119:9090/application.conf

然后尝试通过以下方式运行

Then try to run this via

bin/spark-submit --master mesos://10.3.101.119:7077 --deploy-mode cluster --class ${classname} —-properties-file app.properties ${jar file}

请参见

• 如何传递-D参数或作业是否需要环境变量?
• 从Apache Spark分离日志

• How to pass -D parameter or environment variable to Spark job?
• Separate logs from Apache spark

这篇关于在Mesos集群上运行Spark-Streaming时缺少Java系统属性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！