如何覆盖每个驱动程序星火的log4j.properties？ [英] How to override Spark's log4j.properties per driver?

问题描述

我试图重写星火默认log4j.properties，但还没有任何运气。我尝试了加入以下引发提交：

I'm trying to override Spark's default log4j.properties but haven't had any luck. I tried the adding the following to spark-submit:

--conf "spark.executor.extraJavaOptions=Dlog4j.configuration=/tmp/log4j.properties"  
--conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=/tmp/log4j.properties"

但似乎并没有工作。我使用--files在选项火花提交也试过并且还似乎没有工作。有没有人有日志记录设置，使您拥有每一个驱动log4j.properties文件，而不是使用默认的？

but that didn't seem to work. I also tried using --files option in spark-submit and that also didn't seem to work. Has anyone got logging setup so you have a log4j.properties file per driver and not using the default?

我使用Mesos和马拉松跑星火驱动程序。我不知道该--files选项，我找不到，这不正是什么它是如何使用的任何实例

I'm using Mesos and Marathon to run the Spark driver. I wasn't sure of the --files option and I couldn't find any examples of how it's used and what it does exactly.

我也想提一提，我手动上传log4j.properties文件到我的所有节点是有我的测试变化。

I would also like to mention that I manually uploaded the log4j.properties file to all my nodes that had my changes for testing.

星火版本1.1.0是作为现在。

Version of Spark is 1.1.0 as of right now.

推荐答案

有关司机/壳可以用此设置在 - 驱动程序的Java选项运行时， 火花壳或火花提交脚本。

For the driver/shell you can set this with the --driver-java-options when running spark-shell or spark-submit scripts.

在星火您不能设置 - 配置spark.driver.extraJavaOptions ，因为设置的之后的JVM的开始。当使用火花提交脚本 - 驱动程序的Java选项替换这些选项进入发射的JVM（如的Java -Dblah MyClass的）运行的驱动程序。

In Spark you cannot set --conf spark.driver.extraJavaOptions because that is set after the JVM is started. When using the spark submit scripts --driver-java-options substitutes these options into the launch of the JVM (e.g. java -Dblah MyClass) that runs the driver.

请注意， -Dlog4j.configuration 属性应该是一个有效的URL，因此，如果它从什么地方上的文件系统，使用文件： URL。如果资源变量不能转换为一个URL，例如由于MalformedURLException的，那么log4j的将搜索从classpath中的资源。

Note that the -Dlog4j.configuration property should be a valid URL, so if its from somewhere on your file system use file: URL. If the resource variable cannot be converted to a URL, for example due to a MalformedURLException, then log4j will search for the resource from the classpath.

例如，使用自定义log4j.properties文件;

For example, to use a custom log4j.properties file;

./spark-shell --driver-java-options "-Dlog4j.configuration=file:///etc/spark/my-conf/log4j.warnonly.properties"

这篇关于如何覆盖每个驱动程序星火的log4j.properties？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！