spark-submit，如何指定log4j.properties [英] spark-submit, how to specify log4j.properties

问题描述

在spark-submit中，如何指定log4j.properties?

In spark-submit, how to specify log4j.properties ?

这是我的剧本.我已经尝试了所有组合，甚至只使用一个本地节点.但似乎未加载log4j.properties，所有调试级别的信息都已转储.

Here is my script. I have tried all of combinations and even just use one local node. but looks like the log4j.properties is not loaded, all debug level info was dumped.

current_dir=/tmp
DRIVER_JAVA_OPTIONS="-Dlog4j.configuration=file://${current_dir}/log4j.properties "

spark-submit \
--conf "spark.driver.extraClassPath=$current_dir/lib/*"  \
--conf "spark.driver.extraJavaOptions=-Djava.security.krb5.conf=${current_dir}/config/krb5.conf -Djava.security.auth.login.config=${current_dir}/config/mssqldriver.conf" \
--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file://${curent_dir}/log4j.properties " \
--class "my.AppMain" \
--files ${current_dir}/log4j.properties \
--master local[1] \
--driver-java-options "$DRIVER_JAVA_OPTIONS" \
--num-executors 4 \
--driver-memory 16g \
--executor-cores 10 \
--executor-memory 6g \
$current_dir/my-app-SNAPSHOT-assembly.jar

log4j属性:

log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

log4j.additivity.org=false

log4j.logger.org=WARN
parquet.hadoop=WARN
log4j.logger.com.barcap.eddi=WARN
log4j.logger.com.barcap.mercury=WARN
log4j.logger.yarn=WARN
log4j.logger.io.netty=WARN
log4j.logger.Remoting=WARN   
log4j.logger.org.apache.hadoop=ERROR

# this disables the table creation logging which is so verbose
log4j.logger.hive.ql.parse.ParseDriver=WARN

# this disables pagination nonsense when running in combined mode
log4j.logger.com.barcap.risk.webservice.servlet.PaginationFactory=WARN

推荐答案

如何传递本地log4j.properties文件

正如我从您的脚本中看到的那样，您想要:

How to pass local log4j.properties file

As I see from your script you want to:

1. 将本地log4j.properties传递给执行者
2. 将此文件用于节点的配置.

1. Pass local log4j.properties to executors
2. Use this file for node's configuration.

---

请注意有关--files设置的两件事:

---

Note two things about --files settings:

1. 使用--files上传到spark-cluster的文件将在根目录下可用，因此无需在file:log4j.properties中添加任何路径.
2. -files中列出的文件必须提供绝对路径！

现在修复片段很简单:

current_dir=/tmp
log4j_setting="-Dlog4j.configuration=file:log4j.properties"

spark-submit \
...
--conf "spark.driver.extraJavaOptions=${log4j_setting}" \
--conf "spark.executor.extraJavaOptions=${log4j_setting}" \
--class "my.AppMain" \
--files ${current_dir}/log4j.properties \
...
$current_dir/my-app-SNAPSHOT-assembly.jar

---

需要更多吗?

如果您想了解使用spark-submit时配置日志记录的其他方式，请访问我的其他详细答案: https://stackoverflow.com/a/55596389/1549135

---

Need more?

If you would like to read about other ways of configuring logging while using spark-submit, please visit my other detailed answer: https://stackoverflow.com/a/55596389/1549135

这篇关于spark-submit，如何指定log4j.properties的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！