如何设置默认的火花记录级别? [英] How can set the default spark logging level?
问题描述
我将pyspark应用程序从自己工作站上的pycharm启动到8节点集群.此集群还具有在spark-defaults.conf和spark-env.sh
I launch pyspark applications from pycharm on my own workstation, to a 8 node cluster. This cluster also has settings encoded in spark-defaults.conf and spark-env.sh
这就是我获取spark上下文变量的方式.
This is how I obtain my spark context variable.
spark = SparkSession \
.builder \
.master("spark://stcpgrnlp06p.options-it.com:7087") \
.appName(__SPARK_APP_NAME__) \
.config("spark.executor.memory", "50g") \
.config("spark.eventlog.enabled", "true") \
.config("spark.eventlog.dir", r"/net/share/grid/bin/spark/UAT/SparkLogs/") \
.config("spark.cores.max", 128) \
.config("spark.sql.crossJoin.enabled", "True") \
.config("spark.executor.extraLibraryPath","/net/share/grid/bin/spark/UAT/bin/vertica-jdbc-8.0.0-0.jar") \
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
.config("spark.logConf", "true") \
.getOrCreate()
sc = spark.sparkContext
sc.setLogLevel("INFO")
我想查看日志中正在使用的有效配置.这行
I want to see the effective config that is being used in my log. This line
.config("spark.logConf", "true") \
应该导致spark api将其有效配置作为INFO记录到日志中,但是默认日志级别设置为WARN,因此我看不到任何消息.
should cause the spark api to log its effective config to the log as INFO, but the default log level is set to WARN, and as such I don't see any messages.
设置此行
sc.setLogLevel("INFO")
显示前进的INFO消息,但到那时为时已晚.
shows INFO messages going forward, but its too late by then.
如何设置spark起始的默认日志记录级别?
How can I set the default logging level that spark starts with?
推荐答案
http://spark.apache.org/docs/latest/configuration.html#configuring-logging
配置日志记录
Spark使用log4j进行日志记录. 您可以通过在conf目录中添加log4j.properties文件来配置它.一种开始的方法是复制位于那里的现有log4j.properties.template.
Spark uses log4j for logging. You can configure it by adding a log4j.properties file in the conf directory. One way to start is to copy the existing log4j.properties.template located there.
The following blog about "How to log in spark" https://www.mapr.com/blog/how-log-apache-spark suggest a way to configure log4j, and provide suggestion which includes directing INFO level logs into a file.
这篇关于如何设置默认的火花记录级别?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!