如何停止在 Spark 控制台上显示 INFO 消息? [英] How to stop INFO messages displaying on spark console?
本文介绍了如何停止在 Spark 控制台上显示 INFO 消息?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我想停止在 spark shell 上发送各种消息.
I'd like to stop various messages that are coming on spark shell.
我尝试编辑 log4j.properties
文件以阻止这些消息.
I tried to edit the log4j.properties
file in order to stop these message.
这里是log4j.properties
# Define the root logger with appender file
log4j.rootCategory=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
但是消息仍然显示在控制台上.
But messages are still getting displayed on the console.
以下是一些示例消息
15/01/05 15:11:45 INFO SparkEnv: Registering BlockManagerMaster
15/01/05 15:11:45 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20150105151145-b1ba
15/01/05 15:11:45 INFO MemoryStore: MemoryStore started with capacity 0.0 B.
15/01/05 15:11:45 INFO ConnectionManager: Bound socket to port 44728 with id = ConnectionManagerId(192.168.100.85,44728)
15/01/05 15:11:45 INFO BlockManagerMaster: Trying to register BlockManager
15/01/05 15:11:45 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager 192.168.100.85:44728 with 0.0 B RAM
15/01/05 15:11:45 INFO BlockManagerMaster: Registered BlockManager
15/01/05 15:11:45 INFO HttpServer: Starting HTTP Server
15/01/05 15:11:45 INFO HttpBroadcast: Broadcast server star
我该如何阻止这些?
推荐答案
感谢 @AkhlD 和 @Sachin Janani 建议更改 .conf
文件.
Thanks @AkhlD and @Sachin Janani for suggesting changes in .conf
file.
以下代码解决了我的问题:
Following code solved my issue:
1) 在导入部分添加import org.apache.log4j.{Level, Logger}
2) 在创建 spark 上下文对象后添加以下行,即在 val sc = new SparkContext(conf)
之后:
2) Added following line after creation of spark context object i.e. after val sc = new SparkContext(conf)
:
val rootLogger = Logger.getRootLogger()
rootLogger.setLevel(Level.ERROR)
这篇关于如何停止在 Spark 控制台上显示 INFO 消息?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文