Spark输出:日志样式与进度样式 [英] Spark output: log-style vs progress-style

问题描述

spark-submit输出看起来不同:一个是日志样式"，即，大量的消息流，例如

spark-submit output on two different clusters (both run spark 1.2) look different: one is "log-style", i.e., a voluminous stream of messages like

15/04/06 14:53:13 INFO TaskSetManager: Starting task 262.0 in stage 4.0 (TID 894, XXXXX, PROCESS_LOCAL, 1785 bytes)
15/04/06 14:53:13 INFO TaskSetManager: Finished task 255.0 in stage 4.0 (TID 892) in 155 ms on XXXXX (288/300)
15/04/06 14:53:13 INFO BlockManagerInfo: Added rdd_16_262 in memory on XXXXX:49388 (size: 14.3 MB, free: 1214.5 MB)
15/04/06 14:53:13 INFO TaskSetManager: Finished task 293.0 in stage 4.0 (TID 893) in 156 ms on XXXXX (289/300)
15/04/06 14:53:13 INFO TaskSetManager: Finished task 262.0 in stage 4.0 (TID 894) in 168 ms on XXXXX (290/300)
15/04/06 14:53:16 INFO TaskSetManager: Starting task 1.0 in stage 4.0 (TID 895, ip-10-0-3-92.ec2.internal, NODE_LOCAL, 1785 bytes)
15/04/06 14:53:16 INFO TaskSetManager: Starting task 74.0 in stage 4.0 (TID 896, XXXXX, NODE_LOCAL, 1785 bytes)

和其他进度方式"，即 屏幕底部(可能会因错误而中断).

and the other "progress-style", i.e., a growing progress bar at the bottom of the screen (which may be interrupted by errors, if any).

如何在两种样式之间切换? (基于每个职位或每个团队)

How do I switch between the two styles? (either on a per-job or a per-cluster basis)

我尝试将--conf spark.ui.showConsoleProgress=true传递给 spark-submit无效.

I tried passing --conf spark.ui.showConsoleProgress=true to spark-submit with no effect.

推荐答案

我以前遇到过这种情况，我当时的情况只是因为两个集群之间在conf/log4j.properties中设置了不同的log4j.rootCategory级别.

I have encountered this before, My situation that time is just because different log4j.rootCategory levels are set in conf/log4j.properties between the two clusters.

在具有WARN日志记录级别的集群中发生进度样式"输出，而当我将日志记录级别设置为INFO

The "progress-style" output occurs in the cluster have WARN level of logging, while "Log-style" occurs when I set logging level as INFO

更新(2015-05-10):

在分支1.4.0中，通过SparkContext中的_progressBar启动逻辑进行操作，实际上受两个条件控制:

Come across the _progressBar startup logic in SparkContext, in branch-1.4.0, actually controlled by two conditions:

_progressBar =
  if (_conf.getBoolean("spark.ui.showConsoleProgress", true) && !log.isInfoEnabled) {
    Some(new ConsoleProgressBar(this))
  } else {
    None
  }

因此，要在控制台中启用进度样式输出，必须将spark.ui.showConsoleProgress设置为true并将conf/log4j.properties中的日志级别升级到Not enabling Info，即WARN或ERROR.

Therefore, to enable the progress-style output in Console, you have to both set spark.ui.showConsoleProgress to true and upgrade your log level in conf/log4j.properties to Not enabling Info, i.e, WARN or ERROR.

这篇关于Spark输出:日志样式与进度样式的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！