如何抑制“阶段2 ==="从输出控制台在火花? [英] How to suppress the "Stage 2===>" from the output console in spark?

问题描述

我有数据框，并尝试成功获取非重复计数，并且能够成功获取非重复计数，但是每当执行scala程序时，我都会收到此消息.([Stage 2:=============================> (1 + 1) / 2])我如何才能在控制台中禁止显示此特定消息.

I have dataframe and trying to get distinct count and able to get distinct count successfully but whenever scala program is executing i'm getting this message ([Stage 2:=============================> (1 + 1) / 2])how can i suppress particular this message in console.

val countID = dataDF.select(substring(col("dataDF"),5,7).distinct().count()

推荐答案

您需要将spark.ui.showConsoleProgress设置为false

我在故障单的注释中发现了这一点，以便添加进度条.

I found this in the comments of the ticket for the addition of the progress bar.

https://issues.apache.org/jira/browse/SPARK-4017

尽管我在任何文档中都没有看到它，但确实应该添加它.

I haven't seen it in any of the documentation though, it really should be added.

这篇关于如何抑制“阶段2 ==="从输出控制台在火花?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！