＆是什么QUOT;阶段跳过＆QUOT; Apache中的星火web用户界面是什么意思？ [英] What does "Stage Skipped" mean in Apache Spark web UI?

问题描述

从我星火UI。这是什么被忽略呢？

From my Spark UI. What does it mean by skipped?

推荐答案

通常，它意味着数据已经从缓存中提取，没有必要再执行特定阶段。这是您的DAG一致表示，下一阶段需要改组（ reduceByKey ）。每当有href=\"https://spark.apache.org/docs/1.5.0/programming-guide.html#performance-impact\">自动洗牌参与星火的：

Typically it means that data has been fetched from cache and there was no need to re-execute given stage. It is consistent with your DAG which shows that the next stage requires shuffling (reduceByKey). Whenever there is shuffling involved Spark automatically caches generated data:

洗牌也产生了大量的磁盘上的中间文件。由于星火1.3，这些文件是preserved直到相应的RDDS不再使用，而且垃圾收集。这样做是为了洗牌文件不需要被重新创建如果谱系重新计算。

Shuffle also generates a large number of intermediate files on disk. As of Spark 1.3, these files are preserved until the corresponding RDDs are no longer used and are garbage collected. This is done so the shuffle files don’t need to be re-created if the lineage is re-computed.

这篇关于＆是什么QUOT;阶段跳过＆QUOT; Apache中的星火web用户界面是什么意思？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！