从检查点重新启动后，Spark Streaming选项卡消失 [英] Spark streaming tab disappears after restarting from checkpoint

问题描述

我有一个在集群(Spark 1.6)上运行的Spark Streaming作业，该检查点指向S3.最初启动作业时，我可以看到流"选项卡.但是，当我从检查点重新启动作业时，流"选项卡消失了.该作业仍作为流式作业，我看到批次以配置的批次间隔出现.见下文.

I have a Spark Streaming job running on a cluster (Spark 1.6) which checkpoints to S3. When I start up the job initially, I can see "Streaming" tab. However when I restart the job from checkpoint the Streaming tab disappears. The job still works as a streaming job and I see the batches appear at the configured batch interval. See below.

如果我清除检查点数据，则选项卡返回.我怀疑从检查点重新启动时，流"选项卡未正确注册.

If I clear out the checkpoint data, the tab comes back. I suspect that the Streaming tab is not registered correctly while restarting from a checkpoint.

我查看了

I looked at the Spark Streaming code. Is it possible this flow is not invoked when the application state is deserialised from a checkpoint?

有人知道如何解决此问题吗?

Does anyone know how to fix this?

推荐答案

如果我清除检查点数据，则选项卡返回.我怀疑 从以下位置重新启动时，流"选项卡未正确注册 检查点.

If I clear out the checkpoint data, the tab comes back. I suspect that the Streaming tab is not registered correctly while restarting from a checkpoint.

它被调用，但是直到它完成从S3检查点位置加载所有数据后，流选项卡才出现.如果您的血统很长，则可能需要一些时间来加载.从检查点还原所有数据后，您将看到流式传输"选项卡.

It is invoked, but the streaming tab doesn't appear until it finishes loading all the data from the S3 checkpoint location. If your lineage is long, it may take some time to load. Once all the data is restored from checkpoint, you'll see the streaming tab appear.

这篇关于从检查点重新启动后，Spark Streaming选项卡消失的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！