优雅地关闭火花结构化流 [英] Shutdown spark structured streaming gracefully

问题描述

有一种方法可以通过将属性 spark.streaming.stopGracefullyOnShutdown 设置为 true，然后使用 kill -SIGTERM 命令终止进程来正常关闭 Spark 流.但是，我没有看到此类选项可用于结构化流 (SQLContext.scala).

There is a way to enable graceful shutdown of spark streaming by setting property spark.streaming.stopGracefullyOnShutdown to true and then kill the process with kill -SIGTERM command. However I don't see such option available for structured streaming (SQLContext.scala).

结构化流媒体的关闭过程是否不同?或者只是还没有实施?

Is the shutdown process different in structured streaming? Or is it simply not implemented yet?

推荐答案

此功能尚未实现.但是，spark 结构化 Steaming 的预写日志声称可以毫无问题地恢复状态和偏移量.

This feature is not implemented yet. But the write ahead logs of spark structured steaming claims to recover state and offsets without any issues.

这篇关于优雅地关闭火花结构化流的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！