关机Spark优雅地流式传输 [英] Shutdown spark structured streaming gracefully

问题描述

有一种方法可以通过将属性spark.streaming.stopGracefullyOnShutdown设置为true，然后使用kill -SIGTERM命令杀死该进程来启用正常关闭火花流.但是，我看不到这种选项可用于结构化流(SQLContext.scala).

There is a way to enable graceful shutdown of spark streaming by setting property spark.streaming.stopGracefullyOnShutdown to true and then kill the process with kill -SIGTERM command. However I don't see such option available for structured streaming (SQLContext.scala).

关闭过程在结构化流传输中是否有所不同?还是根本没有实现?

Is the shutdown process different in structured streaming? Or is it simply not implemented yet?

推荐答案

此功能尚未实现.但是，火花结构化蒸汽的预写日志声称可以恢复状态和补偿而没有任何问题.

This feature is not implemented yet. But the write ahead logs of spark structured steaming claims to recover state and offsets without any issues.

这篇关于关机Spark优雅地流式传输的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！