如何优雅地停止运行Spark Streaming应用程序? [英] How to Stop running Spark Streaming application Gracefully?

问题描述

如何停止火花流?我的火花流作业正在持续运行.我想优雅地停下来.

How Do i stop spark streaming? My spark streaming job is running continuously. I want to stop in a graceful manner.

我已经看到以下关闭流应用程序的选项.

I have seen below option to shutdown streaming application.

sparkConf.set("spark.streaming.stopGracefullyOnShutdown","true") 

火花配置:可用属性

但是，如何在正在运行的应用程序上更新此参数?

But, how do i update this parameter on a running application?

推荐答案

看看

Have a look at this blogpost. It it the "nicest" way to gracefully terminate a streaming job I have come across.

如何传递关机信号:

How to pass Shutdown Signal :

现在，我们知道如何确保火花流中的正常关闭.但是，我们如何传递关闭信号来触发流式传输.一个幼稚的选择是在运行驱动程序的屏幕终端上使用CTRL + C命令，但是显然这不是一个好选择.我正在使用的一种解决方案是grep火花流的驱动程序进程并发送SIGTERM信号.当驱动程序收到此信号时，它将启动应用程序的正常关闭.我们可以在一些shell脚本中编写以下命令，然后运行该脚本以传递关闭信号:

Now we know how to ensure graceful shutdown in spark streaming. But how can we pass the shutdown signal to spark streaming. One naive option is to use CTRL+C command at the screen terminal where we run driver program but obviously its not a good option. One solution , which i am using is , grep the driver process of spark streaming and send a SIGTERM signal . When driver gets this signal, it initiates the graceful shutdown of the application. We can write the command as below in some shell script and run the script to pass shutdown signal :

ps -ef |grep spark |grep |awk'{print $ 2}'|xargs kill -SIGTERM

ps -ef | grep spark | grep | awk '{print $2}' | xargs kill -SIGTERM

例如ps -ef |grep spark |grep DataPipelineStreamDriver |awk'{print $ 2}'|xargs kill -SIGTERM

e.g. ps -ef | grep spark | grep DataPipelineStreamDriver | awk '{print $2}' | xargs kill -SIGTERM

这篇关于如何优雅地停止运行Spark Streaming应用程序?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！