带有 Actor 的 Spark Streaming 永不终止 [英] Spark Streaming with Actor Never Terminates

问题描述

将 Spark 1.5 Streaming 与 Actor 接收器一起使用.

Using Spark 1.5 Streaming with an Actor receiver.

val conf = new SparkConf()
    .setMaster("local[4]")
    .setAppName("ModelTest")

val ssc = new StreamingContext(conf, Seconds(2))

val models = ssc.actorStream[Model](Props(...), "ModelReceiver")

models.foreachRDD { rdd => ... }

ssc.start()
ssc.awaitTermination()
// NEVER GETS HERE!

当生成的 Actor 关闭时，代码不会超过 ssc.awaitTermination()

When the generated Actor is shutdown the code will not progress beyond ssc.awaitTermination()

如果我用 Ctrl+C 杀死 SBT，ssc.awaitTermination() 行之后的 println 将完成.

If I kill SBT with Ctrl+C a println after the ssc.awaitTermination() line will complete.

Spark 应该如何终止?

How should Spark be terminated?

推荐答案

正如函数名称所暗示的那样，Spark Streaming 将等待终止是正确的.要终止流应用程序，您可以向该进程发送一个 SIGTERM，例如使用 kill 命令.

You are correct that Spark Streaming will await termination, as the function name hints. To kill a Streaming application you send a SIGTERM to that process, for example by using the kill command.

正如您在 Spark Standalone 文档中所见 你也可以使用 Spark Submit 来终止进程:

As you can also see in the Spark Standalone documentation you can also kill the process using Spark Submit:

./bin/spark-class org.apache.spark.deploy.Client kill <master url> <driver ID>

您可以通过调用sys.ShutdownHookThread 来定义一些要在进程关闭时运行的代码.

You can define some code that you want to run when the process is shutting down, by calling sys.ShutdownHookThread.

sys.ShutdownHookThread {
  log.info("Stopping Spark Streaming...")
  ssc.stop(stopSparkContext = true, stopGracefully = true)
  log.info("Shutting down the application...")
}

这篇关于带有 Actor 的 Spark Streaming 永不终止的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！