如果驱动程序崩溃，该怎么办? [英] What happens if the driver program crashes?

问题描述

我了解工作节点如何容错，但是如果您的驱动程序由于某些意外原因而崩溃怎么办? (掉电/内存问题等).

I understand how worker nodes are fault tolerant, but what happens if your driver program crashes for some unexpected reason? (power down / memory issue etc).

我想您将失去所有工作，因为读取结果的代码不再运行，或者Spark是否以某种方式知道如何重新启动它?如果可以，怎么办?

I would imagine you will lose all work, as the code reading the results is not running anymore, or does Spark somehow know how to restart it? If so how?

推荐答案

@zsxwing指出，这取决于您如何运行驱动程序.除了在yarn中运行之外，您还可以使用集群的部署模式来运行您的作业(这是spark-submit的参数).在Spark Streaming中，您指定--supervise，Spark将为您重新启动作业.详细信息在 Spark流式传输中指南.

As @zsxwing points out it depends on how you run your driver. In addition to running in yarn, you can also run your job with a deploy mode of cluster (this is a parameter to spark-submit). In Spark Streaming you specify --supervise and Spark will restart the job for you. The details are in the Spark Streaming Guide.

这篇关于如果驱动程序崩溃，该怎么办?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！