如何在Yarn上配置应用程序驱动程序的自动重启 [英] How to configure automatic restart of the application driver on Yarn

问题描述

来自Spark编程指南

要从驱动程序故障中自动恢复，用于运行流应用程序的部署基础结构必须监视驱动程序进程，并在驱动程序出现故障时重新启动.不同的集群管理器具有不同的工具来实现这一目标.

Spark Standalon

Spark Standalon

• Spark Standalone -可以提交Spark应用程序驱动程序以在Spark Standalone群集中运行(请参阅群集部署模式)，也就是说，该应用程序驱动程序本身可以在一个工作程序节点上运行.此外，可以指示独立集群管理器监督驱动程序，并在驱动程序由于非零退出代码或由于运行该驱动程序的节点故障而失败时重新启动它.有关更多详细信息，请参见Spark Standalone指南中的群集模式和监督.
• YARN -纱线支持类似的机制，可以自动重新启动应用程序.请参阅YARN文档以获取更多详细信息. ....
  • Spark Standalone - A Spark application driver can be submitted to run within the Spark Standalone cluster (see cluster deploy mode), that is, the application driver itself runs on one of the worker nodes. Furthermore, the Standalone cluster manager can be instructed to supervise the driver, and relaunch it if the driver fails either due to non-zero exit code, or due to failure of the node running the driver. See cluster mode and supervise in the Spark Standalone guide for more details.
  • YARN - Yarn supports a similar mechanism for automatically restarting an application. Please refer to YARN documentation for more details. ....

    所以，问题是如何支持Spark的自动重启 在纱线上流式传输.

  So, the question is how to support the auto-restart for Spark Streaming on Yarn.

  陶

  推荐答案

  您正在寻找的是在纱线群集模式"下启动应用程序的一组指令: https://spark.apache.org/docs/latest/running-on -yarn.html

  What you are looking for is the set of instructions to launch your application in yarn "cluster mode" : https://spark.apache.org/docs/latest/running-on-yarn.html

  这意味着您的驱动程序应用程序在YARN上的群集上运行(而不是在本地计算机上).因此，如果失败，可以由YARN重新启动.

  This means that your driver application runs on the cluster on YARN (not on your local machine). As such it can be restarted by YARN if it fails.

  这篇关于如何在Yarn上配置应用程序驱动程序的自动重启的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！