在EMR群集上运行的Spark作业. system.exit(0)用于正常完成工作，但仍然无法执行EMR [英] Spark jobs running on EMR cluster. system.exit(0) used to gracefully completion of job but still Step on EMR fails

问题描述

在火花作业中.我正在使用if file not found system.exit(0).它应该可以正常完成工作.在本地已成功完成.但是当我运行EMR时.步骤失败.

解决方案

EMR使用 解决方案

EMR uses YARN for cluster management and launching Spark applications. So when you're running a Spark app with --deploy-mode: cluster in EMR, the Spark application code is not running in a JVM on its own, but is rather executed by the ApplicationMaster class.

Browsing through the ApplicationMaster code can explain what happens when you try to execute System.exit(). The user application is launched in startUserApplication, and then the finish method is called after the user application returns. However, when you call System.exit(0), what is executed instead is a shutdown hook which sees that your code hasn't finished successfully, and marks it as an EXIT_EARLY failure. It also has this useful comment:

  // The default state of ApplicationMaster is failed if it is invoked by shut down hook.
  // This behavior is different compared to 1.x version.
  // If user application is exited ahead of time by calling System.exit(N), here mark
  // this application as failed with EXIT_EARLY. For a good shutdown, user shouldn't call
  // System.exit(0) to terminate the application.

这篇关于在EMR群集上运行的Spark作业. system.exit(0)用于正常完成工作，但仍然无法执行EMR的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！