错误时纱线上的火花运行两次 [英] spark on yarn run double times when error

问题描述

我使用在纱线上产生火花的模型，当我遇到问题时，火花将自动重新启动.

I use the model that spark on yarn,when i meet a problem the spark would restart automatic.

无论成功或失败，我都希望准确运行一次.

I want to run exact once whatever successful or fail.

是否可以设置任何conf或api?

Is there any conf or api can set?

我正在使用Spark 1.5版.

I'm using spark version 1.5.

推荐答案

您必须将 spark.yarn.maxAppAttempts 属性设置为1.默认值为 yarn.resourcemanager.am..max-attempts ，默认为2.

You have to set spark.yarn.maxAppAttempts property to 1. Default value for this is yarn.resourcemanager.am.max-attempts which is by default 2.

通过代码设置属性:

SparkConf conf = new SparkConf();
conf.set("spark.yarn.maxAppAttempts", "1");

通过spark-submit提交作业时设置:

Set when submitting the job via spark-submit:

-conf spark.yarn.maxAppAttempts = 1

--conf spark.yarn.maxAppAttempts=1

这篇关于错误时纱线上的火花运行两次的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！