将自定义退出代码从yarn-cluster模式spark传递到CLI [英] pass custom exitcode from yarn-cluster mode spark to CLI

问题描述

我通过提交火花启动了纱线簇模式的火花作业. 为了指示部分失败等，我想将退出代码从驱动程序传递到调用spark-submit的脚本.

I started a yarn cluster mode spark job through spark-submit. To indicate partial failure etc I want to pass exitcode from driver to script calling spark-submit.

我同时尝试了System.exit和在驱动程序中抛出SparkUserAppException，但是在这两种情况下，CLI仅得到1，而不是我传递的退出代码.

I tried both, System.exit and throwing SparkUserAppException in driver, but in both cases CLI only got 1, not what exitcode I passed.

我认为不可能传递自定义退出代码，因为驱动程序传递的任何退出代码都将转换为纱线状态，而纱线会将失败的退出代码转换为1或失败.

I think it is impossible to pass custom exitcode, since any exitcode passed by driver will be converted to yarn status and yarn will convert any failed exitCode to 1 or failed.

推荐答案

您可以将退出代码保存在输出文件中(在HDFS或本地FS上)，并使脚本等待文件出现，读取并继续.这绝对不是一个优雅的方法，但是它可以帮助您继续前进. 保存文件时，请注意该位置的权限.您的火花过程必须具有RW访问权限.

You can save the exit code in the output file (on HDFS or local FS) and make your script wait for this file appearance, read and proceed. This is definitely is not an elegant way, but it may help you to proceed. When saving file, pay attention to the permissions to this location. Your spark process has to have RW access.

这篇关于将自定义退出代码从yarn-cluster模式spark传递到CLI的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！