提交后如何退出火花提交 [英] How to exit spark-submit after the submission

问题描述

使用spark-submit(YARN模式)提交Spark Streaming程序时 它会不断轮询状态，永远不会退出

When submitting spark streaming program using spark-submit(YARN mode) it keep polling the status and never exit

提交火花后，spark-submit中是否有退出选项?

Is there any option in spark-submit to exit after the submission?

===为什么这麻烦我了===

===why this trouble me===

流式传输程序将永远运行，并且我不需要状态更新

The streaming program will run forever and i don't need the status update

如果我手动启动它，可以按ctrl + c停止它 但是我有很多流媒体上下文要启动，我需要使用脚本启动它们

I can ctrl+c to stop it if i start it manually but i have lots of streaming context to start and i need to start them using script

我可以将spark-submit程序放在后台， 但是创建了许多后台Java进程后，对应的用户将无法运行任何其他Java进程，因为JVM无法创建GC线程

I can put the spark-submit program in background, but after lots of background java process created, the user corresponding to, will not able to run any other java process because JVM cannot create GC thread

推荐答案

有趣.我从来没有想过这个问题.不确定是否有干净的方法可以执行此操作，但是我只是简单地取消了机器上的提交过程，然后继续执行纱线作业，直到您专门停止它为止.因此，您可以创建一个脚本，该脚本执行spark提交然后将其杀死.当您实际上想停止工作时，请使用yarn -kill.脏但有效.

Interesting. I never thought about this issue. Not sure there is a clean way to do this, but I simply kill the submit process on the machine and the yarn job continues to run until you stop it specifically. So you can create a script that execute the spark submit and then kills it. When you will actually wanna stop the job use yarn -kill. Dirty but works.

这篇关于提交后如何退出火花提交的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！