Spark将其作为另一个用户提交给纱线 [英] Spark submit to yarn as a another user

问题描述

是否有可能向纱线群集提交点火作业，并且可以通过命令行或罐子内部选择哪个用户将拥有该作业？

spark-submit将从包含用户的脚本启动。

PS：如果集群具有kerberos配置（并且脚本为keytab ）？

解决方案

对于非kerberized集群： export HADOOP_USER_NAME =在提交Spark作业之前，zorro 会执行该操作。

事后如果您想恢复取消设置HADOOP_USER_NAME 复制到其他shell脚本（或交互式shell会话）中的默认凭据。

对于 kerberized集群，假装另一个帐户而不摧毁其他工作/会话的干净方式（可能取决于您的默认票证）将在这一行......

  export KRB5CCNAME = FILE：/ tmp / krb5cc_ $ （id -u）_temp _ $$ 
 kinit -kt〜/ .protectedDir / zorro.keytab [e-mail] 
 spark-submit ........... 
 kdestroy 
  

Is it possible to submit a spark job to a yarn cluster and choose, either with the command line or inside the jar, which user will "own" the job?

The spark-submit will be launch from a script containing the user.

PS: is it still possible if the cluster has a kerberos configuration (and the script a keytab) ?

解决方案

For a non-kerberized cluster: export HADOOP_USER_NAME=zorro before submitting the Spark job will do the trick.
Make sure to unset HADOOP_USER_NAME afterwards, if you want to revert to your default credentials in the rest of the shell script (or in your interactive shell session).

For a kerberized cluster, the clean way to impersonate another account without trashing your other jobs/sessions (that probably depend on your default ticket) would be something in this line...

export KRB5CCNAME=FILE:/tmp/krb5cc_$(id -u)_temp_$$
kinit -kt ~/.protectedDir/zorro.keytab [e-mail]
spark-submit ...........
kdestroy

这篇关于Spark将其作为另一个用户提交给纱线的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！