如何在执行spark pgm时指定spark.authenticate.secret值 [英] How to specify spark.authenticate.secret value while executing spark pgm

问题描述

当我使用deploy-mode:client启动我的spark程序时，出现了此错误.

I got this error when I launched my spark program with deploy-mode : client.

java.lang.illegalargumentexception:错误:必须通过spark.authenticate.secret配置指定秘密密钥.

java.lang.illegalargumentexception: error: a secret key must be specified via the spark.authenticate.secret config.

部署模式:客户端主人:纱.

deploy-mode: client master: YARN.

如何解决此错误?我不知道.

How do I resolve this error? I have no clue.

推荐答案

在纱线群集模式下，我得到了类似的错误，以下是对我的帮助:

I got similar error in yarn-cluster mode, following helped me :

https://issues.apache.org/jira/browse/SPARK-23476

如果将spark.authenticate = true指定为集群范围的配置，则必须添加以下内容--conf"spark.authenticate = false"；--conf"spark.shuffle.service.enabled = false"--conf"spark.dynamicAllocation.enabled = false"；--conf"spark.network.crypto.enabled = false"--conf"spark.authenticate.enableSaslEncryption = false"；在spark-submit命令中.

If spark.authenticate=true is specified as a cluster wide config, then the following has to be added --conf "spark.authenticate=false" --conf "spark.shuffle.service.enabled=false" --conf "spark.dynamicAllocation.enabled=false" --conf "spark.network.crypto.enabled=false" --conf "spark.authenticate.enableSaslEncryption=false" in the spark-submit command.

这篇关于如何在执行spark pgm时指定spark.authenticate.secret值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！