当使用主纱线运行时，必须在环境中设置HADOOP_CONF_DIR或YARN_CONF_DIR [英] When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment

问题描述

线程main中的异常java.lang.Exception：当使用master运行时'yarn'HADOOP_CONF_DIR或YARN_CONF_DIR必须在环境中设置。

我不确定环境在哪里（什么特定文件？）。我尝试过使用：

  export HADOOP_CONF_DIR = $ HADOOP_HOME / etc / hadoop $ b $ export YARN_CONF_DIR = $ HADOOP_HOME / etc / hadoop 
  

在bash_profile中，但这似乎没有帮助。

谢谢！

解决方案

需要添加以下行到 spark-env.sh

 导出HADOOP_CONF_DIR = $ HADOOP_HOME / etc / hadoop 
  

注意：检查$ HADOOP_HOME / etc / hadoop是否正确你的环境。并且spark-env.sh也包含HADOOP_HOME的导出。

I am trying to run Spark using yarn and I am running into this error:

Exception in thread "main" java.lang.Exception: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.

I am not sure where the "environment" is (what specific file?). I tried using:

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

in the bash_profile, but this doesn't seem to help.

Thank you!

解决方案

While running spark using Yarn, you need to add following line in to spark-env.sh

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

Note: check $HADOOP_HOME/etc/hadoop is correct one in your environment. And spark-env.sh contains export of HADOOP_HOME as well.

这篇关于当使用主纱线运行时，必须在环境中设置HADOOP_CONF_DIR或YARN_CONF_DIR的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！