Hadoop环境变量 [英] Hadoop environment variables

问题描述

我试图在Mac上调试单个节点Hadoop集群的一些问题。在所有的设置文档中，它说要添加：

  export HADOOP_OPTS = -  Djava.security.krb5.realm = OX.AC .UK -Djava.security.krb5.kdc = kdc0.ox.ac.uk：kdc1.ox.ac.uk
  

删除此错误：

 无法从SCDynamicStore加载领域信息
  

这有效，但它似乎只适用于标准输出。当我检查我的Hadoop日志目录时，在job _ ### / atempt _ ### / stderr下面，错误仍然存​​在：

  2013-02-08 09：58：23.662 java [2772：1903]无法从SCDynamicStore加载领域信息
  

我很难将RVM Rubies加载到Hadoop环境中以使用Hadoop流式执行Ruby代码。 STDOUT正在打印RVM已加载并使用正确的Ruby / gemset，但使用了STDERR日志：

  env：ruby_noexec_wrapper：没有这样的文件或目录
  

有什么方法可以找出Hadoop实际用于执行作业的路径，或者如果它在这里调用一些其他环境？

进一步背景：

我使用的是Hadoop 1.1.1通过自制软件。它的设置方式非常类似于 INSTALLING HADOOP ON MAC OSX LION 并调试 wukong 3.0.0的实现作为执行Hadoop作业的包装。

解决方案

回答我自己的问题，让其他人可以找到它。

我似乎在我的hadoop-env中加载了rvm，但添加后我必须没有重新启动集群。为了确保您的rubies和gemset已加载，请将标准rvm子句添加到hadoop-env.sh。例如：

  [[-s/Users/ScotterC/.rvm/scripts/rvm]]&&来源/Users/ScotterC/.rvm/scripts/rvm
  

并确保重新启动群集所以它拿起来了。奇怪的是，没有重新启动，我的日志会显示它正在加载rvm，但它显然没有执行该红宝石，它是相应的gemfiles。重新启动后它工作。

I'm trying to debug some issues with a single node Hadoop cluster on my Mac. In all the setup docs it says to add:

export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"

to remove this error:

Unable to load realm info from SCDynamicStore

This works, but it only seems to work for STDOUT. When I check my Hadoop logs directory, under "job_###/atempt_###/stderr" the error is still there:

2013-02-08 09:58:23.662 java[2772:1903] Unable to load realm info from SCDynamicStore

I'm having great difficulty loading RVM Rubies into the Hadoop environment to execute Ruby code with Hadoop streaming. STDOUT is printing that RVM is loaded and using the right Ruby/gemset but my STDERR logs:

env: ruby_noexec_wrapper: No such file or directory

Is there some way to find out what path Hadoop is actually using to execute the jobs, or if it's invoking some other environment here?

Further background:

I'm using Hadoop 1.1.1 installed via Homebrew. It's setup in a manner very similar to "INSTALLING HADOOP ON MAC OSX LION" and debugging an implementation of wukong 3.0.0 as the wrapper for executing Hadoop jobs.

解决方案

To answer my own question so other's can find it.

I appeared to be loading rvm in my hadoop-env but I must have not restarted the cluster after adding it. To make sure your rubies and gemsets are loaded, add the standard rvm clause to hadoop-env.sh. Something like:

[[ -s "/Users/ScotterC/.rvm/scripts/rvm" ]] && source "/Users/ScotterC/.rvm/scripts/rvm"

And make sure to restart the cluster so it picks it up. Oddly enough, without restarting, my logs would show that it was loading rvm but it clearly wasn't executing that ruby and it's respective gemfiles. After restarting it worked.

这篇关于Hadoop环境变量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！