Windows上的Hadoop + HBase集群：找不到winutils [英] Hadoop+HBase cluster on windows: winutils not found

问题描述

我在Windows上尝试使用Hadoop 2.20和HBase 0.98设置一个完全分布式的4节点开发群集。我在Windows上构建了Hadoop，最近在Windows上构建了HBase。

我们已成功运行Hadoop安装指南中的wordcount示例，以及自定义WebHDFS作业。

当我试图从我的主机启动hbase（./bin/start）时，我在HBase上运行cygwin。 -hbase.sh），我得到以下错误：

  2014-04-17 16：22：08,599 ERROR [main] util.Shell：无法在hadoop二进制路径中找到winutils二进制
 java.io.IOException：无法在Hadoop二进制文件中找到可执行文件null\bin\winutils.exe。 
 at org.apache.hadoop.util.Shell.getQualifiedBinPath（Shell.java:278）
 at org.apache.hadoop.util.Shell.getWinUtilsPath（Shell.java:300）
 at org.apache.hadoop.util.Shell。< clinit>（Shell.java:293）
 at org.apache.hadoop.util.StringUtils。< clinit>（StringUtils.java:76）
 at org.apache.hadoop.conf.Configuration.getStrings（Configuration.java:1514）
 at org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps（ZKConfig.java:113）
在org.apache.hadoop.hbase.zookeeper.ZKServerTool.main（ZKServerTool.java:46）
  

看看Shell.java源，这里设置为null，似乎是HADOOP_HOME环境变量。在hadoop下D：/ hadoop，和HBase下的cygwin根目录下C：/ cygwin / root / usr / local / hbase，cygwin $ HADOOP_HOME变量是/ cygdrive / d / hadoop /，Windows系统环境变量％HADOOP_HOME％是D：\hadoop。看起来像这两个变量，变量应该被正确找到...

也可能相关：我运行的Windows Server 2012 x64。

编辑：我已验证在D：\hadoop\bin\中实际上有一个winutils.exe。

解决方案

我们找到了。因此，在Hadoop的Shell.java中，你会发现有两个选项来传递Hadoop路径。

  //首先检查带有JVM作用域的Dflag hadoop.home.dir 
 String home = System.getProperty（hadoop.home.dir）; 
 
 //回到system / user-global env变量
 if（home == null）{
 home = System.getenv（HADOOP_HOME）;经过试验和错误，我们发现在HBase选项（HBase的hbase-2）中， env.sh，HBASE_OPTS变量），您需要在此选项中添加Windows（！）路径到Hadoop。在我们的例子中，我们需要添加 -Dhadoop.home.dir = D：/ hadoop 。
 
 
 
给任何碰巧碰到这个的人）。
 
I'm trying to set up a fully-distributed 4-node dev cluster with Hadoop 2.20 and HBase 0.98 on Windows. I've built Hadoop on Windows successfully, and more recently, also build HBase on Windows. 

We have successfully ran the wordcount example from the Hadoop installation guide, as well as a custom WebHDFS job. As HBase fully-distributed on Windows isn't supported yet, I'm running HBase under cygwin.

When trying to start hbase from my master (./bin/start-hbase.sh), I get the following error:
2014-04-17 16:22:08,599 ERROR [main] util.Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
    at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
    at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293)
    at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
    at org.apache.hadoop.conf.Configuration.getStrings(Configuration.java:1514)
    at org.apache.hadoop.hbase.zookeeper.ZKConfig.makeZKProps(ZKConfig.java:113)
    at org.apache.hadoop.hbase.zookeeper.ZKServerTool.main(ZKServerTool.java:46)
Looking at the Shell.java source, what is here set as null, seems to be the HADOOP_HOME environment variable. With hadoop under D:/hadoop, and HBase under cygwin root at C:/cygwin/root/usr/local/hbase, the cygwin $HADOOP_HOME variable is /cygdrive/d/hadoop/, and the Windows system environment variable %HADOOP_HOME% is D:\hadoop . Seems to me like with those two variables, the variable should be found correctly...

Also potentially relevant: I'm running Windows Server 2012 x64.

Edit: I have verified that there actually is a winutils.exe in D:\hadoop\bin\ .
解决方案
We've found it. So, in Hadoop's Shell.java, you'll find that there are two options to communicate the Hadoop-path.
// first check the Dflag hadoop.home.dir with JVM scope
String home = System.getProperty("hadoop.home.dir");

// fall back to the system/user-global env variable
if (home == null) {
  home = System.getenv("HADOOP_HOME");
}
After trial and error, we found that in the HBase options (HBase's hbase-env.sh, HBASE_OPTS variable), you'll need to add in this option with the Windows(!) path to Hadoop. In our case, we needed to add -Dhadoop.home.dir=D:/hadoop .

Good luck to anyone else who happens to stumble across this ;).

                        
这篇关于Windows上的Hadoop + HBase集群：找不到winutils的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！