Datanode在单机上的Hadoop中失败 [英] Datanode failing in Hadoop on single Machine

问题描述

我在ubuntu 12.04 LTS上使用以下教程设置和配置sudo节点hadoop环境
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu -linux-multi-node-cluster /＃formatting-the-hdfs-filesystem-via-the-namenode

输入hadoop / bin $ start- all.sh
一切正常，然后我检查了Jps
，然后NameNode，JobTracker，TaskTracker，SecondaryNode已经启动，但DataNode未启动...

<如果有任何知道如何解决这个问题，请让我知道..

解决方案

p>

java.io.IOException：不兼容的namespaceIDs

如果您看到错误java.io.IOException：Incompatible namespaceIDs in DataNode（日志/ hadoop-hduser-datanode-.log）的日志，很可能是你（前身为HADOOP-1212）。

完整的错误在我的机器上看起来像这样：
... ERROR org .apache.hadoop.dfs.DataNode：java.io.IOException：/ app / hadoop / tmp / dfs / data中的不兼容namespaceIDs：namenode namespaceID = 308967713; datanode namespaceID = 113030094
at org.apache.hadoop.dfs.DataStorage.doTransition（DataStorage.java:281）
at org.apache.hadoop.dfs.DataStorage.recoverTransitionRead（DataStorage.java:121）
at org.apache.hadoop.dfs.DataNode.startDataNode（DataNode.java:230）
at org.apache.hadoop.dfs.DataNode。（DataNode.java:199）
at org.apache.hadoop.dfs.DataNode.makeInstance（DataNode.java:1202）
位于org.apache.hadoop.dfs.DataNode.run（DataNode.java:1146）
位于org.apache。 hadoop.dfs.DataNode.createDataNode（DataNode.java:1167）
at org.apache.hadoop.dfs.DataNode.main（DataNode.java:1326）
t此刻，似乎有两种解决方法如下所述。

解决方法1：从头开始
我可以证明以下步骤解决了此错误，但副作用不会令您开心（我也不）。我找到的原始解决方法是：

停止集群
删除有问题的DataNode上的数据目录：该目录由dfs.data.dir指定在conf / hdfs-site.xml中;如果你遵循本教程，相关目录是/ app / hadoop / tmp / dfs / data
重新格式化NameNode（注意：在此过程中，所有HDFS数据都会丢失）
重新启动集群
删除所有HDFS数据并从头开始听起来不是一个好主意（在初始设置/测试期间它可能没问题），您可以尝试第二种方法。

解决方法2：更新有问题的DataNode的名称空间ID
非常感谢Jared Stehler提供的以下建议。我还没有自己测试过，但随时可以尝试一下，并向我发送您的反馈。此解决方法是微创，因为您只需编辑有问题的DataNode上的一个文件：

停止DataNode
编辑/ current中的namespaceID的值/ VERSION来匹配当前NameNode的值
重新启动DataNode
如果您按照教程中的说明操作，相关文件的完整路径为：

NameNode：/ app / hadoop / tmp / dfs / name / current / VERSION
DataNode：/ app / hadoop / tmp / dfs / data / current / VERSION（background：dfs.data.dir is by默认设置为$ {hadoop.tmp.dir} / dfs / data，并且我们在本教程中将hadoop.tmp.dir设置为/ app / hadoop / tmp）。

问题的解决方案在以下站点中明确给出：

http://www.michael-noll.com/tutorials/running-hadoop-on -ubuntu-Linux的多节点集群/＃java的IO-ioexceptio n-incompatible-namespaceids

I set up and configured sudo node hadoop environment on ubuntu 12.04 LTS using following tutorial http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/#formatting-the-hdfs-filesystem-via-the-namenode

After typing hadoop/bin $ start-all.sh everything going fine then i checked the Jps then NameNode, JobTracker ,TaskTracker,SecondaryNode have been started but DataNode not started ...

If any know how to resolve this issue please let me know..

解决方案

ya i resolved it...

java.io.IOException: Incompatible namespaceIDs

If you see the error java.io.IOException: Incompatible namespaceIDs in the logs of a DataNode (logs/hadoop-hduser-datanode-.log), chances are you are affected by issue HDFS-107 (formerly known as HADOOP-1212).

The full error looked like this on my machines: ... ERROR org.apache.hadoop.dfs.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode namespaceID = 308967713; datanode namespaceID = 113030094 at org.apache.hadoop.dfs.DataStorage.doTransition(DataStorage.java:281) at org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:121) at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:230) at org.apache.hadoop.dfs.DataNode.(DataNode.java:199) at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:1202) at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1146) at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:1167) at org.apache.hadoop.dfs.DataNode.main(DataNode.java:1326) t the moment, there seem to be two workarounds as described below.

Workaround 1: Start from scratch I can testify that the following steps solve this error, but the side effects won’t make you happy (me neither). The crude workaround I have found is to:

Stop the cluster Delete the data directory on the problematic DataNode: the directory is specified by dfs.data.dir in conf/hdfs-site.xml; if you followed this tutorial, the relevant directory is /app/hadoop/tmp/dfs/data Reformat the NameNode (NOTE: all HDFS data is lost during this process!) Restart the cluster When deleting all the HDFS data and starting from scratch does not sound like a good idea (it might be ok during the initial setup/testing), you might give the second approach a try.

Workaround 2: Updating namespaceID of problematic DataNodes Big thanks to Jared Stehler for the following suggestion. I have not tested it myself yet, but feel free to try it out and send me your feedback. This workaround is "minimally invasive" as you only have to edit one file on the problematic DataNodes:

Stop the DataNode Edit the value of namespaceID in /current/VERSION to match the value of the current NameNode Restart the DataNode If you followed the instructions in my tutorials, the full path of the relevant files are:

NameNode: /app/hadoop/tmp/dfs/name/current/VERSION DataNode: /app/hadoop/tmp/dfs/data/current/VERSION (background: dfs.data.dir is by default set to ${hadoop.tmp.dir}/dfs/data, and we set hadoop.tmp.dir in this tutorial to /app/hadoop/tmp).

The solution for the problem is clearly given in the following site:

http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/#java-io-ioexception-incompatible-namespaceids

这篇关于Datanode在单机上的Hadoop中失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！