Hadoop多节点集群.数据节点无法正常工作 [英] Hadoop Multinode cluster. Data node not working properly

问题描述

我正在将hadoop部署为多节点集群(分布式模式).但是每个数据节点都有不同的群集ID.

I'm deploying hadoop as a multi node cluster (distributed mode). But each data node is having different different cluster id.

在slave1上，

java.io.IOException: Incompatible clusterIDs in /home/pushuser1/hadoop/tmp/dfs/data: namenode clusterID = CID-c72a7d30-ec64-4e4f-9a80-e6f9b6b1d78c; datanode clusterID = CID-2ecca585-6672-476e-9931-4cfef9946c3b

在slave2上，

java.io.IOException: Incompatible clusterIDs in /home/pushuser1/hadoop/tmp/dfs/data: namenode clusterID = CID-c72a7d30-ec64-4e4f-9a80-e6f9b6b1d78c; datanode clusterID = CID-e24b0548-2d8d-4aa4-9b8c-a336193c006e

我也遵循了此链接Datanode无法正确启动，但是我不知道哪个集群ID我选.如果选择任何一个，则数据节点在该计算机上启动，但不在另一台计算机上启动.而且，当我使用基本命令(hadoop namenode-格式)格式化namenode时，每个从属节点上的datanode都启动了，但是主机上的namenode却没有启动.

I followed this link as well Datanode not starts correctly but I dont know which cluster id I should pick. If I pick any then data node starts on that machine but not on another one. And also when I format namenode using basic command (hadoop namenode - format), datanodes on each slave nodes are started but then namenode on master machine doesn't get started.

推荐答案

datanode和namenode的集群ID应该匹配，然后只有datanode才能有效地与namenode通信.如果您使用namenode格式，则会为namenodes分配新的ClusterID，则datanodes中的ClusterID将不匹配.

ClusterIDs of datanodes and namenodes should match, then only datanodes can effectively communicate with namenode. If you do namenode format new ClusterID will be assigned for namenodes then ClusterIDs in datanodes won't match.

您可以在/home/pushuser1/hadoop/tmp/dfs/data/current/(数据节点目录)以及namenode目录(/home/pushuser1/hadoop/)中找到一个 VERSION 文件tmp/dfs/name/current/，基于您为dfs.namenode.name.dir指定的值，该值包含ClusterID.

You can locate a VERSION files in your /home/pushuser1/hadoop/tmp/dfs/data/current/ (datanode directory ) as well as namenode directory(/home/pushuser1/hadoop/tmp/dfs/name/current/ based on the value your specified for dfs.namenode.name.dir) that contains the ClusterID.

如果您准备好格式化hdfs namenode，请停止所有HDFS服务，清除以下目录中的所有文件

If you are ready for format your hdfs namenode, Stop all HDFS services, Clear out all files inside the following directories

rm -rf /home/pushuser1/hadoop/tmp/dfs/data/*  (Need to execute on all data nodes)
rm -rf /home/pushuser1/hadoop/tmp/dfs/name/*

并再次格式化hdfs( hadoop namenode -format )

and format hdfs again (hadoop namenode -format )

这篇关于Hadoop多节点集群.数据节点无法正常工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！