Hadoop集群中的节点数 [英] Number of nodes in Hadoop cluster

问题描述

当我启动集群时，我的控制台是响应。

  hduser @ hadoop-master：/ usr / local / hadoop $ /usr/local/hadoop/sbin/start-dfs.sh 
启动namenodes on [ hadoop-master] 
 hadoop-master：启动namenode，记录到/usr/local/hadoop/logs/hadoop-hduser-namenode-hadoop-master.out 
 hadoop-master：启动datanode，记录到/ usr / local / hadoop / logs / hadoop-hduser-datanode-hadoop -a / child.out 
启动辅助名称节点[0.0.0.0] 
 0.0.0.0：启动secondarynamenode，记录到/usr/local/hadoop/logs/hadoop-hduser-secondarynamenode-hadoop-master.out 
 hduser @ hadoop-master：/ usr / local / hadoop $ jps 
 21079 NameNode 
 21258 DataNode 
 21479 SecondaryNameNode 
 21600 Jps 
 hduser @ hadoop-master： / usr / local / hadoop $ /usr/local/hadoop/sbin/start-yarn.sh 
启动yarn守护进程
启动resourcemanager，记录到/usr/local/hadoop/logs/yarn-hduser-resourcemanager-hadoop-master.out 
 hadoop-child：启动nodemanager，记录到/ usr / local /hadoop/logs/yarn-hduser-nodemanager-hadoop-child.out 
 hadoop-master：启动nodemanager，记录到/usr/local/hadoop/logs/yarn-hduser-nodemanager-hadoop-master.out 
 hduser @ hadoop-master：/ usr / local / hadoop $ jps 
 21079 NameNode 
 21258 DataNode 
 22117 Jps 
 21815 NodeManager 
 21479 SecondaryNameNode 
 21658 ResourceManager 
  

您可以看到在hadoop-child机器中启动的datanode。

现在，当我尝试获取所有节点信息时。

  hduser @ hadoop-master：/ usr / local / hadoop $ bin / hdfs dfsadmin -report 
已配置容量：21103243264（19.65 GB）
现有容量：17825124352（16.60 GB）
剩余的DFS：17821085696（16.60 GB）
使用的DFS：4038656（3.85 MB）
已使用DFS％：0.02％
在复制块中：0 
具有损坏复制块的块：0 
缺失块：0 
缺失块（复制因子1 ）：0 
 
 --------------------------------------- ---------- 
实时数据节点（1）：
 
名称：127.0.0.1:50010（本地主机）
主机名：localhost 
停用状态：正常
配置容量：21103243264（19.65 GB）
使用的DFS：4038656（3.85 MB）
使用的非DFS：3278118912（3.05 GB）
剩余的DFS：17821085696（16.60 GB）
使用的DFS百分比：0.02％
剩余的DFS百分比：84.45％
已配置的高速缓存容量：0（0 B）
使用的高速缓存：0（0 B）
缓存剩余数：0（0 B）
使用的缓存％：100.00％ 
 Cache剩余百分比：0.00％
 Xceivers：1 
最后一次联系：Sun Feb 26 17:13:04 UTC 2017 
  

我需要在这里查看主人和孩子的信息。我有1个主节点和1个子节点。

********************修复***** ** b
$ b

基于Frank这是如何修复的。

1. 编辑了/ etc / config文件并提供了主和子ip地址。

更改了下面2行

  127.0.0.1 localhost hadoop-master 
 961.118.98.183 hadoop-child 
  

到（在两个节点中）

  127.0.0.1 localhost 
 961.118.99.251 hadoop-master 
 961.118.98.183 hadoop-child 
  

2.Restart数据节点应该被重新格式化。
$ b $ pre $ 格式将只删除元数据，数据节点的数据目录将会仍然使用旧的namenode的身份，这将导致datanode失败（所以删除目录）。

请问这个命令。
1）停止集群
2）rm -rf / path / to / datanode / data / dir（在两个节点中）
3）hadoop namenode -format
4）START cluste


解决方案在设置多节点环境时，必须明确定义 namenode 和 resourcemanager 地址。
$ b

在两个节点中将此属性添加到 core-site.xml 中，

 < property> 
<名称> fs.defaultFS< / name> 
<值> hdfs：// hadoop-master：8020< /值> 
< / property> 
  

同样在 yarn-site.xml ，

 < property> 
< name> yarn.resourcemanager.hostname< / name> 
<值> hadoop-master< /值> 
< / property> 
  

请确保在 / etc / hosts 所有节点的文件，IP地址和主机名映射完成。

注意：如果服务正在运行，请停止它们并在再次启动服务前添加这些属性。

Im trying to setup a Hadoop Multi node cluster.

When I start my cluster here is my response in console..

hduser@hadoop-master:/usr/local/hadoop$ /usr/local/hadoop/sbin/start-dfs.sh
Starting namenodes on [hadoop-master]
hadoop-master: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hduser-namenode-hadoop-master.out
hadoop-master: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hduser-datanode-hadoop-master.out
hadoop-child: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hduser-datanode-hadoop-child.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hduser-secondarynamenode-hadoop-master.out
hduser@hadoop-master:/usr/local/hadoop$ jps
21079 NameNode
21258 DataNode
21479 SecondaryNameNode
21600 Jps
hduser@hadoop-master:/usr/local/hadoop$ /usr/local/hadoop/sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-hadoop-master.out
hadoop-child: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-hadoop-child.out
hadoop-master: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-hadoop-master.out
hduser@hadoop-master:/usr/local/hadoop$ jps
21079 NameNode
21258 DataNode
22117 Jps
21815 NodeManager
21479 SecondaryNameNode
21658 ResourceManager

You can see a datanode being started in hadoop-child machine.

Now, when I try to get all the node information. I don't see all the nodes being displayed.

hduser@hadoop-master:/usr/local/hadoop$ bin/hdfs dfsadmin -report
Configured Capacity: 21103243264 (19.65 GB)
Present Capacity: 17825124352 (16.60 GB)
DFS Remaining: 17821085696 (16.60 GB)
DFS Used: 4038656 (3.85 MB)
DFS Used%: 0.02%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (1):

Name: 127.0.0.1:50010 (localhost)
Hostname: localhost
Decommission Status : Normal
Configured Capacity: 21103243264 (19.65 GB)
DFS Used: 4038656 (3.85 MB)
Non DFS Used: 3278118912 (3.05 GB)
DFS Remaining: 17821085696 (16.60 GB)
DFS Used%: 0.02%
DFS Remaining%: 84.45%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Feb 26 17:13:04 UTC 2017

I need to view information of both master and child here. I have 1 master and 1 child node.

********************Fix*******

Based on Frank this is how it was fixed..

1. Edited the /etc/config file and gave both the master and child ipaddress.

changed the below 2 line

127.0.0.1 localhost hadoop-master
961.118.98.183 hadoop-child

to (in both nodes)

127.0.0.1 localhost 
961.118.99.251 hadoop-master
961.118.98.183 hadoop-child

2.Restart the cluster in the following order..the datanode should be reformatted.

format will remove only the meta, the datanode's data directories will still be using the old namenode's identity which wil cause the datanode to fail(so delete the directories).

Can you please this order.. 
1) stop the cluster 
2) rm -rf /path/to/datanode/data/dir (in both nodes) 
3) hadoop namenode -format
4) START cluste

解决方案

When setting up a multinode environment, the namenode and the resourcemanager addresses must be explicitly defined.

Add this property to core-site.xml in both the nodes,

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://hadoop-master:8020</value>
</property>

Also this in yarn-site.xml,

<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>hadoop-master</value>
</property>

Make sure in the /etc/hosts file of all the nodes, the IP address and hostname mapping is done.

Note: If the services are running, stop them and add these properties before starting them again.

这篇关于Hadoop集群中的节点数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！