如何为Hadoop生态系统配置主机文件 [英] How to configure hosts file for Hadoop ecosystem

问题描述

！[输入图像描述] [1]这个问题可能看起来很明显，但由于hadoop集群中主机文件配置不当，我多次遇到它。

任何人都可以描述如何为hadoop和类似的环境使用（如cloudera）设置主机文件和其他相关网络配置。

特别是当我必须添加无论是主机名还是FQDN

更新

以下是一个主机文件主机名为cdh4hdm的机器拥有hadoop Master的角色

  127.0.0.1 cdh4hdm localhost 
＃127.0.1.1 cdh4hdm 
 
＃以下几行适用于支持IPv6的主机
 
 172.26.43.40 cdh4hdm.imp.co.in kdc1 
 172.26.43.41 cdh4hbm.imp.co。 in 
 172.26.43.42 cdh4s1.imp.co.in 
 172.26.43.43 cdh4s2.imp.co.in 
 172.26.43.44 cdh4s3.imp.co.in 
 172.26.43.45 cdh4s4.imp.co.in 
 
 :: 1 ip6-localhost ip6-loopback 
 fe00 :: 0 ip6-localnet 
 ff00 :: 0 ip6-mcastprefix 
 ff02 :: 1 ip6-allnodes 
 ff02 :: 2 ip6-allrouters 
  

请参阅附加图像

在群集中，一些节点正在获取FQDN，另一些获取主机名。

主机名的IP不正确，显示127.0.0.1而不是主机IP

请建议 p>

解决方案

对于 UBUNTU

Hadoop集群的其他配置

为所有群集机器提供主机名，这样做可以将/ etc / hostname文件中的主机名添加为

  hostname-of-machine 
  

主机，主机文件应该是这样的：

hosts
$ b

  127.0.0.1本地主机
＃127.0.1.1本地主机
 
<主机的IP地址> FQDN主机名other_name 
 172.26.43.10 cdh4hdm.domain.com cdh4hdm kdc1 
 172.26.43.11 cdh4hbm.domain.com cdh4hbm 
 172.26.43.12 cdh4s1.domain.com cdh4s1 
 172.26.43.13 cdh4s2.domain.com cdh4s2 
 172.26.43.14 cdh4s3.domain.com cdh4s3 
 172.26.43.15 cdh4s4.domain.com cdh4s4 
  

注意：确保注释行127.0.1.1 localhost它可能会在zookeeper和cluster中创建问题。

在/etc/resolv.conf中添加DNS服务器IP

resolve.conf

 搜索domain.com 
名称服务器10.0.1.1 
  

验证配置检查hostfile，并且您应该能够通过主机名ping所有机器

要检查所有机器上的主机名和FQDN，请运行以下命令： / p>

 主机名//应该返回主机名
主机名-f // Fully Qualifi ed主机名
主机名-d //域名
  

对于RHEL，所有命令都是相同的主机名。

Source1 a>和 Source2

![enter image description here][1]The question may seem pretty obvious, but I have faced it many times, due to bad configuration of hosts file on a hadoop cluster.

Can anyone describe how to setup hosts file and other related network configuration for hadoop and similar environment usage (like cloudera).

Specially when i have to add both the hostname and FQDN

Update

Here is the host file of one of the machine from host name cdh4hdm have role of hadoop Master

 127.0.0.1       cdh4hdm        localhost
  #127.0.1.1      cdh4hdm 

 # The following lines are desirable for IPv6 capable hosts

   172.26.43.40    cdh4hdm.imp.co.in            kdc1
   172.26.43.41    cdh4hbm.imp.co.in   
   172.26.43.42    cdh4s1.imp.co.in    
   172.26.43.43    cdh4s2.imp.co.in    
   172.26.43.44    cdh4s3.imp.co.in    
   172.26.43.45    cdh4s4.imp.co.in    

   ::1     ip6-localhost ip6-loopback
   fe00::0 ip6-localnet
   ff00::0 ip6-mcastprefix
   ff02::1 ip6-allnodes
   ff02::2 ip6-allrouters 

Please see image attached

Here on cluster some nodes are getting FQDN and some are getting hostname.

Also IP of hostname is not proper and showing 127.0.0.1 instead of host IP

Please suggest

解决方案

For UBUNTU

Hosts File and other configuration for Hadoop Cluster

Provide hostname to all cluster machines, to do so add hostname in /etc/hostname file as

hostname-of-machine

On all the host, hosts file should be like this:

hosts

127.0.0.1       localhost
#127.0.1.1      localhost

<ip of host>    FQDN                hostname    other_name
172.26.43.10    cdh4hdm.domain.com  cdh4hdm     kdc1
172.26.43.11    cdh4hbm.domain.com  cdh4hbm
172.26.43.12    cdh4s1.domain.com   cdh4s1
172.26.43.13    cdh4s2.domain.com   cdh4s2
172.26.43.14    cdh4s3.domain.com   cdh4s3
172.26.43.15    cdh4s4.domain.com   cdh4s4

Note: Make sure to comment line 127.0.1.1 localhost it may create problem in zookeeper and cluster.

Add DNS server IP in /etc/resolv.conf

resolve.conf

search domain.com
nameserver 10.0.1.1

to verify configuration check hostfile and your should be able to ping all the machines by their hostname

To check hostname and FQDN on all machines run following commands:

hostname        //should return the hostname
hostname -f     //Fully Qualified Hostname
hostname -d     //Domain name

All commands will be same for RHEL except the hostname.

Source1 and Source2

这篇关于如何为Hadoop生态系统配置主机文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！