Hadoop集群-"hadoop"用户ssh通讯 [英] Hadoop Cluster - "hadoop" user ssh communication

问题描述

我正在EC2服务器上设置Hadoop 2.7.3集群-1个NameNode，1个次要NameNode和2个DataNode.

I am setting up Hadoop 2.7.3 cluster on EC2 servers - 1 NameNode, 1 Secondary NameNode and 2 DataNodes.

Hadoop核心使用SSH与从属进行通信以在从属节点上启动进程.

Hadoop core uses SSH for communication with slaves to launch the processes on the slave node.

1. 我们需要在hadoop用户的所有节点上使用相同的SSH密钥吗?
2. 将NameNode复制或添加到Slave节点SSH凭据的最佳实践/理想方法是什么?

推荐答案

我们需要在hadoop用户的所有节点上使用相同的SSH密钥吗?

Do we need to have same SSH keys on all the nodes for the hadoop user?

• 相同的公钥必须位于所有节点上

将NameNode复制或添加到的最佳实践/理想方法是什么 从节点SSH凭据?

What is the best practice/ideal way to copy or add the NameNode to Slave nodes SSH credentials?

根据文档:

Per documentation:

名称节点:少了SSH密码

名称节点和数据节点之间的无密码SSH.让我们 为此，请在名称节点上创建一个公钥-私钥对.

Password-less SSH between the name nodes and the data nodes. Let us create a public-private key pair for this purpose on the namenode.

namenode> ssh-keygen

使用默认值(/home/ubuntu/.ssh/id_rsa)作为键位置，并 按Enter键输入空密码.

Use the default (/home/ubuntu/.ssh/id_rsa) for the key location and hit enter for an empty passphrase.

数据节点:设置公共密钥

公钥保存在/home/ubuntu/.ssh/id_rsa.pub中.我们要 将此文件从namenode复制到每个数据节点，然后附加 内容添加到每个数据节点上的/home/ubuntu/.ssh/authorized_keys.

The public key is saved in /home/ubuntu/.ssh/id_rsa.pub. We need to copy this file from the namenode to each data node and append the contents to /home/ubuntu/.ssh/authorized_keys on each data node.

datanode1> cat id_rsa.pub >> ~/.ssh/authorized_keys
datanode2> cat id_rsa.pub >> ~/.ssh/authorized_keys
datanode3> cat id_rsa.pub >> ~/.ssh/authorized_keys

名称节点:设置SSH配置

SSH将〜/.ssh/config中的配置文件用于各种 参数.如下所示进行设置.再次，替换每个节点的 HostName参数的公共DNS(例如，替换 使用EC2的NameNode的公共DNS).

SSH uses a configuration file located at ~/.ssh/config for various parameters. Set it up as shown below. Again, substitute each node’s Public DNS for the HostName parameter (for example, replace with EC2 Public DNS for NameNode).

Host nnode
  HostName <nnode>
  User ubuntu
  IdentityFile ~/.ssh/id_rsa

Host dnode1
  HostName <dnode1>
  User ubuntu
  IdentityFile ~/.ssh/id_rsa

Host dnode2
  HostName <dnode2>
  User ubuntu
  IdentityFile ~/.ssh/id_rsa

Host dnode3
  HostName <dnode3>
  User ubuntu
  IdentityFile ~/.ssh/id_rsa

这时，请验证无密码操作是否在每个节点上都有效 如下所示(第一次，您将收到一条警告，指出主机是 未知以及是否要连接到它.输入yes，然后按Enter. 此步骤仅需要一次):

At this point, verify that password-less operation works on each node as follows (the first time, you will get a warning that the host is unknown and whether you want to connect to it. Type yes and hit enter. This step is needed once only):

namenode> ssh nnode
namenode> ssh dnode1
namenode> ssh dnode2
namenode> ssh dnode3

这篇关于Hadoop集群-"hadoop"用户ssh通讯的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！