如何将Hadoop配置文件同步到多个节点？ [英] How to sync Hadoop configuration files to multiple nodes?

问题描述

我需要管理只有3台运行Hadoop的Centos机器的集群。因此， scp 足以让我将配置文件复制到其他2台机器。

I uesd to manage a cluster of only 3 Centos machines running Hadoop. So scp is enough for me to copy the configuration files to the other 2 machines.

但是，我必须为超过10台机器设置一个Hadoop集群。使用 scp 来同步文件真的很沮丧。

However, I have to setup a Hadoop cluster to more than 10 machines. It is really frustrated to sync the files so many times using scp.

我想找到一个工具，可以轻松地将文件同步到所有机器。并且机器名称在配置文件中定义，例如：

I want to find a tool that I can easily sync the files to all machines. And the machine names are defined in a config file, such as:

node1
node2
...
node10

感谢。

推荐答案

如果你不想使用Zookeeper，你可以修改 $ HADOOP_HOME / bin / hadoop 中的hadoop脚本， ：

If you do not want to use Zookeeper you can modify your hadoop script in $HADOOP_HOME/bin/hadoop and add something like :

if [ "$COMMAND" == "deployConf" ]; then
  for HOST in `cat $HADOOP_HOME/conf/slaves`
    do
       scp $HADOOP_HOME/conf/mapred-site.xml $HOST:$HADOOP_HOME/conf
       scp $HADOOP_HOME/conf/core-site.xml $HOST:$HADOOP_HOME/conf
       scp $HADOOP_HOME/conf/hdfs-site.xml $HOST:$HADOOP_HOME/conf
    done
    exit 0
fi

这是我现在正在使用的工作。

That's what I'm using now and it does the job.

这篇关于如何将Hadoop配置文件同步到多个节点？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！