hadoop集群仅使用主节点或所有节点 [英] hadoop cluster is using only master node or all nodes

问题描述

我创建了一个 4节点hadoop群集。我启动所有datanode，namenode资源管理器等。

要查找我的所有节点是否在工作，我尝试了以下过程：

第1步。当所有节点都处于活动状态时运行我的程序

第2步。仅当已激活。

两种情况下的完成时间几乎都相同。

所以，我想知道是否有其他方法可以让我知道在运行程序时实际使用了多少个节点。

解决方案

在聊天中讨论。这个问题是由不正确的Hadoop安装引起的，在这两种情况下，作业都是使用LocalJobRunner在本地启动的。

建议：

1. 使用Ambari安装Hadoop（ http://ambari.apache.org/


2. 将平台改为CentOS 6.4 +


3. 使用Oracle JDK 7


4. 和防火墙


5. 熟悉用于运行状况诊断和默认Hadoop WebUI的群集命令

I have created a 4-node hadoop cluster. I start all datanodes,namenode resource manager,etc.

To find whether all of my nodes are working or not, I tried the following procedure:

Step 1. I run my program when all nodes are active
Step 2. I run my program when only master is active.

The completion time in both cases were almost same.

So, I would like to know if there is any other means by which I can know how many nodes are actually used while running the program.

解决方案

Discussed in the chat. The problem is caused by incorrect Hadoop installation, in both cases job was started locally using LocalJobRunner.

As a recommendations:

1. Install Hadoop using Ambari (http://ambari.apache.org/)
2. Change platform to CentOS 6.4+
3. Use Oracle JDK 7
4. Be patient with host names and firewall
5. Get familiar with the cluster commands for health diagnostics and default Hadoop WebUIs

这篇关于hadoop集群仅使用主节点或所有节点的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！