hadoop群集应该运行在相同的硬件上吗？ [英] Should hadoop clusters run on identical hardware?

问题描述

我记得在某处读到Hadoop的性能如果运行的机器与另一台机器非常不同时会显着恶化，但我似乎无法再发现该评论。我正在考虑在不是由我的组直接管理的虚拟机阵列上运行Hadoop集群，并且我需要知道这是否是我的要求。

$ b $所以，我应该坚持所有的机器都具有相同的硬件，还是可以在不同的硬件配置的机器上运行？

谢谢。 / p>

解决方案

下面的文章描述了异构集群如何影响hadoop map-reduce的性能：

在异构集群中，节点的计算能力可能显着变化
。高速节点可以比低速对等节点更快地处理存储在节点的
本地磁盘中的处理数据。在
快速节点完成对其本地输入数据的处理后，节点
必须通过处理位于
或更多远程慢节点中的未处理数据来支持负载共享。当由于
负载共享而传输的数据量非常大时，将未处理的数据
从慢节点移动到快节点的开销成为影响
Hadoop性能的关键问题。

以下参考资料提供了更多详细信息：

1. http://computerresearch.org/stpr/index.php/gjcst/article/ view / 749/658


2. http://www.usenix.org/event/osdi08/tech/full_papers/zaharia/zaharia.pdf
   

它还提供了可以提高异构群集性能或避免此类性能损失的方法。

明智地建议您在同类机器上运行你的集群，但如果这些机器没有非常不同的规格和d性能差异，您应该继续构建您的集群。

对于生产系统，您应该建议使用同质机器。对于开发而言，性能并不重要。

然而，在构建Hadoop集群之后，您应该能够进行基准测试。

I remember reading somewhere that Hadoop's performance deteriorates significantly if the machines it runs on are very different from one another, but I can't seem to find that comment anymore. I am considering running a Hadoop cluster on an array of VMs that is not directly managed by my group, and I need to know if this is a requirement that I should put in my request.

So, should I insist on all of my machines having identical hardware, or is it okay to run on different machines in different hardware configurations?

Thanks.

解决方案

Following papers describes how heterogeneous cluster affect the performance of hadoop map-reduce:

In a heterogeneous cluster, the computing capacities of nodes may vary significantly. A high-speed node can finish processing data stored in a local disk of the node faster than low-speed counterparts. After a fast node complete the processing of its local input data, the node must support load sharing by handling unprocessed data located in one or more remote slow nodes. When the amount of transferred data due to load sharing is very large, the overhead of moving unprocessed data from slow nodes to fast nodes becomes a critical issue affecting Hadoop’s performance.

Following references has more details:

1. http://computerresearch.org/stpr/index.php/gjcst/article/view/749/658
2. http://www.usenix.org/event/osdi08/tech/full_papers/zaharia/zaharia.pdf

It also provides ways in which you could improve the performance on heterogeneous cluster or avoid this performance penalty.

It is wisely suggested that you have homogenous machines on your cluster but if these machines do not have wildly different specifications and performance difference, you should carry on with building your cluster.

For production systems, you should suggest for homogenous machines. For development, performance is not critical.

How ever, you should be able to benchmark your Hadoop cluster after you have built it.

这篇关于hadoop群集应该运行在相同的硬件上吗？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！