Hadoop MapReduce（Yarn）使用具有不同功率/规格的主机 [英] Hadoop MapReduce (Yarn) using hosts with different power/specifications

问题描述

我目前在群集中拥有高功率（cpu / ram）主机，我们正在考虑添加一些优质存储，但是低功耗主机。

我担心的是会降低工作表现。来自新的（较弱的）主机的Map / Reducers将运行得更慢，而更强大的主机将只需等待结果。

有没有办法配置这个在纱线？也许可以为主机设置优先级，或根据每台机器上的内核数量来分配映射器/缩减器。

谢谢，
Horatiu

解决方案

在YARN中，您可以为每个群集工作人员提供不同的硬件资源配置。然后，YARN将决定每个工作人员可以分配多少个任务（容器）来平衡工作负载。换句话说，您可以设置不同数量的内核和内存 yarn-site.xml 分别为每个工人。

这两个配置选项与您相关：

• yarn.nodemanager.resource.cpu-vcores 设置每个当前worker的CPU核心数节点


• yarn.nodemanager.resource.memory-mb 来设置可以使用的物理内存量（以MB为单位）当前工作节点

I currently have high power (cpu/ram) hosts in the cluster and we are considering to add some good storage but low power hosts.

My concern is that it will reduce the jobs performance. Map/Reducers from the new (less powerful) hosts will run slower and the more powerful ones will just have to wait for the result.

Is there a way to configure this in Yarn ? Maybe to set a priority for the hosts or to assign mapper/reducers according to the number of cores on each machines.

Thanks, Horatiu

解决方案

In YARN you can provide a different hardware resources configuration to each of the cluster workers. The YARN will then decide how many tasks (containers) may be allocated at each worker to balance the workload.

In other words you can set different number of cores and memory in yarn-site.xml for each worker separately.

These two configuraton options are relevant for you:

• yarn.nodemanager.resource.cpu-vcores to set number of CPU cores per current worker node
• yarn.nodemanager.resource.memory-mb to set the amount of physical memory (in MB) that can be used at the current worker node

这篇关于Hadoop MapReduce（Yarn）使用具有不同功率/规格的主机的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！