`yarn.scheduler.maximum-allocation-mb` 和 `yarn.nodemanager.resource.memory-mb` 之间的区别? [英] Difference between `yarn.scheduler.maximum-allocation-mb` and `yarn.nodemanager.resource.memory-mb`?

问题描述

yarn.scheduler.maximum-allocation-mb 和 yarn.nodemanager.resource.memory-mb 有什么区别?

我在 yarn-site.xml 中看到了这两个并且我看到了解释 此处.

I see both of these in yarn-site.xml and I see the explanations here.

yarn.scheduler.maximum-allocation-mb 给出以下定义: RM 处每个容器请求的最大分配，以 MB 为单位.高于此值的内存请求将引发 InvalidResourceRequestException. 这是否意味着仅资源管理器上的内存请求受此值限制?

yarn.scheduler.maximum-allocation-mb is given the following definition: The maximum allocation for every container request at the RM, in MBs. Memory requests higher than this will throw a InvalidResourceRequestException. Does this mean memory requests ONLY on the resourcemanager are limited by this value?

并且 yarn.nodemanager.resource.memory-mb 定义了可以为容器分配的物理内存量，以 MB 为单位. 这是否意味着整个集群中所有容器的总和?

And yarn.nodemanager.resource.memory-mb is given definition of Amount of physical memory, in MB, that can be allocated for containers. Does this mean the total amount for all containers across the entire cluster, summed together?

然而，我仍然无法区分这些.这些解释让我觉得它们是一样的.

HOwever, I still cannot discern between these. Those explanations make me think that they are the same.

更令人困惑的是，它们的默认值完全相同:8192 mb.我如何区分这些之间的区别?谢谢.

Even more confusing, their default values are exactly the same: 8192 mb. How do I tell difference between these? Thank you.

推荐答案

假设您正在设置一个集群，其中每台机器都有 48 GB 的 RAM.其中一些 RAM 应保留用于操作系统和其他已安装的应用程序.

Consider in a scenario where you are setting up a cluster where each machine having 48 GB of RAM. Some of this RAM should be reserved for Operating System and other installed applications.

yarn.nodemanager.resource.memory-mb:

可以为容器分配的物理内存量，以 MB 为单位.这意味着 YARN 可以在此节点上使用的内存量，因此此属性应该低于那台机器的总内存.

Amount of physical memory, in MB, that can be allocated for containers. It means the amount of memory YARN can utilize on this node and therefore this property should be lower than the total memory of that machine.

<name>yarn.nodemanager.resource.memory-mb</name>
<value>40960</value> <!-- 40 GB -->

<小时>

下一步是提供 YARN 指导，说明如何将可用的总资源分解为容器.为此，您可以指定为容器分配的最小 RAM 单位.

---

The next step is to provide YARN guidance on how to break up the total resources available into Containers. You do this by specifying the minimum unit of RAM to allocate for a Container.

在yarn-site.xml

<name>yarn.scheduler.minimum-allocation-mb</name> <!-- RAM-per-container ->
 <value>2048</value>

yarn.scheduler.maximum-allocation-mb:

它定义了容器可用的最大内存分配(以 MB 为单位)

It defines the maximum memory allocation available for a container in MB

这意味着RM只能以"yarn.scheduler.minimum-allocation-mb"的增量为容器分配内存，并且不能超过"yarn.scheduler.maximum-allocation-mb" 并且它不应超过节点的总分配内存.

it means RM can only allocate memory to containers in increments of "yarn.scheduler.minimum-allocation-mb" and not exceed "yarn.scheduler.maximum-allocation-mb" and It should not be more then total allocated memory of the Node.

在yarn-site.xml

<name>yarn.scheduler.maximum-allocation-mb</name> <!-Max RAM-per-container->
 <value>8192</value>

对于 MapReduce 应用程序，YARN 在一个容器中处理每个 map 或 reduce 任务，并且在一台机器上可以有多个容器.我们希望在每个节点上最多允许 20 个容器，因此需要(40 GB 总 RAM)/(20 # 个容器)= 由属性 yarn.scheduler.minimum-allocation- 控制的每个容器最小 2 GBmb

For MapReduce applications, YARN processes each map or reduce task in a container and on a single machine there can be number of containers. We want to allow for a maximum of 20 containers on each node, and thus need (40 GB total RAM) / (20 # of containers) = 2 GB minimum per container controlled by property yarn.scheduler.minimum-allocation-mb

我们再次希望限制由属性 "yarn.scheduler.maximum-allocation-mb"

Again we want to restrict maximum memory utilization for a container controlled by property "yarn.scheduler.maximum-allocation-mb"

例如，如果一项作业要求每个地图容器 2049 MB 内存(mapreduce.map.memory.mb=2048 set in mapred-site.xml)，RM 会给它一个 4096MB(2*yarn.scheduler.minimum-allocation-mb) 容器.

For example, if one job is asking for 2049 MB memory per map container(mapreduce.map.memory.mb=2048 set in mapred-site.xml), RM will give it one 4096 MB(2*yarn.scheduler.minimum-allocation-mb) container.

如果您有一个巨大的 MR 作业需要 9999 MB 的地图容器，该作业将被终止并显示错误消息.

If you have a huge MR job which asks for a 9999 MB map container, the job will be killed with the error message.

这篇关于`yarn.scheduler.maximum-allocation-mb` 和 `yarn.nodemanager.resource.memory-mb` 之间的区别?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！