什么是纱线上保留的内存 [英] What is Memory reserved on Yarn

问题描述

我设法在Yarn上启动了一个Spark应用程序。但是，如下所示，emory的用法有点奇怪：

http ：//imgur.com/1k6VvSI

保留内存是什么意思？我怎样才能有效地使用所有可用的内存？

感谢您的提前。

解决方案

查看来自Cloudera的这个博客解释了YARN中的新内存管理。

以下是相关内容：

< blockquote>

...此更改的实现细节可防止应用程序在此新灵活性下挨饿，这是保留容器的概念。想象一下，两个工作正在运行，每个工作都有足够的任务来饱和整个集群。一份工作是希望每个映射器获得1GB，另一份工作是希望映射器获得2GB。假设第一份工作开始并填满整个集群。只要其中一项任务完成，它就会打开一个1GB的插槽。尽管第二份工作值得这个空间，但一份天真的政策将会给予第一份工作，因为这是唯一适合工作的工作。这可能会导致第二份工作无限期地饿死。
为了防止这种不幸的情况，当一个节点上的空间被提供给应用程序时，如果应用程序不能立即使用它，它就会保留它，并且在该节点上没有其他应用程序可以被分配到该节点上的容器，直到预留满足。每个节点可能只有一个保留容器。总预留内存量在ResourceManager UI中报告。数量很大意味着新工作可能需要更长时间才能获得空间。 ,,,

I managed to launch a spark application on Yarn. However emory usage is kind of weird as you can see below :

http://imgur.com/1k6VvSI

What does memory reserved mean ? How can i manage to efficiently use all the memory available ?

Thanks in advance.

解决方案

Check out this blog from Cloudera that explains the new memory management in YARN.

Here's the pertinent bits:

... An implementation detail of this change that prevents applications from starving under this new flexibility is the notion of reserved containers. Imagine two jobs are running that each have enough tasks to saturate more than the entire cluster. One job wants each of its mappers to get 1GB, and another job wants its mappers to get 2GB. Suppose the first job starts and fills up the entire cluster. Whenever one of its task finishes, it will leave open a 1GB slot. Even though the second job deserves the space, a naive policy will give it to the first one because it’s the only job with tasks that fit. This could cause the second job to be starved indefinitely. To prevent this unfortunate situation, when space on a node is offered to an application, if the application cannot immediately use it, it reserves it, and no other application can be allocated a container on that node until the reservation is fulfilled. Each node may have only one reserved container. The total reserved memory amount is reported in the ResourceManager UI. A high number means that it may take longer for new jobs to get space. ,,,

这篇关于什么是纱线上保留的内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！