Apache Hadoop YARN 中“mapreduce.map.memory.mb"和“mapred.map.child.java.opts"之间的关系是什么? [英] What is the relation between 'mapreduce.map.memory.mb' and 'mapred.map.child.java.opts' in Apache Hadoop YARN?
问题描述
我想知道mapreduce.map.memory.mb和mapred.map.child.java.opts参数之间的关系.
I would like to know the relation between the mapreduce.map.memory.mb and mapred.map.child.java.opts parameters.
是mapreduce.map.memory.mb > mapred.map.child.java.opts吗?
推荐答案
mapreduce.map.memory.mb 是 Hadoop 允许分配给映射器的内存上限，以兆字节为单位.默认值为 512.如果超过此限制，Hadoop 将终止映射器并显示如下错误:
mapreduce.map.memory.mb is the upper memory limit that Hadoop allows to be allocated to a mapper, in megabytes. The default is 512. If this limit is exceeded, Hadoop will kill the mapper with an error like this:
容器[pid=container_1406552545451_0009_01_000002,containerID=container_234132_0001_01_000001]正在运行超出物理内存限制.当前使用量:569.1 MB已使用 512 MB 物理内存；使用了 970.1 MB 的 1.0 GB 虚拟内存.杀死容器.
Container[pid=container_1406552545451_0009_01_000002,containerID=container_234132_0001_01_000001] is running beyond physical memory limits. Current usage: 569.1 MB of 512 MB physical memory used; 970.1 MB of 1.0 GB virtual memory used. Killing container.
Hadoop 映射器是一个 Java 进程，每个 Java 进程都有自己的堆内存最大分配设置，通过 mapred.map.child.java.opts(或 Hadoop 中的 mapreduce.map.java.opts)配置2+).如果mapper进程耗尽堆内存，mapper会抛出java内存不足异常:
Hadoop mapper is a java process and each Java process has its own heap memory maximum allocation settings configured via mapred.map.child.java.opts (or mapreduce.map.java.opts in Hadoop 2+). If the mapper process runs out of heap memory, the mapper throws a java out of memory exceptions:
错误:java.lang.RuntimeException:java.lang.OutOfMemoryError
Error: java.lang.RuntimeException: java.lang.OutOfMemoryError
因此，Hadoop 和 Java 设置是相关的.Hadoop 设置更像是一种资源强制/控制，而 Java 更像是一种资源配置.
Thus, the Hadoop and the Java settings are related. The Hadoop setting is more of a resource enforcement/controlling one and the Java is more of a resource configuration one.
Java 堆设置应该小于 Hadoop 容器内存限制，因为我们需要为 Java 代码保留内存.通常，建议为代码保留 20% 的内存.因此，如果设置正确，基于 Java 的 Hadoop 任务永远不会被 Hadoop 杀死，因此您永远不会看到像上面这样的杀死容器"错误.
The Java heap settings should be smaller than the Hadoop container memory limit because we need reserve memory for Java code. Usually, it is recommended to reserve 20% memory for code. So if settings are correct, Java-based Hadoop tasks should never get killed by Hadoop so you should never see the "Killing container" error like above.
如果您遇到 Java 内存不足错误，则必须增加这两个内存设置.
If you experience Java out of memory errors, you have to increase both memory settings.
这篇关于Apache Hadoop YARN 中“mapreduce.map.memory.mb"和“mapred.map.child.java.opts"之间的关系是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
