YARN容器内存未反映在Java Xmx设置中 [英] YARN container memory not reflected in java Xmx setting
问题描述
请帮助我理解我的理解或配置中的错误.
Please help me to understand the error in either my understanding or configuration.
我在YARN上运行Spark,并在yarn-site.xml中将最小容器内存分配设置为8GB:
I am running Spark on YARN, and have set the minimum container memory allocation to 8GB in yarn-site.xml:
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>8192</value>
</property>
我可以在Resource Manager UI中看到此设置:
I can see this setting reflected in the Resource Manager UI:
但是,当我在服务器上ps容器的Java进程时,最大堆大小设置为1024MB,即-Xmx1024m:
However, when I ps the container's java process on the server the max heap size is set to 1024MB i.e. -Xmx1024m:
root 542 535 1 16:18 ? 00:05:58 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -server -Xmx1024m -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1583021363029_0011/container_1583021363029_0011_03_000003/tmp ...
容器的Java进程ID为 542 :
The container's java process id is 542 :
Logs for container_1583021363029_0011_03_000003
0/03/02 16:18:57 INFO executor.CoarseGrainedExecutorBackend: Started daemon with process name: 542@xxxxxxxx.xxxx.xxx
20/03/02 16:18:57 INFO util.SignalUtils: Registered signal handler for TERM
20/03/02 16:18:57 INFO util.SignalUtils: Registered signal handler for HUP
20/03/02 16:18:57 INFO util.SignalUtils: Registered signal handler for INT
推荐答案
Java opts和YARN容器的大小是不同的属性
Java opts and YARN container sizes are distinct properties
最大堆大小设置可以在集群模式下使用
spark.driver.memory
设置,并在客户端模式下通过-driver-memory
命令行选项设置
Maximum heap size settings can be set with
spark.driver.memory
in the cluster mode and through the--driver-memory
command line option in the client mode
这篇关于YARN容器内存未反映在Java Xmx设置中的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!