为什么Java本机内存跟踪中的内部内存增加 [英] Why is Internal memory in java Native Memory Tracking increasing
问题描述
我的应用程序在docker容器中运行，它使用scala并使用"OpenJDK 64位服务器VM(内部版本25.102-b14，混合模式)"，其Xmx设置为16GB，容器内存限制为24Gb，之后运行了一段时间的容器被杀死了:
My application is running in a docker container, it use scala and use "OpenJDK 64-Bit Server VM (build 25.102-b14, mixed mode)", its Xmx is set to be 16GB and container memory limit is 24Gb, after running for some time the container is killed:
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
但是，我在日志中找不到任何"java.lang.OutOfMemoryError:Java堆空间"错误，即使在过去的两周内，在所有48个节点中也没有发现任何错误.因此，这不太可能是普通的堆OOM.
However I can't find any "java.lang.OutOfMemoryError: Java heap space" errors in the log, not even once in last 2 weeks in all 48 nodes. So it's not likely a normal heap OOM.
dmesg输出:
$ dmesg -l err,crit,alert,emerg
STDIN is not a terminal
[1647254.978515] Memory cgroup out of memory: Kill process 10924 (java) score 1652 or sacrifice child
[1647254.989138] Killed process 10924 (java) total-vm:34187148kB, anon-rss:24853120kB, file-rss:23904kB
[1655749.664871] Memory cgroup out of memory: Kill process 1969 (java) score 1652 or sacrifice child
[1655749.675513] Killed process 1969 (java) total-vm:35201940kB, anon-rss:24856624kB, file-rss:24120kB
[1655749.987605] Memory cgroup out of memory: Kill process 2799 (java) score 1656 or sacrifice child
然后我多次运行JCMD，然后再次将其杀死，数据如下所示: 本机内存跟踪:
I then run JCMD multiple times before it is killed again and the data looks like the following: Native Memory Tracking:
总计:保留= 25505339KB，已提交= 25140947KB -Java堆(保留= 16777216KB，已提交= 16777216KB) (mmap:reserved = 16777216KB，committed = 16777216KB)
Total: reserved=25505339KB, committed=25140947KB - Java Heap (reserved=16777216KB, committed=16777216KB) (mmap: reserved=16777216KB, committed=16777216KB)
-
Class(保留= 247996KB，已提交= 93500KB) (#14539类) (malloc = 2236KB#29794) (mmap:reserved = 245760KB，committed = 91264KB)
Class (reserved=247996KB, committed=93500KB) (classes #14539) (malloc=2236KB #29794) (mmap: reserved=245760KB, committed=91264KB)
线程(保留= 1013160KB，已提交= 1013160KB) (线程#1902) (堆栈:保留= 1003956KB，已提交= 1003956KB) (malloc = 6240KB#9523) (arena = 2964KB#3803)
Thread (reserved=1013160KB, committed=1013160KB) (thread #1902) (stack: reserved=1003956KB, committed=1003956KB) (malloc=6240KB #9523) (arena=2964KB #3803)
代码(保留= 263255KB，已提交= 86131KB) (malloc = 13655KB#20964) (mmap:reserved = 249600KB，committed = 72476KB)
Code (reserved=263255KB, committed=86131KB) (malloc=13655KB #20964) (mmap: reserved=249600KB, committed=72476KB)
GC(保留= 776174KB，已提交= 776174KB) (malloc = 120814KB#164310) (mmap:reserved = 655360KB，committed = 655360KB)
GC (reserved=776174KB, committed=776174KB) (malloc=120814KB #164310) (mmap: reserved=655360KB, committed=655360KB)
编译器(保留= 812KB，已提交= 812KB) (malloc = 681KB#1823) (arena = 131KB#3)
Compiler (reserved=812KB, committed=812KB) (malloc=681KB #1823) (arena=131KB #3)
内部(保留= 6366260KB，已提交= 6366256KB) (malloc = 6366256KB#178778) (mmap:reserved = 4KB，committed = 0KB)
Internal (reserved=6366260KB, committed=6366256KB) (malloc=6366256KB #178778) (mmap: reserved=4KB, committed=0KB)
符号(保留= 18391KB，已提交= 18391KB) (malloc = 16242KB#153138) (arena = 2150KB#1)
Symbol (reserved=18391KB, committed=18391KB) (malloc=16242KB #153138) (arena=2150KB #1)
本机内存跟踪(保留= 9002KB，已提交= 9002KB) (malloc = 186KB#2000) (跟踪开销= 8816KB)
Native Memory Tracking (reserved=9002KB, committed=9002KB) (malloc=186KB #2000) (tracking overhead=8816KB)
Arena Chunk(保留= 273KB，已提交= 273KB) (malloc = 273KB)
Arena Chunk (reserved=273KB, committed=273KB) (malloc=273KB)
未知(保留= 32800KB，已提交= 32KB) (mmap:reserved = 32800KB，committed = 32KB)
Unknown (reserved=32800KB, committed=32KB) (mmap: reserved=32800KB, committed=32KB)
我注意到的是这一部分: 内部(保留= 6366260KB，已提交= 6366256KB)
One thing I noticed is this section: Internal (reserved=6366260KB, committed=6366256KB)
它不断增长，导致总内存使用量超过24GB限制.
It keeps growing and causing total memory usage to exceed 24GB limit.
有人见过类似的问题吗?任何人都知道这里的内部存储器是什么，并且它可能会在不释放内存的情况下持续增长的原因是什么?
Anyone has seen similar issue before? and anyone knows what is Internal memory here and what could be the reason that it keeps growing without releasing the memory?
推荐答案
这不是您的问题的答案，只是一种解决方法.
This is not answer to your question, it is just a workaround.
我在Java版本"1.8.0_45"上运行JRuby的Docker容器中也观察到相同的问题.解决方案是显式调用垃圾回收.我完全不知道为什么会这样，但是在GC内部Java内存返回到8MB之后.
I have observed same problem in docker containers running JRuby on java version "1.8.0_45". The solution was to explicitly invoke Garbage Collection. I have absolutely no idea, why this works, but after GC Internal Java memory returned to 8MB.
这篇关于为什么Java本机内存跟踪中的内部内存增加的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
