为什么 JVM 报告的已提交内存比 linux 进程驻留集大小更多? [英] Why does a JVM report more committed memory than the linux process resident set size?
问题描述
在启用本机内存跟踪的情况下运行 Java 应用程序(在 YARN 中)时(-XX:NativeMemoryTracking=detail，请参阅 https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html 和 https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html)，我可以看到JVM在不同类别中使用了多少内存.
我在 jdk 1.8.0_45 上的应用显示:
<上一页>本机内存跟踪:总计:保留=4023326KB，提交=2762382KB- Java 堆(保留=1331200KB，提交=1331200KB)(mmap:保留=1331200KB，提交=1331200KB)- 类(保留=1108143KB，提交=64559KB)(类#8621)(malloc=6319KB #17371)(mmap:保留=1101824KB，提交=58240KB)- 线程(保留=1190668KB，提交=1190668KB)(线程#1154)(堆栈:保留=1185284KB，提交=1185284KB)(malloc=3809KB #5771)(竞技场=1575KB #2306)- 代码(保留=255744KB，提交=38384KB)(malloc=6144KB #8858)(mmap:保留=249600KB，提交=32240KB)- GC(保留=54995KB，提交=54995KB)(malloc=5775KB #217)(mmap:保留=49220KB，提交=49220KB)- 编译器(保留=267KB，提交=267KB)(malloc=137KB #333)(竞技场=131KB #3)- 内部(保留=65106KB，提交=65106KB)(malloc=65074KB #29652)(mmap:保留=32KB，提交=32KB)- 符号(保留=13622KB，提交=13622KB)(malloc=12016KB #128199)(竞技场=1606KB #1)- 本机内存跟踪(保留=3361KB，提交=3361KB)(malloc=287KB #3994)(跟踪开销=3075KB)- Arena Chunk(保留=220KB，提交=220KB)(malloc=220KB)这显示了 2.7GB 的已提交内存，包括 1.3GB 的已分配堆和几乎 1.2GB 的已分配线程堆栈(使用许多线程).
但是，当运行 ps ax -o pid,rss |grep <mypid> 或 top 它仅显示 1.6GB 的 RES/rss 常驻内存.检查交换说没有使用中:
为什么只驻留 1.6GB 内存时 JVM 指示已提交 2.7GB 内存?剩下的都去哪儿了?
我开始怀疑堆栈内存(与 JVM 堆不同)似乎是预先提交的而没有成为常驻内存，并且随着时间的推移，堆栈内存只会驻留到实际堆栈使用的高水位线.
是的，除非另有说明，否则至少在 linux mmap 上是惰性的.匿名页面仅在写入后由物理内存支持(由于 零页面优化，读取不足)
GC 堆内存被复制收集器或预置零(-XX:+AlwaysPreTouch)有效地触及，所以它总是常驻的.线程堆栈otoh 不受此影响.
为了进一步确认，您可以使用 pmap -x <java pid> 并将各种地址范围的 RSS 与来自 NMT 的虚拟内存映射的输出进行交叉引用.
已使用 PROT_NONE 映射保留的内存.这意味着虚拟地址空间范围在内核的 vma 结构中有条目，因此不会被其他 mmap/malloc 调用使用.但是它们仍然会导致页面错误作为 SIGSEGV 转发到进程，即访问它们是错误的.
拥有可供将来使用的连续地址范围很重要，这反过来又简化了指针运算.
Committed-but-not-backed-by-storage 内存已被映射 - 例如 - PROT_READ |PROT_WRITE 但访问它仍然会导致页面错误.但是该页面错误由内核静默处理，方法是使用实际内存支持它并返回执行，就好像什么都没发生一样.
即这是流程本身不会注意到的实现细节/优化.
对概念进行分解:
Used Heap:活动对象根据上次GC占用的内存量
已提交:已使用除 PROT_NONE 以外的其他内容映射的地址范围.由于延迟分配和分页，它们可能有也可能没有物理或交换支持.
保留:已通过 mmap 为特定内存池预先映射的总地址范围.
reserved - commited 差异由 PROT_NONE 映射组成，保证不受物理内存支持
驻留:当前在物理内存中的页面.这意味着代码、堆栈、已提交内存池的一部分以及最近访问的映射文件的一部分以及 JVM 无法控制的分配.
虚拟:所有虚拟地址映射的总和.涵盖已提交、保留的内存池以及映射文件或共享内存.这个数字很少能提供信息，因为 JVM 可以提前保留非常大的地址范围或 mmap 大文件.
When running a Java app (in YARN) with native memory tracking enabled (-XX:NativeMemoryTracking=detail see https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html and https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html), I can see how much memory the JVM is using in different categories.
My app on jdk 1.8.0_45 shows:
Native Memory Tracking:
Total: reserved=4023326KB, committed=2762382KB
- Java Heap (reserved=1331200KB, committed=1331200KB)
(mmap: reserved=1331200KB, committed=1331200KB)
- Class (reserved=1108143KB, committed=64559KB)
(classes #8621)
(malloc=6319KB #17371)
(mmap: reserved=1101824KB, committed=58240KB)
- Thread (reserved=1190668KB, committed=1190668KB)
(thread #1154)
(stack: reserved=1185284KB, committed=1185284KB)
(malloc=3809KB #5771)
(arena=1575KB #2306)
- Code (reserved=255744KB, committed=38384KB)
(malloc=6144KB #8858)
(mmap: reserved=249600KB, committed=32240KB)
- GC (reserved=54995KB, committed=54995KB)
(malloc=5775KB #217)
(mmap: reserved=49220KB, committed=49220KB)
- Compiler (reserved=267KB, committed=267KB)
(malloc=137KB #333)
(arena=131KB #3)
- Internal (reserved=65106KB, committed=65106KB)
(malloc=65074KB #29652)
(mmap: reserved=32KB, committed=32KB)
- Symbol (reserved=13622KB, committed=13622KB)
(malloc=12016KB #128199)
(arena=1606KB #1)
- Native Memory Tracking (reserved=3361KB, committed=3361KB)
(malloc=287KB #3994)
(tracking overhead=3075KB)
- Arena Chunk (reserved=220KB, committed=220KB)
(malloc=220KB)
This shows 2.7GB of committed memory, including 1.3GB of allocated heap and almost 1.2GB of allocated thread stacks (using many threads).
However, when running ps ax -o pid,rss | grep <mypid> or top it shows only 1.6GB of RES/rss resident memory. Checking swap says none in use:
free -m
total used free shared buffers cached
Mem: 129180 99348 29831 0 2689 73024
-/+ buffers/cache: 23633 105546
Swap: 15624 0 15624
Why does the JVM indicate 2.7GB memory is committed when only 1.6GB is resident? Where did the rest go?
I'm beginning to suspect that stack memory (unlike the JVM heap) seems to be precommitted without becoming resident and over time becomes resident only up to the high water mark of actual stack usage.
Yes, at least on linux mmap is lazy unless told otherwise. Anonymous pages are only backed by physical memory once they're written to (reads are not sufficient due to the zero-page optimization)
GC heap memory effectively gets touched by the copying collector or by pre-zeroing (-XX:+AlwaysPreTouch), so it'll always be resident. Thread stacks otoh aren't affected by this.
For further confirmation you can use pmap -x <java pid> and cross-reference the RSS of various address ranges with the output from the virtual memory map from NMT.
Reserved memory has been mmaped with PROT_NONE. Which means the virtual address space ranges have entries in the kernel's vma structs and thus will not be used by other mmap/malloc calls. But they will still cause page faults being forwarded to the process as SIGSEGV, i.e. accessing them is an error.
This is important to have contiguous address ranges available for future use, which in turn simplifies pointer arithmetic.
Committed-but-not-backed-by-storage memory has been mapped with - for example - PROT_READ | PROT_WRITE but accessing it still causes a page fault. But that page fault is silently handled by the kernel by backing it with actual memory and returning to execution as if nothing happened.
I.e. it's an implementation detail/optimization that won't be noticed by the process itself.
To give a breakdown of the concepts:
Used Heap: the amount of memory occupied by live objects according to the last GC
Committed: Address ranges that have been mapped with something other than PROT_NONE. They may or may not be backed by physical or swap due to lazy allocation and paging.
Reserved: The total address range that has been pre-mapped via mmap for a particular memory pool.
The reserved − committed difference consists of PROT_NONE mappings, which are guaranteed to not be backed by physical memory
Resident: Pages which are currently in physical ram. This means code, stacks, part of the committed memory pools but also portions of mmaped files which have recently been accessed and allocations outside the control of the JVM.
Virtual: The sum of all virtual address mappings. Covers committed, reserved memory pools but also mapped files or shared memory. This number is rarely informative since the JVM can reserve very large address ranges in advance or mmap large files.
这篇关于为什么 JVM 报告的已提交内存比 linux 进程驻留集大小更多?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
