为什么 JVM 报告的提交内存比 linux 进程驻留集大小多? [英] Why does a JVM report more committed memory than the linux process resident set size?
问题描述
在启用本机内存跟踪的情况下运行 Java 应用程序(在 YARN 中)时(-XX:NativeMemoryTracking=detail 参见 https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html 和 https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html)，我可以看到JVM在不同类别中使用了多少内存.
我在 jdk 1.8.0_45 上的应用程序显示:
<前>本机内存跟踪:总计:保留=4023326KB，提交=2762382KB- Java 堆(保留=1331200KB，提交=1331200KB)(mmap:保留=1331200KB，提交=1331200KB)- 类(保留=1108143KB，提交=64559KB)(课程#8621)(malloc=6319KB #17371)(mmap:保留=1101824KB，提交=58240KB)- 线程(保留=1190668KB，提交=1190668KB)(线程#1154)(堆栈:保留=1185284KB，提交=1185284KB)(malloc=3809KB #5771)(竞技场=1575KB #2306)- 代码(保留=255744KB，提交=38384KB)(malloc=6144KB #8858)(mmap:保留=249600KB，提交=32240KB)- GC(保留=54995KB，提交=54995KB)(malloc=5775KB #217)(mmap:保留=49220KB，提交=49220KB)- 编译器(保留=267KB，提交=267KB)(malloc=137KB #333)(竞技场=131KB #3)- 内部(保留=65106KB，提交=65106KB)(malloc=65074KB #29652)(mmap:保留=32KB，提交=32KB)- 符号(保留=13622KB，提交=13622KB)(malloc=12016KB #128199)(竞技场=1606KB #1)- 本机内存跟踪(保留=3361KB，提交=3361KB)(malloc=287KB #3994)(跟踪开销=3075KB)- Arena Chunk(保留=220KB，提交=220KB)(malloc=220KB)这显示了 2.7GB 的已提交内存，包括 1.3GB 的已分配堆和近 1.2GB 的已分配线程堆栈(使用多个线程).
然而，当运行 ps ax -o pid,rss |grep <mypid> 或 top 它只显示 1.6GB 的 RES/rss 常驻内存.检查交换显示没有使用:
为什么只有 1.6GB 常驻时 JVM 显示提交了 2.7GB 内存?剩下的都去哪儿了?
我开始怀疑堆栈内存(与 JVM 堆不同)似乎是预先提交的，而不会成为常驻内存，并且随着时间的推移，只会在实际堆栈使用的高水位线之前成为常驻内存.
是的，除非另有说明，否则至少在 linux 上 mmap 是懒惰的.匿名页面仅在写入后由物理内存支持(由于 零页优化，读取次数不足))
GC 堆内存有效地被复制收集器或预置零 (-XX:+AlwaysPreTouch) 触及，因此它将始终驻留.线程堆栈 otoh 不受此影响.
为了进一步确认，您可以使用 pmap -x 并交叉引用各种地址范围的 RSS 与来自 NMT 的虚拟内存映射的输出.
保留内存已使用 PROT_NONE 进行映射.这意味着虚拟地址空间范围在内核的 vma 结构中有条目，因此不会被其他 mmap/malloc 调用使用.但是它们仍然会导致页面错误作为 SIGSEGV 被转发到进程，即访问它们是一个错误.
为将来使用提供连续的地址范围很重要，这反过来又简化了指针运算.
已提交但未支持的存储内存已映射为 - 例如 - PROT_READ |PROT_WRITE 但访问它仍然会导致页面错误.但是该页面错误由内核通过使用实际内存支持它并返回执行，就像什么也没发生一样，默默地处理.
即这是流程本身不会注意到的实现细节/优化.
对概念进行细分:
Used Heap:根据上次 GC 的存活对象占用的内存量
已提交:已使用 PROT_NONE 以外的其他内容映射的地址范围.由于延迟分配和分页，它们可能会也可能不会得到物理或交换的支持.
保留:已通过 mmap 为特定内存池预先映射的总地址范围.
reserved -mitted 差异由 PROT_NONE 映射组成，这些映射保证不受物理内存的支持
常驻:当前在物理内存中的页面.这意味着代码、堆栈、已提交内存池的一部分以及最近访问过的 mmaped 文件的部分以及 JVM 控制之外的分配.
Virtual:所有虚拟地址映射的总和.涵盖已提交的、保留的内存池以及映射文件或共享内存.这个数字很少提供信息，因为 JVM 可以提前保留非常大的地址范围或 mmap 大文件.
When running a Java app (in YARN) with native memory tracking enabled (-XX:NativeMemoryTracking=detail see https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html and https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html), I can see how much memory the JVM is using in different categories.
My app on jdk 1.8.0_45 shows:
Native Memory Tracking:
Total: reserved=4023326KB, committed=2762382KB
- Java Heap (reserved=1331200KB, committed=1331200KB)
(mmap: reserved=1331200KB, committed=1331200KB)
- Class (reserved=1108143KB, committed=64559KB)
(classes #8621)
(malloc=6319KB #17371)
(mmap: reserved=1101824KB, committed=58240KB)
- Thread (reserved=1190668KB, committed=1190668KB)
(thread #1154)
(stack: reserved=1185284KB, committed=1185284KB)
(malloc=3809KB #5771)
(arena=1575KB #2306)
- Code (reserved=255744KB, committed=38384KB)
(malloc=6144KB #8858)
(mmap: reserved=249600KB, committed=32240KB)
- GC (reserved=54995KB, committed=54995KB)
(malloc=5775KB #217)
(mmap: reserved=49220KB, committed=49220KB)
- Compiler (reserved=267KB, committed=267KB)
(malloc=137KB #333)
(arena=131KB #3)
- Internal (reserved=65106KB, committed=65106KB)
(malloc=65074KB #29652)
(mmap: reserved=32KB, committed=32KB)
- Symbol (reserved=13622KB, committed=13622KB)
(malloc=12016KB #128199)
(arena=1606KB #1)
- Native Memory Tracking (reserved=3361KB, committed=3361KB)
(malloc=287KB #3994)
(tracking overhead=3075KB)
- Arena Chunk (reserved=220KB, committed=220KB)
(malloc=220KB)
This shows 2.7GB of committed memory, including 1.3GB of allocated heap and almost 1.2GB of allocated thread stacks (using many threads).
However, when running ps ax -o pid,rss | grep <mypid> or top it shows only 1.6GB of RES/rss resident memory. Checking swap says none in use:
free -m
total used free shared buffers cached
Mem: 129180 99348 29831 0 2689 73024
-/+ buffers/cache: 23633 105546
Swap: 15624 0 15624
Why does the JVM indicate 2.7GB memory is committed when only 1.6GB is resident? Where did the rest go?
I'm beginning to suspect that stack memory (unlike the JVM heap) seems to be precommitted without becoming resident and over time becomes resident only up to the high water mark of actual stack usage.
Yes, at least on linux mmap is lazy unless told otherwise. Anonymous pages are only backed by physical memory once they're written to (reads are not sufficient due to the zero-page optimization)
GC heap memory effectively gets touched by the copying collector or by pre-zeroing (-XX:+AlwaysPreTouch), so it'll always be resident. Thread stacks otoh aren't affected by this.
For further confirmation you can use pmap -x <java pid> and cross-reference the RSS of various address ranges with the output from the virtual memory map from NMT.
Reserved memory has been mmaped with PROT_NONE. Which means the virtual address space ranges have entries in the kernel's vma structs and thus will not be used by other mmap/malloc calls. But they will still cause page faults being forwarded to the process as SIGSEGV, i.e. accessing them is an error.
This is important to have contiguous address ranges available for future use, which in turn simplifies pointer arithmetic.
Committed-but-not-backed-by-storage memory has been mapped with - for example - PROT_READ | PROT_WRITE but accessing it still causes a page fault. But that page fault is silently handled by the kernel by backing it with actual memory and returning to execution as if nothing happened.
I.e. it's an implementation detail/optimization that won't be noticed by the process itself.
To give a breakdown of the concepts:
Used Heap: the amount of memory occupied by live objects according to the last GC
Committed: Address ranges that have been mapped with something other than PROT_NONE. They may or may not be backed by physical or swap due to lazy allocation and paging.
Reserved: The total address range that has been pre-mapped via mmap for a particular memory pool.
The reserved − committed difference consists of PROT_NONE mappings, which are guaranteed to not be backed by physical memory
Resident: Pages which are currently in physical ram. This means code, stacks, part of the committed memory pools but also portions of mmaped files which have recently been accessed and allocations outside the control of the JVM.
Virtual: The sum of all virtual address mappings. Covers committed, reserved memory pools but also mapped files or shared memory. This number is rarely informative since the JVM can reserve very large address ranges in advance or mmap large files.
这篇关于为什么 JVM 报告的提交内存比 linux 进程驻留集大小多?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
