为什么JVM会报告比linux进程常驻集大小更多的已提交内存? [英] Why does a JVM report more committed memory than the linux process resident set size?
问题描述
运行启用了本地内存跟踪的Java应用程序(在YARN中)(-XX:NativeMemoryTracking=detail请参见
When running a Java app (in YARN) with native memory tracking enabled (-XX:NativeMemoryTracking=detail see https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html and https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html), I can see how much memory the JVM is using in different categories.
我在jdk 1.8.0_45上的应用显示:
My app on jdk 1.8.0_45 shows:
Native Memory Tracking:
Total: reserved=4023326KB, committed=2762382KB
- Java Heap (reserved=1331200KB, committed=1331200KB)
(mmap: reserved=1331200KB, committed=1331200KB)
- Class (reserved=1108143KB, committed=64559KB)
(classes #8621)
(malloc=6319KB #17371)
(mmap: reserved=1101824KB, committed=58240KB)
- Thread (reserved=1190668KB, committed=1190668KB)
(thread #1154)
(stack: reserved=1185284KB, committed=1185284KB)
(malloc=3809KB #5771)
(arena=1575KB #2306)
- Code (reserved=255744KB, committed=38384KB)
(malloc=6144KB #8858)
(mmap: reserved=249600KB, committed=32240KB)
- GC (reserved=54995KB, committed=54995KB)
(malloc=5775KB #217)
(mmap: reserved=49220KB, committed=49220KB)
- Compiler (reserved=267KB, committed=267KB)
(malloc=137KB #333)
(arena=131KB #3)
- Internal (reserved=65106KB, committed=65106KB)
(malloc=65074KB #29652)
(mmap: reserved=32KB, committed=32KB)
- Symbol (reserved=13622KB, committed=13622KB)
(malloc=12016KB #128199)
(arena=1606KB #1)
- Native Memory Tracking (reserved=3361KB, committed=3361KB)
(malloc=287KB #3994)
(tracking overhead=3075KB)
- Arena Chunk (reserved=220KB, committed=220KB)
(malloc=220KB)
这显示了2.7GB的已承诺内存，包括1.3GB的已分配堆和近1.2GB的已分配线程堆栈(使用许多线程).
This shows 2.7GB of committed memory, including 1.3GB of allocated heap and almost 1.2GB of allocated thread stacks (using many threads).
但是，当运行ps ax -o pid,rss | grep <mypid>或top时，它仅显示1.6GB的RES/rss常驻内存.检查交换未使用:
However, when running ps ax -o pid,rss | grep <mypid> or top it shows only 1.6GB of RES/rss resident memory. Checking swap says none in use:
free -m
total used free shared buffers cached
Mem: 129180 99348 29831 0 2689 73024
-/+ buffers/cache: 23633 105546
Swap: 15624 0 15624
为什么当仅驻留1.6GB内存时，JVM会指示2.7GB内存已提交?剩下的去了哪里?
Why does the JVM indicate 2.7GB memory is committed when only 1.6GB is resident? Where did the rest go?
推荐答案
我开始怀疑堆栈内存(与JVM堆不同)似乎已预先提交但没有驻留，并且随着时间的流逝，它们只能驻留到实际堆栈使用率的最高水位.
I'm beginning to suspect that stack memory (unlike the JVM heap) seems to be precommitted without becoming resident and over time becomes resident only up to the high water mark of actual stack usage.
是的，除非另有说明，至少在Linux mmap上是懒惰的.页面仅在写入后才获得物理内存的支持(由于零页面优化)
Yes, at least on linux mmap is lazy unless told otherwise. Pages are only backed by physical memory once they're written to (reads are not sufficient due to the zero-page optimization)
GC堆内存有效地被复制收集器或通过预调零(-XX:+AlwaysPreTouch)触及，因此它将始终处于驻留状态.线程堆栈otoh不受此影响.
GC heap memory effectively gets touched by the copying collector or by pre-zeroing (-XX:+AlwaysPreTouch), so it'll always be resident. Thread stacks otoh aren't affected by this.
为进一步确认，您可以使用pmap -x <java pid>并将各种地址范围的RSS与NMT虚拟内存映射的输出交叉引用.
For further confirmation you can use pmap -x <java pid> and cross-reference the RSS of various address ranges with the output from the virtual memory map from NMT.
已使用PROT_NONE映射保留的内存.这意味着虚拟地址空间范围在内核的vma结构中具有条目，因此不会被其他mmap/malloc调用使用.但是它们仍然会导致页面错误以SIGSEGV的形式转发到进程，即访问它们是一个错误.
Reserved memory has been mmaped with PROT_NONE. Which means the virtual address space ranges have entries in the kernel's vma structs and thus will not be used by other mmap/malloc calls. But they will still cause page faults being forwarded to the process as SIGSEGV, i.e. accessing them is an error.
这对于使连续的地址范围可供将来使用很重要，这反过来又简化了指针运算.
This is important to have contiguous address ranges available for future use, which in turn simplifies pointer arithmetic.
已提交但未由存储支持的内存已映射-例如-PROT_READ | PROT_WRITE，但访问它仍会导致页面错误.但是，该页面错误由内核静默处理，方法是将其与实际内存一起备份并返回执行，就好像什么都没有发生一样.
即这是实现细节/优化，过程本身不会注意到.
Committed-but-not-backed-by-storage memory has been mapped with - for example - PROT_READ | PROT_WRITE but accessing it still causes a page fault. But that page fault is silently handled by the kernel by backing it with actual memory and returning to execution as if nothing happened.
I.e. it's an implementation detail/optimization that won't be noticed by the process itself.
给出概念的细分:
已用堆:根据上一个GC，活动对象占用的内存量
Used Heap: the amount of memory occupied by live objects according to the last GC
已提交:已使用除PROT_NONE之外的其他内容映射的地址范围.由于延迟分配和分页，它们可能会或可能不会受到物理或交换的支持.
Committed: Address ranges that have been mapped with something other than PROT_NONE. They may or may not be backed by physical or swap due to lazy allocation and paging.
保留:已通过mmap预映射到特定内存池的总地址范围.
reserved-commit 差异由PROT_NONE映射组成，保证不会受到物理内存的支持
Reserved: The total address range that has been pre-mapped via mmap for a particular memory pool.
The reserved − committed difference consists of PROT_NONE mappings, which are guaranteed to not be backed by physical memory
居民:当前处于物理内存中的页面.这意味着代码，堆栈，已提交的内存池的一部分，以及最近已被访问的mmaped文件的一部分，以及在JVM控制范围之外的分配.
Resident: Pages which are currently in physical ram. This means code, stacks, part of the committed memory pools but also portions of mmaped files which have recently been accessed and allocations outside the control of the JVM.
虚拟:所有虚拟地址映射的总和.涵盖已提交，保留的内存池，还包括映射的文件或共享内存.这个数字很少提供信息，因为JVM可以提前保留很大的地址范围或mmap大型文件.
Virtual: The sum of all virtual address mappings. Covers committed, reserved memory pools but also mapped files or shared memory. This number is rarely informative since the JVM can reserve very large address ranges in advance or mmap large files.
这篇关于为什么JVM会报告比linux进程常驻集大小更多的已提交内存?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
